What is the difference between the Trinity theory and Modalism?

To this we must add that, in the orthodox understanding of the Trinity, the difference in relations has no practical implication

No, actually the relations are of immense practical implication. In Trinitarianism God is inherently a relational being, but in modalism God is not; any relational characteristics of God only get exhibited after the universe is created. Whether this is a change in God, or just that God's previously unexercised characteristics are exercised with the creation probably depends on the individual modalist.

If God is not inherently relational, as must be the case with modalism, then:

1. God never loved another before he created. And while Trinitarians can teach that the reason for God creating is that the Fathers love for the Son and Spirit overflows, the modalists do not have this motivation.

2. Likewise, God the Father is not inherently and eternally Father. For Trinitarians, the Fatherhood of the Father gives rise to his Fatherly creation and care over us. We can say that human fathers are imitations of the heavenly father, but modalists must say that God is Father to the Son only in appearance, not reality.

3. God is not an eternally communicative being. For the Trinitarian, God's revelation of himself to humanity, as well as the creation of human language, all flow out of God's nature as a communicative being, who has eternally related not just with emotions (as in the first point), but also in communication, albeit in ways beyond our understanding. For the modalist, God never communicated with any other being before he created one. Rather than God's revelation of himself being a natural continuation of the Father's self-expression to the Son, and the Son to the Spirit and so on, God's revelation must be a new expressive act.

4. Jesus cannot truly intercede with the Father (as I asked about here). Trinitarians believe that Jesus intercedes with the Father on our behalf. He can do this because he is a distinct person, and he is the only competent mediator because only God can truly mediate with himself. But for modalists, the "Son" interceding with the "Father" is only an act. This then puts the doctrine of the propitiation of the wrath of God into question, for if the Son and the Father are one person, then why act like the Father has wrath towards people that must be dealt with before they can approach him, when the Son has no such issue?

   (Trinitarians who teach that the Father and Son share one mind and will do have a related problem: how the Son can represent people to the Father when they share one mind, as the Father's enmity to people should be shared by the Son. I will add a link to my thesis on this question in the theology of Calvin later.)

In modalism, there is no distinction between God the Father, God the Son, and God the Holy Spirit. Instead, the distinctions are "energies" or "modes" of God, not unlike the Hindu conception of avatars of a singular, unitarian godhead. It should be emphasized that these modes or energies are always expressed in how God interacts with creation, rather than how God interacts with or is related to Himself.

The Monarchians properly so-called (Modalists) exaggerated the oneness of the Father and the Son so as to make them but one Person; thus the distinctions in the Holy Trinity are energies or modes, not Persons: God the Father appears on earth as Son; hence it seemed to their opponents that Monarchians made the Father suffer and die. [1]

Under the orthodox catholic understanding of the Trinity, the persons are distinct with relation to one another, yet indistinct in their essence. Aquinas parses this out over a number of entries into the Prima Pars of his Summa Theologiae.

To begin with, we must understand that the relations between persons subsists within the Divine Nature, yet, owing to God's nature as Pure Act, relations in Him are essential, not accidental. Hence it follows that the persons are distinct from one another essentially, but not from the essence of God. They are the essence of God (Divine Simplicity).

I answer that, The truth of this question is quite clear if we consider the divine simplicity. For it was shown above (I:3:3) that the divine simplicity requires that in God essence is the same as "suppositum," which in intellectual substances is nothing else than person. But a difficulty seems to arise from the fact that while the divine persons are multiplied, the essence nevertheless retains its unity. And because, as Boethius says (De Trin. i), "relation multiplies the Trinity of persons," some have thought that in God essence and person differ, forasmuch as they held the relations to be "adjacent"; considering only in the relations the idea of "reference to another," and not the relations as realities. But as it was shown above (I:28:2) in creatures relations are accidental, whereas in God they are the divine essence itself. Thence it follows that in God essence is not really distinct from person; and yet that the persons are really distinguished from each other. For person, as above stated (I:29:4, signifies relation as subsisting in the divine nature. But relation as referred to the essence does not differ therefrom really, but only in our way of thinking; while as referred to an opposite relation, it has a real distinction by virtue of that opposition. Thus there are one essence and three persons. [2]

We can build on this if we understand another point in Aquinas. The Divine essence is in the persons, yet again, owing to Divine Simplicity, the persons are the Divine essence. So the essence is in them and they are the essence. Furthermore, it follows that the Divine essence both is in God and is God. From here we see clearly why it must be the case that the persons of God are not distinct in nature. All three of them have the Divine Essence in them, and at the same time are the Divine Essence.

I answer that, Different opinions have been held on this point. Some have said that the properties are not the persons, nor in the persons; and these have thought thus owing to the mode of signification of the relations, which do not indeed signify existence "in" something, but rather existence "towards" something. Whence, they styled the relations "assistant," as above explained (I:28:2. But since relation, considered as really existing in God, is the divine essence Itself, and the essence is the same as person, as appears from what was said above (I:39:1), relation must necessarily be the same as person.

Others, therefore, considering this identity, said that the properties were indeed the persons; but not "in" the persons; for, they said, there are no properties in God except in our way of speaking, as stated above (I:32:2). We must, however, say that there are properties in God; as we have shown (I:32:2. These are designated by abstract terms, being forms, as it were, of the persons. So, since the nature of a form requires it to be "in" that of which it is the form, we must say that the properties are in the persons, and yet that they are the persons; as we say that the essence is in God, and yet is God. [3]

So then the only question left is what makes the persons distinct from one another? What makes the distinction real? The answer is that they are distinct only in their relation to one another.

Sometimes, however, this regard to another, signified by relation, is to be found only in the apprehension of reason comparing one thing to another, and this is a logical relation only; as, for instance, when reason compares man to animal as the species to the genus. But when something proceeds from a principle of the same nature, then both the one proceeding and the source of procession, agree in the same order; and then they have real relations to each other. Therefore as the divine processions are in the identity of the same nature, as above explained (I:27:2 and I:27:4), these relations, according to the divine processions, are necessarily real relations. [4]

So there is a relation between the Father and the Son, a relation between the Father and the Holy Ghost, and a relation between the Son and the Holy Ghost.

I answer that, It must be said that the Holy Ghost is from the Son. For if He were not from Him, He could in no wise be personally distinguished from Him; as appears from what has been said above (I:28:3; I:30:2). For it cannot be said that the divine Persons are distinguished from each other in any absolute sense; for it would follow that there would not be one essence of the three persons: since everything that is spoken of God in an absolute sense, belongs to the unity of essence. Therefore it must be said that the divine persons are distinguished from each other only by the relations. Now the relations cannot distinguish the persons except forasmuch as they are opposite relations; which appears from the fact that the Father has two relations, by one of which He is related to the Son, and by the other to the Holy Ghost; but these are not opposite relations, and therefore they do not make two persons, but belong only to the one person of the Father. If therefore in the Son and the Holy Ghost there were two relations only, whereby each of them were related to the Father, these relations would not be opposite to each other, as neither would be the two relations whereby the Father is related to them. Hence, as the person of the Father is one, it would follow that the person of the Son and of the Holy Ghost would be one, having two relations opposed to the two relations of the Father. But this is heretical since it destroys the Faith in the Trinity. Therefore the Son and the Holy Ghost must be related to each other by opposite relations. Now there cannot be in God any relations opposed to each other, except relations of origin, as proved above (I:28:44). And opposite relations of origin are to be understood as of a "principle," and of what is "from the principle." Therefore we must conclude that it is necessary to say that either the Son is from the Holy Ghost; which no one says; or that the Holy Ghost is from the Son, as we confess. [5]

For additional reading, Question 27 gets into what the divine processions are in God which generate and spirate the Son and Holy Ghost. This generation and spiration are loosely speaking the cause of difference in the persons, because it is by generation and being generated (or begotten) and by spirating and proceeding that these opposite relations come to be.

In conclusion

So this is the difference between modalism and orthodox trinitarianism. Trinitarians hold that there is a real distinction between the persons, but not the nature, of God. Aquinas defends this by explaining that the relations of the persons of the Trinity, by which the principle (The Father) generates (The Son) and spirates (The Holy Ghost) them, are the "cause" (loosely speaking) of the opposed relations in the persons which form the distinction. Whereas, modalists hold that the three modes of the Trinity are indeed a singular person, having no difference within them among one another, that is, no relation, but merely in mode of interaction with creation.

The question of the eternity of these relations (or not) is not important to the distinction between modalism and trinitarianism. Modalists deny that the relations are real, whether implicitly or explicitly.

Modalism (also known as Sabellianism) : There is only one person in God, who represents himself in the roles of three persons.

Sabellius argued that the Father, the Son, and the Spirit are 'masks' or personae worn by the one divine person. Like an actor on a stage, God could appear sometimes as the Father, other times as the Son, and other times as the Spirit. However, these are not actually three different actors. He was excommunicated by the bishop of Rome in AD 220.

The Trinity : Throughout God's mighty acts in history we meet three distinct persons who are nevertheless identified as God. God is "one in essence, three in persons." In every external act of the Godhead, the Trinity is undivided and yet each person's agency is distinct.

Source for the above definitions - Pilgrim Theology by Michael Horton, pp469, 95 & 475

Sabellians so identified Christ with the Father that he appears to be merely a mode of the Father's existence, hence those who took the same position in the following century being called 'modalists'.

Our English word 'person' comes from Latin and it has lost something in its translation from Greek. The original Greek word 'ousia' meant essence or substance. The usual Latin for that is 'substantia', meaning essence or substance. The Greek word 'hypostasis' meant person, or a second meaning was substance. The usual Latin is 'persona', meaning person, or a second meaning of actor, or role. The Greek word 'prosopon' meant face or mask, with a second meaning of person. The usual Latin for that is persona, and that is what we get our understanding of 'person' from.

But if we went back to the original Greek of the Bible New Testament, we would see that essence or substance was the meaning, and it is that language that the Trinity doctrine is all about. Most English speaking people today only think of it as an individual person.

The Greeks described the Trinity as 'mia ousia en trisin hypostasesi' = one substance (essence) in three subsistenes [persons]." Unfortunately, that could be misunderstood as saying, "one essence in three substances", which would be 3 gods. When the Latins then said, 'una substantia in tribus personis' = one substance in three persons", they could be misunderstood as saying one 'hypostasis' (person) in three roles. That was the error of Sabellian modalism, which is with us today in the form of Oneness Pentecostalism.

We are 16 centuries removed from this. Yet the teaching back then remains the same as today - "Three - Father, Son and Holy Spirit - are God, yet God is not three, but One."

Source for the above - Heresies and Orthodoxy in the History of the Church, Harold O.J. Brown pp 63, 128-130 (Hendrickson 1998)

The point I wish to make is that going from koine Greek to Latin, something got lost in translation. And Latin to English lost a bit more. There's nothing like sticking to the original koine Greek that the NT was written in, but it was never going to be easy, getting a verbal handle on the awesomeness of Deity. Actually, the Trinity doctrine expresses wherein the mystery of Deity lies, more than tries to explain it!

Modalists try to explain the Trinity by viewing the three Persons as different modes of the one God. They teach no distinct individual Persons in the Godhead (as with orthodox Christianity). For the Modalist, Christ is not only God, he is the Father himself. This claims that the fundamental unity and oneness of God does not permit a second (or a third) Person can share the titles of deity (which the Bible clearly assigns to Jesus Christ and the Holy Spirit).

Modalism is a word used to try to explain the Trinity while preserving the oneness of God. Modalism frequently reappears over the centuries (right up to the present one), often found in modern fundamentalistic circles that insist on the deity of Christ but only in the sense that God reveals Himself under different aspects, or modes, in different ages - as the Father in creation and the giving of the Law; as the Son in Jesus Christ; as the Holy Spirit after Christ's ascension. In so doing, Modalists 'lose' the role of Jesus representing us to the Father. It is a form of Docetism, claiming that the Son, as Christ, only appeared to be human. Theologically, the teaching then becomes a Christ who was fully God, but who only appeared to be a man - which renders null and void the biblical doctrine of Christ fully representing humanity because he was fully human himself.

Final point - you say relations do not make a difference here. But relationship within the Godhead is everything. How the Son relates to the Father and vice versa, and how the Holy Spirit relates to Father and Son, is to be worked out in every Christian's life and witness. It affects Christian marriage, family, and congregation, and all our dealings with all others. Relationship is massively important, so that if we misunderstand how Father, Son and Holy Spirit relate in the Godhead, we will be all at sea here on earth in our relationship with Deity, and with others. The final word is what I've read here:

In the nature of what Deity is, that nature is possessed by a Person, who is named Jesus Christ. He shares that nature with his Father. In relationship – regarding person – his Father is greater than he. But in the matter of nature – eternal nature – his nature is equal to that of the Father and he is perfectly one with the Father. As well as this, Jesus Christ, born of woman, also possesses human nature.

The Divine relationship is within one Spirit – the Divine Person who is the Holy Spirit. All that passes between the Father and the Son – and between the son and the Father – does so in one Spirit. That Spirit is a Divine Person sharing the same divine nature as the Father and the Son.

Fulness is one of the attributes of Deity. And since that attribute is shared, then it follows that there must be – and there indeed is – a perfection of unity within Deity. For each shares the attribute of fulness. Everything – absolutely everything – is filled by Deity. By all that is Deity. Thus, there is – absolutely – a shared perfection of unity. In all things. This is sheer logic.

Only once was that perfect bond within Deity ever broken… “My God, my God – why hast thou forsaken me?” This appalling breach within everlasting Deity was caused by the bearing of sins. And by the resolving of the matter of sin, by means of death. This – the awful rift in the most perfect of relationships – was accepted and agreed upon by Deity in unanimous counsel, before the earth was created, for it was foreseen by the wisdom of Deity that it would be necessary. And yet Deity still created humanity. What revelation this is : it is wonderful! (The Everlasting Gospel by Nigel Johnstone, p43)

The Trinity doctrine shows that three persons share divine nature. The Father and the Son share the one divine nature, with absolute unity of the Spirit in that nature. This is a million [spiritual] miles from modalism.