Is a belief that the Father, Son and Holy Spirit are not three persons, but one, Modalism?

Possibly, if you are not an extreme Monarchianist (denier of Persons in the Trinity).

Modalism is

A cluster of Trinitarian heresies of the second and third centuries claiming that there is only one person in God, but that this one person manifests himself in three different ways or modes, e.g., as creator (Father), redeemer (Son), and sanctifier (Holy Spirit). (Etym. Latin modus, way, manner, method.)

See Pohle The Divine Trinity: A Dogmatic Treatise pt. 1, ch. 2, §1, Art. 2 "The Modalism of Sabellius".

The modalist view of the Trinity in summary

Believing that the Father, Son, and Holy Spirit are God as one person, not three, is modalism only if Father, Son, and Holy Spirit are seen as different "modes," or roles, or ways of appearing, of God.

In general, modalism involves the idea that there is no real or permanent distinction between the Father, Son, and Holy Spirit, but that each of them is simply a different way that God appears to human beings under different circumstances or to accomplish different tasks.

Emanuel Swedenborg's view of the Trinity

The eighteenth century scientist, philosopher, and theologian Emanuel Swedenborg (1688–1772) offered a concept of God in which Father, Son, and Holy Spirit are one God, in one person, but are distinct and permanent parts of God, not simply different ways that God appears to human beings, or God functioning to accomplish different tasks.

Swedenborg expresses the nature of the Trinity in one Person of God in two primary ways.

More abstractly:

1. The Father is the divine love
2. The Son is the divine wisdom
3. The Holy Spirit is the divine power, or action

More concretely:

1. The Father is the divine soul
2. The Son is the divine body
3. The Holy Spirit is the divine action

How to picture Swedenborg's view of the Trinity

The more concrete version is also the easier one to picture, since it uses the analogy of a human being, created in the image and likeness of God (Genesis 1:26–27), to illustrate the nature of the Trinity in God. Each one of us is a single human being. Yet we have different "parts," the most basic being our soul, our body, and our actions.

In the very same way, Swedenborg said, God is a single divine Person, but has three "parts" which are analogous to our soul, body, and actions. The divine soul is the divine love, the divine body is the divine wisdom, and the divine action is everything God says and does from divine love through divine wisdom.

For a fuller explanation of Swedenborg's concept of a Trinity in one person of God, please see my article:

Who is God? Who is Jesus Christ? What about that Holy Spirit?

Swedenborg does not actually apply the term "parts" to the Father, Son, and Holy Spirit, hence the quotation marks. Rather, he uses the Latin term essentialia, which is difficult to translate into English. It has been translated as "essentials," and more recently "essential components." The basic meaning of an "essential" of God is "something without which God is not God."

This can be illustrated by the human soul, body, and actions. These are not exactly "parts" of a human being. However, without all three of them together, a human being is not a human being. They are therefore the basic "essentials" for us to be a human being.

Swedenborg's view of the Trinity is not modalist

What they are most definitely not, however, is different ways that we appear to other people under different circumstances, or different expressions of us to accomplish different tasks, as in the modalist conception of God. Nor are they different roles we may play, as the modalist conception of God can also be expressed.

Rather, all of our appearances, tasks, and roles are the body acting from the soul in order to accomplish the soul's purposes. In other words, every one of our "modes" or "appearances" or "roles" proceeds from the soul through the body and into action.

In the very same way, in Swedenborg's conception of the Father, Son, and Holy Spirit, every mode, appearance, role, and action of God is the Son, or the divine wisdom, acting from the Father, or the divine love, to accomplish the purposes of God's love. These are not three persons of God, but rather three essentials that are present in all of God's appearances, words, and actions.

This, according to Swedenborg, is the meaning of Jesus' statement:

No one has ever seen God, but the one and only Son, who is himself God and is in closest relationship with the Father, has made him known. (John 1:18, New International Version)

In other words, the Father is not a "mode" or "appearance" of God, because we never see the Father at all, except through the Son. Every appearance of God is through the Son, meaning that the Son is the only "mode" of God, if that means an "appearance" of God, or God acting to accomplish particular tasks, such as redemption or sanctification. And the Holy Spirit is God speaking and acting.

For a fuller explanation of the distinction between Swedenborg's Trinity and modalism, please see my answer to this question here on Christianity StackExchange:

What's the difference, if any, between the Swedenborgian and Oneness Pentecostal doctrines of God?

Conclusion: Believing that the Father, Son, and Holy Spirit are one person, not three persons, does not necessarily involve modalism

To return to the original question, then: No, believing that the Father, the Son and the Holy Spirit are God, but not as three persons but as one, does not necessarily mean you believe in modalism. It could mean that. But there is at least one view, the one presented by Emanuel Swedenborg, in which the Father, Son, and Holy Spirit are one person, and which rejects the modalist conception of God.

Options to consider

In the history of conceptions regarding God, Jesus, and the Holy Spirit there have been names commonly assigned to some of them. Over time, those named conceptions have developed a measure of stability and precision as shown in the Wikipedia entries dedicated to them.

A very good survey of non-Trinitarian conceptions can be found in the Wikipedia entry Nontrinitarianism. Narrowing the field of options to those conceiving God the Father, Jesus, and the Holy Spirit as one person gives us a few options:

1. Monarchianism emphasizes God the Father as one, and this view has two types:

   • Modalistic monarchianism (or simply, Modalism), 'considers God to be one while appearing and working through the different "modes" of Father, Son, and Holy Spirit.' This is the view of Oneness Pentecostalism.
   • Dynamic monarchianism (aka Adoptionism) conceives that God the Father "adopted" Jesus to be God at either baptism, resurrection, or ascension, and that Jesus remained divine since then. In this view, Jesus is not co-eternal with the Father. Viewed another way, Adoptionism is a kind of Subordinationism where Jesus has less status than the Father. This is the view of the now extinct Ebionites and a few minor groups throughout history.

2. Unitarianism differs from Modalism that Unitarians teach that Jesus is not God, but an inspired man and a great teacher. (see GotQuestions entry). This is the view of the Unitarian churches, whose theology is described here.

3. Sabellianism is similar to Modalism in its conception. Encyclopaedia Britannica characterizes it as a "more developed and less naive form of Modalistic Monarchianism". The different name is to identify the origin of the teaching as coming from Sabellius, a theologian and priest from the 3rd century, and as an Eastern Church heresy. New World Encyclopedia has a good treatment on it.

4. Emanuel Swedenborg's view of the Trinity, expounded in Lee Woofenden's article Who is God? Who is Jesus Christ? What about that Holy Spirit?. See also Mr. Woofenden's answer contrasting this view with modalism.

What you believe

Since you specify that 1) all three are God and 2) all three is one person, then what you believe can either be Modalistic monarchianism or Sabellianism or Swedenborg's view unless I failed to name other conceptions that can fit your search criteria.

Further study

One very good book about the historical development of the doctrine of Trinity, various ways of conceptualizing the Trinity, as well as practical consequences to other doctrines is The Trinity - A Guide for the Perplexed by an Anglican priest, Paul M. Collins, reviewed here. Here's a relevant quote from Chapter 2 (Moments of Interpretation) in a section "A pre-Nicene ecology of language":

Monarchianism can be said to have been used to safeguard the unity of God. Dynamic or adoptionist Monarchianism, attributed to Paul of Samosata, understood that the power (dynamis) of God had descended on Jesus, inspired him and given him divine honour. Some would argue that such an understanding can be read out of the Apostles’ Creed. There were other forms of Monarchianism: such as modalism, patripassianism and Sabellianism, in which the three names used in relation to the Godhead: Father, Son and Spirit represented one hypostasis or persona, which was manifest in a succession of modes. There was no understanding of any real distinction within the divine being. These understandings of the divine led to the asking of crucial questions about how God in se was related to the revelation and redemption given. It is out of these concerns and the perceived threats of misunderstanding or heresy that the imperatives towards definition emerge.