This will be a lengthy answer as I will give you all the pertinent Scriptures along with my understanding of each beginning with the following:

All Scripture is quoted from the King James translation unless otherwise noted.

Luke 1:26 And in the sixth month the angel Gabriel was sent from God unto a city of Galilee, named Nazareth,

Shortly after this Mary went and spent six months with Elizabeth, before returning to Nazareth. Until her return no one in Nazareth knew she was pregnant. Shortly after her return she and Joseph were married, and so most in Nazareth probably assumed that Joseph was the natural father of the child, and had been with Mary before she left to be with Elizabeth.

We know that the wedding took place shortly after her return since she was only three months away from delivery when she returned. She delivered in Bethlehem, and that was probably not close to their marriage, since the proclamation would have been given with time for travel and such to be accomplished.

Luke 2:1 through 4 And it came to pass in those days, that there went out a decree from Caesar Augustus, that all the world should be taxed. And all went to be taxed, every one into his own city. And Joseph also went up from Galilee, out of the city of Nazareth, into Judaea, unto the city of David, which is called Bethlehem; (because he was of the house and lineage of David:)

After the Shepherds came and worshipped Jesus Luke says:

Luke 2:19 But Mary kept all these things, and pondered them in her heart.

This seems to me to indicate that this was given to the Apostles by Mary herself, and not from historical accounts.

Joseph and Mary were very devout as indicated by the following Scriptures:

Luke 2:22 And when the days of her purification according to the law of Moses were accomplished, they brought him to Jerusalem, to present him to the Lord;

Luke 2:41 Now his parents went to Jerusalem every year at the feast of the passover.

Now let's take a look at the wise men and their visit and see what we can glean from that:

Matthew 2:1 through 3 Now when Jesus was born in Bethlehem of Judaea in the days of Herod the king, behold, there came wise men from the east to Jerusalem, Saying, Where is he that is born King of the Jews? for we have seen his star in the east, and are come to worship him. When Herod the king had heard these things, he was troubled, and all Jerusalem with him.

And from this Scripture we learn a couple of things:

Matthew 2:11 And when they were come into the house, they saw the young child with Mary his mother, and fell down, and worshipped him: and when they had opened their treasures, they presented unto him gifts; gold, and frankincense, and myrrh.

1. By the time that the Wise men arrived in Bethlehem Jesus is no longer referred to as a baby, but a young child.

2. They had stayed in Bethlehem long enough for Joseph to find enough work to get them a house to live in.

They then lived for several years in Egypt awaiting the Lord telling them it was now safe to return to Israel.

Matthew 2:13 through 15 And when they were departed, behold, the angel of the Lord appeareth to Joseph in a dream, saying, Arise, and take the young child and his mother, and flee into Egypt, and be thou there until I bring thee word: for Herod will seek the young child to destroy him. When he arose, he took the young child and his mother by night, and departed into Egypt: And was there until the death of Herod: that it might be fulfilled which was spoken of the Lord by the prophet, saying, Out of Egypt have I called my son.

This would have been a period of several years, until the death of King Herod. Then God sent an Angel to Joseph again to tell him it was now safe to return to Israel.

Matthew 2:20 and 21 Saying, Arise, and take the young child and his mother, and go into the land of Israel: for they are dead which sought the young child's life. And he arose, and took the young child and his mother, and came into the land of Israel.

Upon their return Joseph decided to return to Nazareth to protect Jesus, This was probably because he felt secure in Nazareth where there were friends and relatives.

Matthew 2:22 and 23 But when he heard that Archelaus did reign in Judaea in the room of his father Herod, he was afraid to go thither: notwithstanding, being warned of God in a dream, he turned aside into the parts of Galilee: And he came and dwelt in a city called Nazareth: that it might be fulfilled which was spoken by the prophets, He shall be called a Nazarene.

And when Jesus was 12 years old he announced that he was the son of God, but was not believed.

Luke 2:48 through 51 And when they saw him, they were amazed: and his mother said unto him, Son, why hast thou thus dealt with us? behold, thy father and I have sought thee sorrowing. And he said unto them, How is it that ye sought me? wist ye not that I must be about my Father's business? And they understood not the saying which he spake unto them. And he went down with them, and came to Nazareth, and was subject unto them: but his mother kept all these sayings in her heart.

So we see that from the time Mary was about nine months pregnant until Jesus was several years old, they traveled quite a bit and any memories about the incident of Jesus conception would have long faded, since most would have considered her pregnancy due to Joseph anyway.

Before the birth

Mary was the first to know that God would be the Father of her Son; Luke 1:34-35:

‘How will this be,’ Mary asked the angel, ‘since I am a virgin?’

The angel answered, ‘The Holy Spirit will come on you, and the power of the Most High will overshadow you. So the holy one to be born will be called the Son of God.

And Joseph knew also before the birth; Matthew 1:20:

But after he had considered this, an angel of the Lord appeared to him in a dream and said, ‘Joseph son of David, do not be afraid to take Mary home as your wife, because what is conceived in her is from the Holy Spirit.

Jesus' childhood

Jesus made no effort to keep this a secret; Luke 2:49:

‘Why were you searching for me?’ he asked. ‘Didn’t you know I had to be in my Father’s house?’

Jesus' ministry

Again, Jesus was quite open about His Divine Father, both to His apostles; John 14:2:

My Father’s house has many rooms; if that were not so, would I have told you that I am going there to prepare a place for you?

and in public; John 2:16:

To those who sold doves he said, ‘Get these out of here! Stop turning my Father’s house into a market!’

and He even gave hints to those who wanted Him dead; John 8:19:

Then they asked him, ‘Where is your father?’

‘You do not know me or my Father,’ Jesus replied. ‘If you knew me, you would know my Father also.’

these being people who probably did not believe this story, since they apparently thought Jesus was illegitimate; John 8:41b:

‘We are not illegitimate children,’ they protested. ‘The only Father we have is God himself.’

After Jesus' ministry

Those writing after the event left us in no doubt; Matthew 1:23:

‘The virgin will conceive and give birth to a son, and they will call him Immanuel’ (which means ‘God with us’).

and Luke 3:23:

Now Jesus himself was about thirty years old when he began his ministry. He was the son, so it was thought, of Joseph, the son of Heli,...

Conclusion

I don't think there was any conscious effort to conceal Jesus' Father as being God, not Joseph. However, at the time who would have believed it?

If Mary withheld this mystery from Joseph her spouse, it is more than reasonable to believe that those who knew of it [Joseph, Elizabeth, Mary's parents] left it to God to reveal it at his appointed time.

To the General Public, no! as the following passages reveal:

Mt 13:55 (RSVCE)
55 Is not this the carpenter’s son? Is not his mother called Mary? And are not his brethren[a] James and Joseph and Simon and Judas?

Lk 4:22 (RSVCE)
22 And all spoke well of him, and wondered at the gracious words which proceeded out of his mouth; and they said, “Is not this Joseph’s son?”

Jn 1:45 (RSVCE)
45 Philip found Nathan′a-el, and said to him, “We have found him of whom Moses in the law and also the prophets wrote, Jesus of Nazareth, the son of Joseph.”

It required divine revelation for Elizabeth and Joseph to know.

Why wasn't it scandalous for Mary to be pregnant while betrothed to Joseph?

This is answered by the two-stage process in marriage among the Jews. The betrothed were considered husband and wife and could have marital relations [cf. Tb 7:9 (RSVCE) - Tb 8 and Mt 1:20 (RSVCE): Before God, Mary is considered Joseph's wife].

This is another example of the interplay in human [salvation] history between God's will and free human actions.