Upvote:6
Well, Buddhism separated itself out as a separate system of thought with the Buddha himself. Jainism must have been a distinct tradition by that time too because the Buddha interacted with the Jains and they are mentioned in the Suttas where they are called the Niganthas. Their leader at that time was Mahavira, who in the Suttas is called Nigantha Nataputta, but they may have existed for quite some time before that.
It seems that there may have been a lot of different ascetic groups that had a lot of the ideas of the Niganthas without actually being a part of the movement proper, as it seemed that Jain ideas of karma had diffused into the general religious climate of the ascetics. For example, it seems that when the Buddha was practicing self mortification for six years, he was acting under this sort of general theory of karma as the basis of his practice.
There were other movements and systems of thought as well that would eventually become part of Hinduism, but it is important to remember that at that time they were very much separate traditions that didn't see themselves as part of the same religion or spiritual heritage. They were very diverse and didn't join together as Hinduism until at the very earliest 200 A.D. These traditions are essentially became the six Astika schools of Hindu philosophy, but it is very unclear as to whether or not these groups accepted the Vedas at that particular time.
Some of them very likely didn't see themselves as Vedic systems of belief at that time and only subscribed to the Vedas when the schools of philosophy began to see themselves as part of a common heritage. Only the Mimamsa school was explicitly based on the Vedas for example, and some of them, like the Samkhya, Yoga, and Vedanta schools had received a great deal of influence from the various ascetic movements and possibly considered themselves to be part of the ascetic tradition rather than the Vedic tradition.
In addition to these distinctive traditions, you had the general folk religion of the people that was largely unorganized. People would venerate Nagas in caves and the various tree spirits by leaving offerings before them. These devotional customs had nothing to do with the Vedic rituals of the day, but would eventually be incorporated into the devotional practices of Hinduism.
I hope that gives a good overview.