Hard not to control my breathing

While you control your breathing or focus to your whole body, you losing to focus the breath at nose's tip point. So the right practice is not control your breath or focus to your whole body, but just discerns your breath at the nose's tip point. And do not focus the nose's tip or nose's tip touching, just focus on the entire breath that touching at only nose's tip, not whole respiratory:

The ānāpānassati-pabba is sequences of practice:

1. Monitoring the breath:

"Breathing in long, he discerns, 'I am breathing in long'; or breathing out long, he discerns, 'I am breathing out long.'

2. Then the breath going to be fainter and shorter:

Or breathing in short, he discerns, 'I am breathing in short'; or breathing out short, he discerns, 'I am breathing out short.'

3. Then the breath maybe disappear because of the practitioner's weak mindfulness, breath still going on but he can't see it. So he has to train his mindfulness to be stronger mindful on fainter and shorter breath by making more focus on breath's touching point, the tip of nose:

He trains himself, 'I must sensitively discerns the entire breathing when I will breathe' He trains himself, 'I must sensitively discerns the entire breathing when I will breathe.'

He trains himself, 'I will breathe in calming breath.' He trains himself, 'I will breathe out calming breath.'

For more the tricks of 1st step, when the practitioner forget to focus on breath, the 2nd step, when the breath disappear, you can see in the path of purification, ānāpānassatikathā, CH. VIII (9) Mindfulness of Breathing 259.

There is a trick in mahāsatipaṭṭhānasutta ānāpānapabba's commentary, too.

Note 1: Touching is not breath. Breath appear because of breath touching at nose's tip. So, the practitioner focusing on only breath at only nose's tip, not on touching or nose's tip.

Note 2: Also, breath appear at the whole respiratory, such as throat or lung, but don't focus on it, because the practitioner practicing concentration meditation, so too much breath touching point will make the practitioner distraction.

Don't try to force or control anything, observe and just note it and continue.

It's not that much important how your body reacts to your meditation, but how you deal with these bodily reactions. Yes, there will always be some bodily reactions that will distract you from meditation, but the point is to not get distracted and just remain in the "present moment".

With practice you will be less distracted, until you won't be distracted any more. Practice diligently and you shell rip the rewards soon.

That sounds like bad advise if you are practicing insight meditation. This answer assumes the practice in question is insight meditation.

"You" don't control the breath.

I had this same question when I was starting out and the teacher basically said "You don't control the breath so there is no problem, just keep seeing things as they are moment by moment".

For me, it was just a matter of losing my wrong view that I had to be concerned that I'm controlling the breath. If I'm thinking that I'm controlling the breath then I would focus my mindful awareness on this thinking(instead of worrying that I was controlling the breathing).

For the record I'm new to meditation. I'm trying to observe my breathing without controlling it. That was the guidance from someone I met at a buddhist meeting.

On objective of meditation is to calm the fabrication. By controlling you are, in some instances, you will be creating fabrications.

Though sometimes I just realize that I do change the frequency and deepness of it.

This you should be careful not to do. There is a stage in Anapanasati you have to calm the breath, but in this instance also you just have to intention to calm but not the action to calm, hence you are not doing any action to calm it.

I don't want to do it, just happens. Is it just me or it will be better with practice?

If it happens naturally then it is OK. The depth of the breath change generally becoming shorter but some cases if there through disturbances this might become deeper.

Also see:

• Anapanasati: Mindfulness with Breathing - Unveiling the Secrets of Life by Ven Buddhadasa Bhikkhu
• Ānâpāna,sati Sutta

A simple and practical solution for occurance of own control of breathing is intentionally stop fabricated breathing for few moments and observe. Then you will be able to sense that the body (spontaneous neural discharge from the brain to nerves that innervate respiratory muscles) automatically start breathing process after few seconds of tolarance period. Now you can start observing the natural breathing. Most teachers dictate to make several deep breaths before start the meditation in order to make an exact understanding of the touching point in the breathing channel. This practice leads for some novice practioners to unknowingly continue the same practice of fabricated breathing. But we should always endeavour to 'let it go' and become just an observer without interventions or conclusions. This may be called 'Choiceless Awareness'. May the Noble Triple Gem Bless You