Upvote:5
You are asking two different things here.
You can certainly re-enter Australia if you magically appear at Immigration there - you're still AU citizen, and you will be let in eventually (there is the possibility of a fine though).
However traveling to Australia would be much more difficult:
It is unlikely that the airline would check you in, especially if it is a non-direct flight - and as Dorothy mentioned above, even with direct flights they are reluctant because there is a possibility of a heavy fine. Note that the airline does not know whether you're still Australian citizen (as maybe you renounced it and didn't surrender your old passport). Also even a direct flight could possibly be diverted to another country due to weather or mechanical failure, and having such a passenger would make things quite complicated for the airline.
EU passport control is not likely to let you out with an expired passport if they notice it.