What of Augustine's ecclesiology was rejected in the Reformation?

The ecclesiology of Augustine and the ecclesiology of the Reformers were both very much products of the times they lived in:

In Augustine's case as well as sourcing a basic understanding on ecclesiology from scripture and tradition, any development of his thinking in this area was greatly influenced by the problems the Church had been facing - especially in its dealings with false teachers. When he considered "heretics" and schismatics, it was evident to him that they had wandered into error - their contradictions of the truth guarded by "the one true church" led him to believe that there could be no salvation outside the (visible) Catholic church - to be safeguarded from error, it would be necessary to remain within the fold and consequently submit to the established governing authorities within the visible institution.

The Reformers were faced with an entirely different scenario: In standing for their conviction of what the truth was, they were actually put out of the church. Despite largely agreeing with Augustine on what this truth constituted - especially in respect to soteriology - they found themselves in variance with the authority of the visible established church. This led them to understand that (the Spirit of) Antichrist had not just been in operation within the visible church - leading some into error and becoming heretics and schismatics as in days gone by - but had seized control of the reins as it were - this was a radically different situation. It necessarily generated an ecclesiology that emphasized the invisible nature of the (true) catholic Church organized under Christ's headship against that of a visible human institution lead by men who, while claiming Apostolic succession for their authority, denied this authority through their life conduct and their false doctrine. Consider:

I feel much freer now that I am certain the pope is the Antichrist - Martin Luther

I posit that at least one of the reasons he felt more free is that it helped him clarify his own ecclesiology which in its previous form would have been inadequate to explain the necessity of defending what he saw as essential truth against the attacks of the (visible) Church hierarchy.

Some assert that the distinction between the visible and invisible church is present in The City of God, chapter 35 in particular. I would say that the concept of the (true) Catholic Church being invisible is at best only present in seed form in that passage. It's entirely possible that other doctrines like "the nature of Christ's Church as found in the Bible is one, universal, visible, historical, concrete/locatable/identifiable, hierarchical, sacramental, infallible Church; and as the Creeds state that visible Church is 'One, Holy, Catholic, and Apostolic';" would completely suppress any such seed growing in to the Reformers' "maturer" version of the doctrine. To what extent Augustine adopted these aspects of (what is now current) Catholic ecclesiology is difficult to determine, but I think in his various arguments against heretics, there is evidence that he had at least enough of these concepts resident in his ecclesiology to prevent the seed from growing to maturity. Furthermore, in Schaff's preface to The City of God, he writes:

[Augustine] confines the Kingdom of God to the narrow limits of the Jewish theocracy and the visible Catholic Church.

The doctrine Augustine is espousing is basically the implications of the parable of the wheat and the tares (Matthew 13) - which the current RC church of course accepts as scripture and is why (though they believe in a salvific nature of baptism) they teach against "assurance of salvation" as evangelicals would understand that phrase. So evidently, accepting the "seed" as true is insufficient for the complete Reformation view to be fully developed - other factors are at play.

It's an interesting hypothetical to ask "what would Augustine do?" given the situation in operation in the 16th century. We can speculate regarding what he taught regarding his understanding of scripture, tradition and the situation he faced in his day, but the reality is that we just don't know.

Summary: The aspect of Augustine's ecclesiology that promoted an unreflective submission to the authority (teaching and governing) of the visible Church regardless of it's condition (he couldn't conceive that the institution could be corrupted to the extent that ancient Israel had been prior to being carried away into exile) was necessarily rejected by the Reformers as they faced the reality of being put out of the visible church while steadfastly holding to the truth that they saw in scripture. This was the departure point, and the ecclesiology they subsequently developed (although varied in visible forms according to varied interpretions of scripture and reliance on tradition) was based on the understanding of the 'true' catholic Church of Christ being invisible in form.