Papal Infallibility and teaching heresy?

What would happen if the Pope taught a heresy?

That is the question.

Starting with relevant Q & A from the Penny Catechism 83., 92. & 93.

1. What is the ninth article of the Creed?
   The ninth article of the Creed is, 'the Holy Catholic Church; the Communion of Saints'.

2. Is the Pope infallible?
   The Pope is infallible.

3. What do you mean when you say that the Pope is infallible?
   When I say that the Pope is infallible, I mean that the Pope cannot err when, as Shepherd and Teacher of all Christians, he defines a doctrine concerning faith or morals, to be held by the whole Church.

Please see also Infallibility | New Advent.

This article Heresy | New Advent says:

The pope himself, if notoriously guilty of heresy, would cease to be pope because he would cease to be a member of the Church.

It is plainly logical then that when a heretical Pope attempts to define ex cathedra a heretical doctrine concerning faith or morals, to be held by the whole Church, he would not be doing so as the Pope who cannot err, because, he would not be Pope, having ceased to be Pope the moment he embraced his heresy.

The often circulated theory that the Pope would be taken out if he attempted to define a heretical doctrine, has never been the teaching of the Church and it is purely speculative (apart from being silly). For one, what God [the Holy Spirit] will do, belongs to God [the Holy Spirit].

Closing:

Even the uneducated grandma who lives her Catholic Faith, knows it (cf. the Vincentian Canon), therefore it would not be a mystery to anyone what the Pope was attempting to do. What do they faithful do? The remnant will just continue to follow and live the Faith handed down to them from the Apostles through the Church teaching over the ages. [cf. Dt 30:11-20 (RSVCE): '[...] is not too hard for you, neither is it far off. It is not in heaven, [...].'].

As for the novel teaching of the 'Pope', clean up follows his pontificate.

cf. 2 Co 11:4 (RSVCE) and Ga 1:6-10 (RSVCE):

There Is No Other Gospel
6 I am astonished that you are so quickly deserting him who called you in the grace of Christ and turning to a different gospel—[a] 7 not that there is another gospel, but there are some who trouble you and want to pervert the gospel of Christ. 8 But even if we, or an angel from heaven, should preach to you a gospel contrary to that which we preached to you, let him be accursed. 9 As we have said before, so now I say again, If any one is preaching to you a gospel contrary to that which you received, let him be accursed.

Footnotes:
a. 1.6 After the greeting there is no commendation, as was usual, but rather strong rebuke.

PS From the mind of a convert to Catholicism:

Take a large view of the faith of Christians during the centuries before Constantine established their religion. Is there any family likeness in it to Protestantism? Look at it as existing during that period in different countries, and is it not one and the same, and a reiteration of itself, as well as singularly unlike Reformed Christianity? Hermas with his visions, Ignatius with his dogmatism, Irenaeus with his praise of tradition and of the Roman See, Clement with his allegory and mysticism, Cyprian with his 'Out of the Church is no salvation', and Methodius with his praise of virginity, all of them writers between the first and fourth centuries, and witnesses of faith of Rome, Africa, Gaul, Asia Minor, Syria, and Egypt, certainly do not represent the opinions of Luther and Calvin. They stretch over the whole of Christendom; they are consistent with each other; they coalesce into one religion; but it is not religion of the Reformation. - The Catholic Church is fundamentally unchanged. H.S. I, 402-3 | 8. The Church, Visible and Invisible | The Mind of [Bl.] Cardinal Newman, from his own writings | Compiled by Charles Stephen Dessain | Catholic Truth Society.

Being a hypothetical question, you're bound to get a lot of personal opinion. I'll give it my best shot though.

In a nutshell, Catholics (as far as personal experience and research) are told they must obey the Pope because of his responsibility to serve and protect the Church and it's people. So if a Pope were to snap and go full-on heretic, some would say they are obligated to follow what the Heretic Pope was teaching.

Luckily, Pope or not, we are given information on what should be done (and what will be done) with people who teach heresy and warp God's Word.

Gal 1:8 But though we, or an angel from heaven, preach any other gospel unto you than that which we have preached unto you, let him be accursed.

Titus 3:10 As for a person who stirs up division, after warning him once and then twice, have nothing more to do with him, knowing that such a person is warped and sinful; he is self-condemned.

As well as

Proverb 30:6 Add thou not unto His words, lest He reprove thee, and thou be found a liar.

We see from the very very few examples provided above the ultimate price for heresy. Revelation gives us an excellent warning on adding or taking from the Word, and many dispute whether it targets Revelation itself or the entire Bible. Personally, I believe the latter, as God would have knowledge (being All-Knowing) that His Word would have been formed into one book.

Either you are removed from the Church as a whole, and/or you are rebuked by the Lord Himself. I know this question is hypothetical, so I answered it best I could within the question itself.

Pope Honorius was condemned as a heretic by Pope Saint Leo II, an ecumenical council and subsequent popes affirmed the anathemas. The historical record therefore shows that a pope can fall into heresy. Pope Paul IV taught that if a pope falls into heresy he loses the papal office. He said in order to be pope a man must be Catholic. If he ceases to be Catholic by becoming a heretic he is no longer pope. A man can be a bachelor or married. He can go from one condition to the other, but not be both simultaneously. Many Catholics today believe the Church teaches the pope can not fall into heresy. This false as shown by the historical record. If the pope falls into heresy the Apostolic see is empty or vacant. Papal infallibility is an entirely different issue.

Heretics lose their office in the Church.

Pope Paul IV, Bull Cum ex Apostolatus Officio, Feb. 15, 1559: “1… Remembering also that, where danger is greater, it must more fully and more diligently be counteracted, We have been concerned lest false prophets or others, even if they have only secular jurisdiction, should wretchedly ensnare the souls of the simple, and drag with them into perdition, destruction and damnation countless peoples committed to their care and rule, either in spiritual or in temporal matters; and We have been concerned also lest it may befall Us to see the abomination of desolation, which was spoken of by the prophet Daniel, in the holy place. In view of this, Our desire has been to fulfill our Pastoral duty, insofar as, with the help of God, We are able, so as to arrest the foxes who are occupying themselves in the destruction of the vineyard of the Lord and to keep the wolves from the sheepfolds, lest We seem to be dumb watchdogs that cannot bark and lest We perish with the wicked husbandman and be compared with the hireling…

1. In addition, [by this Our Constitution, which is to remain valid in perpetuity We enact, determine, decree and define:-] that if ever at any time it shall appear that any Bishop, even if he be acting as an Archbishop, Patriarch or Primate; or any Cardinal of the aforesaid Roman Church, or, as has already been mentioned, any legate, or even the Roman Pontiff, prior to his promotion or his elevation as Cardinal or Roman Pontiff, has deviated from the Catholic Faith or fallen into some heresy: (i) the promotion or elevation, even if it shall have been uncontested and by the unanimous assent of all the Cardinals, shall be null, void and worthless;

Like Affable Geek states...this is more historically concrete than one might think.

I think the basic 2 questions you are getting at (someone correct me if I'm wrong) are:

Question 1 - What would (should) happen if a Roman Pope promulgates, decrees, or endorses a heretical doctrine?

Answer - He would be anathametized as an heretic in an Ecumenical Council:

Pope Honorius I of Rome was the Pope of the Church of Rome from 625 to 638. While successful in missionary and administrative activities he is remembered for his condemnation as a heretic at the Sixth Ecumenical Council of 680/681, after his death.

It was in his support of Patriarch Sergius I during the Monothelite controversy that Pope Honorius gained his notoriety in history. About the year 634, Patr. Sergius I raised, in a letter to Honorius, the use of the expression "one operation/one will" in an attempt to reconcile the differences between the Orthodox and the Monophysites following the condemnation of Monophysitism at the Fourth Ecumenical Council in Chalcedon. The term was raised when emperor Heraclius made use of the expression in refuting the Monophysites during a visit to Armenia. Its use was questioned and then referred to Patr. Sergius who, not ready to make a decision, referred the question to Pope Honorius. Honorius, in his reply to Sergius, while concurring on the questionability of the expression, and also of "two operations" as being Nestorian, left his explanation concerning Monothelitism unclear and sounding as if he endorsed Monophysitism while giving support to Sergius...

...[T]o settle the issue, in 680, the Sixth Ecumenical Council met in Constantinople under the auspices of Eastern Roman emperor Constantine IV. The council condemned Monothelitism and, after deliberating over the doctrinal letters of Sergius and Honorius that were found quite foreign to the apostolic doctrines, anathematized as heretics Honorius and Sergius as well as Cyrus of Alexandria, Paul II, Peter of Constantinople, and Theodore of Pharan. (Orthodoxwiki)

Question 2: Are Catholics required to accept whatever the Pope promulgates, decrees, or endorses as a divinely revealed dogma of Truth no matter what?

Answer - Yes...according to Vatican I at least:

Wherefore we teach and declare that, by divine ordinance, the Roman Church possesses a pre-eminence of ordinary power over every other Church, and that this jurisdictional power of the Roman Pontiff is both episcopal and immediate. Both clergy and faithful, of whatever rite and dignity, both singly and collectively, are bound to submit to this power by the duty of hierarchical subordination and true obedience, and this not only in matters concerning faith and morals, but also in those which regard the discipline and government of the Church throughout the world...

...[T]his is the teaching of the Catholic truth, and no one can depart from it without endangering his faith and salvation...

...[S]ince the Roman Pontiff, by the divine right of the apostolic primacy, governs the whole Church, we likewise teach and declare that he is the supreme judge of the faithful, and that in all cases which fall under ecclesiastical jurisdiction recourse may be had to his judgment. The sentence of the Apostolic See (than which there is no higher authority) is not subject to revision by anyone, nor may anyone lawfully pass judgment thereupon. And so they stray from the genuine path of truth who maintain that it is lawful to appeal from the judgments of the Roman pontiffs to an ecumenical council as if this were an authority superior to the Roman Pontiff...

...[T]herefore, faithfully adhering to the tradition received from the beginning of the christian faith, to the glory of God our savior, for the exaltation of the Catholic religion and for the salvation of the christian people, with the approval of the Sacred Council, we teach and define as a divinely revealed dogma that when the Roman Pontiff speaks EX CATHEDRA, that is, when, in the exercise of his office as shepherd and teacher of all Christians, in virtue of his supreme apostolic authority, he defines a doctrine concerning faith or morals to be held by the whole Church, he possesses, by the divine assistance promised to him in blessed Peter, that infallibility which the divine Redeemer willed his Church to enjoy in defining doctrine concerning faith or morals. Therefore, such definitions of the Roman Pontiff are of themselves, and not by the consent of the Church, irreformable. (Pastor Aeternus)

This question brings attention to the fact that the Roman Catholic Church perches upon this very doctrine, namely, Papal Infallibility. In other words, it is this doctrine that separates the Church from all other Churches - especially the Eastern Churches.

Rome claims that whenever former Popes made heretical statements or accepted heretical doctrines...they were not using all of the required ingredients for being infallible.

How would a situation similar to that of Pope Honorius' be explained or justified by Rome in the future?

Not sure...

This is not completely hypothetical.

In the mid-300s, Pope Liberius may have signed a statement supporting what later became known as Arian heresy. That he did so in exile and possibly under torture is mitigation, and the story is complex, but it is a more concrete version of this question. In any event, there was clear pressure from the Emperor that came into play.

A similar situation occurred in the 600s with the anathemazation of Pope Honorious I in regards to Christ's nature and monothelitism. There were questions about whether or not the letter was spoken ex cathedra, and ultimately, this became a big discussion point when the dogma of papal infallability was promulgated.

The answer in both cases is basically that the church, having later repudiated his position, went to great lengths to say that the statement was not made under a condition in which papal infallibility is present. This article by Carroll explains the exact nuances of Liberius. Vatcan I in the 1870's, formulating the doctrine of papal infallability said that Honorious never spoke ex cathedra.

This basically would suggest that a mitigated sedevacantism is a logical possibility, but that it is ultimately a “nuclear weapon” option that must be avoided at all costs.

Most likely, any other such heresy later pronounced by the Magisterium would need to prove a similar circumstance.