Could the Pope be removed from office for lapses in judgment?

No. As Jesus notes, you aren't taken from an office because you are a hypocrite or don't live according to your own teaching which it is your duty to teach:

Matthew 23:1-4 (DRB) Then Jesus spoke to the multitudes and to his disciples, 2 Saying: The scribes and the Pharisees have sitten on the chair of Moses. 3 All things therefore whatsoever they shall say to you, observe and do: but according to their works do ye not; for they say, and do not. 4 For they bind heavy and insupportable burdens, and lay them on men's shoulders; but with a finger of their own they will not move them.

They still have the episcopal power of 'binding and loosing'¹ despite their hypocrisy. If by the question "What would happen if the Pope were to abuse his power?" you mean he could use his legitimate authority illegitimately or illicitly, as the Pharisees and Scribes, then of course this is possible. But if you mean, 'What would happen if the Pope were to abuse infalliblity' as though it were some superpower, and not simply Christ's promise of protection against such specific kind of abuses of universal authority, then you are asking the Catholic, 'What if Jesus lied,' which is of course a non-question for Catholics and most anyone who identifies as Christian.

The Catholic's obligation is to obey proper authorities (any bishop, and to a lesser extent their helpers the priests) in anything that is not contrary to morality/the Faith, i.e., immoral. If a bishop asks you to do anything that doesn't harm your salvation or isn't equivalent to asking you to deny or otherwise distort a truth of the Faith, you are supposed to, indeed obliged to per Jesus' words. But God gave us reason. In the name of obedience and of humility, both key characteristics of Christ, we should respect the authority for its own sake, since it comes directly from God, even if its channel, a bishop, has moved it here or there. See 1 Peter 2:18-23.

Since Catholics are obliged to obey the bishop in all things except for sin, and since no one human being except Jesus Christ knows the hearts and minds of men, there is no possibility for such a case to be detected even if it theoretically existed.

There is no limit on 'how destructive' an administrative or disciplinary decision is.

For example, "The Pope instructs Catholics to practice open idolatry" as an example is extreme, but for a real life example, a Pope once taught it's OK to baptize literally in the name of Christ, instead of the formula given by Christ Himself which is meant by 'in the name of Christ' (i.e. 'into the name of Christ' in contradistinction to the baptism of John, not literally 'I baptize you in the name of Christ') and used for all time in the Church. But not within what we would call the kind of teaching to which infallibility applies or has ever been said to apply. It simply didn't constitute such. Destructive? Very possibly, quite potentially, yes. But not to be helped by an uncatholic (and actually anti-catholic) doctrine of absolute/universally infallible popes. It's just bad judgement. A mistake.

There is simply no doctrine of perfect judgement 24/7 for all popes in all times. Infalliblity is a very specific, and very 'meta,' doctrine. As mentioned before, the only real reason infallibility applies to the Pope, and hinges on the Pope ultimately (in the case of an Ecumenical Council being ratified) is because he is the only bishop with universal jurisdiction/authority, and therefore the only one to whom such a protection against binding, universally, some false doctrine, could be given or makes sense for, and indeed is necessary for (otherwise the gates of Hell could indeed prevail against the Church, if a fallible decision on interpretation of the Faith was made the universal Faith of His Church and thus all newcomers to Christ after its definition).

You can't 'impeach' a Pope for changing something which doesn't pertain to the Faith. That's his right. It's your obligation to obey: IF it isn't equivalent to asking you to abandon some part of the Faith or morality in the process, in which case the obligation becomes one of refusal to obey a sin.

Then there is the fact that the Popes and other bishops will all be judged according to the morality and reasonablness of their decisions, and how that impacted the souls under their care (Heb 13:17; Jas 3:1). But this is distinct from here on earth, while alive, where unless sinful, all their decisions are theirs to make, and the faithful to obey.

^Footnotes

^{1 Mt 16:19; Mt 18:18; Cf. Jewish Encyclopedia, Binding and Loosing.}