How do the 3 Protestant traditions deal with the two types of sin mentioned in 1 John 5:16-17?

TL;DR: it's not the sin itself which is "unto death" or "not unto death", but the attitude of the sinner that makes it so.

We know that we are all guilty of sin:

For all have sinned, and come short of the glory of God — Romans 3:23

We know that sin leads to death:

For the wages of sin is death; but the gift of God is eternal life through Jesus Christ our Lord. — Romans 6:23

Looking at 1 John, we see that sin is simply disobeying God's commandments, and it is our duty to refrain from sin:

Whosoever committeth sin transgresseth also the law: for sin is the transgression of the law. — 1 John 3:4

For this is the love of God, that we keep his commandments: and his commandments are not grievous. — 1 John 5:3

There is no distinction between "mortal sins" and "venal sins"; either one obeys God or one sins.

There is however, a distinction in how we sin.

If any man see his brother sin a sin which is not unto death, he shall ask, and he shall give him life for them that sin not unto death. There is a sin unto death: I do not say that he shall pray for it. All unrighteousness is sin: and there is a sin not unto death. — 1 John 5:16–17

People can sin during a moment of weakness. They will feel guilt and will regret what they've done. A fellow Christian can help someone in that situation, praying for them and helping with their repentance (which is mostly about being able to forgive oneself after changing one's attitude and resolving not to do it again).

But people can also sin deliberately, knowing but not caring that it is wrong. Without guilt and repentance they cannot be forgiven, and any prayers for them will be in vain. (This is the equivalent of Matthew 12:31's unpardonable sin; as long as we reject God, we cannot be forgiven.)

Here is a specific non-Catholic teaching:

It may seem that the statement "all unrighteousness is sin" in verse 17 contradicts "there is sin not leading to death," especially in light of the fact that Romans 6:23 clearly teaches that all sin results in death. However, we have to study all of the scriptures about sin and salvation together. Jesus taught that sin would result in death, but He also explained how a sinner could escape death. Using the tragic death of 18 people killed in a building collapse, He explained in Luke 13:3-5 that everyone will perish unless he or she repents.

Any sin can result in death, and it will unless a sinner repents. Conversely, no sin needs to lead to eternal death if a person turns from it, seeks God's forgiveness through Christ's sacrifice and commits to obedience from that point on.

So a sin not leading to death is a sin where the sinner is not hardened and unrepentant.

Our booklet Transforming Your Life: The Process of Conversion gives a very helpful explanation about the process of repentance, forgiveness, receiving the gift of God's Holy Spirit and continuing faithfully in the Christian way of life.

And specifically, this chapter: — What is "a sin not unto death"? | United Church of God

Here is a quote from Dr. Bruce Milne, formerly lecturer in Biblical and Historical Theology at Spurgeon’s College, London. The Foreword to his book is by J.I. Packer. I believe this is still the current Reformed Protestant view:

Recent interpretation sees the sin as essentially Christological. Jesus distinguished between sin against the Spirit and sin ‘against the Son of Man’ (Matthew 12:32 [1]) before his death, resurrection and outpouring of the Spirit at Pentecost. Prior to the first Easter the ‘Son of Man’ was an enigmatic, hidden revelation of God. Failure to recognize Jesus during his earthly mission (e.g. his own family, Mark 3:21[2]), was less serious than deliberately attributing his entire mission, its good works in particular, to Satan – of which the Pharisees were guilty. With Pentecost, the distinction vanishes. Jesus is demonstrated as Son of God with power in resurrection and the gospel of the cross is preached in the power of the Spirit. Rejection of this message and of the Christ it enshrines is rejection of the Spirit who bears testimony to its truth (Hebrews 10:29). This sin is unpardonable if continued in, for it places one beyond the only hope of redemption. John calls it a ‘sin that leads to death’ (1 John 5:16). Source: Dr. Bruce Milne: ‘Know the Truth’ 1982 (Chapter 10, page 109)

[1] Matthew 12:31-32: And so I tell you, every sin and blasphemy will be forgiven men, but the blasphemy against the Spirit will not be forgiven. Anyone who speaks a word against the Son of Man will be forgiven, but anyone who speaks against the Holy Spirit will not be forgiven, either in this age or in the age to come.

[2] Mark 3:21: When his family heard about this, they went to take charge of him, for they said “He is out of his mind.” (This was when the teachers of the law accused Jesus of being possessed by Beelzebub – see verses 22-29.)

I found an exposition on 1 John 5:16-18 by Charles Spurgeon, a Reformed Protestant minister, at the end of a sermon he delivered in June 1911. He explains how it is not possible for a believer who is born of God to commit the sin that leads unto death:

All unrighteousness is sin: and there is a sin not unto death. We know that whosoever is born of God sinneth not; but he that is begotten of God keepeth himself, and that wicked one toucheth him not. He who has committed the sin which is unto death has no desire for forgiveness, he will never repent, he will never seek faith in Christ, but he will continue hardened and unbelieving. He will henceforth never be the subject of holy influences, for he has crossed over into that dark region of despair where hope and mercy never come.

Perhaps some of you think that you have committed that unpardonable sin and are at this moment grieving over it. If so, it is clear that you cannot have committed that sin, or else you could not grieve over it. If you have any fear concerning it, you have not committed that sin which is unto death, for even fear is a sign of life.

Whoever repents of sin and trusts in Jesus Christ is freely and fully forgiven, therefore it is clear that he has not committed a sin which will not be forgiven. There is much in this passage to make us prayerful and watchful, but there is nothing here to make a single troubled heart feel anything like despair. He that is born again, born from above, can never commit this unpardonable sin. He is kept from it—“that wicked one” cannot even touch him, for he is preserved by sovereign grace against this dreadful damage to his soul...

So, “all unrighteousness is sin,” and you are warned to keep clear of it. “There is a sin unto death,” but you are not told what that sin is on purpose that you may, by the grace of God, keep clear of sin altogether. (Page 9) http://www.spurgeongems.org/sermon/chs3252.pdf

From the above it is clear that a Christian who has repented of sin and is born from above can not commit the “sin unto death”. In another sermon (October 1865) Charles Spurgeon had these words of comfort to believers who were afraid that they had committed the unpardonable sin, the sin that leads to death:

“Yes,” says one, “but I believe I have committed the unpardonable sin.” My dear brother, I believe you have not, but I want you to call one thing to remembrance, and that is that the unpardonable sin is a sin which is unto death. Now a sin which is unto death means a sin which brings death on the conscience. The man who commits it never has any conscience afterwards—he is dead there. Now, you have some feeling; you have enough life to wish to be saved from sin; you have enough life to long to be washed in the precious blood of Jesus. You have not committed the unpardonable sin, therefore have hope. “All manner of sin and blasphemy shall be forgiven unto men.” “But,” you reply, “Oh, I cannot repent! My heart is so hard.” Call to remembrance that Jesus Christ is exalted to give repentance and remission of sins, and you may come to Him to get repentance, and need not bring it to Him. Come without any repentance, and ask Him to give it to you, and He will give it. Rest assured there is no fear whatever that if the soul seeks softness and tenderness, it has that softness and tenderness in a measure even now, and will have it to the fullest extent before long. “Oh, but,” you say, “I have a general unfitness and incapacity for being saved.” Then, dear friend, I want you to call this to remembrance, that Jesus Christ has a general fitness and a general capacity for saving sinners. I do not know what you need, but I do know Christ has it. I do not know the full extent of your disease, but I do know Christ is the Physician who can cure it. I do not know how hard, and stubborn, and stolid, and ignorant, and blind, and dead your nature may be, but I do know that “Christ is able to save unto the uttermost those who come unto God by Him.

I have laboured to speak comfortable words and words in season, and I have tried to speak them in homely language, too. But, O Comforter, what can we do without You? YOU must cheer our sadness. To comfort souls is God’s own work! Let us conclude, then, with the words of the Saviour’s promise, “If I go away, I will send you another Comforter, who shall abide with you forever.” And let our prayer be that He would abide with us to His own glory and to our comfort forevermore. Amen.” (Page 8) https://www.spurgeongems.org/sermon/chs654.pdf

Since the sin that leads to death is the sin of rejection of Christ (which means the rejection of the Holy Spirit) then it follows that all who are born again and who are indwelt with the Holy Spirit have not committed the sin that leads to spiritual death and ultimately, eternal death.

EDIT:You may find this answer given by Nathaniel (is protesting) informative: How do Protestants understand the "unforgivable" sin?