Prior to 1963 did a British nobleman ever disclaim or renounce his title? What would have happened if someone had tried?

Just to add one more interesting example to the already very comprehensive answers:-

In 1660 (so after the resolution of the House of Lords referred to in Ecu91's answer), Robert Villiers surrendered the Earldom of Buckingham and Viscountcy of Purbeck to the King in return for a fine. He had attempted to renounce the title unilaterally after the death of the previous Viscount Purbeck in 1658, as he was viewed as illegitimate. However, two years later he was ordered to stand trial as Viscount Purbeck before the House of Lords - apparently due to "remarks he made at the time of Charles I’s trial":

Although he at first stood on his privilege as a Member of the Commons, he was brought to the bar, and charged with treasonable and atheistical words in 1649. Danvers did not deny the words, but insisted that he was not a peer, or, if he was, that with five small children and £5,000 of debts, his estate, reduced to under £1,000 p.a., was inadequate to support the title. He was discharged on £10,000 bail on 27 July. Meanwhile in the Lower House, the elections committee was asked to recommend whether a new writ should be issued for Malmesbury, but no report was made and no writ issued, although Danvers was alleged to have said that ‘he only sits in the present Parliament for formality’s sake, and never goes, for he can do his country no good’. During Michaelmas term he levied a fine to the King for all his titles, and on 27 Dec. he took the oaths to the restored monarchy and entered into a further bond of £5,000 to attempt nothing against it.
_{The History of Parliament: the House of Commons 1660-1690, ed. B.D. Henning, 1983}

After Villiers' death, in response to a petition from his son in 1678, the House of Lords resolved and adjudged that "no Fine now levied, or at any Time hereafter to be levied, to the King, can bar such Title of Honour, or the Right of any Person claiming such Title under him that levied, or shall levy, such Fine". (HLJ Vol. 13: 18 June 1678)

It has not happened for many centuries, but it did happen, principally in order to bring about a change in the title rather than to surrender it forever.

Roger le Bygod, fifth Earl of Norfolk (1245-1306) surrendered his title to Edward I in 1302. It was regranted to him and the heirs of his body. Since he had no children, the effect was to disinherit his younger brother. After Roger's death, the earldom was recreated for Edward II's brother Thomas.

John Neville (1431-1471) was created Earl of Northumberland in 1465, but in 1469 the letters patent were formally cancelled; he was created Marquis of Montague instead. The earldom went to Henry Percy, the son of the third Earl, who had been stripped of the title by attainder.

Charles Brandon (1484-1545) was created Viscount Lisle in anticipation of a marriage to Elizabeth Grey, Viscountess Lisle. Because he married Henry VIII's sister Mary instead, he surrendered the viscountcy (and was created Duke of Suffolk in the mean time).

These examples come from Peerage Law in England by Francis Beaufort Palmer (1907), which contains various other examples as well.

In 1641, a dispute over the barony of Grey of Ruthyn led the House of Lords to resolve unanimously:

That no Peer of this Realm can drown or extinguish his Honour (but that it descend to his Descendants), neither by Surrender, Grant, Fine, nor any other Conveyance to the King.

Since then, this has been the firm legal position. A deprivation of title can come about by Act of Parliament, but only by such an Act - as for example the Titles Deprivation Act 1917 which stripped peerages from people who fought against the UK in the First World War.

The closest case is probably that of Thomas Moreton FitzHardinge Berkeley, born 1796, de jure 6th Earl of Berkeley.

Upon the death of his father (Frederick Berkeley, the 5th Earl) in 1810, Thomas' four older brothers born before the 1795 marriage of his parents were ruled illegitimate (and thus ineligible to inherit the title) by the Lords' Committee on Privilege. This made Thomas the de jure Earl despite being a minor. While still a minor "his oldest brother and mother ran things"; Upon majority he signed all responsibility for the earldom over to his oldest brother and declined to exercise any privilege related to the title. The title then went dormant until his death childless in 1882, and the accession of the 7th Earl. (The state of dormancy, I believe, officially occurs upon failure to attend Parliament in response to a writ of summons.)

So in regards "What would have happened if someone had tried?": Life went on. Parliament ruled the earldom dormant until the disclaimer died decades later; and upon that death the title and privileges were inherited by the closest (legitimate) male relative.

Absent a means to legally disclaim the title, this is likely the closest one could come to doing so.