In modern times, what would happen to a Roman Catholic monk if he were to violate his monastic vows?

What is a vow?

A vow is defined as a promise made to God. The promise is binding, and so differs from a simple resolution which is a present purpose to do or omit certain things in the future.

What happens when one breaks their vow?

Unlike the simple breach of a promise made to a man, a failure to give to God what has been promised Him is a matter of importance, a very serious offence. ...committing sacrilege in the widest sense of the word.

They have broken it, what do they do now?

Unless the vow is annulled or one is dispensed from the obligation to fulfill it, after seeking God's forgiveness in the Sacrament of Reconciliation or Penance, the person who made the vow is still under the obligation to fulfill it.

Please see:

• Vows | New Advent
• Promises and vows | Catechism of the Catholic Church, 2101-2103

Since the vow is between the individual and God, I do not see the why and the how of the involvement the individual's monastic order or what the relevance of modern times would be upon the breaking of the vow.

Example of a vow and its fulfillment in the Old Testament: Jephthah’s Vow and Jephthah’s Daughter.

To answer the O.P.'s question directly, the maximum penalty for breaking monastic vows is essentially expulsion from the monastery (which entails a dispensation from the vows).

For the benefit of readers, in the Catholic Church, “monks” are those men who live in a monastic community, which entails a certain separation from the world and a dedication to prayer and work—the so called “contemplative” vocation—as opposed to active apostolate. The equivalent for women are called “nuns.” For example, the Benedictines, Cistercians, and Trappists are orders of monks and nuns. There are, however, other entities in the Church that share with monasticism a special consecration to God, but entail a more active ministry; these groups include religious congregations (for example, the Jesuits), secular institutes of consecrated life, and societies of apostolic life (for example, the Paulist Fathers).

As a practical matter, however, membership in all these groups entails certain promises or vows, which are dealt with in essentially the same way.

By “monastic vows” the Church means the solemn vows of poverty, chastity, and obedience that monks make when they formally enter their monastery. The Code of Canon Law (CIC) describes vows as follows:

A vow, that is, a deliberate and free promise made to God about a possible and better good, must be fulfilled by reason of the virtue of religion (Can. 1191 §1).

In the case of monks, they take public and solemn vows, as described by Can. 1192:

§1. A vow is public if a legitimate superior accepts it in the name of the Church; otherwise, it is private.

§2. A vow is solemn if the Church has recognized it as such [as in the case of monastic orders]; otherwise, it is simple.

The taking of these vows constitutes what is called religious profession, described in Canon Law as follows:

By religious profession, members assume the observance of the three evangelical counsels [i.e., poverty, chastity, and obedience] by public vow, are consecrated to God through the ministry of the Church, and are incorporated into the institute with the rights and duties defined by law (Can. 654).

Note that religious profession takes places in two stages: temporary profession and perpetual profession:

Temporary profession is to be made for a period defined in proper law; it is not to be less than three years nor longer than six (Can. 655).

When the period for which profession was made has elapsed, a religious [here, referring to anyone who takes these kinds of vows: monks, nuns, members of religious orders, and so on] who freely petitions and is judged suitable is to be admitted to renewal of profession or to perpetual profession; otherwise, the religious is to depart (Can. 657 §1.).

What happens to a religious or consecrated person when he commits a serious fault against his vows depends on whether he has only made his temporary profession, and also on the seriousness of the offense.

The general practice nowadays, in the face of such a violation, would be simply to dismiss a monk or religious who has made only his temporary profession, or else simply not admit him to perpetual profession. (In the former case, the superior general is authorized to dispense his subjects from their temporary vows; see Can. 688.)

For monks and religious who have made their perpetual profession, there is generally more of an attempt to try to rectify the situation before proceeding to dismissal, because only the gravest of causes should lead to an indult of departure from an institute (Can. 691 §1).

In very extreme cases (spelled out in Canons 694-704), monks and religious can be expelled from their institute.

Offenses that lead to dismissal by their very commission

• Notoriously defecting from the Catholic Faith (Can. 694).
• Attempting marriage (Can. 694).

Offenses that generally require dismissal

• Other grave offenses against the sixth commandment, such as concubinage, abuse of minors, and so on (Can. 1395).

• Committing homicide (Can. 1397).

• Procuring [i.e., directly helping someone to have] an abortion (Can. 1398).

Other possible reasons for dismissal

Canon 696 §1 details other reasons for dismissal:

A member can also be dismissed for other causes provided that they are grave, external, imputable, and juridically proven such as: habitual neglect of the obligations of consecrated life; repeated violations of the sacred bonds; stubborn disobedience to the legitimate prescripts of superiors in a grave matter; grave scandal arising from the culpable behavior of the member; stubborn upholding or diffusion of doctrines condemned by the magisterium of the Church; public adherence to ideologies infected by materialism or atheism; the illegitimate absence mentioned in ⇒ can. 665, §2, lasting six months; other causes of similar gravity which the proper law of the institute may determine.

In summary, monks (and other religious, consecrated persons, and members of societies of apostolic life) face in essence a maximum penalty of expulsion from their institute or society (and, if they are priests or deacons, also reduction to the lay state) for grave offenses against their vows.

They would not be obliged to copy manuscripts as a penance, and would certainly not be subject to cruel punishments (such as being walled up). If such things occurred in the past, they are now entirely forbidden by Canon Law.

There are several different sets of rules for different communities of monks; and the specific penalties for the monks are dependent on the rule.

Most communities of monks or nuns are governed by one of two sets of rules: the Rule of St. Augustine (of Hippo), and the Rule of St. Benedict (of Nursia). There are also governing documents such as the Statutes of the Carthusians, but these are the two primary documents. Since this particular monk was a Benedictine, let's look at the penalties of the Benedictine Rule.

St. Benedict of Nursia, who lived in the early sixth century, developed a rule by which the monks under his care could live. The Order of St. Benedict has a translation of the Rule online, and the Rule has several sections dealing with how monks are to be disciplined. If a monk is a troublemaker, his superior (probably a prior) is enjoined to discuss the problem with him privately. If the behavior is still a problem, the superior will talk privately with him again. If that doesn't help, he will be called out in front of the community and rebuked publicly. If even that doesn't solve the problem, the monk will be excommunicated: that is, excluded from the community (see Chapter 23 of the Rule).

St. Benedict envisioned two kinds of excommunication: for less serious and more serious faults. For less serious faults, the monk (or sister; the same Rule applies to both male and female communities)

shall take [his] food alone after the community meal, so that if they eat at the sixth hour, for instance, that [brother] shall eat at the ninth, while if they eat at the ninth hour [he] shall eat in the evening, until by a suitable satisfaction [he] obtains pardon.

("Rule of St. Benedict", Chapter 24)

If a more serious fault is involved, the monk is excluded both from the common meal time and from common worship time. Further, no one is allowed to talk to him or even approach him:

Let none of the brethren join him either for company or for conversation. Let him be alone at the work assigned him, abiding in penitential sorrow and pondering that terrible sentence of the Apostle where he says that a man of that kind is handed over for the destruction of the flesh, that the spirit may be saved in the day of the Lord (1 Cor. 5:5). Let him take his meals alone in the measure and at the hour which the Abbot shall consider suitable for him.

("Rule", Chapter 25)

The abbot of the monastery is enjoined to send spiritual help to the monk, in the hope of helping him and solving the problem. He is also entitled to use corporal punishment ("the rod") in severe cases. I highly doubt this punishment is used any longer, though I have no evidence either way.

The (second to) last resort is to have the community pray for the afflicted brother, in the hope that the Lord will help where the Abbot cannot; but if even this doesn't work, the abbot is told to expel the monk from the monastery:

But if [he] is not healed even in this way, then let the [Abbot] use the knife of amputation, according to the Apostle's words, "Expel the evil one from your midst" (1 Cor. 5:13), and again, "If the faithless one departs, let [him] depart" (1 Cor. 7:15) lest one diseased sheep contaminate the whole flock.

("Rule", Chapter 28)

I don't see anything so harsh as walling up a monk alive.