Can a priest add his own prayers to the Mass after the dismissal?

I think the answer is partly in this communication by the Office for the Liturgical Celebrations of the Supreme Pontiff, titled "The Priest in the Concluding Rites of the Mass", issued in 2010.

First, let us recall the Rite in the Ordinary Form. As the text says, this is:

"The concluding rites consist of 1) Brief announcements, if they are necessary; 2) The priest's greeting and blessing, which on certain days and occasions is enriched and expressed in the prayer over the People or another more solemn formula; 3) The dismissal of the people by the deacon or the priest, so that each may go out to do good works, praising and blessing God; 4) The kissing of the altar by the priest and the deacon, followed by a profound bow to the altar by the priest, the deacon, and the other ministers"

Thus, the final blessing could be expanded with the Prayer over the People (e.g see prayers for Lent here), or other "more solemn" formula.

The same text then goes on to present the goal of such Rite, where it is clear that this is a brief one. Some quotes:

Of what has been said, it turns out that the two columns that sustain the Rites of Conclusion of the Mass are the blessing and the dismissal. ... From these biblical elements stems the Christian liturgical use of blessing, which has the meaning of "asking God for his gifts on his creatures, and of thanking him for the gifts already received". ...

And:

In the Rites of Conclusion of the Holy Mass the priest is still carrying out a priestly task, namely, of mediation between God and the faithful people. It is not only a question of greeting one another and agreeing to meet the next time, remembering the commitment perhaps during the week. The priest here invokes on the people the divine blessing, while in the name of the people he thanks God for the gifts already received by his kindness. Here also he acts "in persona Christi." Because of this, he does not say in the plural "may the omnipotent God bless us," or "the Mass is ended, let us go in peace." He speaks in the name of the Person of Christ and as minister of the Church, because of this he imparts the blessing, while invoking it, and he sends the faithful to the daily mission of life: "may God bless you" and "Go in peace." Through him, Christ and the Church charge the baptized with giving this daily witness of the Gospel.

From the above it is clear that after communion, the final rites are to be brief, and still in a very solemn mood, with the Priest acting as a mediator. Surely the priest has the capacity to insert "another more solemn formula", but this does not mean everything. For example, to insert here a rather long set of prayers like the Rosary or to perform the Stations of the Cross during Lent would, in my view, conflict with the solemnity and purpose of the rite. In fact, in my experience these rather long prayers are usually celebrated before the mass.

Now, when it comes to the Extraordinary Form of the mass, things might be slightly different. For example, it is common in this form to have, as part of the Concluding Rites, some rather long extra prayers. Blessing prayers are longer, and also the first verses of Gospel of John are read (see full prayers here). If the mass is a Low Mass, some final prayers to Mary and the Archangel Michael (known as Leonine prayers, and found here) can be added once the mass is finished. Whether more prayers can be added or not, I refer you to the 1962 missal (see in particular page 66, although text is in Latin).

Does a priest have the right to prolong Mass with a novena? This is how the original question was worded before it was edited. I would like to address this specific nuance of the question at this moment.

After the dismissal the Mass is over and one is free to leave or stay!

Novenas attached to the celebration of the Mass should definitely be avoided, when improvised by a pastor of some parish church. There have been occasions when pastors have added a novena immediately after the Gospel or homily. When pastors add a novena or other prayers at the end of the Mass, the faithful should be given the opportunity to leave the church prior to the commencement of such prayers, unless they have been order by proper authority and for a just cause. Some people may have to work immediately after Mass, of which I am in that number.

Distinct from and in harmony with the Liturgy (13)

The objective difference between pious exercises and devotional practices should always be clear in expressions of worship. Hence, the formulae proper to pious exercises should not be commingled with the liturgical actions. Acts of devotion and piety are external to the celebration of the Holy Eucharist, and of the other sacraments.

On the one hand, a superimposing of pious and devotional practices on the Liturgy so as to differentiate their language, rhythm, course, and theological emphasis from those of the corresponding liturgical action, must be avoided, while any form of competition with or opposition to the liturgical actions, where such exists, must also be resolved. Thus, precedence must always be given to Sunday, Solemnities, and to the liturgical seasons and days.

Since, on the other [hand], pious practices must conserve their proper style, simplicity and language, attempts to impose forms of "liturgical celebration" on them are always to be avoided. - Directory on Popular Piety and the Liturgy (Vatican City)

Therefore it is incorrect to mingle any devotional exercise such as a novena or non-liturgical litanies within the context of the Mass; this mixing respects neither the nature of the Eucharistic celebration nor the essence of the pious exercise. Novenas or non-liturgical litanies may, however, be recited immediately before or after Mass. Thus one is free to follow the specific devotions in question or not, as they are not part of the Mass.

The Prayers after the Mass of the Extraordinary Rite of the Mass are the norm that has been instituted uniquely by past Supreme Pontiffs and make up a part of that Rite, yet are also said after the Mass has ended.

As for the Universal Prayers (Prayers of the Faithful) bishops enjoy a greater flexibility with the Sacred Liturgy within their Diocese. The local ordinary (bishop) may for example ask all the parishes of his diocese to pray for a specific intention at the "Prayers of the Faithful". In addition, a priest may add an intention to the "Prayer of the Faithful" which as the presider of the liturgy (Mass) he may also do before the concluding prayer.

In a letter to all parishioners in his diocese, Bishop Jenky asked that the powerful St. Michael’s prayer be included in all General Intercessions during each Sunday Mass. In his letter he writes, “The HHS Mandate is a bigoted and blatant attack on the First Amendment rights of every believing Catholic. I ask that every parish, school, hospital, Newman Center, and religious house in this Diocese insert the prayer to St. Michael the Archangel into the Sunday General Intercessions just before their concluding prayer. It is God’s invincible Archangel who commands the heavenly hosts, and it is the enemies of God who will ultimately be defeated. This prayer should be announced as: A Prayer to Saint Michael the Archangel for the Freedom of the Catholic Church in America.” - Time to Bring Back St. Michael’s Prayer

Another bishop has asked for prayers to defeat evil in the world at General Intercessions of the Faithful at all the Masses of his diocese.

General Intercession petitions

And so, I have asked for a particular petition or petitions to be added into the General Intercessions at all Sunday and Solemnity Masses, and for the petitions to be offered always to the Father and through the Son, but also with a particular prayer to St. Michael the Archangel, who in Scripture is called “the great prince,” and “guardian of [God’s] people (Daniel 12:1).” And who, through Scripture, we see as the great warrior angel who defeats Satan himself and drives him into hell.

Thus, as a local Church, we will intensify our prayers for the particular intention of the defeat of evil in this world and for the triumph of good. - Bishop asks for prayers to defeat evil in world

The bishop of my diocese has asked for the "prayer for vocations" to be said at all Sunday Masses.

A priest may only use the prayer options and formulae that are permitted in the Rubrics, he may not improvise on the rules.