Indisputably, Jansenists considered Calvinists heretics, denied imputation of righteousness, and affirmed Catholic doctrine on all non-soteriological matters. Jansenists additionally claimed that they (in alleged contrast to Calvinism) affirmed man's theoretical freedom not to sin and man's cooperation with grace. Some Jansenists also denied eternal security. Let's take a further look at each of these things.

Undisputed differences with Calvinism

Before we address the meat of the question, let's examine the undisputed differences between Jansenism and Calvinism:

1. All non-soteriological issues. In God Owes Us Nothing, page 91, Leszek Kolakowski writes, "In such important matters as the interpretation of the Eucharist and of penance, the sacrament of priesthood (to which they attached special weight), the apostolic succession, the cult of saints and of the Holy Virgin (prominent in Saint-Cyran), and the very concept of the Church, including the hierarchy and the papacy, they were emphatically, unequivocally Roman."

2. Imputation of righteousness. In Iustitia Dei, page 284, Alister McGrath writes, "The entire post-Tridentine Catholic tradition (including those otherwise considered heterodox, such as Baianists and Jansenists) continued to regard justification as a process in which man was made righteous, involving the actualisation rather than the imputation of righteousness."

3. The truth or error of Calvinism. As you noted, when the Jansenists were accused of Calvinism, Jansenists did not argue that Calvinism was correct -- they argued that they were not actually Calvinists. This goes back to Jansen himself, who was a proud Catholic and wrote tracts against the prominent Calvinist Gisbert Voet, though it was not on soteriological matters but "on the question of the legitimacy of a Protestant mission in conquered Catholic lands," according to Hans van Ruler in the Dictionary of Seventeenth and Eighteenth-Century Dutch Philosophers.

Alleged differences with Calvinism

After the church condemned "five propositions" attributed to Jansen in his book Augustinus, Jansenists circulated a tract (titled in Latin, Brevissima quinque Popositionum Distinctio, or in French, Distinction Abrégée des Cinq Propositions) which claimed that the five propositions were actually ambiguous and could be interpreted in orthodox or heretical ways. The tract contained three columns. Under each of the five propositions, they explicated two possible heretical interpretations (on one side, the Calvinist interpretation, and on the other, the Semi-Pelagian, which they attributed to the Molinists), and one "orthodox" interpretation, that is, the Jansenist interpretation. Unfortunately, I have been unable to find the tract in English or any summary that goes beyond the facts I've just recounted. But I was able to find other descriptions of Jansenists' claims about their differences with Calvinism.

Nigel Abercrombie's book The Origins of Jansenism is perhaps the most thorough English-language treatment of the movement. The first section discusses Jansen's book Augustinus. On page 151, discussing Augustinus, Volume 3, Book 8, Abercrombie says:

Jansen denies that his system is identical with that of Calvin, and supports his denial with three ancient sophistries: the first, that man could resist either delectatio 'if he would' (si velit), but he never does, because the delectationes determine his will; the second, that man 'co-operates' with grace, but only in the sense that the effect of grace is an action performed by man; the third, that when we sin, the avoidance of sin is 'in our power', in the sense already explained in the third book de Gratia Christi.

Since I don't understand the difference between the first and third "sophistry," and since Abercrombie elsewhere seems to conflate the two, I'll treat them under the same header. In addition to these points, it's worth mentioning the intra-Jansenist debate on eternal security.

Theoretical freedom not to sin

Before we can understand the distinctions Jansen made about theoretical freedom, we have to understand his theology of sin. Essentially, he taught that the will is either directed toward earthly or heavenly pleasures (or "delectations") at all times, and whichever presents the stronger pull in the individual will be victorious. See The Condemnation of Jansenism by George Tiffany:

Jansenius taught, as did Baius, that supernatural grace was part of the very essence of man. Hence, when grace was lost due to original sin, the nature of man was essentially impaired. If man's nature, then, is essentially corrupt, the will is no longer master of its decisions. Its freedom is only freedom from external violence, not freedom from necessity. The will is then incapable of doing good and cannot resist the grace of God. It must always obey the strongest impression or, what Jansenius calls, the "delectatio victrix". This means that we must act according to that which gives us most pleasure. Jansenius took this doctrine from St. Augustine, but the sense of St. Augustine as seen from the context is this: if we put our happiness in virtue or vice, then virtue or vice will be the ruler of our lives. Due to his theory of knowledge by divine illumination, St. Augustine emphasized to a great extent the part of the will in knowledge. Accordingly, illumination and the consequent delectation affected the will in the process of knowledge. Jansenius overemphasized this. He took the words of St. Augustine to mean that the will is hovering between two attractions and that the stronger is always victorious and draws the assent of the will.

Let's revisit Abercrombie's summary of Jansen's thought on this matter from page 151:

Man could resist either delectatio 'if he would' (si velit), but he never does, because the delectationes determine his will. ... When we sin, the avoidance of sin is 'in our power', in the sense already explained in the third book de Gratia Christi.

Here then is how Abercrombie summarizes Jansen's arguments on the subject "in the third book." From page 146:

[Jansen says that] God sometimes commands what is impossible, ... 'not only to infidels, or to "blinded and hardened" men, but also to faithful and just men who will and try to perform them with all the power at their present disposal; and that the grace whereby these precepts might be made possible, is lacking to them.' ... Those who possess [faith and charity] would be able to perform any commandment if their will were set free by efficacious grace for this purpose; they are therefore said to be able to fulfil the commandments 'if they will'.

Cooperation with grace

Later in the book, Abercrombie discusses later Jansenists' defenses of their orthodoxy. On page 216, as he recounts the arguments that the Jansenist Antoine Arnauld's Second Apology gives in response to Bishop Isaac Habert:

Habert had declared (following the Louvain Theses) that Jansen no more admitted human co-operation than did Calvin. Arnauld replies that the accusation is entirely false, and rests upon no authority, and quotes a few short passages of Augustinus which assert human co-operation. For the first time, however, Arnauld's defence is short and undeveloped--for the reason that Jansen, like Calvin himself, admitted indeed a physical co-operation of human activity in the production of the act, but allowed no more.

It appears that Arnauld and Jansen taught that man cooperates with his own sanctification. Calvinists sometimes speak in much the same way, so it would seem that if there's a real difference between them on this point, it's that Jansenism identifies sanctification with justification (along with the rest of Catholicism) and Calvinism asserts that there is an important distinction between the two doctrines. This is a significant difference, but perhaps it is not the same difference that Arnauld thought it was.

Is there eternal security?

In The Religious Origins of the French Revolution, Dale Van Kley recounts an ongoing internecine debate between several Jansenists, including Antoine Arnauld, Laurent Boursier, Jean-Baptiste Le Sesne de Ménilles d'Étemare, and Pierre Nicole. The subject of the debate was the proper balance between fear and confidence in a life of faith. Those who believed that confidence should prevail affirmed perseverance of the saints and assurance of salvation, whereas those on the side of fear denied both doctrines.

Van Kley says writes of Arnauld's denial of eternal security on page 102:

The clinching argument of Arnauld's nine-hundred page polemic against the decisions of Dordrecht was that the Calvinist doctrine of the perseverance of the saints undermined fear and gave rise to a "false assurance" of salvation, proving that the "supposed Reformation of the Calvinists, far from being a work of the Holy Spirit, can only be regarded as the work of a demon."

However, on page 104, the affirmation most Jansenists gave (implicitly or explicitly) to the doctrine comes into focus:

From the thought of God's "supreme majesty," on the one hand, Boursier derived the quasi-Calvinist comfort of being among the elect to whom God vouchsafed the gift of hope while delegating to charity the role of providing some additional "marks" of salvation. And from a consideration of God's "infinite mercy" as displayed in the crucifixion, on the other hand, he also derived confidence that spoke directly to the heart, whose independent testimony in the matter he defended. ... Boursier did not go out of his way to explain the difference between this testimony of the heart and the inner testimony of the Holy Spirit that Arnauld had so excoriated when appealed to by Calvinists. But even if Boursier's and d'Etemare's many qualifications be deemed sufficient to distinguish their position from Calvinist confidence, their emphasis was all on the side of confidence and against fear--a far cry, in any case, from the emphases of Arnauld and Nicole. ... To the extent that the Jansenist theological community had become more "confident," it also became a little more Calvinist, at least by the standards of its own understanding of Calvinist theology.