Did the people who initiated the trans-Atlantic slave trade think they were doing something new?

Slavery in Americas didn't appear in United States in the 18th century; it originates much earlier in Spanish and Portuguese colonies. So we should look whether slavery existed in Christian Iberian kingdoms before the discovery of the New World. And it surely did.

Slavery distinct from mere serfdom existed in Europe in medieval era without interruptions. Unlike serfdom which was represented as local tenants still enjoying certain freedoms, slavery was largely associated with captives from war and raiding which were deemed unfit for ransoming. Slave markets traded people who were often captured thousands miles away whereas serfs usually lived in the same place for generations. Slaves were considered property like cattle, unlike serfs who still enjoyed some important amount of personal freedom.

The major slave traders in the medieval Christendom were Genoa and Venice who used their access to the Mediterranean and Black Sea slave markets. In fact slaves of Slavic origin constituted such a large portion of the medieval slaves that it's usually assumed that the very word "slave" is a derivative of the "Slav". The eastern markets also supplied large number of the captives from steppes and the Caucasus. All those people constituded very large proportion of the slaves in medieval Iberian kingdoms, which you can read e.g. in "La esclavitud en Valencia durante la baja edad media (1375-1425)" by Francisco Javier Marzal Palacios. In addition, during Muslim-Christian wars, both sides practiced enslavement of captives in part to pay for the war, e.g. in 1147 almost 10000 Muslim women and children from Almeria were sold to Genoese slavers.

However, towards the end of the Reconquista and especially with the conquest of Ceuta the Iberian kingdoms, primarily Portugal, gained access to the African West Coast; and European participation in the Sub-Saharan slave trade begins. In 1444 the first large group of the African slaves was brought into Europe by Lançarote de Freitas. By 1452 the first sugar plantations start appearing in Madeira.

By the time Christopher Columbus discovered the New World, a system with slaves captured from the West African Coast brought to European owned sugar plantations was already operating in the Old World. This system already began to transfer to the New World at the beginning of the 16th century - the first large group of 4000 African slaves was sent to New Spain in 1518. Reliance on African slaves increased as the Native American population declined largely due to epidemics but also from the overexploitation on encomiendas.

So, no: Slavery by itself was nothing new, and it was separate from serfdom. It dates before the discovery of the Americas. What was novel was its widespread use on plantations. It was caused by sudden access to new productive lands and direct access to many slaves.

The history of Russian serfdom is a completely separate topic, I'm afraid, and had a VERY different nature than the Atlantic Slave trade. Until the very end when some centralized agricultural estates appear it was not associated with high production plantation economy. Russian serfs at least officially never were considered a simple property of their lords, but subjects of the Tsar given to dvoryans (Russian nobles) for their support, and until the 18th century, the government opposed the trade of serfs. It took the ugliest form in the times of Catherine the Great when nobles were freed from any responsibilities and given excessive rights over the lives of their serfs. Still, they enjoyed certain personal freedoms in comparison with European and American slaves.

As for rationalization, the following is largely my opinion (and I'm not a historian to be authoritative). In the medieval period, religious differences were extremely important. It was much easier to accept enslavement of Muslim, "pagan" or "wrong Christian" captive rather than fellow Christian. You should remember that it was a time of many conflicts with religious justifications. Crimes against humanity were done by practically everyone then. Muslims were eagerly enslaving Christians (and vice versa) both in war and piracy. But in general, if you were in a weak position anywhere, you very likely faced imprisonment, slavery or death. This hostile world only further supports the mentality where all "others" are considered natural enemies. This produced a comfortable atmosphere for slave markets to operate as those "others" are acceptable targets. The decline of this justification and transition to straightforward racism is probably connected to the rise of the European dominance and much improved order in world affairs (for Europeans at least).