Did every medieval village in Europe have a manor, castle, or other type of lord's residence?

Looking at late 11th century England, the Domesday Book commissioned in 1085-6 by William I records a division:

• of England into Counties;

  Counties were the primary structural element of Domesday Book. There were 31 counties in Great Domesday.

• of Counties into Fiefs;

  ..., a fief being the conventional term for the manors held directly from the Crown by a single landowner in a single county in 1086.

• of Fiefs into Hundreds;

  Hundreds were the main administrative subdivisions of a county, with a significant role in financial, military, judicial, and political matters, centred upon the Hundred court, which met monthly.

• of Hundreds into Vills; and

  The Domesday vill was the smallest unit in the administrative system, an area often equivalent to the parish in the south or to the township in the north. ....

  A vill might be part of a manor, might correspond to a manor, or might contain two or more manors.

• of Vills into Manors

  Manors are the atomic units of Domesday Book, the smallest discrete elements. .... Vills were administrative units; manors were units of ownership.

So the answer now hinges on your intended meaning of village.

The vill was an administrative unit, not a type of settlement. It might contain a village, but it might equally contain hamlets or scattered farms or some combination of village, hamlet and farm.

Until a generation ago it was generally assumed that many, perhaps the majority of Domesday vills were nucleated villages, and that nucleated villages were an ancient feature of the countryside in the eleventh century, as old as the Anglo-Saxon settlement of the fifth and sixth centuries. This is no longer credited. There is as yet no consensus as to the date at which nucleated villages began to emerge on a significant scale, though the period between the tenth and twelfth centuries is held to be the most probable.

Do you intend to refer to the administrative division of a county, or to a geographically isolated (relatively self-sufficient) cluster of households? The latter, in most instances of the time, would more properly be referred to as a hamlet:

In British geography, a hamlet is considered smaller than a village and distinctly without a church (one road with houses either side)..

There would have been at least one hamlet for each manor, plus isolated households distinct from any hamlet because, then as now, many farmers would prefer living closer to the land they cultivate than to the nearest community.

Further, the Domesday Book records some:

• 30,000 manors;

• About 13,400 [named] individual places

• 48 castles including Windsor

• 112 boroughs

• Over 300 parish churches

• Around 6,000 mills (but no windmills)

• Numerous markets [60 noted, though inconsistently recorded]

• And some industry including salt pans, lead working, quarries and potteries

So if your intended meaning of village is in regards some level of economic infrastructure, such as a market, mill, castle, or town, then except for mills there are two or more orders of magnitude fewer such than manors.

Details of individual manor entries can be looked up here, as for Stedham (annotated as being a very large manor both in population and income). Note that the suffix "ham" refers to it being a hamlet at original name adoption.

Households: 23 villagers. 16 smallholders. 10 slaves. [Heads of Household only - not a census count - so likely about 1/5 of actual population.]

Terminology:

Villager, or Villein
In economic terms, the villagers were indistinguishable from Freemen or freemen. They were the most substantial group among the unfree peasantry, possessing on average 30 acres of land and two plough oxen.

Bordar, or Smallholder
Smallholders formed the second largest group among the peasantry, constituting almost a third of the recorded population. .... On average, they possessed 5 acres of land and might have a share in the villagers' plough teams, ....