Whose colors/coat-of-arms did men-of-arms wear in a feudal army, 14th century?

Referring to the Black Book of Edward IV - it's drawn from his own household accounts, so the limits to the retainers allowed are the limits Edward himself set. Remember that Edward had only managed to get, hold, then regain the throne by force-of-arms but he was very well aware that the nobles had their own retainers and that it was possible to lose the throne again. Warwick had a huge number of retainers, well into the hundreds. The limitations on the numbers of retainers were an attempt to control the issue of lords having private armies as large as they could afford and attacking each other if they disagreed with something. As for colours - lords had their own livery. For example, the House of York, in the person of the Duke of York (father to Edward IV and Richard III) used murray (a sort of deep burgundy-red) and blue. There might also be a sigil and a coat-of-arms. Richard III's personal emblem was the White Boar, but his brother Edward's was the "Sunne in Splendor" - a sort of star-burst. The lord himself would use his colours and his sigil. His employed retainers - the men employed directly by himself to be his armsmen - would probably wear his colours - so for example, a heavy padded jacket made up of four sections of cloth in two of the lord's colours, with the diagonal-opposites being in matching colours. They would also likely wear his sigil (eg White Boar) sewn onto their breast. During battle, they would start grouped together under a banner displaying the colours and/or the sigil. However, over time, the lower lords didn't always have a standing army - it was expensive. So they would either hire professional arms-men when they needed them or they would gather (volunteers or strong-armed) peasants in from the villages they controlled. These peasants would have an arming/padded jack (heavyly padded jacket) if they were lucky. They would be very unlikely to have the lord's colours. So in a small skirmish between two small lords, you have your two sigil banners and other than that, very little way of telling who fought for whom. In larger battles, you might have a mix of peasants in their own gear/mercs who might have put the sigil on for ease and would have decent gear/liveried retainers. Where your higher lord needs back-up (perhaps to bring his own strength up to a level required by HIS higher lord), he will send to the lower lords to provide their men. So you have groups of peasants under their manorial lord's banner standing in larger groups headed by the liveried retainers of the upper lord under HIS banner with the upper lord's own peasants in the mix.

Most Middle Ages battles were bloody messes, in part because of the difficulty in determining friend and foe. Not helped by confusion over banners: at the Battle of Barnet (wars of the Roses) in poor weather, the Earl of Oxford's men (Lancastrian) attacked the men of Lord Hastings (York) and chased them off the field. In the time it took Oxford to get his men back under control, the battle-line veered around. As he returned, he unknowingly came up behind his own side, right behind the position held by John Montagu. Montagu's men mistook Oxford's banner of a "streaming star" for Edward IV's "Sunne in Splendor" and attacked Oxford. As Montagu had been on Edward's side at one point, Oxford's men assumed Montagu had turned sides again. They called "treason" and panic spread through the Lancastrian side. Edward attacked, Montagu was killed and the Lancastrians were utterly defeated, including the death of Montagu's older brother the famous Earl of Warwick. This defeat, with the deaths of the two brothers, led almost directly to the following defeat of the Lancastrian Prince Edward and the death of Henry VI, leaving the future Henry VII as the only Lancastrian with any chance of the throne. So the fact that English troops fought without any ready identifiers, and the fact that knowing your enemy very often relied on whether or not you recognised their badge, had a very large bearing on the rulership of England. Had Edward IV lost at Barnet, Henry VI would probably have been put back on the throne and Henry VII would perhaps never have ruled, as Henry VI had a son and heir.

Ordinary soldiers did not wear emblems or colors. Units had pennants or flags. Knights might have an emblem, but that would normally be the house of the knight, not anyone else. Here is a picture illustrating a battle from the 100 Years War. As you can see they use flags and pennants:

Englishmen, as well as their Gascon allies wore the red St George's cross stitched over; front and back so as to distinguish each other. Anyone found 'posing' with one who wasn't one of them scored a death sentence. It was an ordinance given by Richard II that every member of the army, lord and archer must wear it over their armour/clothing.

Sources: Hundred Years War, Volume 3 by Johnathan Sumption

The following quotes, from English Medieval Knight 1400-1500 By Christopher Gravett on Google Books, states that retained Men-At-Arms would have worn their lords colours.

Great lords employed knights and men-at-arms in private retinues, indeed sometimes so many that they formed private armies. Under this system of ‘livery and maintenance’, the retainers wore their lord’s coat with his livery colours, usually the two principal colours from his coat of arms, and they were maintained at his expense.

It is worth mentioning that a Knight that was the retainer of one lord could have their own retainers.

The Black Book of Edward IV gives the following guide to the maximum number of retainers allowed by various ranks of noblemen:

King: 600 Viscount: 80

Duke: 240 Baron: 40

Marquis: 200 Knight: 16

Earl: 140

The book also states that a Lord would ask his retainers to supply men (in the form of men at arms and archers) for an army. That Lord, would then ask the same of his retainers who would in turn gather their retainers - presumably this chain would continue downwards until the required number of men had been gathered.