Was Walden Pond ever the primary source for rail transported ice?

...was Walden Pond ever used for large scale ice harvesting involving rail transport?

It seems we can rely on Henry David Thoreau's account to verify this. Thoreau mentions your railroad (emphasis mine):

I have thought that Walden Pond would be a good place for business, not solely on account of the railroad and the ice trade; it offers advantages which it may not be good policy to divulge; it is a good port and a good foundation.

again here:

The Fitchburg Railroad touches the pond about a hundred rods south of where I dwell. I usually go to the village along its causeway, and am, as it were, related to society by this link. The men on the freight trains, who go over the whole length of the road, bow to me as to an old acquaintance, they pass me so often, and apparently they take me for an employee; and so I am. I too would fain be a track-repairer somewhere in the orbit of the earth.

Note the Fitchburg Railroad is the exact same railroad mentioned in the OPs link concerning Fresh Pond. Some details concerning the production force and figures of the ice trade:

To speak literally, a hundred Irishmen, with Yankee overseers, came from Cambridge every day to get out the ice. They divided it into cakes by methods too well known to require description, and these, being sledded to the shore, were rapidly hauled off on to an ice platform, and raised by grappling irons and block and tackle, worked by horses, on to a stack, as surely as so many barrels of flour, and there placed evenly side by side, and row upon row, as if they formed the solid base of an obelisk designed to pierce the clouds. They told me that in a good day they could get out a thousand tons, which was the yield of about one acre.

A thousand tons per day would fill 30 of the above pictured boxcars. They gave estimates on transport loss:

They calculated that not twenty-five per cent of this would reach its destination, and that two or three per cent would be wasted in the cars. However, a still greater part of this heap had a different destiny from what was intended; for, either because the ice was found not to keep so well as was expected, containing more air than usual, or for some other reason, it never got to market. This heap, made in the winter of ’46–7 and estimated to contain ten thousand tons, was finally covered with hay and boards; and though it was unroofed the following July, and a part of it carried off, the rest remaining exposed to the sun, it stood over that summer and the next winter, and was not quite melted till September 1848. Thus the pond recovered the greater part.

So 10,000 Tons of ice were left stored on the pond, and took two years to melt. So yes there was a railroad and large scale ice extraction at Walden Pond.

Was Walden pond or Fresh pond the primary source?

That probably will vary year by year. It would appear that many different ponds were used. From an article at cambridgehistory.org

Beginning in 1805, Tudor harvested ice from across Massachusetts at places including Haggett's Pond in Andover, Spy Pond in Arlington, Sandy Pond in Ayer, Fresh Pond in Cambridge, Walden Pond in Concord, Suntaug Lake in Lynnfield, Spot Pond and Doleful Pond in Stoneham, Lake Quannapowitt in Wakefield, Horn Pond in Woburn, and Wenham Lake in Wenham.

A different source, an 1888 publication of the US Census Office gives us some details concerning the Ice business around Boston in 1888, which certainly indicates Fresh Pond as a major producer at that time, and doesn't even mention Walden Pond by name:

I found another publication, Report of the Secretary of Agriculture, by the US Dept. of Agriculture gives us some production numbers for the year 1864.

This fits the questions 1860s timeframe, and again shows Fresh pond as the top listed producer for the area, and once more no mention of Walden Pond.

Still just snapshots of information(1864 and 1888), but my inclination is that, Fresh Pond would appear to consistently be the top producer of Ice for this region and timeframe. The distribution of the ice collection across so many ponds would also seem to dispute the contention that the ice could have come 'mostly' from any 'particularly suitable pond'. (The top producer only accounting for about 30% of the production listed for 1864.)