Here are a couple of stories on the the subject, (from an adjudicator’s point of view.).
From Naval History & Heritage Command:
1)
It has often been claimed that the "brass monkey" was a holder or storage rack in which cannon balls (or shot) were stacked on a ship. Supposedly when the "monkey" with its stack of cannon ball became cold, the contraction of iron cannon balls led to the balls falling through or off of the "monkey." This explanation appears to be a legend of the sea without historical justification. In actuality, ready service shot was kept on the gun or spar decks in shot racks (also known as shot garlands in the Royal Navy) which consisted of longitudinal wooden planks with holes bored into them, into which round shot (cannon balls) were inserted for ready use by the guncrew. These shot racks or garlands are discussed in: Longridge, C. Nepean. The Anatomy of Nelson's Ships. (Annapolis MD: Naval Institute Press, 1981): 64. A top view of shot garlands on the upper deck of a ship-of-the-line is depicted in The Visual Dictionary of Ships and Sailing. New York: Dorling Kindersley, 1991): 17.
2)
The pyramid display of cannon balls was just that, a display, often found in front of the gates of naval and military bases. They were for show only, no one but a blithering idiot would stack cannon balls on a ship. The first time the ship pitched or rolled, the watch on duty would be ankle deep in rolling 18 pound or heavier cast iron spheres. Only 18 pound and heavier cannon balls were kept in racks. Nine pounders and lighter were kept in shot garlands (think hammocks) that were tied to the ships' rails.
So this really leaves it open to interpret the expression. This makes me a fence sitter.