But Round Peg/ Square hole is an accepted variant
http://www.answers.com/topic/square-peg-in-a-round-hole
http://www.idiomdictionary.com/definition/square-peg-in-a-round-hole.html
http://dictionary.reference.com/browse/you+can't+fit+a+round+peg+in+a+square+hole
The phrase is an idiom and none of the sources I can find connect the phrase with wooden pegs in timber structures since these wooden pegged structures do fit. The expression is an idiom to represent things which simply do not and could not fit or belong together..(and if the use of square pegs on round holes you described was widely known, and a common practice, this phrase would not work as an idiom). Its use dates from about 1800 and that is long long after pegged joints were first made in timber structures....
Idiom (Latin: idioma, "special property", f. Greek: ἰδίωμα – idiōma, "special feature, special phrasing", f. Greek: ἴδιος – idios, "one’s own") is an expression, word, or phrase that has a figurative meaning that is comprehended in regard to a common use of that expression that is separate from the literal meaning or definition of the words of which it is made
Therefore Round Hole/ Square peg or Square Hole/ round peg are both suitable idioms for a person who does not suit a particular job.
But as far as I Knew pegged Joints were made with roughly round pegs. Certainly when I have seen them used the end entering the hole is pretty well round, although the remainder may be more irregular