I've never noticed the phrase before, but I've found a few references online.
From urbandictionary.com:
A midwestern colloquialism meaning that you could even drive a person who had no manner of sensory perception up the wall because you are beyond all natural levels of irritating.
From lifeblogcarpediem.wordpress.com:
Across the years, I’ve occasionally used an expression that is a fair assessment for someone who has the propensity to go on and on when you were actually done with the conversation fifteen minutes earlier. I describe it as: “She could drive a wooden man crazy.”
From city-data.com:
My Granny said, "I swan" which resembled "I declare!" She also said about something annoying, "That would drive a wooden man crazy." I use that one myself.