Why would any group charge a wooden wall? To smash it to flinders with a battering ram or something? The whole army probably isn't going to charge simultaneously, just a group of sappers. They'll get a rude surprise when they encounter the invisible stone wall too early, probably drop their ram (assuming it's a carried ram), but then recover early.
If the entire army is intent on scaling the wall, they'd probably only lose a few moments of inaction, then use their ladders to scale the wall they can't see but are now aware of.
Greg