Well, let's do a quick math check, eh?
Let's say an average miner has a Strength score of 14. That gives him an average damage of 1d8+3 per 6 seconds.
The chance that he damages the wall at all in any given round is 3 / 8, or ~38%.
When he does manage to damage the wall, he'll roll 10 damage on average, which means he'll get 2 points past the hardness. His average damage per round, then, 0.75.
Therefore, it will take a standard miner, on average, 20 rounds, or two minutes, to put a 5' by 5' by 1" depression in a wall of rock.
This translates into roughly 4 hours to dig a 5' by 5' by 10' tunnel for a standard miner. It seems a little fast, to me, but then we're dealing with D&D, so a little bit of unreality seems fine.
Note that if you make the above-mentioned miner slightly weaker (Strength 12 or 13), it takes a full 24 hours of work to dig the same tunnel.
So, how do those numbers sound to you?