Sounds like a typical "mole hunt."

I have put smoke bombs in their holes, and that makes them leave for a while, but they always come back.

A gardening site (I think it was Ed Hume Seeds) has a non-lethal remedy that is said to be effective and isn't terribly harmful to anything. If I recall, it was basically a small amount of dish soap mixed in with water and distributed across the affected area. Nowhere near as devious (fun) as some of the other methods being described, but I think I will give this a try this spring.