Much has to do with the type of worm you're using.

They will all ball up at times. First off, you need to get away from a sinking worm and move to a floater so it doesn't beat in the rocks as much.

Then, you gotta look for worms of a better material and stiffness for this application.

http://www.dannyjoeslures.com are the best I've found out of dozens and dozens I've tried.