Try using non buoyant worms, most likely the worms that you are fishing are either neutrally buoyant or are fully buoyant.

I fish non buoyant or neutrally buoyant worms under a float.

Good luck.

P.S. the easiest way too test the buoyancy of your worm/lure is too put it in a bucket and see where it ends up after resting.