Let them go They have 20 miles of river to cover! If you try to race a side drifter, he'll take ya right to your take out
I've got no problem fishing with side drifters... you see em once, then they are gone.