Personally, I keep it where it should be all the time. Shouldn't hurt the reel any stripping line. Messing with the drag while fighting a fish almost always ends in disappointing results.

Also remember that it may not have been any fault of the drag setting ... tick a rock on a backcast at the hook might break even under the lightest setting ... part of the bugrod game