The key term here is "restricted manueverability"
with regards to the fishing vessel. Generally,
"while trolling (fishing)" implies commercial vessels. However, if trolling with downrigger gear caused "restricted manueverability" the fishing vessel may have the Right of Way. It would be a tough Coast Guard call, so stay out of the way of sailboats not under power.

Spud