This has come up many times over the years. A couple of relevant points:

1) Lead shot causes harm to waterfowl because they directly ingest it, and they also grind it in there gizzard so that finely ground lead comes into contact with stomach acid, allowing it to come into solution and poison them. Fish a) do not ingest the lead, and b) do not have gizzards to grind it.

2) Lead shot and sinkers introduced into rivers has a higher specific gravity than the rocks and sand around it, which means it will sift itself into the lowest level of the substrate, becoming unavailable to wildlife. Think about how gold, another heavy mineral, collects itself in a stream. You have to dig for it.

3) Lead is pretty insoluble in water. It doesn't just leach away, to my knowledge. I hope there is a chemist out there that can quantify this, but lead doesn't just dissolve into the water like, say, salt does.

So, based on these points, I don't get two worried about the sinkers I lose. Instead, I worry about the litter, sixpack holders, and monofilament line that I encounter on the shore, which demonstrably does injure wildlife.