Eviljam-
Fish get a variety of parasites (not unlike fleas, lice, etc on mammals). A common one seen on trout in freshwater is a parasitic copepod. This critter attaches itself to the outside of the fish (concentrations around the gills, mouth, base of fins, etc.). It is typically about the size of a grain of rice and is pale whitish/tanish in color. It is pretty common in many of our waters (lakes and streams).

I suspect that is what your fish had. They do not affect the quality of the meat and are usually easily removed in the cleaning process. The fish also seem to tolerate fairly high numbers of them - not uncommon to see dozens to even a hundred or more on an individual fish. Tend to see more during the summer/fall and they are worse some years than others.

Tight lines
Curt