When you catch one of the little ones, take note of its mouth size and be sure to use something bigger than it can swallow. You could try using a white fly with sparse red yarn tail and doused with scent. Hook size does make a difference between being pestered by the little ones and getting the bigger ones.