It isn't just the size that matters. The hull design is very important in regard to how it handles the rougher water. Generally speaking, if the water is rough, you have the chance for water to come over the bow. Width of the boat also matters. You can't necessarily avoid another boat going by you leaving a huge wake while you are fishing. You need to understand the scenarios before you get into them!

I suggest you need some test rides in moderate chop without your family. Find out what you're willing to deal with before you buy the boat. Leave some room for error.

My 20 ft boat seems small when the wind/tide picks up, but when it is flat, you could be out in PS in a dinghy.

Good luck and have fun looking.