Marine Tex is the way to go. If it is a through hole you are trying to fill I apply marine tex from the inside of the transom. Let it dry and then fill the rest of the hole with 5200 from the outside. When the 5200 is dry I cap the hole with some more marine tex. If you even want to get carried away; sand down the marine tex and apply gelcoat to match the color of your boat. If done right you can't even tell where the hole was. Anytime you put a screw into a glass boat make sure it has a bead of 5200 on it. The stuff is permanent and is easy to work with.

I recently replaced the outboard bracket on my boat. I pulled out all of the thru-bolts and hooked the bracket up to a boom. We had the boat lifted off the trailer a good three inches for about 4 hours before the 5200 seal finally gave way. The most impressive thing was it did not rip any gelcoat off the transom. After witnessing that display of strength 5200 is teh sh1t!