I always keep mine on, but it mounts directly to the transom. My buddy has his motor on a bracket, but leaves his connected, also.

I don't think there's a problem with it unless you started driving over a lot of really rough, bumpy roads. That might cause some added stress to the transom, especially on a glass boat.

However, if it's mounted correctly with the proper backing I don't see where it would hurt to trailer it all together. As added precaution, I guess you could tie it up securely to avoid any extra movement and take some stress off the bracket and transom.