I think a lot of drivers don't like being "stuck" behind a vehicle that is towing anything and so they will try to pass you by. Which is fine except then some will pull in front of you and slow down. As far as passing while towing - I can easily keep up with traffic and if someone is doing 55 while the traffic around them is going the legal limit of 65, I am going to pass them. But generally I try to stay in the right lane out of the way. Plus you can't be in the left lane for any length of time while towing a trailer or you could run the risk of picking up a ticket.
Towing something definately gives one a different perspective on driving.