The answer sorta depends on what State you're in and whether the river is considered "navigable" by the Army Corps of Engineers. In many states, the OHWM is the limit, as already explained by several folks. In other states (e.g., New York I think), the landowner owns the land under the river but not the water. That means you can take a boat down the river but you can't walk along the riverbank (even if in the OHWM) or set an anchor without permission.

Angler behavior doesn't change the limits of State jurisdiction but it sure does change landowner attitudes towards those anglers trying to exercise their access rights. I try to practice good public relations with landowners knowing that my actions might make life easier for the next guy/gal trying to fish that stretch of river.