Well, first off, once you've rinced the brush after shaving, do you give it a good squeeze and/or shake to get out hairs? If you don't do that a lot of water will be left in the brush.

However,...