Fell the stubble with a WTG pass and you should not have too much trouble against the grain. Hairs on lip and chin are the toughest. Is your blade sharp enough?