Do you have really coarse facial hair?? It sounds like you may need to strop before carrying on to your neck and chin. Or it may be down to technique like Rick says.

I've used my straight 4...