The irritation may be from the angle or pressure you are using in that area. If you get a comfortable shave on your sideburn/cheek area but not your neck start playing with angles and pressure.

My...