The irritation if your blade is properly honed is more from technique, yes you are exfoliating but it should not be that drastic.

Lighten up on the pressure and use the shallowest angle that cuts...