Most probably you have some chipping near the toe. When you shave, you are shaving with the middle of the blade, but when cutting your sideburns you are using the tip and more pressure.

Even...