I haven't noticed any difference in difficulty between shaving off a 2 day beard and a 4 day beard (although the 4 day beard is quite rare for me)