Stop shaving against the grain. Across the grain should be sufficient particularly in the areas you are talking about. If you need it closer, take another across the grain pass.