Let it air-dry for a while instead of coating it with oil immediately, you could lock in moisture under the oil. If you shave somewhat regularly I doubt that it's the oil drying up. I'm not even sure...