I'd ditch the olive oil soap olive oil can kill shaving lather. Get a glycerine based soap this will add to the glide. Proraso do one especially made for a pre shave wash.
It will take a little time...