Last night we upgraded some servers and one of them could not be accessed using SSH. I checked it on the vCenter and found it entered emergency mode and showed some messages on the screen.
The error messages told that the /sysroot could not be mounted when the system was booted and tried to mount the /dev/sda2 partition while the filesystem type fsck.LVM2_member was unknown. We searched something and run the fsck on the / filesystem while it was clean. While the old kernel still worked well and could be booted. We tried to remove the reinstall the new kernel while still got the same error.
It was weird as it worked well using the old kernel. We did some checking and found there was something wrong in the /etc/fstab config file. The partition /dev/sda2 was a PV while it marked as root files system in the /etc/fstab and another correct line to mount /dev/mapper/rootvg-rootlv as root file system was commented out. We corrected this issue and rebooted but still got the same error.
So we continue to check the grub config file /boot/grub2/grub.cfg and found the root device was /dev/sda2! We updated it to /dev/mapper/rootvg-rootlv and this time it was booted ... still to the emergency mode! But the error messages were changed so we know it was another issue. According to the prompt, we run 'journalctl -xb' to get the boot log and found the datalv got problem and a manual fsck was needed. We fixed it manually and rebooted it again and this time we got the login prompt finally. 
The root cause of first issue should be about the wrong config in the /etc/fstab. As /dev/sda2 was marked for root file system, the new kernel was installed and set the root device to /dev/sda2 in the grub config file and so it tried to mount /dev/sda2 to /sysroot and failed as it was a PV. The second issue was simple about filesystem itself.
