2/13/2020 12:19 AM

8/13/2023 6:06 PM

Should you encounter a SEM or LEM appliance that is giving an fsck error when booting or get stuck on either of the screens similar to below then you can use this guide to attempt to repair the file system. 

Cause 1: 

Cause 2: systemd-fsck[460]: /dev/sda5 contains a file system with errors, check forced.



Cause 3:  Buffer I/O error on dev sda5, logical block, lost async  page write



*Note: As per cause 2 screenshot, the fsck command was running on /dev/sda5. After reboot, let it run for about 30 to 50 minutes (depends on the size of the disk allocation) and see if that boots up normal.
 
Do note that this may not resolve the file system error and it is possible a redeployment of the appliance may be required.

Prior to following any of the instructions listed below you MUST do the following:

• Verify that the customer has a snapshot of the SEM in its current state. We want a snapshot event if the appliance is broken/not booting.
• Send the following disclaimer to the customer:

  The SEM is in a state where the appliance will not boot and is in an unknown state due to an environmental event. We are using a process that may resolve the disk problem, but it may render the appliance unusable.   

  I agree that Solarwinds has advised me that there is a possibility of data loss and that this step may not work. If this step does not work, we must redeploy the appliance. 

  Please reply "I approve" via email if you accept.
   

If the customer does not agree or cannot do either step then do not proceed with the following steps. Instead create a new dev case and send it up through your AE.

Log & Event Manager

Possible Causes: 
- Resizing VM disk while the appliance is still running. 
- Resizing VM disk when the disk is full and the snapshots are still in place. 
- Underlying VM hard-drive issue with corrupted blocks
- SEM OS Partition is accidentally deleted from vmware V-Center while disk resizing - To rebuild SEM, you can refer to this KB - Unable to boot SEM - Operating System not found

Prerequisites

• You must follow the steps listed in the Overview section for taking a snapshot and documenting customer acceptance BEFORE doing any of the below steps.
• Download Debian NetInst ISO
  • A direct link is here: https://cdimage.debian.org/debian-cd/current/amd64/iso-cd/debian-12.1.0-amd64-netinst.iso
  • Should the above link be broken, navigate to https://cdimage.debian.org/debian-cd/current/amd64/iso-cd/ and download the file that matches the filename debian-X.Y.Z-amd64-netinst.iso where X.Y.Z matches the current release of Debian Linux (as of August 13th, 2023 the version is 12.1.0)
• Some of these steps will differ depending on the specific hypervisor its version the customer is running. This guide assumes VMware vSphere using the VMware Remote Console (VMRC)

SEM OS Partition/Disk is accidentally deleted from vmware V-Center:

• Refer: Restore SEM functionality from unbootable state or perform data migration

Booting ISO

1. Mount ISO on VM's Virtual DVD drive
2. Boot into the Virtual Machine BIOS
   
3. Change boot order to "boot CD-ROM" first
   
4. Save changes and exit the BIOS
   
5. Reboot VM

Starting Rescue Mode

1. Debian Installer Loads
   
2. Select 'Advanced Options' and hit *enter* on keyboard
3. Select 'Graphical Rescue Mode' and hit *enter* on keyboard
   

Configure Rescue Mode

1. On "Select Language" Select or Leave on 'English' and click *Continue*
   
2. On "Select Your Location" Select or Leave on 'United States' and click *Continue* 
   
3. On "Configure The Keyboard" Select or Leave on 'American English' and click *Continue* 
   
4. On "Configure The Network" Leave hostname as-is and click *Continue* 
   
5. On "Configure The Network" Leave domain as-is and click *Continue*
   
6. On "Configure The Clock" Leave as-is and click *Continue*
7. On "Enter Rescue Mode" choose 'Do not use a root file system' and click *Continue*
8. On "Enter Rescue Mode" choose 'Execute Shell In The Installer Environment' and click *Continue*
9. On "Enter Rescue Mode" Click *Continue*

Running `fsck` in Rescue Mode

You should now have a BASH shell to run `fsck` commands in.
Depending on the disk space allocated to the Appliance its possible for the `fsck` command to take a long time to run

!!Check VM if it has 1 disk or 2 disks
Depending on which version the appliance was deployed you will need to run the following different commands:

For Dual Disk SEM (6.4 to latest):

• fsck.ext4 -y /dev/sda1
• fsck.ext4 -y /dev/sda2
• fsck.ext4 -y /dev/sda4
• fsck.ext4 -y /dev/sdb1

For Single Disk SEM (versions below 6.4 - EOL):

• fsck.ext4 -y /dev/sda1
• fsck.ext4 -y /dev/sda2
• fsck.ext4 -y /dev/sda5
• fsck.ext4 -y /dev/sda6

Cleaning/Finishing Up

1. Click *Go Back*
2. Select 'Reboot' and click *continue*
3. VM will reboot and start the Debian ISO again
4. Power Off VM
5. Remove ISO from Virtual DVD drive
6. Power On VM

If `fsck` was successful, then the appliance should boot to the bluescreen normally. You will want to check the manager.log and make sure there are no errors starting up the Manager service.