Repair an instance in rescue mode

Rescue mode starts an instance from a rescue image on its host instead of from its own disk. The instance's disk stays attached without being started, so you can mount it and repair it from outside. Use rescue mode when the instance no longer boots, when you cannot sign in to it, or to copy data off a disk with problems.

Rescue mode changes only how the instance boots. Its disk, addresses, snapshots and backups stay as they are.

• The instance is not suspended.
• Nothing on the instance needs to keep running. Entering rescue mode restarts the instance at once, and its services stay down until you leave rescue mode.
• You work in the rescue image through the console. Rescue images have no SSH sign-in of their own.

1. In the dashboard, go to Cloud Compute, open the instance and select the Rescue tab.
2. Under Rescue media on this host, select the image to boot, such as SystemRescue. Each card shows the image's name, version and file.
3. Select Enter rescue mode…, then Reboot into rescue.

We restart the instance from the rescue image. If the instance was stopped, we start it. The Rescue mode card then shows In rescue, and the header shows a badge with Rescue and the image's file name.

While the instance is in rescue mode, every start and restart boots the rescue image again. Leave rescue mode to boot from the disk.

These steps are for Linux instances.

1. Open the Console tab. SystemRescue signs you in as root without a password.

2. Find the instance's disk and its partitions:

   Shell

   lsblk -f
   

   The disk is usually /dev/sda, or /dev/vda on some instances. The rescue image and the instance's cloud-init drive show as CD drives, such as sr0, with the filesystem iso9660. The root filesystem is normally the largest partition. The commands below use /dev/sda1, so replace it with yours.

3. To check a filesystem that will not mount or that the instance reports as damaged, run the check before you mount it. Use e2fsck for ext4 and xfs_repair for xfs, as lsblk -f shows:

   Shell

   e2fsck -f /dev/sda1
   

4. Mount the root filesystem and switch into it:

   Shell

   mount /dev/sda1 /mnt
   for d in dev proc sys; do mount --bind /$d /mnt/$d; done
   chroot /mnt
   

   You now work inside the instance's own system. For example, run passwd ubuntu to set a new password for the ubuntu user. Use the user name the Access tab shows under Username.

5. When you are done, leave the chroot and unmount everything:

   Shell

   exit
   umount -R /mnt
   

Rescue images do not run cloud-init, so the rescue system may come up without the instance's address. Check with ip addr. If the interface has no public address, give it the settings the instance uses itself:

1. Mount the root filesystem at /mnt, as above.

2. Find the file that holds the instance's network settings by searching for its address, and read the prefix length and gateway in it:

   Shell

   grep -rsl 203.0.113.10 /mnt/etc/
   

3. Set the same address and gateway by hand. Replace ens18 with the interface name that ip link shows:

   Shell

   ip link set ens18 up
   ip addr add 203.0.113.10/24 dev ens18
   ip route add default via 203.0.113.1 dev ens18 onlink
   

Then copy the files to a machine you control, for example:

rsync -a /mnt/var/www/ alex@backup.example.com:/srv/restore/

1. On the Rescue tab, select Leave rescue mode….
2. Select Reboot from disk.

We remove the rescue image, restore the boot order and restart the instance from its own disk.

The Rescue mode card shows Normal boot, the badge leaves the header, and the instance runs from its own disk with the changes you made.

No rescue images on this host

The instance's host has no rescue image. Contact support and ask us to add one.

Could not list the rescue media

We could not reach the instance's host. Select Try again. If the error stays, contact support.

Already in rescue mode or Not in rescue mode

The page is out of date. Reload it to see the instance's current mode.

Instance is suspended

A suspended instance cannot enter or leave rescue mode. The alert at the top of the page says why the instance is suspended. You can leave rescue mode once the suspension ends.

The instance boots the rescue image after a restart

The instance is still in rescue mode. Select Leave rescue mode… on the Rescue tab.

You cannot sign in with the new password after you leave rescue mode

On AlmaLinux, Rocky Linux, CentOS Stream and Fedora, enter rescue mode again and run touch /.autorelabel in the chroot, as the note above describes.

• Use an instance's console explains the display you work in.
• Manage an instance's password, SSH keys and hostname resets the password without rescue mode while the instance still runs.
• Rebuild an instance starts again from a fresh image. Leave rescue mode first.
• Back up and restore an instance puts back a whole disk from a backup.

GET /api/v1/client/vps/{uuid}/rescue/media lists the ISO images on the instance's host. Each has a volid, its storage, its size in bytes and its format.

curl https://api.coritan.com/api/v1/client/vps/$INSTANCE_UUID/rescue/media \
  -H "Authorization: Bearer $CORITAN_TOKEN"

POST /api/v1/client/vps/{uuid}/rescue/enter restarts the instance from the image you name in iso_volid, a volid from that list:

curl -X POST https://api.coritan.com/api/v1/client/vps/$INSTANCE_UUID/rescue/enter \
  -H "Authorization: Bearer $CORITAN_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"iso_volid": "local:iso/systemrescue.iso"}'

It returns once the instance has restarted, with {"status": "rescue", "iso_volid": "local:iso/systemrescue.iso", "drive": "ide3"}. A value without its storage prefix answers 400 with iso_volid is required (e.g. local:iso/rescue.iso), and an instance already in rescue mode answers 409 with Already in rescue mode.

POST /api/v1/client/vps/{uuid}/rescue/exit takes no body. It restarts the instance from its disk and answers {"status": "normal"}, or 409 with Not in rescue mode.

GET /api/v1/client/vps/{uuid} shows the mode: rescue_mode is 1 in rescue mode and 0 otherwise, and rescue_iso names the image.

Entering and leaving answer 403 with Instance is suspended on a suspended instance. Listing the media and both changes answer 409 with Instance is migrating while we move the instance to another host.