Expanding /var/lib/docker Storage on a Vast.ai Host

For: Vast.ai host operators who need more space for /var/lib/docker.
There are two different approaches, depending on your current setup. Pick the one that matches your situation:

Both options share the same first three moves: confirm the machine is idle, stop/mask the same four services, and back up /etc/fstab. Those shared steps are written out once below, then each option continues on its own path.

Step 0: Confirm the machine is idle

"Idle" means no active rental, not just a container that happens to be stopped. A stopped container can still belong to an active rental, and the renter can start it again at any moment, including mid-migration.
Check this on the machine's card in the Vast.ai console: look at the Occ, #Running, and #Stored fields. All of them need to read 0 (e.g. Occ: x, #Running: D: 0, I: 0, R: 0, #Stored: D: 0, I: 0, R: 0) before you can be sure there's truly no instance on the machine.

If you must proceed while a rental is still technically active (even if stopped), notify the renter of the maintenance window first, run docker ps -a to confirm no container is running, and work as quickly as possible within that window.

Always keep a backup before deleting anything. If something goes wrong midway, you need to be able to roll back to the original state.

Step 1: Record the current state and gather information

# See if a new disk is already detected (only relevant for Option B, e.g. /dev/sdb, /dev/nvme1n1...)
lsblk

# Check current size of /var/lib/docker
sudo du -sh /var/lib/docker

# Check free disk space
df -h

If you're following Option B, make sure you clearly identify which device is the new disk (e.g. /dev/sdb1) before continuing. Using the wrong device can destroy data on another disk.

Step 2: Stop and mask services

Goal: make sure nothing writes to /var/lib/docker during the work, and that none of these services restart on their own mid-way.

sudo systemctl stop vastai
sudo systemctl stop docker docker.socket containerd
sudo systemctl mask vastai docker docker.socket containerd

If you hit this error on the vastai mask command:

Failed to mask unit: File /etc/systemd/system/vastai.service already exists.

This means there's a real service file (not a symlink) at that path, so mask can't overwrite it automatically. Don't use --force. Instead:

sudo cp /etc/systemd/system/vastai.service /etc/systemd/system/vastai.service.bak
sudo cp /etc/systemd/system/vastai.service.bak /root/vastai.service.bak
sudo rm /etc/systemd/system/vastai.service
sudo systemctl daemon-reload
sudo systemctl mask vastai

Then verify all four are stopped and masked:

sudo systemctl status vastai docker docker.socket containerd

All four should show inactive (dead) and Loaded: masked before you continue. If any still shows active (running), stop here and do not proceed until it's fully stopped. For Option B, use sudo lsof +D /var/lib/docker to see which process is holding it.

Step 3: Back up /etc/fstab

sudo cp /etc/fstab /etc/fstab.bak-$(date +%Y%m%d)

Option A doesn't need to edit fstab afterwards, but this backup is a cheap safety net either way.

Option A: Grow the loop-mounted image file in place

Use this when /docker-xfs.img (or similar) is a loop-mounted file sitting inside an existing ext4 partition that already has enough free space. This is simpler and lower-risk than a full migration since there's no data copy involved.
Note: these steps involve general Linux disk/filesystem administration, which is technically outside the support scope Vast can officially cover.

A1. Unmount the XFS loop mount

sudo umount /var/lib/docker

A2. Grow the backing image file

Decide your target size (leave comfortable headroom out of your free space) and extend the file, e.g.:

sudo truncate -s 2T /docker-xfs.img

truncate just extends the file's logical size. It doesn't touch the XFS filesystem inside yet.

A3. Remount

Your existing /etc/fstab line doesn't need to change, since it references the file path, not a size, so loop,pquota,nofail stays exactly as is:

sudo systemctl daemon-reload
sudo mount -a

A4. Grow the XFS filesystem to fill the enlarged loop device

sudo xfs_growfs /var/lib/docker

This is the step that actually expands the usable space. Project quotas (pquota) remain intact through an xfs_growfs resize, with no special handling needed.

A5. Verify

df -h /var/lib/docker

You should now see the new total size, with your existing Docker data untouched.
Once verified, skip to Step 4: Unmask and restart services below.

Option B: Migrate to a brand-new disk

Use this when you're moving /var/lib/docker to a physically different disk or partition. Estimated downtime depends mostly on how much data you currently have in /var/lib/docker (the copy step is the slowest part).

B1. Copy the entire Docker data directory to a temporary location

This is the most important step. Do not skip it, since this is your only backup if something goes wrong in later steps.

# Create a temporary folder to hold the backup (change /mnt/backup_docker if that disk doesn't have enough space)
sudo mkdir -p /mnt/backup_docker

# Copy everything, preserving permissions, symlinks, and timestamps
sudo rsync -aHAX --info=progress2 /var/lib/docker/ /mnt/backup_docker/

If rsync isn't installed: sudo apt install rsync -y and re-run. cp -a /var/lib/docker /mnt/backup_docker also works, but rsync is safer and shows progress more clearly.
Once the copy finishes, verify the size on both sides roughly matches:

sudo du -sh /var/lib/docker
sudo du -sh /mnt/backup_docker

Only continue once these two numbers are close to each other.

B2. Prepare the new disk (if it doesn't already have a filesystem)

# WARNING: this wipes all data on the target device. Double-check the device name first
sudo mkfs.xfs /dev/sdb1

XFS is recommended since Vast.ai relies on it for quota support. If the disk already has a filesystem and important data, skip formatting. Just confirm the filesystem type with sudo blkid /dev/sdb1.

B3. Unmount the current /var/lib/docker

If it's currently mounted as its own separate disk/partition (check with mount | grep docker):

sudo umount /var/lib/docker

If this reports target is busy, go back to Step 2 and confirm Docker/containerd/Vast.ai are fully stopped.

B4. Update /etc/fstab to point to the new disk

Get the UUID (preferred over /dev/sdX, since device names can change after reboot):

sudo blkid /dev/sdb1

Add a line to /etc/fstab (replace with your actual UUID):

UUID=1234abcd-56ef-78gh-90ij-klmnopqrstuv /var/lib/docker xfs rw,auto,pquota 0 0

The pquota option is required. Vast.ai uses it to enforce per-container disk quotas. Do not remove it.

B5. Confirm /var/lib/docker is empty, then mount the new disk

ls -la /var/lib/docker

This should be empty. If you see leftover files, stop and investigate before continuing.

sudo systemctl daemon-reload
sudo mount -a

If you see a hint that fstab was modified but systemd still uses the old version, run daemon-reload again, then mount -a again.
Verify:

df -h /var/lib/docker
mount | grep docker

B6. Copy the Docker data from the backup into the new disk

sudo rsync -aHAX --info=progress2 /mnt/backup_docker/ /var/lib/docker/
sudo du -sh /var/lib/docker

Step 4: Unmask and restart services

sudo systemctl unmask containerd docker.socket docker vastai

If you had to delete vastai.service in Step 2, restore it first:

sudo cp /etc/systemd/system/vastai.service.bak /etc/systemd/system/vastai.service
sudo systemctl daemon-reload

Then start everything back up in order:

sudo systemctl start containerd docker
sudo systemctl enable vastai
sudo systemctl start vastai

Check that everything is healthy:

sudo systemctl status containerd docker vastai

All three should show active (running).

Step 5: Refresh machine info with Vast.ai

sudo python3 /var/lib/vastai_kaalia/send_mach_info.py

This tells the Vast.ai platform about the machine's updated disk configuration.

Step 6: Final verification

# Confirm Docker is using the expected directory
sudo docker info | grep -i "Docker Root Dir"

# Confirm Docker works normally
sudo docker ps
sudo docker pull vastai/test:speedtest

# Watch the Kaalia log for a few minutes to make sure there's no crash loop or connection error
sudo journalctl -u vastai -f --no-pager

If docker pull succeeds and the Kaalia log shows no errors, the work is complete.

Rollback (if something goes wrong midway)

# 1. Stop the services
sudo systemctl stop docker containerd vastai

# 2. Unmount (if already remounted)
sudo umount /var/lib/docker

# 3. Restore the old fstab
sudo cp /etc/fstab.bak-<the date you backed it up> /etc/fstab

# 4. Remount so /var/lib/docker serves data from the original location again
sudo mount -a

# 5. Restore vastai.service if it was deleted
sudo cp /etc/systemd/system/vastai.service.bak /etc/systemd/system/vastai.service
sudo systemctl daemon-reload

# 6. Restart the services
sudo systemctl start docker containerd vastai

For Option A, "the original state" just means the image file before truncate. Since nothing was copied or deleted, rollback mainly consists of restoring fstab/services and, if needed, restoring from your pre-work backup of the Docker data.
For Option B, rollback also relies on the /mnt/backup_docker copy made in Step B1. That's your source of truth if the new disk or the second rsync pass has a problem.

Updated on: 02/09/2026