Sometimes, a Linux filesystem becomes corrupted, system files are damaged, or some crucial files get lost. This often happens, regardless of which filesystem (ext2, ext3, ext4, jfs, reiserfs, reiser4, or xfs) is used. There are many possible reasons, such as:
- Unstable hardware, for example, memory or hard drive problem
- Overheat, power surge, quake or another environmental disaster
- Buggy software, such as a bug in the kernel or the filesystem driver
- Compromised security, for example, network intrusion or attack
- Worm or virus infection
Files in Linux systems can be categorized into the following three:
- Verifiable System Files
In Linux systems that are managed by packages (such as Debian and Ubuntu), these files are installed by packages and make up the bulk of the filesystem. These files reside in such directories as /bin, /lib, /sbin and /usr. They are usually static, which means they don't normally change except when the system is updated, or locally compiled binaries are installed. - Changeable System Files
These files are auxiliary system files for system configuration, initialization or customization, and system data (such as logs and cache). They reside in /boot, /etc, /opt, /srv and /var. - User Data
These files are created and used by superuser (a.k.a root) and normal users, or software-generated during casual user activities. Typically, they are in /home, /media, /mnt and /root.
This post focuses on verifiable system files (installed by packages). When the filesystem becomes corrupted (but not completely unreadable), it is possible to verify and restore the system integrity by using package checksums. Before you continue, make sure to fsck the filesystem.
e2fsck -r -v /dev/sda7
In this example, /dev/sda7 points to an ext2 partition we're going to check. Be aware that you cannot fsck a mounted filesystem. Therefore, boot with a Debian Live CD (or a Ubuntu CD) and run fsck. After you've performed fsck, there may be some files created in the /lost+found directory. We'll deal with them later. First, mount the filesystem.
mount -t ext2 /dev/sda7 /mnt
Go to /var/lib/dpkg/info. Then, concatenate all the md5sums files. Most, if not all, Debian and Ubuntu packages come with a md5sum file that we can use to check the integrity of the package and the files installed by the package.
cd /var/lib/dpkg/info
cat *.md5sums | sort > /dev/shm/all.md5
all.md5 has md5 checksums of all the files installed on the system. Now, check the files on the Debian/Ubuntu system against the concatenated md5sums file.
cd /
md5sum -c /dev/shm/all.md5 > /dev/shm/check.txt 2>&1
/dev/shm/check.txt now contains the results of the integrity check. It looks like this:
bin/bash: OK
bin/bunzip2: OK
bin/bzcat: FAILED
In this example, /bin/bzcat is damaged. To find all the missing or damaged files, use a command like this one:
grep -v ': OK$' /dev/shm/check.txt
Let's reinstall this file. First, find out which package this file belongs to.
dpkg -S /bin/bzcat
We'll see the following result.
bzip2: /bin/bzcat
Now we know that we need to reinstall bzip2. Let's download the package.
dpkg -p bzip2 | grep 'Filename: '
This command will let us know the name of the package to download. Use wget to download it.
wget ftp://ftp.us.debian.org/debian/pool/main/b/bzip2/bzip2_1.0.5-4_i386.deb
You can just reinstall the package.
dpkg -i bzip2_1.0.5-4_i386.deb
Or, you can just extract one file:
dpkg --fsys-tarfile bzip2_1.0.5-4_i386.deb | tar xf - ./bin/bzcat
Alternatively,
dpkg --fsys-tarfile bzip2_1.0.5-4_i386.deb | tar xOf - ./bin/bzcat > /mnt/bin/bzcat
To restore a file from the /lost+found directory, you can also use the MD5SUMS file. First, run md5sum on files in /lost+found.
cd /lost+found
md5sum *
You may get an output like this.
9aaa2176d20c1b1203e3abbac55a2513 #124531
To find out what #124531 file is originally, find its md5 checksum from the all.md5 file above.
grep 9aaa /dev/shm/all.md5
You'll get a result like this.
9aaa2176d20c1b1203e3abbac55a2513 bin/bzip2
Now you can just move it to its place.
mv \#124531 /mnt/bin/bzip2
After you restore all damaged files and restore files from /lost+found, you can find missing files in the system. Go to /var/lib/dpkg/info again and concatenate all the list files.
cd /var/lib/dpkg/info
cat *.list | sort | uniq > /dev/shm/all.txt
The .list files in the /var/lib/dpkg/info directore show the list of files installed by packages. Let's find what's missing from the system.
cd /
for f in $(cat /dev/shm/all.txt ); do test -e "$f" || echo "$f" >> /dev/shm/nonexist.txt ; done
The file /dev/shm/nonexist.txt will show which files are missing from the system. You can then replace the missing files as done previously.
