Wednesday, July 28, 2010

Checking Integrity of A Debian/Ubuntu System

Sometimes, a Linux filesystem becomes corrupted, system files are damaged, or some crucial files get lost. This often happens, regardless of which filesystem (ext2, ext3, ext4, jfs, reiserfs, reiser4, or xfs) is used. There are many possible reasons, such as:



  • Unstable hardware, for example, memory or hard drive problem
  • Overheat, power surge, quake or another environmental disaster
  • Buggy software, such as a bug in the kernel or the filesystem driver
  • Compromised security, for example, network intrusion or attack
  • Worm or virus infection


Files in Linux systems can be categorized into the following three:




  1. Verifiable System Files

    In Linux systems that are managed by packages (such as Debian and Ubuntu), these files are installed by packages and make up the bulk of the filesystem. These files reside in such directories as /bin, /lib, /sbin and /usr. They are usually static, which means they don't normally change except when the system is updated, or locally compiled binaries are installed.
  2. Changeable System Files

    These files are auxiliary system files for system configuration, initialization or customization, and system data (such as logs and cache). They reside in /boot, /etc, /opt, /srv and /var.
  3. User Data

    These files are created and used by superuser (a.k.a root) and normal users, or software-generated during casual user activities. Typically, they are in /home, /media, /mnt and /root.


This post focuses on verifiable system files (installed by packages). When the filesystem becomes corrupted (but not completely unreadable), it is possible to verify and restore the system integrity by using package checksums. Before you continue, make sure to fsck the filesystem.



e2fsck -r -v /dev/sda7


In this example, /dev/sda7 points to an ext2 partition we're going to check. Be aware that you cannot fsck a mounted filesystem. Therefore, boot with a Debian Live CD (or a Ubuntu CD) and run fsck. After you've performed fsck, there may be some files created in the /lost+found directory. We'll deal with them later. First, mount the filesystem.



mount -t ext2 /dev/sda7 /mnt


Go to /var/lib/dpkg/info. Then, concatenate all the md5sums files. Most, if not all, Debian and Ubuntu packages come with a md5sum file that we can use to check the integrity of the package and the files installed by the package.



cd /var/lib/dpkg/info
cat *.md5sums | sort > /dev/shm/all.md5


all.md5 has md5 checksums of all the files installed on the system. Now, check the files on the Debian/Ubuntu system against the concatenated md5sums file.



cd /
md5sum -c /dev/shm/all.md5 > /dev/shm/check.txt 2>&1


/dev/shm/check.txt now contains the results of the integrity check. It looks like this:



bin/bash: OK
bin/bunzip2: OK
bin/bzcat: FAILED


In this example, /bin/bzcat is damaged. To find all the missing or damaged files, use a command like this one:



grep -v ': OK$' /dev/shm/check.txt


Let's reinstall this file. First, find out which package this file belongs to.



dpkg -S /bin/bzcat


We'll see the following result.



bzip2: /bin/bzcat


Now we know that we need to reinstall bzip2. Let's download the package.



dpkg -p bzip2 | grep 'Filename: '


This command will let us know the name of the package to download. Use wget to download it.



wget ftp://ftp.us.debian.org/debian/pool/main/b/bzip2/bzip2_1.0.5-4_i386.deb


You can just reinstall the package.



dpkg -i bzip2_1.0.5-4_i386.deb


Or, you can just extract one file:



dpkg --fsys-tarfile bzip2_1.0.5-4_i386.deb | tar xf - ./bin/bzcat


Alternatively,



dpkg --fsys-tarfile bzip2_1.0.5-4_i386.deb | tar xOf - ./bin/bzcat > /mnt/bin/bzcat


To restore a file from the /lost+found directory, you can also use the MD5SUMS file. First, run md5sum on files in /lost+found.



cd /lost+found
md5sum *


You may get an output like this.



9aaa2176d20c1b1203e3abbac55a2513  #124531


To find out what #124531 file is originally, find its md5 checksum from the all.md5 file above.



grep 9aaa /dev/shm/all.md5


You'll get a result like this.



9aaa2176d20c1b1203e3abbac55a2513  bin/bzip2


Now you can just move it to its place.



mv \#124531 /mnt/bin/bzip2


After you restore all damaged files and restore files from /lost+found, you can find missing files in the system. Go to /var/lib/dpkg/info again and concatenate all the list files.



cd /var/lib/dpkg/info
cat *.list | sort | uniq > /dev/shm/all.txt


The .list files in the /var/lib/dpkg/info directore show the list of files installed by packages. Let's find what's missing from the system.



cd /
for f in $(cat /dev/shm/all.txt ); do test -e "$f" || echo "$f" >> /dev/shm/nonexist.txt ; done


The file /dev/shm/nonexist.txt will show which files are missing from the system. You can then replace the missing files as done previously.

2 comments:

  1. FYI - there is a utility called 'debsums' that automatically checks packages against their MD5 digests. According to the documentation with it, it's been around since 1997.

    ReplyDelete
  2. Thanks , that was exactly what I was looking for while reading the excellent book:

    "Hardening Linux"
    by James Turnbull

    ReplyDelete

About This Blog

KBlog logo This blog is about current events and issues concerning general population. Thanks for visiting the blog and posting your comments.

© Contents by KBlog

© Blogger template by Emporium Digital 2008

Followers

Total Pageviews

icon
Powered By Blogger