11 Oct 09 Linux Consultant – How to recover a compromised server

As a security consultant I often have to deal with machines that are already compromised. The ‘official’ standpoint is always to wipe the machine alltogether, reinstall your OS, and restore your data and configurations from the backups that you obviously have.

The above not always being possible, and as a second best alternative, you’ll have to recover the machine.

The first thing to do is compare each command line utility to that of a known good identical system before using it, so you can rely on the results that it returns. A hacker will often drop a modified ‘ls’, ‘lsmod’, ‘ps’ and various other tools onto your system to hide the various other things that he may have installed.

You’ll need to use md5sum and ls to check the size and checksum of each utility before you use it, although of course, md5sum and ls themselves could be hardcoded with predefined responses. You could also use ’strings’ to check the ASCII contents of those tools, although the ’strings’ could just as easily be rigged. If you’re that paranoid, you’ve got no choice but to wipe the machine alltogether.

So firstly, check the integrity, of each of your core utilities. If your Debian 5.0 with the latest updates installed system was compromised, you’ll need to check against another Debian 5.0 system with the same updates and tools installed. Or, if you can find a listing online somewhere of what binaries should be what sizes and have what MD5s then you should be fine.

Once you have confirmed your ‘md5sum’ utility, you should be able to just start comparing MD5s and not worrying about file sizes and strings. Check your package management utilities and check that you’re happy with them, then apt-get install rkhunter this will check a number of issues. There are other ‘root kit hunters’ that you can use as well if you wish. Once this has been run, check your ps utility and ensure that it is as you expect. Then once done just run ps auxw and check each running process in the same way. Assuming that all of that is done and has not shown up anything, all is good so far. If something has been found and one of your binaries is compromised. Assuming your package manager is in good order, dpkg -P <package> and reinstall. If it is a core package that can not be removed/purged without affecting the rest of the system, then just scp over a new binary. Check again that the libc6 version and package version is IDENTICAL, and check of course that scp itself is in good order.

At this point, we can assume that your binaries themselves are in good order. Check for any new SUID utilities with find / -perm +4000 and once done, firstly make sure that everything on that list is as expected, and secondly, double check your md5sums of each and everyone of those.

This all being OK, continue to check by looking at your /etc/passwd, /etc/group and /etc/shadow files checking for user accounts that you don’t recognise. Then check syslog, wtmp, lastlog, etc, and check the IPs and last logins of each account. Also check directories such as /tmp/ especially with ls -al to check for directories beginning with a ‘.’ which would otherwise be hidden.

If everything above returns success, then it’s unlikely that your system was directly compromised. There is always the chance that your web application or database was compromised, but then that’s outside of the scope of this article. In short though, check your webserver log files as that should give you the information on what was compromised, and how it was done. Obviously ensure that any 3rd party software that you may be using such as wordpress, vBulletin, etc, etc are always up to the latest version.

Edit/Addition:
In response to a reader’s comments, I would add that should you be able to remove the network connection to the compromised machine and still access it, then do. Your login and anything you type could be being sent to an attacker without you even realising it.

Additionally, there is no point in simply recovering a hacked server without knowing how it was compromised in the first place. Arguably you should have worked it out though by following the steps above.

Tags: apt-get, compromised, debian, dpkg, find, hacker, lsmod, md5, md5sum, ps, rkhunter, Security Consultant, strings, tools

01 Apr 09 Creating an MD5 on Linux with md5sum

An MD5 is a type of Hash, also, a Checksum.

An MD5 hash is a one way verification sum which can be used to verify a string or contents of a file. Once you have a file and an MD5 checksum, the recipient of the file can also perform an MD5 calculation to ensure that the file’s contents are unchanged. They may have been changed maliciously such as in the case of a binary file, or simply by data corruption. An MD5 is NOT a type of encryption. It can not be reversed.

In the case that you know the length of the data, say between 5 and 8 characters for a password, you can attempt to brute force (try every combination automatically until something hits) the password. For that reason passwords are often salted before being MD5′d however salts and their purpose are outside the scope of this article.

We can use PHP to do the following:

<?php
$string = “teststring”;
$checksum = md5($string);
echo “The checksum is: ” . $checksum . “\n”;
?>

The output is:

The checksum is: d67c5cbf5b01c9f91932e3b8def5e5f8

We can also use the ‘md5sum’ linux command and pipe input to it via STDIN.

echo “teststring”|md5sum

test:~# echo “teststring”|md5sum
50be80a7a199c13e2bb09e2e745ba233  -

Why is the output of this md5sum different to that above? Well, ‘echo’ automatically adds a newline to the string to make it “teststring\n”. We can surpress this with -n:

test:~# echo -n “teststring”|md5sum
d67c5cbf5b01c9f91932e3b8def5e5f8  -

We can also run the md5sum command against a file:

test:~# md5sum /bin/bash
c8770eb0a3f2b6088914b4bc29301113  /bin/bash

Tags: Linux, md5, md5sum, PHP