The Debian administration team and security experts are finally
able to pinpoint the method used to break-in into four project
machines. However, the person who did this has not yet been
uncovered.
The package archives were not altered by the intruder.
The Debian administration and security teams have checked these
archives (security, us, non-us) quite early on in the investigation
and re-installation process. That’s why the project was able to
open up the security archive again and confirm that the stable
update (3.0r2) wasn’t compromised.
If the project had anticipated to get compromised at the same
time the stable update was implemented, the involved people would
have postponed it. However, the updated packages were already
installed in the stable archive and mirror servers at the time the
break-ins were discovered, so it wasn’t possible to hold it back
anymore.
Several methods based on different control data were used to
verify the packages and to ensure that the archives weren’t altered
by the attacker:
- externally stored lists of MD5 sums accumulated over the past
weeks on not compromised machines - digitally signed .changes files from external
debian-devel-changes archives on not compromised machines - digitally signed .changes files on the respective archive
servers - externally stored mirror log files
Timeline
Below is the timeline of discovery and recovery of the
compromised machines. All times are in UTC. Some times are only
estimates since our conversation did not contain exact
timestamps.
Sep 28 01:33 Linus Torvalds releases 2.6.0-test6 with
do_brk() fix
Oct 02 05:18 Marcello Tosatti applies do_brk() boundary check
Nov 19 17:00 Attacker logs into klecker with sniffed password
Nov 19 17:08 Root-kit installed on klecker
Nov 19 17:20 Attacker logs into master with same sniffed
password
Nov 19 17:47 Root-kit installed on master
Nov 19 18:30 Attacker logs into murphy with service account from
master
Nov 19 18:35 Root-kit installed on murphy
Nov 19 19:25 Oopses on murphy start
Nov 20 05:38 Oopses on master start
Nov 20 20:00 Discovery of Oopses on master and murphy
Nov 20 20:54 Root-kit installed on gluck
Nov 20 22:00 Confirmation that debian.org was compromised
Nov 21 00:00 Deactivation of all accounts
Nov 21 00:34 Shut down security.debian.org
Nov 21 04:00 Shut down gluck (www, cvs, people, ddtp)
Nov 21 08:30 Point www.debian.org to www.de.debian.org
Nov 21 10:45 Public announcement
Nov 21 16:47 Developer information updated
Nov 21 17:10 Shut down murphy (lists)
Nov 22 02:41 security.debian.org is back online
Nov 25 07:40 lists.debian.org is back online
Nov 28 22:39 Linux 2.4.23 released
Discovery
On the evening (GMT) of Thursday, November 20th, the admin team
noticed several kernel oopses on master. Since that system was
running without problems for a long time, the system was about to
be taken into maintenance for deeper investigation of potential
hardware problems. However, at the same time, a second machine,
murphy, was experiencing exactly the same problems, which made the
admins suspicious.
Also, klecker, murphy and gluck have “Advanced Intrusion
Detection Environment” (package aide) installed to monitor
filesystem changes and at around the same time it started warning
that /sbin/init had been replaced and that the mtime and ctime
values for /usr/lib/locale/en_US had changed.
Further investigation revealed the cause for both these problems
to be the SucKIT root-kit. It includes password sniffing and
detection evasion capabilities (i.e. tools to hide processes and
files) which are installed directly into the kernel, which in turn
caused the oopses that were noticed.
Detailed Attack Analysis
On Wednesday, November 19th, at approximately 5pm GMT, a sniffed
password was used to log into an unprivileged developer account on
the host klecker (.debian.org). The attacker then retrieved the
source code through HTTP for an (at that time) unknown local kernel
exploit and gained root permissions via this exploit. Afterwards,
the SucKIT root-kit was installed.
The same account and password data were then used to log into
the machine master, to gain root permissions with the same exploit
and also to install the SucKIT root-kit.
The attacker then tried to get access to the host murphy with
the same account. This failed because murphy is a restricted
machine and its only purpose is to act as list server to which only
a small subset of developers can log into. Since the initial login
attempt didn’t work the person used his root access on master to
access an administrative account which was used for backup purposes
and gained access to murphy as well. The SucKIT root-kit was
installed on this host as well.
On the next day the attacker used a password sniffed on master
to log into gluck, get root there and also install the SucKIT
root-kit.
The forensic analysis revealed exact dates and times when the
program /sbin/init was overwritten and the root-kit installed. The
analysts also discovered the executable file which was used to gain
root access on the machines, which was protected and obfuscated
with Burneye. Upon unwrapping and disassembling the exploit,
security experts discovered which kernel bug was utilised.
An integer overflow in the brk system call was exploited to
overwrite kernel memory (change page protection bits). By doing so
the attacker gained full control about the kernel memory space and
was able to alter any value in memory.
Even though this kernel bug was discovered in September by
Andrew Morton and already fixed in recent pre-release kernels since
October, its security implication wasn’t considered that severe.
Hence, no security advisories were issued by any vendor. However,
after it was discovered to be used as a local root exploit the
Common Vulnerabilities and Exposures project has assigned
CAN-2003-0961 to this problem. It is fixed in Linux 2.4.23 which
was released last weekend and in the Debian advisory DSA 403.
Linux 2.2.x is not vulnerable to this exploit because boundary
checking is done before. It is also believed that Sparc and PA-RISC
kernels are not vulnerable since user and kernel addresses are
stored in different address spaces on these architectures.
Please understand that we cannot give away the used exploit to
random people who we don’t know. So please don’t ask us about
it.
Recovery
After the machines were shut down, images of the compromised
hard disks were created and stored on a separate machine. They were
distributed to the people doing the forensic analysis. The three
machines in the US (master, murphy, gluck) were reinstalled
afterwards and their services re-instated one by one after
investigation by the relevant service admin.
On klecker, however, this was postponed for a scheduled
maintenance so the security archive could be brought online again
sooner than the other services. At that time we also didn’t have
console access to klecker, so recovery had to be done remotely.
After a disk-image was made via serial console login to a local
machine on a firewalled network connection, the root-kit was
removed, the kernel exchanged and hardened, binaries double-checked
and the security archive verified against several different
external sources. This machine will be re-installed in the next few
weeks.
As a security precaution all developer accounts were disabled in
LDAP and SSH keys removed on the more important machines, so that
no more machines could be compromised. This, however, effectively
disabled just about any public Debian work that involved uploading
files and accessing the CVS repositories.
All passwords used on quantz (i.e. all Alioth, arch and
subversion passwords) have been invalidated as well. All SSH
authorized keys have been removed as well. Please use the lost
password system to receive a new password at:
https://alioth.debian.org/account/lostpw.php
When all services are running again and the machines are
sufficiently secured, LDAP will be reset so that developers can
create a new password again (<http://db.debian.org/password.html>).
It can’t currently be predicted when this will happen, though.
Upon recovery SSH was re-installed on the compromised machines.
Hence, there are new RSA host keys and key fingerprints for these
hosts. The keys will be included in LDAP as soon as they are
created and can be taken from <http://db.debian.org/machines.cgi>.
Consequences
!! Renew your passwords! !!
Since passwords were sniffed on the compromised hosts, any
outgoing connection that involved a password is to be considered
compromised as well, i.e. the password should be considered known
to the attacker. It should therefore be changed immediately.
Additionally, if somebody had access to a Debian machine and was
using the same password or passphrase on other machines or keys we
strongly advise to change the password or passphrase respectively
as soon as possible.
If an SSH key was generated or stored on one of these machines
and was used to log into other machines (i.e. by installing it in
.ssh/authorized_keys), it should be removed as well.
The secret GnuPG/PGP keys which were found on debian.org
machines were also removed from the Debian keyrings and thus
deactivated.
Developers who are worried about their own machines should at
least run chkrootkit and watch its output. Matt Taggert maintains a
backport of the current version for woody at the following
address:
deb http://lackof.org/taggart/debian woody/chkrootkit
main
deb-src http://lackof.org/taggart/debian woody/chkrootkit
main
Additionally, a detailed list of precaution issues is provided
by Wichert Akkerman and Matt Taggart at:
http://www.wiggy.net/debian/developer-securing/
SucKIT Root-Kit
SucKIT is a root-kit presented in Phrack issue 58, article 0x07
(“Linux on-the-fly kernel patching without LKM”, by sd &
devik). This is a fully working root-kit that is loaded through
/dev/kmem, i.e. it does not need a kernel with support for loadable
kernel modules. It provides a password protected remote access
connect-back shell initiated by a spoofed packet (bypassing most
firewall configurations), and can hide processes, files and
connections.
Usually, SucKIT is launched as /sbin/init at system bootup,
forks to install itself into the kernel, start up a backdoor, and
launches a copy of the original “init” binary from the parent (with
pid 1). Any subsequent executions of /sbin/init are redirected to
the original init.
TESO’s Burneye Protection
Burneye is a means of obfuscating ELF binaries on the UNIX
platform presented in Phrack issue 58, article 0x05 (“Armouring the
ELF: Binary encryption on the UNIX platform”, by grugq & scut).
Using tools like TESO’s Burneye, an attacker can alter an
executable program to encrypt its true purpose, hiding it from
firewall filters, intrusion detection systems, anti-virus software
and the prying eyes of investigators.
Thanks
- James Troup and Ryan Murray for their general work on all
hosts - Adam Heath and Brian Wolfe for their work on master and
murphy - Wichert Akkerman for his work on klecker
- Dann Frazier and Matt Taggart for their work on gluck
- Michael Stone and Robert van der Meulen for their forensics
work - Marcus Meissner for disassembling the used exploit
- Jaakko Niemi for his work on checking and re-enabling
lists.debian.org - Colin Watson for his work on checking and re-enabling
bugs.debian.org - Josip Rodin for his work on checking and re-enabling the lists
web archives