Tom Kemp's Centrify Blog

Fannie Mae Incident Reveals Need to Manage and Monitor UNIX Root Access

Monday, February 2, 2009

Even though most of us in the IT industry know about the threat of insider attacks, it was still shocking to read the recent headlines that a former UNIX engineer at mortgage giant Fannie Mae was charged in federal court with planting a logic bomb that would have effectively shut down all 4,000 servers at Fannie. An FBI affidavit filed in the case reveals that this logic bomb planted in a script would have "caused millions of dollars of damage and reduced if not shutdown operations" at Fannie for at least one week. The affidavit, which refers to Fannie Mae as company "ABC," later states that "if this script were executed, the total damage would include cleaning out and restoring all 4,000 ABC servers, restoring and securing the automation of mortgages, and restoring all data that was erased." Very serious stuff. I am going to use this blog post to give some details from the affidavit regarding what happened at Fannie Mae and discuss how Centrify's products can help enterprises avoid something like this happening to them.

So here's what happened at Fannie: on October 24, 2008, a UNIX engineer was terminated because two weeks before he had "erroneously created a computer script that changed the settings on the Unix servers without the proper authority of his supervisor." He was terminated between 1 and 1:30 p.m., and was told between 2 and 2:30 pm that "all of his ABC equipment to include his badge, laptop, etc., needed to be turned in by the end of the day." He did not turn in his laptop until 4:45 pm and presumably left the building around 5 pm. What is interesting is that once the engineer was terminated, his computer access was not immediately terminated — he was terminated at 1pm but his account was still active until late that evening.

This time window gave him the unfortunate opportunity to modify a script which he did at 2:53 p.m. when he ssh'ed into a development server as root. He proceeded to modify a legitimate maintenance script with some very destructive logic that would run on all 4,000 UNIX systems that run Fannie's operations. According to the FBI Affidavit the script would do the following on January 31, 2009:

"The script would then run .z.sh which would create a list of production, contingency, and backup servers and then run .a.sh on all of ABC servers. Script .a.sh would disable all logins and clear out all logs, including all the logs that revealed MAKWANA's access to the dsysadmOI server on October 24, 2008 thus eliminating his "footprints." If a ABC engineer attempted to login a message "Server Graveyard" would be displayed. This script would also remove the root password appliance access to the server by updating the password and shadow files so no one could change the root password from the password appliance. Next .a.sh would build a list of all servers that contain ABC data and wipe out all of the data and replacing the data with zeros. This would also destroy the backup software on the servers making the restoration of data more difficult because new operating systems would have to be installed on all servers before any restoration could begin. The script would also remove all "High Availability" software from any critical server. And finally this script would power off all servers, disabling the ability to remotely turn on a server. Subsequently, the only way to turn the servers back on was physically getting to a datacenter to turn the servers back on. This script would propagate itself out to all 4,000 ABC servers, thus damaging all ABC data."

As you can see, a very nasty script — replacing all the data with zeroes!! — and it seems he had more than a few minutes to write and embed this script. Basically it would have knocked Fannie offline for a week and the remediation would have included "cleaning out and restoring all 4,000 ABC servers, restoring and securing the automation of mortgages, and restoring all data that was erased."

Luckily a week after this script was planted, another engineer found this logic bomb embedded in a standard maintenance script — and it was "only by chance" that this logic bomb had been found. Fannie Mae then turned things over to the FBI and now we see this indictment.

So given this stark example of an insider attack, how can Centrify and our Centrify Suite help an enterprise avoid a situation that Fannie avoided "only by chance"? Here's a few ways that come to mind:

  • Because DirectControl introduces the paradigm of having users and IT staff authentication (i.e. login to) non-Microsoft systems (e.g. UNIX, Linux, SAP, etc.) using their Active Directory credentials, it helps organizations break the bad habit of IT staff sharing the UNIX root password (which is what this person had access to) and the password to other privileged accounts, and instead have personnel log in with an account that clearly identifies who the user is. This means when an IT staffer is terminated, you don't have to change the root password on 4000 systems, as they were not sharing the root account to begin with. It also provides the ability to immediately disable a given user's access across the enterprise (i.e. all UNIX and Windows systems) in one place — Active Directory. This is key because in most instances access to UNIX systems is typically managed locally (i.e. /etc/passwd), and it may take many hours (if not days) to go around and check and turn off a user's potential access to 4,000 systems. In the case of Fannie it took over 10 hours to disable the UNIX engineer's access, well after he had left the building.
  • DirectControl not only controls who can authenticate to your mission-critical UNIX systems, but its patented Zone technology also provides granular access control to further limit which users can authenticate to which particular systems or groups of systems.
  • Our DirectAuthorize solution provides centralized, role-based privilege management features help you manage and enforce fine-grained control over user access and privileges on UNIX and Linux systems. By controlling how users access systems and what they can do, DirectAuthorize enables you to lock down sensitive systems and eliminate uncontrolled use of root accounts and passwords.
  • Finally, our DirectAudit solution helps you spot suspicious activity by showing which users accessed what systems, what commands they executed, and what changes they made to key files and data. In the case of Fannie, the UNIX engineer's every keystroke and the actual text of the script he was editing would have been captured and the FBI would be able to "playback" (like Tivo or a VCR) in court the session in which the UNIX engineer created the logic bomb and implanted it in the maintenance script. Check out this blog post on additional ways the Centrify solution can audit UNIX and Linux systems.

I think this incident will further heighten the awareness and need for tools to enable superuser privilege management, which ironically I blogged about just a few weeks before this incident was reported. And definitely check out this webinar we did with Gartner that has an in-depth discussion of how organizations that cannot definitively link individual users' access to and activity on key systems risk security breaches or audit failures — which "only by chance" was avoided by Fannie.

< Previous Article: Burton Group on Active Directory Bridge Products
> Next Article: DirectControl Named a Top Mac Client Management Solution