UNIX-Based Investigations

UNIX File System Analysis

Briefly reviewing the UNIX file system, each disk drive is divided into one or more disk partitions with each containing a single file system. Within each file system is a list of inodes and a set of data blocks. Each inode holds almost all of the information that describes an individual file, including the size of the file, the location of disk blocks, etc. Inode numbers and their corresponding file names are stored in directory entries.

Data blocks are blocks of regularly sized data. The UNIX file system divides any data request to or from a file into logical blocks of data that correspond to the physical blocks on the disk. The downside of this methodology is that the data blocks are not necessarily contiguous to one another. Typically, the UNIX file system uses 8 KB as the size of its logical data blocks.

In UNIX when a file is deleted, the name remains in the directory, but the inode number, the name to which the name points, is removed. The inode itself is changed; consequently, the ctime is updated and the data block location is erased. As a file is deleted, UNIX decrements the inode's internal link to zero.

Removing all directory entry file name inode pairs performs this erase action. When the inode is deleted, the kernel marks its resources as available. The inode still contains all the data about the file and remains until it has been reallocated and overwritten. Having inodes containing some data but having a link count of zero reveals deleted inodes. Without having the file's content available, investigators can learn much about the file with only the metadata remaining in the directory entries and inodes.

To mount a file system, the UNIX kernel needs the sizes and locations of the file system metadata. The first piece of metadata is the super-block and it is stored at a known drive location. The super-block contains information such as the number of inodes and data blocks, size of a data block, etc. Predicated on the information contained in the super-block, the kernel is able to calculate the locations and sizes of the inode table and data portion of the disk. Inodes and data blocks are clustered together in groups scattered across the hard drive media. Usually UNIX maintains more than one super-block, one inode table, and one block array in the event of a disastrous data loss.

Undeleting UNIX

Undeleting UNIX files is different than handling Windows restoration issues. UNIX can be configured to make frequent backups, and it is not unusual for backups to be made hourly. Hopefully, there will be an easily accessed backup and it may be stored in /class/.snapshot. Examining this file will reveal that it is a backup of the home directory from several points in the past. It is possible to copy from this directory the items lost or corrupted in the regular home directory.

There is a UNIX command that should find which directory the home account is stored in: find/class/.snapshot/hourly.0 -name $USER -prune 2>/dev/null

Backups may be in the following path examples:

• /class/.snapshot/hourly.0/01

• /class/.snapshot/hourly.1/01

• /class/.snapshot/weekly.0/01

• /class/.snapshot/weekly.1/01

Retrieving lost e-mail is a very similar process. UNIX makes copies of e-mail in a similar manner as it makes a copy of the home directory. Access to copies of e-mail can be accomplished through /var/mail/.snapshot. By reviewing this directory, backups of deleted e-mail may be seen and captured. Be mindful that viewing e-mail contained in this folder will require it to be loaded into an e-mail client.

Data Hiding Techniques

Data "deleted" from UNIX files can be located by investigators with hex editors searching for specific files and file extensions. However, there is a technique where malicious individuals may attempt to "hide" data from forensic investigations. For the most part, it is effective in systems not using the Berkley Fast File System or FFS.

Experience Note

FFS deprecated the bad data blocks inode, preventing individuals from hiding data in there.

By way of explanation, the bad blocks inode has been used to reference data blocks occupying bad sectors on the target disk thereby preventing these data blocks from being rewritten by live files. If investigators run the file system checker utility, fsck, it is possible that the file system can been seen as having been radically altered.

The first inode that is capable of allocating block resources on an ext2 file system is the bad blocks inode (inode 1) and not the root inode (inode 2). Because of this positioning, it is possible to store data on the blocks allocated to the bad blocks inode and have it hidden from many forensic tools. Malicious individuals have tools that will allow them to exploit flaws in the UNIX file system and store data outside the view of forensic tools. It is for this reason that investigators should look at the physical level of UNIX bad inode blocks before moving on to other areas.

Coroner's Toolkit

Responding to reports of critical incidents happening on UNIX-based systems, investigators establish a timeline of events beginning from the last time the system was stable and uncorrupted through the actual event and until it was discovered and brought to a halt. The Coroner's Toolkit is a collection of utilities that attempt to gather and analyze data in the target UNIX-based system where investigators can accomplish their goals.

The Coroner's Toolkit contains the following data-gathering tools:

• grave-robber, the main data-gathering program

• file, Ian Darwin's file command

• icat, copies a file by inode number

• ils, list file system inode information

• lastcomm, a portable lastcomm command

• mactime, the MAC time file system reporter

• md5, the RSA-based MD5 digital signature tool

• pcat, copies the address space of a running process

• unrm, uncovers unallocated blocks from a raw UNIX file system

• Lazarus, attempts to resurrect deleted files or data from raw data

Within the Coroner's Toolkit, available at www.porcupine.org/forensics/, there is a tool called unrm that can emit all the unallocated data blocks on a UNIX file system. It functions by reading the list of free data blocks, locating each logical block, and looking to see if they contain any blocks of unallocated data. The unrm should be used if investigators are looking for a specific file known to be deleted.

Another effective data recovery tool is Lazarus. Lazarus attempts to give unstructured data some structure that can be viewed and manipulated. Lazarus depends on UNIX never writing file data except in well-defined boundaries. UNIX generally writes files in contiguous data blocks, when possible, attempting to boost performance. For this reason, UNIX should never need a defragmenting utility, unlike some other popular operating systems. Lazarus maps the disk that is created and provides visibility into the disk seeing the data by content type. Lazarus is a very comprehensive program in that it takes a very broad view of deleted files.

Using unrm and Lazarus will fill significant amounts of disk space on the forensic machine. Because Lazarus takes a large view of its work, it does not run in just a few minutes, so investigators are advised to let it run for several hours, if not days, before being able to see the results.

Using unrm and Lazarus is not as easy as using a Windows file recovery tool, using them is a time consuming and laborious process. It would best be described as a "hit and miss" process.

Experience Note

There is an interesting series of short articles about UNIX/Linux file system recoveries located at recover.sourceforge.net/linux.

Success rates restoring deleted UNIX files are spotty at best. The easiest way to restore deleted files from a UNIX system is to access an uncorrupted backup copy. UNIX administrators should back up all critical files on a regular basis observing the risk manager's axiom… the more important a file is, the more often it should be backed up.

Hiding Files

Users will go to many extremes to cover their tracks. Regrettably, employees and attackers employ a variety of ways to hide their nefarious acts by concealing files, including:

• Placing them in storage facilities located outside the workplace

• Secreting data in hidden hard disk partitions hoping no one will think to look there

• Encrypting data partitions of their hard drives with complex algorithms

• Encrypting data through the use of steganography

Web sites provide storage for users and may be accessed from any system with Internet access. For a small fee, files may be securely and privately stored on remote sites ready for access at some future date. In order for investigators to locate such services on target machines, it is recommended that a thorough review is made of pertinent files resident on subject's computer. Look for URLs in the history, temporary, and bookmark files indicating that files may have been stored there. If you have a very sophisticated user who could conceal his activities, check the user logs where access is made from the inside to the outside network for online storage sites.

In the case of having files, storage media, and hard drives encrypted by a password, investigators may try to obtain access by running a password cracking application. It is important to identify the application such as Word, Excel, or WordPerfect, as many cracking applications are specific to individual programs. With password protected files, it may be possible to run a tool like John the Ripper and brute force the password.

With encrypted hard drives, if investigators can identify the protection application, there are some manufacturers that provide a universal password that permits access in the event the hard drive owner forgot her password. By way of application, the encryption key is based on the password. If the drive's content is completely encrypted, consequently, working at a physical level is not going to provide any insight. Having the password is the only way to access an encrypted drive.

Steganography

Steganography is a concealment application where information is hidden in plain sight. By secreting data in an otherwise innocuous multimedia object (usually image or sound files) called carriers, steganography can hide information remaining essentially impervious to detection. Steganographic applications accomplish their task by first hiding the data's existence in the carrier and then encrypting it. Detecting these data-carrier files is a two-prong problem; first the multimedia file containing the data must be identified and then the data must be deciphered. With a workstation containing possibly hundreds or even thousands of such files, the task is formidable indeed.

Investigators can possibly determine if they have a steganography user by locating the program on the workstation's target hard drive or locating it at another location. Finding such an application might indicate the user has encrypted files with hidden data in them. Placing image or sound files in a steganographic application will result in a password prompt. This prompt does not necessarily mean the target file contains hidden data, so these tools cannot be used to screen potential files for hidden data. Most steganographic applications will prompt for passwords whether the file has data concealed in it or not.

There is one tool that claims to be able to detect steganographically hidden .jpg files and is available at www.outguess.org/download.php.

A wide variety of steganography tools are available at members.tripod.com/steganography/stego/software.html.

Creative investigators may well be served to obtain the passwords from the owner before attempting to brute force passwords in an effort to decipher encrypted files and drives. Failing this effort, using a technique such as the installation of a keyboard monitor to capture all the user's keystrokes may prove to be the answer. Outside of law enforcement, employers may implement keystroke monitors where employees do not have any reasonable expectation of privacy in their workstation or systems use.

Experience Note

Within the statutory constraints applicable to law enforcement agencies, search warrants and court ordered wiretaps govern the use of keyboard monitors.

Software keyboard monitors and similar tools made by Spector Soft are available at www.spector.com.

Hardware keyboard monitors are available at www.keykatcher.com/index.htm.

Locating hardware keyboard monitors is a matter of tracing desktop cabling. These are small cylindrical or box-shaped devices that are placed in-series with cables connecting keyboards to desktop towers or between the desktop and the network. Investigators should be aware that keykatcher makes a keyboard that acts as a keyboard monitor eliminating the inline device.

Finding hardware keyboard monitor information is available at www.spy-cop.com/keyloggerremoval.htm.

Finding software keyboard monitor software information is available at www.spy-cop.com/spycop-free-product.htm.

Before using such a technique, it would be prudent to consult with prosecutors or corporate legal counsel. Also, consult with your legal counsel before attempting to provide the results of a keyboard monitor to law officers.

Strong Encrypted Protections

There seems to be a growing use of encryption with very strong protection features. While there are many talented investigators and analysts in the world today, the conversion of encrypted data to plain text data in most cases is virtually impossible. Unless data may be captured from the target keyboard through legal means, investigators have to accept the idea there will be information that cannot be accessed within the limits of current technology and resources.

File Recovery Alternatives for UNIX/Linux

There is an alternative file recovery utility for Ext2 files systems used in Linux and some flavors of UNIX. It uses proprietary technology and flexible settings providing control over the data recovery process. It is called R-Linux and is available at www.rtt.com/RLinux.shtml. It is interesting to note that R-Linux is a Windows-based utility used to recover Linux data.

Responding to Windows NT Incidents

There is an old adage: "you've got to use the right tool for the right job." Responders must have the right tools in anticipation of the most common set of circumstances, so they are not looking around for their tools when precious time is wasting and profitability is declining.

Tools in the Tool Bag

In Windows operating system environment, there are two basic types of utility applications, those based on a Graphical User Interface (GUI) and those that are based on command line interface (CLI).

Following is a list of tools that are available at www.sysinternals.com:

• PsTools v1.56: The PsTools suite includes command-line utilities for listing the processes running on local or remote computers, running processes remotely, rebooting computers, dumping event logs, and more.

• Tokenmon v1.01: View security-related activity, including logon, logoff, privilege usage, and impersonation with this monitoring tool.

• Filemon v4.34: This monitoring tool lets you see all file system activity in realtime. It works on all versions of WinNT/2K, Windows 9x/Me, Windows XP 64-bit Edition, and Linux.

• PSLoggedon: An applet that displays both the locally logged on users and users logged on via resources for either the local computer or a remote one.

• TCPView v2.22: See all open TCP and UDP endpoints on Windows NT, 2000, and XP. TCPView displays the name of the process that owns each endpoint. Full source to the command-line version of this tool, netstat, is included.

• NTFSDOS Professional v4.0: Full read/write access to NTFS drives from DOS.

If investigators are going to use floppy disks or CDs, they must be rendered write-protected after writing.

There are several schools of thought concerning the use of tools in responding to critical incidents. Some responders have experienced vigorous cross examinations at the hands of knowledgeable attorneys where they did not keep copies of their programs and tools so these they could be examined by the opposing side's experts. Because this seems to be a current trend in qualifying witnesses, you must be sensitive to this tactic and ensure the versions of your tools are logged as part of your investigation. Investigators should maintain versions of all relevant tools so these tools can be produced when necessary.

Storing the Data

During the course of the response, there will be a lot of information gathered from the system. Consider the area where the incident has occurred as a crime scene because if investigators take the most restrictive posture when they respond, then should the matter proceed to court, their evidence should be introduced.

All media intended to be used to duplicate evidence must be cleansed using software intended for that exact purpose. This cleansing process includes all blank CDs, zip disks, jazz disks, tapes, floppies, hard drives, etc.

Experience Note

Arriving on the scene is not the time to begin your preparations. Do you really want to take the stand and testify to your lack of professional diligence?

To Turn Off or not to Turn Off

If responders arrive at the scene before the system has been turned off, they might consider efforts to collect valuable evidence that could be lost otherwise. It is a matter of priorities. They should be included in the decision to be made by senior managers as part of the response posture. The balance is this one, if turning off the system will stop the progress of any further damage and whether turning off the system will likely result in the loss of evidence. Response postures should be certain to error on the side of caution and turn off relevant systems containing spreading damage. Following the firefighter model, it is a matter of business sense to contain the damage before worrying about evidence.

If the decision to keep the victim-system online, here is a list of items that should be considered as volatile and might disappear when the system is turned off:

• List of users logged onto the system

• List of currently running processes

• List of currently open ports

• List of currently listening services on their respective ports

• List of systems currently connected to the target system

When investigators approach the target system, they should have a plan outlining their general activities. Before anything actually takes place, an activity log should be initiated and maintained documenting all steps and their results. Log entries should include any and all tools deployed, system and application commands, who performed the action, the date/time/place, etc.

Essentially there are two reasons for maintaining an activity log, to gather information that will permit the reconstruction of the response-activities at a later date and protect the organization by demonstrating the responders exercised professional due diligence. More than once, logs have effectively answered legal and policy-compliance challenges.

System Users

If the response posture requires that an investigation proceed while the system is still active and the attacker is online, using the program, Psloggedon, written by Mark Russinovich, www.sysinternals.com, shows all users connected locally and remotely. If the system offers dial-up remote access, the investigators should determine the user accounts having remote-access privileges on the target system at the time of the incident. Depending on the number of logged on accounts, investigators may wish to remove the telephone lines, disconnecting online activity.

There is a command line tool, as part of the Remote Access Service, RAS, called rasusers that can be used to determine the users that have remote access to the target system. Rasusers is available at http://wettberg.home.texas.net/rasusers.htm.

Open Ports and Listening Services

The next step may provide one of the most significant steps in the real-time investigation. Determine what are the open ports and listening services. A handy tool, fport, is available from the Foundstone Web site at www.foundstone.com. This tool will show all listening processes.

The display format for fport is:

• Process identification

• Process name

• Port, Protocol

• Path

Forensic Investigation: Not Exactly a Needle in a Haystack

These are some logical areas that may interest an investigator in locating digital evidence:

• File space. This refers to blocks on the drive that either are assigned to an active file or assigned to the file system depending on the structure such as FAT (Windows) or inode (UNIX). Of course viewing interesting files from file space is merely a matter of using a disk editor, locating the file, and copying the file to another media for viewing by the investigator. In this fashion, the original media does not suffer from being changed.

• Slack space. This is the space made up of the file system blocks that are partially used by the operating system. Slack space is prevalent in file systems that have written to a sector, then overwritten that space with the newly written information not occupying the entire sector creating a slack space containing data from the previous data. Tools like EnCase or a disk editor will allow investigators to see the "junk" contained in the slack space. Slack space seldom contains enough information to see the entire file, however there is often enough information to interest investigators. File names, file extensions, and pieces of text files are the usual finds.

• RAM space. RAM space is the term used to describe empty space between the data and the end of the sector. If there is an empty space, the operating system selects information from the data currently in RAM and writes it there. It can be similar to slack space in appearance.

  Experience Note

  An investigator conducting an analysis on a target hard drive was able to effectively refute allegations made by a defendant that he had never installed pirated software on his workstation. The defendant had installed a number of expensive applications on his workstation and deleted them and attempted to write over the disk space. However, there were enough data left in the slack space to demonstrate he had indeed installed these applications. The most incriminating evidence was the extensions of the application's files.

• Unallocated file space. Any unclaimed sector falling within an active partition or not.

• Unclaimed sectors can often be restored by Undelete utilities depending on the operating system and if the unallocated file space is partially overwritten or not.

Physical Level Search

Investigators should consider begin looking at the raw data contained on the target media. Often these analyses are performed with tools like a disk editor or EnCase. With the forensically correct duplicated software, many experienced investigators will perform these principle processes:

• String search

• Slack space

• Free space examination

All analysis operations must be performed on the forensic image or the restored image of the evidence. Never perform examinations on the original evidence.

There is a frequently pursued avenue in running string searches to produce lists of data; for example:

• All e-mail addresses

• All Web site URLs

• All gif and jpeg file extensions

• String searches matching specific words

• String search

Experience Note

There is a very handy DOS-based program called SearchString written by Dan Mares. It is available at www.maresware.com. This tool provides the context of the string search hit as well as the location being the byte offset from the beginning of the file. By inputting the specific string to be searched, this tool will scan the target media and produce the relative location of the item.

Also, most disk editors have well-developed string search capabilities. Many experienced investigators use disk editors to search for file extensions that are pertinent to the case, e.g., eml, png, gif, jpg, doc, txt, or exe.

File Slack and Free Space

Depending on the operating system's file system, there will be residue that can be located and examined when looking for evidence. File residue basically falls into two categories, file slack and free space.

Free space is that space located on a hard drive that is not allocated to a file. It can be space that has never been allocated to a file or space that is considered unallocated. This unallocated condition usually occurs after a file has been deleted. Unallocated file space occurring after a file has been deleted will often contain remnants of the deleted file. Fragmented data previously written could still reside in these areas and not be easily accessible to the everyday user. In order to gain visibility into these areas, it is necessary to work on the physical level.

In the case of slack space, this occurs when data is written to a storage medium in measures that fail to completely fill the block size as it is defined by the operating system. Investigators attempting to look into this area for evidence will also have to work beneath the operating system at the physical level of the medium.

Experience Note

An employee had been downloading obscene images to his work-station and subsequently deleting them. After a time, he performed word processing and other types of work thinking these had overwritten the images he had previously downloaded and would make viewing the images impossible. Fragments of these images and their file extensions were contained within the slack space and unallocated file space of his workstation hard drive. After forensically imaging the hard drive, investigators peered into slack areas using a disk editor. Investigators were aware that most photographic-quality image files have extensions such as .gif, .jpeg, and .png. They merely used the find function of the disk editor to perform a string search for these extensions. Experience and training taught them that deleted files in DOS-based operating systems are preceded by the σ character (lower-case sigma) and are listed with a hexadecimal value of E5h. They easily located the deleted files. After completing their search, they were able to identify the nature of the deleted files by their names and extensions and even recover some of the image fragments.

DOS-Based Operating Systems File Deletions

The file deletion process in DOS-based operating systems is a two-step process. In the first phase, the operating system marks the file entry with a lower-case sigma character× σ. This character has a hexadecimal value of E5h. In phase two, it clears the FAT chain marking all data blocks as empty. In principle, many operating systems handle file deletions in similar fashion.

Using an undelete utility, like Norton's Utility suite, the file recovery software searches the file directory tree for file names beginning with σ and labeled with the value of hexadecimal E5h. Once found, the utility starts at the file cluster offset that is specified in the directory entry. If the file cluster is not claimed by another file in the block allocation table (FAT), then the utility will indicate the file has a good chance of recovery. Many commercial file recovery utilities will reconstruct the deleted file by replacing the sigma character with another recognizable character and rebuild the FAT table. In processing, the utility looks to the file size specified in the directory entry and determines if that block is free. If it is possible, the program will advise that the file has a good chance of being recovered.

Reading E-Mail Headers

As it appears in your e-mail client, it seems that e-mail is passed directly from the sender to the recipient without any intermediate steps. Typically, an e-mail passes through at least four computers in its route. In the case of an ISP whose users connect via dial-up, DSL, Cable Internet, or T1, the client is the user's machine and the actual mail server belongs to the client's ISP. To review the process, when a user sends e-mail, she normally composes the message on her workstation and sends it off to either the mail server located within the company of the ISP. At this point, her workstation usually keeps a copy of the e-mail in the send folder. Even if she deletes the contents of the send folder, the e-mail will reside in the deleted folder until she deletes them from this folder.

Experience Note

It is possible that the e-mail client is configured to automatically empty the deleted folder, but as you have seen, there are ways to recover deleted files.

From her workstation, the e-mail server receives it and the server begins to look for the recipient's e-mail server, exchanging information packets with this server and eventually delivering the e-mail message. It does not really matter whether she is sending her e-mail through the Internet or merely within her own organization. For practical purposes, the process is basically the same. This e-mail will reside on this server until the recipient accesses his e-mail client and reads the e-mail. Of course, there are times depending on the type of e-mail configuration and the type of e-mail server, the e-mail server retains a copy of the e-mail or downloads the e-mail to the recipient's e-mail client located on the workstation. It is very possible that although the e-mail was downloaded to the recipient's workstation and the account emptied of the e-mail, there is a copy of the e-mail located on the e-mail server's backup storage. Tenacious investigators will pursue the chances of obtaining a copy of the e-mail from one of the many e-mail servers involved in the message transmission and receipt.