Forming a Critical Incident Response Team

In many descriptions you will see the words "Critical Incident Response Team" associated with critical incidents. Many incident response efforts are unsuccessful, not for lack of planning, but because many mistakes were made in creating a team that was not staffed with knowledgeable, dedicated employees. Many organizations use checklist methods of emergency response because of legal or policy mandates where senior managers think their systems' security is guaranteed because they mark a box. Feeling they have met all legal and policy requirements, they are lulled into a false sense of security.

Experience Note

Locks only keep honest people, well, honest. They will not stop a knowledgeable, persistent thief. When visiting a small police department, a visiting dignitary was shown the department's new gymnasium and locker room. She noticed padlocks on the locker doors and asked the commander giving the tour, the reason why. Without missing a beat, the commander remarked the locks were present on the doors to, "keep honest people honest." Even in the police department, they were respectful of each other's belongings but they kept them secure by locking them up.

Security controls have the purpose of making unauthorized entry so unattractive and difficult, they compel attackers to go elsewhere. The only truly effective security systems are those that render important systems inoperable. Of course, that condition is ridiculous. Systems security before, during, and after a critical incident exists as part of the whole picture of good business practice. It ensures uptime, efficiency providing critical systems needed for daily business operations. The purpose of security is to preserve what belongs to the organization from being stolen, deleted, or modified. So, what happens when an attacker, inside or outside the organization, causes a critical incident?

Most organizations have long understood the importance of having fire suppression equipment installed in data-centers, emergency exits, and employee training for emergencies. These same organizations have extensive information security measures with firewalls, DMZs, VPNs, and physical security. Safeguards, like these, have the purpose of maintaining the organization's property and reputation in the community.

Regrettably, critical incident response and management are often neglected until a catastrophe actually strikes and the organization finds itself scrambling to recover.

Experience Note

Critical incident management is determining which assets are needed to sustain profitability (profitability means the organization is accomplishing its goals), establishing policies and procedures addressing employee conduct, compliance audits and mechanisms to actually address crises when they occur.

CIRT

CIRTs should be composed of team members with specific roles supported by specialized training and experience. The CIRT must have a function-point or coordinator where all reports of critical incidents are made. The function-point is usually an individual senior employee or member of a business unit having significant managerial and business experience. She possesses a clear understanding of the organization's goals and objectives, and probably participates in the drafting of the business' operational plans sometime in her career.

It is not expected this person would have a complete knowledge of the organization's mission, goals, policies, and procedures, but it is important that she have sufficient knowledge. For the function-point person to deliver services, she must be available 24 hours, holidays, and weekends. Contact may be accomplished through telephone or other expedient means.

Under practical circumstances, it is immaterial whether the organization decides to use its own in-house talent or delegates the responsibility to outside consultants. The first contact is the employee who receives information relating to the critical incident and makes several important decisions relating to it:

• Does an actual critical incident exist?

• Where is the critical incident occurring?

• What is the extent of the damage?

• Has the damage been contained or is it continuing?

• Do I need to triage the damage at this moment?

• What resources do I need to deploy at this moment?

• Do I have the resources to address this crisis at this time?

• Do I have sufficient information to deliver a meaningful report to senior managers?

• When should I notify law enforcement authorities?

Using Outside Consultants

One of the greatest advantages in using outside consultants (commercial CIRTs) is that of overall reduced cost. This is particularly true in smaller organizations where their operational demands are less than larger organizations. In many cases, contract consultants specializing in critical incident response deal with a wide variety of matters resulting in a high degree of expertise. Additionally, many of their team members have specialties such as UNIX, Windows, or specific programming languages usually not available to employees of smaller businesses.

These are the advantages of commercial CIRTs:

• Most commercial firms have the ability to respond in a matter of hours depending on travel times.

• They provide 24-hour support and are in constant contact.

• They can offer full-service response-posture, as their services usually include forensic duplication and examination, litigation support, expert testimony, technical support, policy formulation, and legal expertise.

• Commercial CIRTs can provide mock-incident response training. Participants address imaginary, but logical, scenarios and interact with personnel, data, and facilities.

• Keeping abreast of current attack-trends. Commercial CIRTs are able to track attacker trends and tailor their response-posture to their clients. By assigning technically trained account executives to clients, they anticipate malicious behavior and are prepared to marshal resources accordingly.

Commercial CIRTs vary greatly in their abilities. Senior managers should do their homework before signing contracts for service.

• Be certain to ask for references from several recent customers and do not hesitate to ask for individual employee's qualifications and experience levels.

• Contact their references. Ask the most important question, "would you hire them again?"

• Determine their reputation in the business community by contact entities such as the Better Business Bureau to ascertain if complaints have been filed and are unresolved.

• Depending on circumstances, ask for financial references.

• Determine if they have bonding in the event of future legal action.

Using In-House Talent

The primary reason for initiating and developing an in-house CIRT is the ability to address emergencies observing the organization's policies and procedures. Staffed with employees, CIRT capability can be directed to address emergencies meeting cultural and internal needs. Because critical incidents often involve sensitive or political matters, in-house talents are more likely to address them in a fashion most advantageous to the organization.

In many cases, internal CIRTs are funded through the corporate offices or on a charge-back basis to the individual business units. Some CIRTs are funded through corporate headquarters paying salaries and other recurring expenses while the individual business units pay for the on-site expenses such as travel, lodging, or other expenses.

Here are a few advantages of the internal CIRTs:

• Direct support. Internal CIRTs will provide emergency response to affected business units with greater specific business-practice knowledge than commercial CIRTs. Generally, they have greater sensitivity to corporate culture than equivalent outside firms.

• Risk management, policy, and audit support. Although these functions are usually addressed by an organization's business units having an internal CIRT can provide invaluable input to heightened awareness and effectiveness. After all, the CIRT has a high vantage point from which to gauge their interaction and deliver this experience to risk managers, policy writers, and auditors on a continuing basis.

• Emergency drill participation. An internal CIRT can participate in emergency exercises testing the full range of recovery, resumption, and critical incident response capabilities. An emergency exercise consisting of an unannounced test can measure the effectiveness of personnel, equipment, and procedures. Postmortem critiques, conducted among employees, are generally more productive and sensitive than sessions involving outsiders.

Ad Hoc CIRTs

This is a concept that has gained a lot of favor in the past few years for smaller businesses. Ad hoc CIRTs are developed utilizing existing talent, and where deficiencies are identified training is vigorously sought. For the most part, ad hoc CIRTs are composed of specially trained employees that have regularly assigned duties and when emergencies strike, they form their response team. For this concept to avoid being stillborn, it must have fanatical senior management sponsorship.

Here are a few suggestions for getting an ad hoc version of CIRT off the ground:

• Identify key technical employees that are qualified to address critical incidents. Such experts would include the IT manager, senior systems administrator, senior engineers, legal advisor, risk manager, human resources unit, etc.

• Draft response policies and procedures for the CIRT to screen initial reports, criteria for activation, response activities, and post-incident critique.

• Obtain senior management agreements with a minimum number of hours of participation on an annual basis for CIRT members. Provide financial incentives to employees for CIRT participation.

• Provide specialized training to CIRT members. This should be training that is complementary with their skills.

• Seek to train toward professional certifications. This is one of those incentive areas where CIRT participants can receive certifications qualifying them for advancement.

CIRT Requirements and Roles

As in any plan, the best place to start is with your deliverables and requirements. Experienced planners actually begin at the end by asking, "What is it we need the CIRT to do?" The most basic requirement for an incident response team is providing support and direction in successfully resolving critical incidents with a minimum degree of business disruption.

Basically, CIRTs are support units intended to provide critical incident response support to the organization as a whole and to the affected business unit specifically.

In this tasking, CIRTs usually serve in these potential roles:

• Direct hands-on emergency response where CIRT members are actively engaged in the containment and restoration of critical IT functions. The full-version of this activity is for the CIRT to assume complete response responsibility. Taking this posture potentially alienates employees already assigned to the affected business units. However, in the event of severe circumstances and if mandated by senior managers, this approach is efficient and effective.

• However, this role can suffer from a conflict of loyalties, as the CIRT is sometimes regarded as "big brother" when it appears on the scene and immediately takes control of the situation.

• Advisory/Shared role. In this role, the CIRT acts as a trusted advisor sharing response activities with the affected business unit. There is less conflict of loyalties in this role, meaning responsibilities are shared between entities.

Added CIRT Responsibilities

Because senior managers view full-time CIRTs as responding only when needed, sometimes they get the reputation of having little if anything to do unless they are responding to a crisis. Their perceived usefulness can be expanded by accepting added responsibilities:

• Acting as a problem screening unit. In this capacity, the CIRT acts as a unit where software patches, tools, and updated software versions are tried and tested before being applied. The practical side of this task rests in the CIRT being able to patch a corrupted system and know the patch they are applying has been tested. There is confidence this patch will not conflict with existing systems and is free from malicious code. Additionally, the CIRT acts like a clearinghouse for recurring or particularly troublesome system problems. They work closely with help desk coordinators and system administrators where any indications of critical incidents are reported and a determination is made if the CIRT should be activated either as an entire unit or in part.

• Coordinate inside emergency efforts and establish liaison with outside agencies. The CIRT coordinates the emergency response efforts of all organizational units in the event of a crisis and works to actively facilitate their drill and actual crises. The CIRT is tasked with the development of liaison with law enforcement and regulatory and legal entities. It actively seeks to participate in such entities as NIPC (National Infrastructure Protection Center), Infragard, HTCIA (High Tech Crime Investigators Association), ISACA (Information Security Audit and Control Association), ISSA (Information Systems Security Association), and the ACFE (Association of Certified Fraud Examiners).

• Provide training inside the organization and to outside entities. CIRT members should be in a very good position to deliver specialized training and increased awareness as one of their proactive jobs. Consider having the CIRT author technical articles in professional periodicals thereby benefiting them by having to do the research and delivering information to other professionals. Through this means, team members learn about developments and emerging trends while potentially providing a valuable service to their constituents and their organization.

CIRT Funding

Funding CIRTs, as are most business matters, is merely a matter of funding. Sometimes developing resources is more a matter of convincing bean counters of their value than anything else.

Here are a few basics to consider when developing your CIRT:

• CIRT as part of the IT function. Locating a CIRT as part of the organization's IT function can greatly facilitate productive interaction between the lessons learned as a result of responding to emergencies and improving development processes. Placing the CIRT as part of the IT function creates avenues of communication between responders and systems staff.

• Business units may benefit by having the CIRT as part of their operation. For example, the systems development unit could greatly benefit by having the knowledge and skills of the CIRT integrated as part of their operation. Having the CIRT as part of the IT audit unit could provide increased granularity and direction in audit programs.

• Corporate headquarters may wish to fund the cost of the CIRT's activities charged as an overhead cost to each of its business units. In this fashion, the cost of having the CIRT is spread to all affected business units, saving each unit from having to make preparations and fund critical response programs. In this fashion, there is an avoidance of duplicating response efforts between headquarters and the individual business units saving time and money. By adopting the "big picture" view, it allows the CIRT to respond to emergencies on the corporate level where trouble spots can be more readily recognized and addressed.

Who Does the CIRT Support?

The quick answer to this question is everybody. However, for a CIRT to adequately function, it must understand the people it serves mission and goals. For CIRT managers, it is suggested they track units calling for their services so they may gear their response accordingly. It is likely the same business units are requesting services time and again; consequently, it is important for CIRT to service their requests as if they were favored clients. For example, if the business units primarily supported by the CIRT consisted of systems users rather than findings produced by IT audit reports, CIRT's response would be less technical than the response delivered to the auditor's findings. Responding to the auditors would probably require more forensic skills than responding to worms and viruses encountered by users.

CIRT Communications

CIRT members should be mindful that their clients are the business units they service. Misplaced, flippant, or capricious remarks return poor dividends. Communications between the CIRT and the units it supports is not just something that is casually performed; it must be a matter of deliberation and coordinated efforts.

CIRTs should have specific communications goals when measuring their success:

• Timeliness. CIRTs must deliver information in a timely fashion. The means by which the information is transmitted may be e-mail, telephone calls, faxes, voice mail, company Web sites, memos, conferences and workshops, working groups, seminars, and bulletin boards. Basically, employees served by the CIRT should have information as soon as it is discovered. For example, the CIRT becomes aware that the BUGBEAR.exe virus is in the wild. The most efficient way to deliver a message warning about the proliferation and damage this virus can cause is by sending a voice mail message to each employee warning that e-mail attachments should not be opened. Sending an e-mail to each employee may result in the information arriving too late, as the employee may be checking e-mail by opening an attachment before getting to the CIRT warning. Timely and credible warnings will go a long way to developing the CIRT's position and credibility in the organization.

• Relevant communications are a must for a CIRT. If the units supported by them are primarily Windows platforms, it does little good to deliver information about UNIX and OS/2. Communications should be crafted so they are meaningful to their recipients.

• Digestibility. The intended audience must understand CIRT communications. For example, if the CIRT primarily serves workstation users, the CIRT should not craft exhaustive communications dealing with the technical aspects of UNIX server configurations. Granted, there might be readers who enjoy the finer aspects of server configurations, but the broader appeal will be to the majority of the users. Reserve specialized information to specialized employees.

  CIRTs should be mindful that there are many levels of employees that are going to read their material, including managers. Including a brief executive summary at the beginning of the communication is appreciated. Depending on the audience, it may serve to have two or even three versions of a communication to be disseminated. One version would be delivered to the general user population, one version to be sent to the managers, and another version intended for the technical staff.

• Accuracy of communications. Few actions will work to destroy the credibility of any business unit faster than to disseminate incorrect information. Get the facts, and get them straight before transmitting information to anyone. Every phrase and every term should be carefully scrutinized for accuracy before going out. CIRT managers must read the communication for technical accuracy and understanding. Of course, CIRT communications should be professional and courteous. This is not the place for colorless humor or sarcasm. Part of communication's accuracy is the assurance the intended audience receives them.

CIRTs should develop out-of-band communications. This means that CIRTs, their constituency, and management should know when and how to use OOBC. OOBC efforts require advance arrangements and coordination within a response team. CIRTs should analyze the organization's current communication structure and devise private alternate channels. OOBC may include private cellular telephones, text pagers, wireless equipment such as PDAs, out-of-business-area telephone communications, registered mail, encrypted e-mail, etc. CIRTs must ensure that each OOBC system is periodically tested and achieves acceptable levels of security.

Types of Malicious Code Attacks

As a matter of course, investigators are tasked to address rogue code attacks. Handling such an attack presents challenges in terms of priorities. For example, certain types of attacks spread very quickly affecting computers on networks in large numbers. The Morris Internet Worm was one such attack.

There are many demands placed on the responders once they are aware of a potential critical incident. The clear first priority is isolating the attack. Infected computers must be isolated from the surrounding system to prevent the infecting code from spreading to other systems. With this done, responders should ask themselves if it is important to determine the origins of the infection, or not. If the network and its connecting systems are cleansed of the rogue code, the evidence of its origin may be erased; however, if the systems containing the code are isolated, they remain inoperable until the investigation is completed, negatively affecting productivity.

Viruses

In a virus attack, it is very likely the virus has infected many systems. The responder's likely priority is isolating the infected machines, clean them, and return them to their users. In these cases, analyzing the virus and its origins is secondary. Tracing the virus is difficult, if not impossible in a large system, but there are some considerations that can be made:

• Make a forensically sound copy of one of the infected hard drives.

• Treat the hard drive as if it were evidence and a crime scene in its own right.

• Make a second forensic copy of the drive and use it as a work copy for future analysis.

• If the first infected computer in the system can be identified by timestamp analysis, it may be possible to identify the origin of the infestation.

Experience Note

It is extremely difficult to identify the person who introduced a virus into a network. Anti-virus software must be constantly updated and users trained not to open e-mail attachments. Usually the best that can be done is to isolate the systems, cleanse them, return the systems to the production environment, and train users.

Trojan Horses and Logic Bombs

In many regards, these examples of malware code are easier to handle than viruses because they are usually confined to one machine. The problem is that they might remain dormant or unused on that machine until a triggering event takes place. Triggering events are items such as calendar dates, keystrokes made in a specific order, or the execution of program code. Once the triggering event takes place, the user discovers the malicious code and administrators take appropriate restoration steps. With the malicious code running, it's going to be fairly obvious where it is located. For example, if there is a logic bomb planted in a billing program, it is possible that users will know it when they try to execute the application and it does its damage.

With the execution of a logic bomb, the damage is done. Depending on the extent of possible legal action, the best solution is to cleanse the victim system and reinstall the software with clean backed-up data. It is possible that evidence could be collected, timestamps compared, and an attacker identified. Conducting such an investigation can be very time-consuming and resource intensive; consequently, it must be determined if it is worth the effort or not.

In the event of a suspected logic bomb hidden in an application, there are some specific steps that can be taken to prevent it from executing and doing its damage. It is important that a forensically sound copy of the suspected drive is made and preserved as evidence. Copy the evidence drive for performing all analyses and return a clean drive to the original system. The best way to locate the malicious code is to compare the victim system, where the malicious code is resident, with a clean backup copy. Investigators should review the date of the last change in the target media and compare it to the date of the last change for the same file in the backup copy. Continue to compare backups as far as is practical and compare dates. If there is a timestamp date change that cannot be explained or is not documented, the altered file is probably the guilty one.

If the investigators can gain visibility into the programmer's code (this may or may not be possible) there are editors that will automate the line by line comparison revealing any changes.

In the event of suspected malicious code, here are a few types of event logging that will help:

• Login and logoff

• File deletion

• Privilege changes

• Access by all root

• Failed login attempts

• Unused or dormant account access

• SU (Switch User) activity in UNIX-based systems

• System reboots

• Remote access of target system

• New user accounts

Things to Do after a Malicious Code Attack

If a system has been the victim of a malicious code attack or denial-of-service attack, either as a result of outside or inside activity, here are a few suggested measures:

• Contain the potential problem. The most efficient way to accomplish this is to simply remove the Ethernet cable from the NIC card, or isolate the affected systems from the rest of the network by some other means. If this cannot be accomplished, disable the power to the target machine. If you do not disconnect the target machine from the network, infections or attackers will cause havoc because they will continue to have access to the system.

  Experience Note

  Managers should not be lulled into the idea that attackers having penetrated a system will not return. Once in, they will continuously attempt to intrude or deny service.

• Preserve the target media as evidence. This probably will mean making a forensic copy of the target drive and preserving it for evidence.

• Cleanse the original drive and return it to service. Make forensic copies of media containing the malicious code and preserve them as evidence. This will preserve the timestamp dates of infection. Cleansing media may consist of media degaussing, reformatting the drive, or launching a forensic erasing tool where the entire drive is overwritten several times destroying the offending code.

• A completely clean reinstallation will be more time consuming, but will ensure correct functionality rather than running the anti-virus software to delete or quarantine the offender.

Digital Bloodhounds

For matters that are going to be pursued to levels of litigation, it is important for investigators to discover the identities of those responsible for causing system damage. In many cases, civil and criminal legal processes can be, and should be, pursued on parallel tracks. It is not unusual for an individual to be criminally charged while the damaged parties file lawsuits to recover their losses. Considering all the different technologies that can be employed to conceal the attacker's identity such as anonymous e-mail re-mailers and compromised server accounts, it would seem attackers have a definite advantage over investigators.

Experience Note

The Director of Risk Management for a credit card company noted that there were chat rooms dedicated to trading and verifying credit card information. After engaging several of the chat room operators in conversations over the course of many weeks, it was discovered many of them resided in countries that had few, if any, laws making credit card fraud a criminal act. Further, it was discovered the subjects knew they were acting criminally and fully acknowledged they could not be extradited due to a lack of international treaties. Consequently, they knew they could steal credit card information, sell it, commit fraud with it, and not suffer any punishment.

IP Addresses

When investigators begin looking for evidence in an attacked a system, the most logical place to begin searching is the IP address. IP addresses create areas of difficulties when locating the offender.

It is possible, and indeed quite likely, that the source IP address is spoofed and not correctly resolved to the attacker.

It is possible, and indeed likely, that the source IP address used for an attack is many hops away from the actual origin of the attack. Experienced attackers will pass through multiple systems before actually launching an attack. It is very common for investigators to have to obtain information from the administrators of multiple systems in the attack-chain before arriving at the attacker's origin.

Experience Note

Much of IP tracing depends on the investigator's skill and luck. This is not to say it is not successful, but realistically it can be difficult and discouraging.

The IP address is assigned to an individual machine. It may be difficult to determine who was using a particular machine at a particular time in a shared environment like a school or library.

UNIX-Based Investigations

UNIX File System Analysis

Briefly reviewing the UNIX file system, each disk drive is divided into one or more disk partitions with each containing a single file system. Within each file system is a list of inodes and a set of data blocks. Each inode holds almost all of the information that describes an individual file, including the size of the file, the location of disk blocks, etc. Inode numbers and their corresponding file names are stored in directory entries.

Data blocks are blocks of regularly sized data. The UNIX file system divides any data request to or from a file into logical blocks of data that correspond to the physical blocks on the disk. The downside of this methodology is that the data blocks are not necessarily contiguous to one another. Typically, the UNIX file system uses 8 KB as the size of its logical data blocks.

In UNIX when a file is deleted, the name remains in the directory, but the inode number, the name to which the name points, is removed. The inode itself is changed; consequently, the ctime is updated and the data block location is erased. As a file is deleted, UNIX decrements the inode's internal link to zero.

Removing all directory entry file name inode pairs performs this erase action. When the inode is deleted, the kernel marks its resources as available. The inode still contains all the data about the file and remains until it has been reallocated and overwritten. Having inodes containing some data but having a link count of zero reveals deleted inodes. Without having the file's content available, investigators can learn much about the file with only the metadata remaining in the directory entries and inodes.

To mount a file system, the UNIX kernel needs the sizes and locations of the file system metadata. The first piece of metadata is the super-block and it is stored at a known drive location. The super-block contains information such as the number of inodes and data blocks, size of a data block, etc. Predicated on the information contained in the super-block, the kernel is able to calculate the locations and sizes of the inode table and data portion of the disk. Inodes and data blocks are clustered together in groups scattered across the hard drive media. Usually UNIX maintains more than one super-block, one inode table, and one block array in the event of a disastrous data loss.

Undeleting UNIX

Undeleting UNIX files is different than handling Windows restoration issues. UNIX can be configured to make frequent backups, and it is not unusual for backups to be made hourly. Hopefully, there will be an easily accessed backup and it may be stored in /class/.snapshot. Examining this file will reveal that it is a backup of the home directory from several points in the past. It is possible to copy from this directory the items lost or corrupted in the regular home directory.

There is a UNIX command that should find which directory the home account is stored in: find/class/.snapshot/hourly.0 -name $USER -prune 2>/dev/null

Backups may be in the following path examples:

• /class/.snapshot/hourly.0/01

• /class/.snapshot/hourly.1/01

• /class/.snapshot/weekly.0/01

• /class/.snapshot/weekly.1/01

Retrieving lost e-mail is a very similar process. UNIX makes copies of e-mail in a similar manner as it makes a copy of the home directory. Access to copies of e-mail can be accomplished through /var/mail/.snapshot. By reviewing this directory, backups of deleted e-mail may be seen and captured. Be mindful that viewing e-mail contained in this folder will require it to be loaded into an e-mail client.

Data Hiding Techniques

Data "deleted" from UNIX files can be located by investigators with hex editors searching for specific files and file extensions. However, there is a technique where malicious individuals may attempt to "hide" data from forensic investigations. For the most part, it is effective in systems not using the Berkley Fast File System or FFS.

Experience Note

FFS deprecated the bad data blocks inode, preventing individuals from hiding data in there.

By way of explanation, the bad blocks inode has been used to reference data blocks occupying bad sectors on the target disk thereby preventing these data blocks from being rewritten by live files. If investigators run the file system checker utility, fsck, it is possible that the file system can been seen as having been radically altered.

The first inode that is capable of allocating block resources on an ext2 file system is the bad blocks inode (inode 1) and not the root inode (inode 2). Because of this positioning, it is possible to store data on the blocks allocated to the bad blocks inode and have it hidden from many forensic tools. Malicious individuals have tools that will allow them to exploit flaws in the UNIX file system and store data outside the view of forensic tools. It is for this reason that investigators should look at the physical level of UNIX bad inode blocks before moving on to other areas.

Coroner's Toolkit

Responding to reports of critical incidents happening on UNIX-based systems, investigators establish a timeline of events beginning from the last time the system was stable and uncorrupted through the actual event and until it was discovered and brought to a halt. The Coroner's Toolkit is a collection of utilities that attempt to gather and analyze data in the target UNIX-based system where investigators can accomplish their goals.

The Coroner's Toolkit contains the following data-gathering tools:

• grave-robber, the main data-gathering program

• file, Ian Darwin's file command

• icat, copies a file by inode number

• ils, list file system inode information

• lastcomm, a portable lastcomm command

• mactime, the MAC time file system reporter

• md5, the RSA-based MD5 digital signature tool

• pcat, copies the address space of a running process

• unrm, uncovers unallocated blocks from a raw UNIX file system

• Lazarus, attempts to resurrect deleted files or data from raw data

Within the Coroner's Toolkit, available at www.porcupine.org/forensics/, there is a tool called unrm that can emit all the unallocated data blocks on a UNIX file system. It functions by reading the list of free data blocks, locating each logical block, and looking to see if they contain any blocks of unallocated data. The unrm should be used if investigators are looking for a specific file known to be deleted.

Another effective data recovery tool is Lazarus. Lazarus attempts to give unstructured data some structure that can be viewed and manipulated. Lazarus depends on UNIX never writing file data except in well-defined boundaries. UNIX generally writes files in contiguous data blocks, when possible, attempting to boost performance. For this reason, UNIX should never need a defragmenting utility, unlike some other popular operating systems. Lazarus maps the disk that is created and provides visibility into the disk seeing the data by content type. Lazarus is a very comprehensive program in that it takes a very broad view of deleted files.

Using unrm and Lazarus will fill significant amounts of disk space on the forensic machine. Because Lazarus takes a large view of its work, it does not run in just a few minutes, so investigators are advised to let it run for several hours, if not days, before being able to see the results.

Using unrm and Lazarus is not as easy as using a Windows file recovery tool, using them is a time consuming and laborious process. It would best be described as a "hit and miss" process.

Experience Note

There is an interesting series of short articles about UNIX/Linux file system recoveries located at recover.sourceforge.net/linux.

Success rates restoring deleted UNIX files are spotty at best. The easiest way to restore deleted files from a UNIX system is to access an uncorrupted backup copy. UNIX administrators should back up all critical files on a regular basis observing the risk manager's axiom… the more important a file is, the more often it should be backed up.

Hiding Files

Users will go to many extremes to cover their tracks. Regrettably, employees and attackers employ a variety of ways to hide their nefarious acts by concealing files, including:

• Placing them in storage facilities located outside the workplace

• Secreting data in hidden hard disk partitions hoping no one will think to look there

• Encrypting data partitions of their hard drives with complex algorithms

• Encrypting data through the use of steganography

Web sites provide storage for users and may be accessed from any system with Internet access. For a small fee, files may be securely and privately stored on remote sites ready for access at some future date. In order for investigators to locate such services on target machines, it is recommended that a thorough review is made of pertinent files resident on subject's computer. Look for URLs in the history, temporary, and bookmark files indicating that files may have been stored there. If you have a very sophisticated user who could conceal his activities, check the user logs where access is made from the inside to the outside network for online storage sites.

In the case of having files, storage media, and hard drives encrypted by a password, investigators may try to obtain access by running a password cracking application. It is important to identify the application such as Word, Excel, or WordPerfect, as many cracking applications are specific to individual programs. With password protected files, it may be possible to run a tool like John the Ripper and brute force the password.

With encrypted hard drives, if investigators can identify the protection application, there are some manufacturers that provide a universal password that permits access in the event the hard drive owner forgot her password. By way of application, the encryption key is based on the password. If the drive's content is completely encrypted, consequently, working at a physical level is not going to provide any insight. Having the password is the only way to access an encrypted drive.

Steganography

Steganography is a concealment application where information is hidden in plain sight. By secreting data in an otherwise innocuous multimedia object (usually image or sound files) called carriers, steganography can hide information remaining essentially impervious to detection. Steganographic applications accomplish their task by first hiding the data's existence in the carrier and then encrypting it. Detecting these data-carrier files is a two-prong problem; first the multimedia file containing the data must be identified and then the data must be deciphered. With a workstation containing possibly hundreds or even thousands of such files, the task is formidable indeed.

Investigators can possibly determine if they have a steganography user by locating the program on the workstation's target hard drive or locating it at another location. Finding such an application might indicate the user has encrypted files with hidden data in them. Placing image or sound files in a steganographic application will result in a password prompt. This prompt does not necessarily mean the target file contains hidden data, so these tools cannot be used to screen potential files for hidden data. Most steganographic applications will prompt for passwords whether the file has data concealed in it or not.

There is one tool that claims to be able to detect steganographically hidden .jpg files and is available at www.outguess.org/download.php.

A wide variety of steganography tools are available at members.tripod.com/steganography/stego/software.html.

Creative investigators may well be served to obtain the passwords from the owner before attempting to brute force passwords in an effort to decipher encrypted files and drives. Failing this effort, using a technique such as the installation of a keyboard monitor to capture all the user's keystrokes may prove to be the answer. Outside of law enforcement, employers may implement keystroke monitors where employees do not have any reasonable expectation of privacy in their workstation or systems use.

Experience Note

Within the statutory constraints applicable to law enforcement agencies, search warrants and court ordered wiretaps govern the use of keyboard monitors.

Software keyboard monitors and similar tools made by Spector Soft are available at www.spector.com.

Hardware keyboard monitors are available at www.keykatcher.com/index.htm.

Locating hardware keyboard monitors is a matter of tracing desktop cabling. These are small cylindrical or box-shaped devices that are placed in-series with cables connecting keyboards to desktop towers or between the desktop and the network. Investigators should be aware that keykatcher makes a keyboard that acts as a keyboard monitor eliminating the inline device.

Finding hardware keyboard monitor information is available at www.spy-cop.com/keyloggerremoval.htm.

Finding software keyboard monitor software information is available at www.spy-cop.com/spycop-free-product.htm.

Before using such a technique, it would be prudent to consult with prosecutors or corporate legal counsel. Also, consult with your legal counsel before attempting to provide the results of a keyboard monitor to law officers.

Strong Encrypted Protections

There seems to be a growing use of encryption with very strong protection features. While there are many talented investigators and analysts in the world today, the conversion of encrypted data to plain text data in most cases is virtually impossible. Unless data may be captured from the target keyboard through legal means, investigators have to accept the idea there will be information that cannot be accessed within the limits of current technology and resources.

File Recovery Alternatives for UNIX/Linux

There is an alternative file recovery utility for Ext2 files systems used in Linux and some flavors of UNIX. It uses proprietary technology and flexible settings providing control over the data recovery process. It is called R-Linux and is available at www.rtt.com/RLinux.shtml. It is interesting to note that R-Linux is a Windows-based utility used to recover Linux data.