Critical Incident Detection: How to Know What Is Serious and What Is Not

The first step in dealing with critical incidents rests with becoming aware that an adverse event has happened. The detection of critical events happens through a variety of avenues:

• Suspected critical incidents may be detected by a review of firewall logs.

• Suspected critical incidents may be detected as suspicious user activity.

• Suspected critical incidents may be detected by intrusion detection systems.

• Suspected critical incidents may be detected by systems administrators.

• Suspected critical incidents may be detected by systems auditors.

• Suspected critical incidents may be reported contacts outside the organization.

• Suspicious events reported by users.

• Suspicious events reported by help desk operators.

In the course of detecting critical incidents, it is important to have the function point, where these activities can reported. Employees should be regularly familiarized with the procedure directing them to the location where they report suspicious activities. Function-points can be employees or a business unit where potential incidents may be reported 24/7. It is their responsibility to accept the report, elicit as much relevant information as possible, record the details, triage the event, and decide to activate the response plan or not. Using an incident reporting checklist ensures all the pertinent information is recorded so an appropriate determination might be made. When eliciting information, get the facts first, then the complainant's speculation and "good-guesses."

Here are some of the key areas for an incident report checklist:

• Current time and date

• Person accepting information

• Person reporting information

• Nature of the critical incident

• When, how, why, where, and who of the critical incident

• Systems involved (software, hardware, employees, etc.)

• Key contact information for reporting employee, including reporting chain

• Describe any need for out-of-band communications

• Describe estimated priority of critical incident

• Recommendations from the reporting party

If you do not know that an incident has taken place, it is difficult if not summarily impossible to determine if your systems have been compromised. The function-point should get as many of the facts as are available at that time. If information is not collected in a timely fashion, it cannot be determined which sensitive data, systems, or networks have been attacked and the extent of damage that has been done to your operation's confidentiality, integrity, and availability. Detecting critical incidents in a timely fashion is imperative. If you can compare the current systems state with the last time you knew the system was uncorrupted; responders should know what is needed to restore operations.

Critical Incident Symptoms

There are some suspicious activities that might escape the notice of Intrusion Detection Systems, firewalls, and less-than-vigilant systems monitoring. Their discovery sometimes depends on attentive administrators, help desk employees, auditors, and log-entry analyses. Here are a few examples:

• Unusual login accounts. These include failed login attempts — attempts to enter dormant or default accounts. In this category is included the new or unusual account not created by administrators. Often this rogue account is a mysterious root account or a privileged user account.

• Unusual account activity during irregular work hours. It seems that attackers often attempt to gain access during hours when administrators are assumed to be least likely to notice their activities. Attackers assume the system may be unattended or poorly staffed at these times.

• Unfamiliar files or applications. Usually these types of programs are "backdoor" programs facilitating the unauthorized access of an attacker. They often take the form of something innocuous such as /etc/inetd.d/ or ".." (this is read as space-dot-dot).

• Unauthorized changes or escalation of file and directory privileges.

• Use of commands not normally related to an employee's job. This is an event that is often revealed when reviewing log entries. Log entries reveal that a user (not an administrator) is executing commands such as extracting downloaded programs using the tar command and subsequently compiling code.

• Presence of unauthorized utilities. Finding password cracking utilities and other tools used by attackers in the system indicate a potential problem.

• Erasures or gaps in log entries. This is another one of those activities that quickly indicates there is trouble afoot. If there are gaps or erasures in logs, it is likely someone has attempted to cover his tracks.

Response Strategy

The objective of creating a response strategy is to determine the most appropriate method responding to the critical incident. Of course, before heading off into the sunrise with technical guns blazing, there are at least three immediate considerations that must be made:

1. Technical factors

2. Business factors

3. Legal requirements

In formulating your response strategy, much depends on the character of the emergency. You do not want to treat arterial bleeding with a gauze and tape when you should be calling a surgeon. Here are some dependent factors that will significantly impact your decision process:

• Are the affected systems impacting profitable operations?

• If information was stolen, what was its level of sensitivity or classification?

• Which business functions are being impacted and at what level?

• Has the incident been contained or is it continuing?

• What is the origin of the emergency? Is it internal or external to the organization?

• Is the critical incident public knowledge?

• What are the legal reporting requirements? Does the law require this matter to be reported immediately to authorities? Who are those authorities? Should this matter be handled as a human resources matter? Should this matter be handled as a civil suit?

• What, if any, steps should be immediately taken to discover the identity of an outside-agency attacker?

• What is the fault tolerance level of the affected systems?

• As of this moment with the incident contained, what are the financial losses?

Critical incidents will vary greatly from being infected with the latest virus to the loss of extremely sensitive information. Depending on the size of the affected company, the theft of sensitive information as credit card information could result in financial ruin. Even in a larger business, a successful class-action suit resulting from negligently failing to safeguard personally identifiable information can result in significant monetary losses and incalculable damage to business reputation.

Information collected during the initial emergency assessment phase will significantly impact the manner in which a response strategy is formulated. Contained within the response strategy are estimates of your organization's ability to respond to the critical incident, public perceptions, legal and regulatory requirements, as well as an impact assessment on critical assets.

Experience Note

Remember "haste makes waste." Responders should act deliberately but not dawdle.

Actions taken in response to critical incidents must be made with some degree of alacrity, but attempting to address matters without a firm set of facts is not likely to be productive.

IP Addressing in Brief

By way of review, Internet Protocol, IP, addresses are those numbers assigned to network devices that serve as their identification. It is a decimal notation that divides a 32-bit address into four 8-bit fields. An IP address consists of the following components: Network ID and Host ID. For example, in the IP address 204.9.205.21, the network ID is 204.9.205 and the host ID is 21.

For practical purposes, Internet IP addresses are divided by classes A, B, and C. Network classes are only applicable in Internet environments as closed networks may assign addresses in any form their naming convention policies allow.

• Class A networks begin at address 1.xxx.xxx.xxx through 126.xxx.xxx.xxx

• Class B networks begin at address 128.xxx.xxx.xxx through 191.xxx.xxx.xxx

• Class C networks begin at address 192.xxx.xxx.xxx though 223.255.255.xxx

• Class D networks begin at address 224.0.0.0 through 239.225.225.225

• Class E networks begin at address 240.0.0.0 through 247.225.225.225

Class D networks are reserved for multicasting and Class E networks are reserved as experimental.

There are other reserved IP addresses for example. IP address 225.2.100.1 is a multicast address to be received by a group of hosts on the network. Multicasting is the transmission of information to specific hosts.

Broadcasting is the transmission of information to be received by all hosts on a network. For example, the IP addresses 255.255.255.255, 192.9.205.255, 180.10.255.255, and 10.255.255.255 are broadcast IP addresses.

A unicast IP address uniquely identifies a specific host on a network. The datagram with a unicast IP address is received and processed by one single host. For example, the IP address 204.10.95.214 is a unicast IP address.

Reviewing DNS

Domain name services are structured in a hierarchy with the highest level being the last component of the DNS address. DNS names can be up to 255 characters long and are not case-sensitive. They must start with a letter and may only consist of letters, digits, and hyphens. DNS was originally introduced in the United States and the final component of an address was intended to indicate the type of organization hosting the domain. Some of the three letter final labels such as .edu, .gov, .mil, and .biz are in common usage today. When resolved, these DNS names will indicate the IP address of the host. The purpose of DNS is to register meaningful names with numeric IP addresses, while the process of reducing a DNS name to its IP registration is called "resolution."

In some cases, there are two letter codes indicating the country of origin as part of the domain name as defined in ISO 3166, available at www.din.de/gremien/nas/nabd/iso3166ma/codlstp1/en_listp1.html.

If a DNS name cannot be resolved locally, the DNS server will communicate with other DNS servers reaching higher-level servers attempting to resolve the name. If it is unsuccessful, it will attempt to contact the ultimate authority or root server for the domain, e.g., .gov, .org, etc. This process is called the recursive resolution of addresses.

Resources

Binary mathematics, IP address architecture, CIDR, DNS, TCP/IP, and network subnetting are topics that could fill volumes and are outside the scope of this book, but here are a few resources that will provide valuable information in these areas:

• A review of IP networks and subnetting is available at www.learntcpip.com

• A review of DNS, domain name service, is available at www.ietf.org/rfc/rfc1034.txt?number=1034

• A review of CIDR is available at www.ietf.org/rfc/rfc1517.txt?number=1517

• A review of IP address formatting is available at www.ietf.org/rfc/rfc1166.txt?number=1166

• A review of TCP/IP protocols is available at www.ietf.org/rfc/rfc1180.txt?number=1180

Locating the Origin of Denial-of-Service Attacks

Denail-of-service attacks, DoS, are extremely difficult to trace to a source IP address. For argument sake, DoS attacks include Distributed denial-of-service, DDoS attacks with the distinction being the DoS emanates from a single IP address where the DDoS attack emanates from multiple, even hundreds of IP addresses. Typically, DoS attacks originate from spoofed or compromised accounts making it virtually impossible to trace them to a single source machine.

Investigators have the option of contacting the system administrators of compromised systems and hope they have adequate logging to take the next step backward in the direction of the perpetrator, but eventually it seems there is a system in the tracing chain that does not have enough information to complete the trace. The path toward the attacker will end there.

Experience Note

Because law enforcement authorities generally have very limited abilities outside their jurisdiction, it may be very helpful for the affected system administrators to contact the system administrators of the previous systems in the chain. Often law enforcement authorities may accept information collected pursuant to the business activities of administrators but may not direct administrators to contact their counterparts, as this would possibly taint the evidence as the officers could only collect the information through a warrant or international treaties.

UNIX Logging

UNIX is a platform that offers much in the way of logging features. Here, as in many systems, it is wise to alter the default-logging configuration.

Experience Note

Of all the preparatory steps that responders can take before a critical incident, adequate logging is probably the most useful. Logging permits the reconstruction of events and is one of those "save your bacon" policies.

The principle file for logging is UNIX syslog. This file stores logging information in one, or more, configurable locations. Logging is configurable through the syslogd file located at /etc/syslog.conf. Within this file, there are two significant configurations, action and selector. Action controls the location that the logging message is stored, while the selector controls the message's priority and facility. Essentially, this means the selector field controls where the logging message is generated, and the severity of the message.

Logging all security-related events is accomplished in this example:

    var/log/syslog

Because attackers will attempt to delete or alter system logging, it is strongly recommended that all messages are logged and stored, remotely avoiding the possibility that an attacker could gain root privileges to the same system where the messages are generated and stored. Configuring systems to log their messages to a remote logging server is accomplished by this example:

 auth.* @xxx.xxx.xxx.xxx

Of course, the xxx.xxx.xxx.xxx is the IP address of the remote logging server.

UNIX is capable of logging the commands that each user executes. This log file is found, depending on the variety of UNIX used, in the/var/adm; /var/log; or/var/adm files. Enabling this level of logging is one that will significantly help emergency responders in their efforts to reconstruct the damaging events on the system.

Experience Note

There is a major consideration for UNIX process tracking in that it will generate large logging files as essentially every action taken by a user is generating an entry. There must be a compromise reached where meaningful logging is created in the event of an emergency, yet there must be some controls placed on the amount of logging as it will soon consume all available storage.

Windows Logging

Enabling Windows NT server logging, in Microsoft's terms, is called "auditing," and must be manually done by the administrator as it is not done by default at installation. It is accomplished by following this path: Start | Programs | Administrative Tools | User Manager | Audit Policy. The following events are the minimum level of logging:

• User logon and logoff

• Security policy changes

• Shutdown and restart

Windows allows the auditing of file and directory permission changes. In order to accomplish this task, merely right-click on the target file or directory and choose Properties from the displayed menu. In this dialog box, choose the Security selection tab and choose Auditing. From this Directory Auditing dialogue box, you can choose the auditing of events surrounding the selected file or directory such as the success or failure of attempts to change privileges.

Remote logging in Windows is accomplished with third-party software: Kiwi Syslog Daemon, available at www.kiwisyslog.com/products.htm.

Application Logging

There are many applications that allow logging of significant events. Each has its own configuration requirements. However, these are several minimum steps that should be considered when implementing application logging:

• Log messages to a directory or file secured so that only authorized employees can access it.

• If possible, log messages to a secure, remote server.

• Record logs on WORM, Write Once Read Many, media.

• Log as much information as will be necessary to reconstruct system events.

• Ensure that all logging includes the relevant IP addresses of inside and outside the organization.

In the case of Microsoft's Internet Information Server (IIS), there are several layers of meaningful auditing. By accessing the Management Console, administrators may view and change the auditing capabilities. At the time of installation, IIS has logging enabled by default but by viewing the Extended Logging Properties dialogue box, you can see the type of logging available: Date, Time, Client IP Address, User Name, Server IP, Cookie, Server Port, and Bytes Sent, to name a few.

Hardening Servers

In a perfect world, all system components would be impregnable and all applications would not need to be protected from attackers by hiding them behind firewalls. Regrettably, most of us do not live in that world, so a system must be fortified against attacks. By taking a few preincident precautions, you can save yourself a lot of time and resources when the emergency occurs.

Here are a few proactive suggestions:

• Ensure all versions of operating systems, applications, and hardware are the most current available.

• Ensure that all software updates have been installed for all operating systems and applications. Changes must be approved and documented.

• Ensure that all services, not absolutely essential for the server's function are disabled or removed.

• Ensure that all unnecessary hardware is removed or disabled.

• Ensure there is a standard installation procedure for all hardware and software.

• Never accept default software configurations for production environments. Always review configurations and thoroughly test systems before placing them in production.

  Experience Note

  Attackers are depending on default configurations for their success. Many attacks can be thwarted by correctly configuring software.

• Backup regularly and include critical data, applications, and configurations.

• Ensure all changes are documented, recorded, and approved before implementation.

Backup Frequently

Procedures must demand that regular and frequent backups take place. Backing up data and system configurations will help you discover attacker-related changes and expedite restoration. Backups are only as good as the originals. Test backup copies by attempting to restore operations by using backed-up copies. Many businesses have copies and when critical incidents occur, they discover their backups are insufficient. If the only available backup copies were made after the system was compromised, they are not going to be much help. However, if good backups were made before the incident, they will be invaluable in determining the changes made to the system by the attacker.

User Security Training

Users, whether employees or not, are the most significant threat to system security. Even the most seasoned systems managers are surprised which system flaws can be exploited either intentionally or by accident. It is for this reason that user education is essential in maintaining system security. Users should know what types of emergency actions are acceptable and those that are not. Employees must be trained to know the basic steps regarding critical incident response.

Bare-essential employee training for critical incidents should include the following:

• In normal business operations, they must recognize those events that are indicative of emergencies.

• In the event of a critical incident, they must know and follow applicable policies and procedures.

• They must know that nothing is more important than lives. Data and physical facilities can be replaced.

• Ensure employees do not take any corrective or restorative steps unless advised by senior managers and CIRT members.

• Ensure employees do not take any investigative steps, unless directed by the CIRT.

System Security Architecture

Arguably, there are numerous steps that can be taken ensuring the security of a system. The most secure system denies all access and privilege but is absurd and unworkable. So a compromise is needed, and one that is basically transparent to the users.

Experience Note

The best system security is transparent to the users.

Network administrators are those who are charged with the responsibility of the system topology, architecture, and secure operation of the network. Administrators are tasked with the day-to-day enforcement of the organization's policies, practices, and standards. Emergency responders are usually dependent on administrators recognizing potential emergencies, standardized configurations, and documentation for their effectiveness. For example, administrators are responsible for the creation and enforcement of policy for packet screens. If they do not have standardized configurations and documentation of access control lists; responders effectiveness will be severely hampered addressing a firewall breach.

Generally, network security is dependent on the following:

• Firewalls consisting of packet screens, inspections, and proxies

• Intrusion detection

• User authentication

• User privileges

• Activity log reviews

• System monitoring

• System audits

• Encryption

Time Stamps

Administrators should make certain that all logs and indeed, all applicable functions are timestamped recording the same time and synchronized for all machines across the network. Using the Network Time Protocol is an efficient way of achieving system-wide temporal parity. Having the same time synchronized with all devices will greatly aid a responder when she is attempting to reconstruct events from log files.

System Monitoring Structure

This seems to be a recurring theme, many businesses have no idea what they have and where it is located. Although having a current hardware and software installation standard particular to each item seems to be obvious. Each time a piece of hardware or software is installed, the authorized person should have a standard installation procedure checklist created specifically for that item governing installation and configuration. From a responder's perspective, fewer things are more frustrating than learning that servers, having the same hardware, software, and function, are configured differently.

Here are some items to have ready when the responder makes contact:

• Ensure all relevant documentation is current and available.

  • Software manuals corresponding to the installed versions

  • Hardware manuals

  • Documentation for software configuration procedures

  • Cabling diagrams

  • Schematics and relevant diagrams

  • Organizational chart, job descriptions, and reporting authority

• Ensure all relevant logs are securely stored and part of the backup procedures.

• Have a current inventory of the locations of hardware and software. The inventory should include items such as:

  • Manufacturer

  • Date of acquisition

  • Serial numbers

  • How the hardware is being used

  • Peripheral equipment attached

  • Names and versions of installed software accompanied by authenticity codes

  • Owner

  • Users

  • Physical location

  • Configuration of hardware

  • Configuration of software

  • Updates installed

• Network topology map. Having a current and accurate topology map is extremely helpful during a critical incident. Under the best of circumstances, the topology map includes relative details as connected hosts, relevant applications, switches, hubs, routers, firewalls, NICs, terminal equipment, location of network storage devices, location and types of cabling or wireless linkage, and open-ended network connections. In order to give responders an accurate picture of the system, this map should also include the physical location and connectivity (including IP addresses and device names) of each device. Many employees will object to performing and updating this document, but it is absolutely necessary in preparing for emergency responses.

Business Issues

In business priorities, often we consider business decisions before any others. For example, when an employee is discovered stealing proprietary information and transmitting it to competitors, the organization goes into self-preservation mode minimizing its risks and preserving its critical assets. Seldom is the offender criminally prosecuted or civilly sued. As a matter of routine course, the offending employee is suspended pending the outcome of an investigation, and, if it is discovered the employee was violating policies, she is dismissed without any future legal action.

The damage an offending employee has done usually spans tangible and the intangible critical assets. Tangible losses are sustained in that valuable information was stolen and passed to competitors. In the avenue of intangible damage, she caused incalculable harm to the business reputation of the company. It is possible the financial losses suffered by the loss of credibility will exceed those from the stolen property.

This is a quandary — should managers legally pursue an offender and risk public scrutiny by airing their seemingly dirty laundry or should they dispose of the matter privately and risk becoming a target for attackers knowing they will not receive serious punishment? Organizations should consider if they fail to legally pursue attackers and criminals with full vigor, they accept current circumstances and fail to deter future attackers. In many cases, the decision is simple to make as laws mandate that suspicious or criminal activities must be reported to law enforcement authorities.

Legal Issues

Adverse legal actions can drive an otherwise well-run business into oblivion. Responders should consult with legal counsel whenever administrative, criminal, or civil proceedings might be the result of employee behavior or an outside originating attack. Considering laws and business positions, it is possible that legal counsel will advise against a particular course of action and suggest alternatives.

Political Issues

Shades of company politics can substantially color the fashion in which an organization handles its crises. Suppose for a moment that it is the atmosphere within an organization to accept all employees at their word, trusting them, it is likely they are provided substantial freedom in their work. In this atmosphere, identifying and handling critical incidents caused by employees would be a matter of little significance. In such environments, few company resources, if any, would be dedicated to addressing critical incidents. However, if the organization has a more realistic culture where it vigorously safeguards critical assets, it will allocate the necessary resources to monitor compliance with its policies and procedures.

Experience Note

Performing an audit on the organization's Chief Legal Officer's workstation, the auditor discovered pornography. After a careful review, it was determined some of this material was in violation of federal laws as well the company's policies. The auditor advised her supervisor and jointly they briefed the Chief Executive Officer presenting examples of the images. It was well known that the CEO and CLO were close friends. Subsequently, the CLO was dismissed and criminally prosecuted.