CIRT Composition: What Kind of Skills and Talent Do I Need for a CIRT?

CIRT core membership should include the senior manager sponsor, IT security program manager, representatives from the legal counsel unit, public relations unit, human resources unit, and the CIRT manager. The CIRT manager should be someone who is a senior employee who has significant knowledge of the organization's operations as well as an employee capable of making sound business decisions.

The IT program manager is the head of the organization's IT security program and might double as the CIRT manager. In the case of a critical incident spanning regions or countries, one IT critical incident manager should be named for each office with all strategic efforts coordinated at the headquarters level. This representative will be responsible for tactical decisions, triage functions, and local resource deployment. It is the IT security program manager, with senior manager's approval, that is responsible for authorizing any release of information about the incident to the press. However, the program manager should not be the individual disclosing information to the press. A public relations unit employee should make contacts with the press. Delegating press responsibility relieves the program manager from having to evade sensitive questions or even having to lie to the press corps. Regardless, the public relations unit is going to be the place where the institutional knowledge and experience in this area is going to be found.

Legal Unit

Activating the CIRT requires an opinion from senior managers and specifically from the legal unit representative that is knowledgeable about the relevant laws dealing with the organization and its functions, intellectual property, information security, and privacy. In the case of CIRT deployment, it is the legal unit's responsibility to ensure that the CIRT does not violate laws and regulations while responding to a critical incident. Knowledgeable and experienced legal advise become particularly important when CIRTs are directed to follow attackers with the objectives of locating, identifying, assisting, apprehension, and prosecution. Legal representatives must be more than attorneys with general knowledge; they must possess a thorough understanding of information technology, business functions and civil, administrative and criminal matters. Through their participation on the CIRT core, they must initiate and develop relationships with law enforcement and regulatory authorities, professional support groups such as NIPC and Infragard. Often, this employee will serve as the primary contact for law enforcement.

Public Relations

Depending on the organization's size and funding, having a public relations unit representative is a decided advantage. This employee addresses all media requests for information and similarly handles authorized press releases. It is expected this employee will have developed relationships with media organizations as well as specific news agency representatives.

Human Resources Unit

A senior representative from the human resources unit must be part of the CIRT core. This person ensures that the CIRT team's response efforts do not violate employees' rights. Also, this person will make certain that appropriate disciplinary standards are applied should an employee be found to be the source of a critical incident. In the event an employee is an unwitting part of an attack, or if the employee is a victim, certain rights might be granted within the scope of their employment. The human resources unit representative is responsible for seeing that an employee's reasonable expectation of privacy is respected or knowing whether an employee is entitled to union representation in the event of an interview.

IT Investigative, Analysis, and Forensic Experts

These CIRT members ensure that the response is performed in a methodical and deliberate fashion, making certain all relevant evidence is properly collected, preserved, and introduced at legal proceedings. CIRTs require their members to participate in addressing crises on an as-needed basis. Key participants should consist of IT security officers, systems administrators, telecommunications equipment specialists, database managers, engineers/software developers, and of course, systems owners.

IT Security Officers

Most organizations have individuals assigned full- or part-time to ensuring the security of systems. Often this employee performs duties in support of auditors, making certain the IT units are in compliance with the organization's policies, procedures, and standards. This employee helps in addressing attacks by knowing how the system was installed and configured before the attack. She will also be the person who provides CIRT with access and interpretation of logs.

Systems Administrators

These employees are the "bread-and-butter" individuals responsible for the day-to-day operation of the system, including hardware, software, and employee interaction with the system. Systems administrators should have in-depth knowledge of the function of the system's hardware, operations, and configurations. Depending on the organization, its culture and function, the systems administrators can provide immeasurable assistance to the CIRT.

Telecommunications Specialists

These employees are the ones who are most knowledgeable about the integration of the various components of the telephone and network border systems, including installation, security, configuration, and operation. Systems administrators sometimes perform this function in smaller organizations. These employees have intimate knowledge of the interaction between the various hardware/software components, cabling, telephone lines, PBXs, terminal equipment, routers, firewalls, gateways, and protocols like X.25 or Frame Relay. They are usually responsible for developing relationships with communications carriers including the interaction between the organization and the carrier's equipment.

Database Managers

Most organizations dealing with substantial amounts of data will employ database managers and administrators. These are the employees who have the responsibility of maintaining the integrity of the database; assessing the impact of proposed changes; and in the event of an attack, determining the effects of deletions, modifications, or additions.

Engineers/Software Developers

These employees have knowledge of the system's platforms and applications and how they interact with the hardware. They are the employees that know if the system is running according to design specifications.

System Owners

It is imperative that the systems owners be part of the CIRT, as it is their responsibility to see that the system personnel, data, and facilities are functioning effectively and efficiently. Owners should know the emergency response/recovery plans and their execution. They will be fully aware of backup and restoration procedures as well as equipment redundancies. Ultimately, the owners are responsible to the other stake-holders and will have to answer questions regarding the attack, including its effect on critical assets.

CIRT Management Skills

Possessing well-developed management skills is the single-most desirable attribute the CIRT team leader can have. When a critical incident arrives, it is incumbent on the CIRT manager to ensure the team has the requisite skills, resources, training, experience, motivation, and attitude. Managing a CIRT is not really very different than managing any business unit that is populated by field-specific experts. CIRT managers do not need to have great technical proficiency, but on the other hand, they should have sufficient knowledge to make qualified decisions concerning team priorities and tactical deployments.

Technical Skills

Technical skills are absolutely essential in determining CIRT's efficiency and effectiveness. There is also a matter of the team's credibility. If the team does not earn a reputation for being able to handle emergencies, they will not be contacted for help and no one will listen to their warnings or advice. CIRT's technical skills should span relevant operating systems (UNIX, Linux, Windows, etc.); networking skills; programming languages such as C++, PERL, Java, XML, and HTML; and hardware equipment such as firewall appliances, routers, etc. Electrical engineering experience is a plus.

Staffing CIRTs with professionals that have skills in all relevant areas is extremely difficult and expensive. Such employees are going to command high salaries and are probably out of reach of most organizations. If this is not within the organization's budget, find individuals who have expertise in one or more areas and task them to work as a team. Teams, permanent and ad hoc, are composed of employees having key skills that mentor others in developing new skills. Foster a team culture of mutual dependence and spirit, it will pay dividends in the future.

Team Skills

These skills are vital in the CIRT's successful operation. Team skills are focused on:

• Having a common vision of the job to be done

• Division of responsibilities

• Ability of seeing the next item to be done without prompting

• Knowing when to tell and when to ask

• Knowing when the task exceeds an individual's skills resulting in getting help from another team member

Developing team skills is a direct result of management skills, so good managers tend to engender good team skills.

Communication Skills

Team members must be able to cooperate and communicate with coworkers as well as write and deliver effective formal presentations. If there are not employees that have technical writing skills, consider hiring technical writers to supplement team skills. Communications skills are so vital to CIRT's success, that if they are absent it is very possible that no amount of technical ability will compensate.

People Skills

In the event of a critical incident response, people skills are some of the most vital skills in the tool bag. There must be a dedicated team spirit in a CIRT when responding to critical incidents. Tempers, egos, and poor judgment cannot coexist in this type of teamwork environment. Being able to get along with team members as well as serving constituents are key elements in successfully addressing emergencies. At times, technical experts gain reputations as being difficult to work with; consequently, gathering team members with people skills can be challenging. In the arena of responding to critical incidents, team members must be adept at soothing a manager's bruised ego or an embarrassed administrator as they go about their work. Casting disparaging remarks about the employees that are responsible for day-to-day system operation certainly does not gain respect.

Incident Reporting

Along with the policy that potential or suspected critical incidents must be reported to the function-point, organizations must develop a standard for reporting emergencies that must be formalized as part of their response procedure. This procedure should include a standard checklist where critical information is elicited from the person reporting the incident.

Experience Note

Do not get excited when fielding a complaint call. Do not request information that really does not have any bearing on the matter at hand; get to the point and collect enough information allowing a requirements assessment to take place and nothing more.

Here is an example of a proposed incident questionnaire:

• Date of the report. Obtain from the person reporting the incident, the time, date, and place the incident was first noticed.

• Duration of the incident. How long did the incident last and what were the indications that something had happened?

• What was the name of the system being attacked?

• Where is the system located?

• What is the operating system and affected applications?

• What was the data stored on the system?

• What was the sensitivity level of the data?

• Provide a detailed description of the incident.

• How widespread is the knowledge of the attack and its details?

• What are the implications of the incident, including adverse effects on the organization?

• Incident reporter's identity, contact information, and emergency contact information for supervisor, senior manager, and system owner.

Incident reporting should be made directly to the organization's function-point that acts as the incident screener and information collector. This employee, or business unit, collects the basic information making a determination whether it should receive a formal CIRT response or be treated as a system anomaly. The information collection form might serve as the front-end of an incident database by tracking their frequency, systems affected, response posture, and improvements.

What Should I Do if I Have Been Hit?

What organizations do in the face of crisis is determined by:

• Type of critical incident

• Its impact

• Anticipated legal actions

• Best way to return to normal operations

In essence, there are two tracks to follow when responding to incidents, one requires careful and detailed coordination where evidence is collected and preserved. The other track is one guided by the overarching philosophy of "let's restore operations as soon as possible and do not worry about evidence."

Response Steps for Legal Actions

In following the "locate and prosecute to the Nth degree" track, these are the basic measures to follow:

• Determine if the emergency is a real incident. This is the most important step for the employee acting as the function-point to take. If there truly is an attack under way, immediate and decisive action is warranted, but if there is merely something developed as a result of a user-error, then administrators should be told to take appropriate action.

• If there is a qualified opinion made by the function-point, terminate attack immediately. The CIRT or a CIRT-directed effort must halt any further damage from occurring to the system's elements. There can be a lot of discussion regarding this step, but the CIRT's actions must be guided by three priorities: personnel, data, and physical facilities. Any attack affecting the confidentiality, integrity, or availability of critical assets must receive immediate attention. Given that terminating an attacker while engaged in a "live" attack will probably result in the loss of amounts of potential evidence, senior managers must decide to create policies that terminate attacks first preserving operations and worry about evidence collection as a secondary matter.

• If there has been a decision to pursue the attacker, with advice of legal counsel, law enforcement authorities must be advised as soon as possible.

• In most cases, law enforcement agencies will not assume responsibility for taking over the emergency. That obligation rests fully with the organization. Rather, officers will work with CIRT members in the investigation and collection of evidence necessary for criminal prosecutions. Depending on the agency and its policies, copies of evidence collected by officers may or may not be provided to the organization's CIRT. Make certain that there are no misunderstandings when officers arrive at the scene.

• For many departments, copies of evidence collected by law officers cannot be provided to the CIRT as a matter of policy. There are many reasons for this policy:

  • Officers collecting evidence can be compelled to testify at civil and administrative hearings where the department does not have an interest.

  • Officers may provide testimony in these proceedings that could later be used to impeach their testimony at criminal proceedings.

  • Departments do not have the resources to provide copies to the organization.

• The collection of evidence for the organization is their responsibility.

• Any legal actions taken or anticipated on the part of the organization should be coordinated with law officers. Failure to do so may have a quelling effect on their criminal prosecution and result in damage to the law officer-organization relationship.

• CIRTs must document each action taken, including the date, time, place, system name, application, operating system, and who participated. Experienced CIRT members often follow the two-employee rule.

• Any action is observed and documented by at least two persons. The reason for the two-person rule is to lessen legal challenges. All notes are considered evidentiary and must be preserved as such.

• Isolate compromised systems from the network. This is one of those initial steps limiting the proliferation of any damage. Taking systems offline is a judgment call on the part of senior managers. Depending on circumstances such as systems redundancy, equipment availability, program availability, and personnel resources, determine if this is a step where affected systems are forensically duplicated and returned to service or not. This is another one of those items to discuss with law enforcement officers as they may wish to collect the forensic copies themselves, and if the organization has qualified employees, they might be directed to create forensic copies and deliver them later to the officers.

• Discover how the attacker gained access to the affected systems. Secure the attacker's access points on all unaffected systems first, then secure the affected systems as a matter of response priorities. It is imperative that the point of attack is discovered and closed. Many times the easiest way to detect the points of entry is to compare the affected systems with "clean" systems.

• There are experts that insist on directing the attacker to a secure system where her attack process can be captured and studied. These processes are frequently known as "honeypots." While honeypots provide a lot of material for study and vulnerability analysis, their value must be weighed very carefully.

• CIRTs must document the state of the compromised systems. Maintaining a system state log is important, memorializing whether the system is in production, offline, ready to be restored to production, or replaced by a redundant system.

• Restore the victim-systems to productivity. After locating the point of entry, compare the attacked systems with the last known system-state unaffected by attacks.

  Experience Note

  Several years ago, attackers successfully invaded systems by exploiting documented vulnerabilities that were unpatched. On gaining unauthorized access, they installed backdoors, then downloaded and updated the systems. By doing this, they precluded others from invading the same systems. The organization was oblivious to the updates and the attack.

• CIRTs should document their time, resource costs, and expenditures. The cost of responding, restoring, and business resumption can form the damage-basis for civil actions in the way of estimated damages along with the cost of the equipment, revenue losses, and employee-time losses. These accumulated costs can have a significant impact during criminal trials and sentencing. Many jurisdictions establish the degree of culpability, length of sentence, and victim restitution based on costs resulting from the defendant's actions.

• CIRT members must secure all affected systems logs, audits, notes, documentation, and any other relevant evidence created or collected at the time of the incident. The evidence collection process actually has its beginning the moment the attack begins and does not cease until litigation is completed. All evidence must be documented as part of a chain of custody schedule with a copy of this document accompanying evidence-items at all times. Error on the side of caution, evidence should be catalogued on a chain of custody and even the chain of custody schedule is regarded as part of the evidence package.

• After-action briefings. This is the presentation made to senior managers where they are briefed about the incident, effects, CIRT actions, legal actions, restoration, and current systems status. In this briefing, senior managers deliver their views about CIRT's efforts, expectations, and results. At this time, it is common for CIRT's constituents to have their say. This is not the place for injured egos and hurt feelings; CIRTs should consider any and all criticisms or praise in the spirit of accomplishment or improvement.

• Postmortem. The CIRT members including full-timers, part-timers, and ad hoc members attend this meeting. Depending on the sensitivity of the discussions, outsiders who participated in the critical incident response should be in attendance. The purpose of this meeting is for CIRT members to critically analyze their performance and deliverables.

CIRT Success Metrics

The likelihood of totally eliminating attacks from outside or inside the organization is zero. CIRTs are similar to fire departments; they have significant support costs but, when activated, they are literally worth their weight in gold. Consequently, crafting a series of success metrics is usually one that is left to the very last minute. Here are a few suggestions that should be considered during the CIRT creation process:

• How many incidents did the CIRT address in a given time period? (Time periods could be measured in months, quarters, or years.)

• What were the estimated amounts of financial damage averted by CIRT intervention?

• What has been the impression of CIRT's technical expertise with their constituency?

• What is the average time and employee resources needed to address each specific incident type?

• What is the documentation completed by individual CIRT members relative to the actions taken with each incident?

• What recognition or awards were presented to the CIRT?

• Postincident feedback from constituency. Basically, this mechanism is one where a questionnaire form is provided to the victim-business unit and the results compiled by the CIRT as part of their success metrics. Particular emphasis in these questionnaires should be placed on the anonymity of the person completing them, if so desired.

• Were significant changes brought to the organization's policies and procedures suggested by the CIRT as a result of their intercession with a critical incident?

CIRT Development Life Cycle

In various forms, CIRTs have been in existence for more than 20 years. In some cases, they have performed magnificently and made substantial contributions to their organizations; while in other cases, they have foundered and sometimes failed. The levels of CIRT competence and success in the organization are tied to their development life cycle. Consequently, these are the stages of the CIRT life cycle:

• Initiation and proposal. Here is the stage where it all begins. Usually, someone makes a proposal to senior managers testing the idea and follows with a written proposal containing:

  • Necessity studies

  • Plan

  • Resource requirements

  • Structure

  • Lines of reporting and authority

  • Staffing

  • Funding

  • Training needs

  • Deliverables

  • Success metrics

  Often the employee who will serve as the unit manager begins a small ad hoc CIRT team as a pilot program. This allows the organization time to get accustomed to the concept and its execution before submitting a formal proposal. Additionally, if immediate success is realized, it makes selling the proposal much easier if a good reputation is already earned. Most employees have not heard of CIRT in this phase and do not have any expectations, yet.

• Developmental. This phase is marked by the formation of the CIRT. Much of their direction will be guided by what is done at this time. In this phase, staffing is selected or recruited, an infrastructure is created, an office site is established, equipment and tools are procured, funding is allocated, duty rosters are developed ensuring that the function-point is available to screen trouble calls at all hours, policies and procedures applicable to the CIRT are instituted, and the team is advertised as operational.

  At this stage, precedence and reputation are going to be earned. When the fledgling CIRT responds, literally every critical eye will be focused on how it performs, how it interacts with managers, and how it interacts with its constituents. Of all times, this is not the one for judgment errors or other failings. The future of the team hinges on its ability to respond quickly and bring the emergency under control with a satisfactory solution. Failing to define and obtain senior management's approval of operational requirements, drafting deficient policies and procedures, forming meaningless outside liaison contacts, and training is staff poorly can quickly spell doom for the team and its effectiveness. On the other hand, if successful the team can move on to the next stage of development.

• Establishment. In this phase startup and development problems are resolved. Constituents know when they should notify the CIRT and know what its course of action is when it arrives. In some instances, CIRTs are loaned or contracted to other organizations to assist in critical incidents. Through contracts and mutual assistance agreements, CIRTs may be deployed at business sites belonging to other organizations on a value-added basis. In this fashion, the cost of their existence is somewhat defrayed.

  In this phase, senior managers have accepted the CIRT and formally recognized its efforts. At some time in this phase, the organization and team members realize the CIRT's existence is indefinite.

  Plans are made for team progress by developing an institutional knowledge base. Team members might be considered promotions, relocation, rotation, or other work assignments. Working with the human resources unit, well-qualified prospective candidates are located and incentives provided, motivating them to consider team membership. The CIRT manager is also anxiously engaged in providing mentors for employees to upgrade training and professional certifications for her employees.

• Postestablishment. This phase includes the expansion of the team to include operations and requirements not part of any previous phases. Usually these activities include the CIRT providing constituency training, delivering presentations as guest-lecturers, authoring articles for peer-review publications, and substantial research and analysis.

Forming a Critical Incident Response Team

In many descriptions you will see the words "Critical Incident Response Team" associated with critical incidents. Many incident response efforts are unsuccessful, not for lack of planning, but because many mistakes were made in creating a team that was not staffed with knowledgeable, dedicated employees. Many organizations use checklist methods of emergency response because of legal or policy mandates where senior managers think their systems' security is guaranteed because they mark a box. Feeling they have met all legal and policy requirements, they are lulled into a false sense of security.

Experience Note

Locks only keep honest people, well, honest. They will not stop a knowledgeable, persistent thief. When visiting a small police department, a visiting dignitary was shown the department's new gymnasium and locker room. She noticed padlocks on the locker doors and asked the commander giving the tour, the reason why. Without missing a beat, the commander remarked the locks were present on the doors to, "keep honest people honest." Even in the police department, they were respectful of each other's belongings but they kept them secure by locking them up.

Security controls have the purpose of making unauthorized entry so unattractive and difficult, they compel attackers to go elsewhere. The only truly effective security systems are those that render important systems inoperable. Of course, that condition is ridiculous. Systems security before, during, and after a critical incident exists as part of the whole picture of good business practice. It ensures uptime, efficiency providing critical systems needed for daily business operations. The purpose of security is to preserve what belongs to the organization from being stolen, deleted, or modified. So, what happens when an attacker, inside or outside the organization, causes a critical incident?

Most organizations have long understood the importance of having fire suppression equipment installed in data-centers, emergency exits, and employee training for emergencies. These same organizations have extensive information security measures with firewalls, DMZs, VPNs, and physical security. Safeguards, like these, have the purpose of maintaining the organization's property and reputation in the community.

Regrettably, critical incident response and management are often neglected until a catastrophe actually strikes and the organization finds itself scrambling to recover.

Experience Note

Critical incident management is determining which assets are needed to sustain profitability (profitability means the organization is accomplishing its goals), establishing policies and procedures addressing employee conduct, compliance audits and mechanisms to actually address crises when they occur.

CIRT

CIRTs should be composed of team members with specific roles supported by specialized training and experience. The CIRT must have a function-point or coordinator where all reports of critical incidents are made. The function-point is usually an individual senior employee or member of a business unit having significant managerial and business experience. She possesses a clear understanding of the organization's goals and objectives, and probably participates in the drafting of the business' operational plans sometime in her career.

It is not expected this person would have a complete knowledge of the organization's mission, goals, policies, and procedures, but it is important that she have sufficient knowledge. For the function-point person to deliver services, she must be available 24 hours, holidays, and weekends. Contact may be accomplished through telephone or other expedient means.

Under practical circumstances, it is immaterial whether the organization decides to use its own in-house talent or delegates the responsibility to outside consultants. The first contact is the employee who receives information relating to the critical incident and makes several important decisions relating to it:

• Does an actual critical incident exist?

• Where is the critical incident occurring?

• What is the extent of the damage?

• Has the damage been contained or is it continuing?

• Do I need to triage the damage at this moment?

• What resources do I need to deploy at this moment?

• Do I have the resources to address this crisis at this time?

• Do I have sufficient information to deliver a meaningful report to senior managers?

• When should I notify law enforcement authorities?

Using Outside Consultants

One of the greatest advantages in using outside consultants (commercial CIRTs) is that of overall reduced cost. This is particularly true in smaller organizations where their operational demands are less than larger organizations. In many cases, contract consultants specializing in critical incident response deal with a wide variety of matters resulting in a high degree of expertise. Additionally, many of their team members have specialties such as UNIX, Windows, or specific programming languages usually not available to employees of smaller businesses.

These are the advantages of commercial CIRTs:

• Most commercial firms have the ability to respond in a matter of hours depending on travel times.

• They provide 24-hour support and are in constant contact.

• They can offer full-service response-posture, as their services usually include forensic duplication and examination, litigation support, expert testimony, technical support, policy formulation, and legal expertise.

• Commercial CIRTs can provide mock-incident response training. Participants address imaginary, but logical, scenarios and interact with personnel, data, and facilities.

• Keeping abreast of current attack-trends. Commercial CIRTs are able to track attacker trends and tailor their response-posture to their clients. By assigning technically trained account executives to clients, they anticipate malicious behavior and are prepared to marshal resources accordingly.

Commercial CIRTs vary greatly in their abilities. Senior managers should do their homework before signing contracts for service.

• Be certain to ask for references from several recent customers and do not hesitate to ask for individual employee's qualifications and experience levels.

• Contact their references. Ask the most important question, "would you hire them again?"

• Determine their reputation in the business community by contact entities such as the Better Business Bureau to ascertain if complaints have been filed and are unresolved.

• Depending on circumstances, ask for financial references.

• Determine if they have bonding in the event of future legal action.

Using In-House Talent

The primary reason for initiating and developing an in-house CIRT is the ability to address emergencies observing the organization's policies and procedures. Staffed with employees, CIRT capability can be directed to address emergencies meeting cultural and internal needs. Because critical incidents often involve sensitive or political matters, in-house talents are more likely to address them in a fashion most advantageous to the organization.

In many cases, internal CIRTs are funded through the corporate offices or on a charge-back basis to the individual business units. Some CIRTs are funded through corporate headquarters paying salaries and other recurring expenses while the individual business units pay for the on-site expenses such as travel, lodging, or other expenses.

Here are a few advantages of the internal CIRTs:

• Direct support. Internal CIRTs will provide emergency response to affected business units with greater specific business-practice knowledge than commercial CIRTs. Generally, they have greater sensitivity to corporate culture than equivalent outside firms.

• Risk management, policy, and audit support. Although these functions are usually addressed by an organization's business units having an internal CIRT can provide invaluable input to heightened awareness and effectiveness. After all, the CIRT has a high vantage point from which to gauge their interaction and deliver this experience to risk managers, policy writers, and auditors on a continuing basis.

• Emergency drill participation. An internal CIRT can participate in emergency exercises testing the full range of recovery, resumption, and critical incident response capabilities. An emergency exercise consisting of an unannounced test can measure the effectiveness of personnel, equipment, and procedures. Postmortem critiques, conducted among employees, are generally more productive and sensitive than sessions involving outsiders.

Ad Hoc CIRTs

This is a concept that has gained a lot of favor in the past few years for smaller businesses. Ad hoc CIRTs are developed utilizing existing talent, and where deficiencies are identified training is vigorously sought. For the most part, ad hoc CIRTs are composed of specially trained employees that have regularly assigned duties and when emergencies strike, they form their response team. For this concept to avoid being stillborn, it must have fanatical senior management sponsorship.

Here are a few suggestions for getting an ad hoc version of CIRT off the ground:

• Identify key technical employees that are qualified to address critical incidents. Such experts would include the IT manager, senior systems administrator, senior engineers, legal advisor, risk manager, human resources unit, etc.

• Draft response policies and procedures for the CIRT to screen initial reports, criteria for activation, response activities, and post-incident critique.

• Obtain senior management agreements with a minimum number of hours of participation on an annual basis for CIRT members. Provide financial incentives to employees for CIRT participation.

• Provide specialized training to CIRT members. This should be training that is complementary with their skills.

• Seek to train toward professional certifications. This is one of those incentive areas where CIRT participants can receive certifications qualifying them for advancement.

CIRT Requirements and Roles

As in any plan, the best place to start is with your deliverables and requirements. Experienced planners actually begin at the end by asking, "What is it we need the CIRT to do?" The most basic requirement for an incident response team is providing support and direction in successfully resolving critical incidents with a minimum degree of business disruption.

Basically, CIRTs are support units intended to provide critical incident response support to the organization as a whole and to the affected business unit specifically.

In this tasking, CIRTs usually serve in these potential roles:

• Direct hands-on emergency response where CIRT members are actively engaged in the containment and restoration of critical IT functions. The full-version of this activity is for the CIRT to assume complete response responsibility. Taking this posture potentially alienates employees already assigned to the affected business units. However, in the event of severe circumstances and if mandated by senior managers, this approach is efficient and effective.

• However, this role can suffer from a conflict of loyalties, as the CIRT is sometimes regarded as "big brother" when it appears on the scene and immediately takes control of the situation.

• Advisory/Shared role. In this role, the CIRT acts as a trusted advisor sharing response activities with the affected business unit. There is less conflict of loyalties in this role, meaning responsibilities are shared between entities.

Added CIRT Responsibilities

Because senior managers view full-time CIRTs as responding only when needed, sometimes they get the reputation of having little if anything to do unless they are responding to a crisis. Their perceived usefulness can be expanded by accepting added responsibilities:

• Acting as a problem screening unit. In this capacity, the CIRT acts as a unit where software patches, tools, and updated software versions are tried and tested before being applied. The practical side of this task rests in the CIRT being able to patch a corrupted system and know the patch they are applying has been tested. There is confidence this patch will not conflict with existing systems and is free from malicious code. Additionally, the CIRT acts like a clearinghouse for recurring or particularly troublesome system problems. They work closely with help desk coordinators and system administrators where any indications of critical incidents are reported and a determination is made if the CIRT should be activated either as an entire unit or in part.

• Coordinate inside emergency efforts and establish liaison with outside agencies. The CIRT coordinates the emergency response efforts of all organizational units in the event of a crisis and works to actively facilitate their drill and actual crises. The CIRT is tasked with the development of liaison with law enforcement and regulatory and legal entities. It actively seeks to participate in such entities as NIPC (National Infrastructure Protection Center), Infragard, HTCIA (High Tech Crime Investigators Association), ISACA (Information Security Audit and Control Association), ISSA (Information Systems Security Association), and the ACFE (Association of Certified Fraud Examiners).

• Provide training inside the organization and to outside entities. CIRT members should be in a very good position to deliver specialized training and increased awareness as one of their proactive jobs. Consider having the CIRT author technical articles in professional periodicals thereby benefiting them by having to do the research and delivering information to other professionals. Through this means, team members learn about developments and emerging trends while potentially providing a valuable service to their constituents and their organization.

CIRT Funding

Funding CIRTs, as are most business matters, is merely a matter of funding. Sometimes developing resources is more a matter of convincing bean counters of their value than anything else.

Here are a few basics to consider when developing your CIRT:

• CIRT as part of the IT function. Locating a CIRT as part of the organization's IT function can greatly facilitate productive interaction between the lessons learned as a result of responding to emergencies and improving development processes. Placing the CIRT as part of the IT function creates avenues of communication between responders and systems staff.

• Business units may benefit by having the CIRT as part of their operation. For example, the systems development unit could greatly benefit by having the knowledge and skills of the CIRT integrated as part of their operation. Having the CIRT as part of the IT audit unit could provide increased granularity and direction in audit programs.

• Corporate headquarters may wish to fund the cost of the CIRT's activities charged as an overhead cost to each of its business units. In this fashion, the cost of having the CIRT is spread to all affected business units, saving each unit from having to make preparations and fund critical response programs. In this fashion, there is an avoidance of duplicating response efforts between headquarters and the individual business units saving time and money. By adopting the "big picture" view, it allows the CIRT to respond to emergencies on the corporate level where trouble spots can be more readily recognized and addressed.

Who Does the CIRT Support?

The quick answer to this question is everybody. However, for a CIRT to adequately function, it must understand the people it serves mission and goals. For CIRT managers, it is suggested they track units calling for their services so they may gear their response accordingly. It is likely the same business units are requesting services time and again; consequently, it is important for CIRT to service their requests as if they were favored clients. For example, if the business units primarily supported by the CIRT consisted of systems users rather than findings produced by IT audit reports, CIRT's response would be less technical than the response delivered to the auditor's findings. Responding to the auditors would probably require more forensic skills than responding to worms and viruses encountered by users.

CIRT Communications

CIRT members should be mindful that their clients are the business units they service. Misplaced, flippant, or capricious remarks return poor dividends. Communications between the CIRT and the units it supports is not just something that is casually performed; it must be a matter of deliberation and coordinated efforts.

CIRTs should have specific communications goals when measuring their success:

• Timeliness. CIRTs must deliver information in a timely fashion. The means by which the information is transmitted may be e-mail, telephone calls, faxes, voice mail, company Web sites, memos, conferences and workshops, working groups, seminars, and bulletin boards. Basically, employees served by the CIRT should have information as soon as it is discovered. For example, the CIRT becomes aware that the BUGBEAR.exe virus is in the wild. The most efficient way to deliver a message warning about the proliferation and damage this virus can cause is by sending a voice mail message to each employee warning that e-mail attachments should not be opened. Sending an e-mail to each employee may result in the information arriving too late, as the employee may be checking e-mail by opening an attachment before getting to the CIRT warning. Timely and credible warnings will go a long way to developing the CIRT's position and credibility in the organization.

• Relevant communications are a must for a CIRT. If the units supported by them are primarily Windows platforms, it does little good to deliver information about UNIX and OS/2. Communications should be crafted so they are meaningful to their recipients.

• Digestibility. The intended audience must understand CIRT communications. For example, if the CIRT primarily serves workstation users, the CIRT should not craft exhaustive communications dealing with the technical aspects of UNIX server configurations. Granted, there might be readers who enjoy the finer aspects of server configurations, but the broader appeal will be to the majority of the users. Reserve specialized information to specialized employees.

  CIRTs should be mindful that there are many levels of employees that are going to read their material, including managers. Including a brief executive summary at the beginning of the communication is appreciated. Depending on the audience, it may serve to have two or even three versions of a communication to be disseminated. One version would be delivered to the general user population, one version to be sent to the managers, and another version intended for the technical staff.

• Accuracy of communications. Few actions will work to destroy the credibility of any business unit faster than to disseminate incorrect information. Get the facts, and get them straight before transmitting information to anyone. Every phrase and every term should be carefully scrutinized for accuracy before going out. CIRT managers must read the communication for technical accuracy and understanding. Of course, CIRT communications should be professional and courteous. This is not the place for colorless humor or sarcasm. Part of communication's accuracy is the assurance the intended audience receives them.

CIRTs should develop out-of-band communications. This means that CIRTs, their constituency, and management should know when and how to use OOBC. OOBC efforts require advance arrangements and coordination within a response team. CIRTs should analyze the organization's current communication structure and devise private alternate channels. OOBC may include private cellular telephones, text pagers, wireless equipment such as PDAs, out-of-business-area telephone communications, registered mail, encrypted e-mail, etc. CIRTs must ensure that each OOBC system is periodically tested and achieves acceptable levels of security.

Critical Incident Response and CIRT Development

Critical Incident Management

In modern organizations, the combination of easily available data, poorly administered safeguards, and malicious individuals make systems vulnerable and attractive to attacks. Almost daily, we hear of businesses being robbed of critical information assets or suffering outages through virus infections or denial-of-service attacks. Computer networks are still relatively new, having their birth only 30 years ago. It sometimes seems hard to put in perspective, but the vaunted Information Highway was just getting its feet of the ground in the early 1980s. And, as information became an extremely valuable commodity, the exploitation of vulnerabilities seemed to keep pace with the growth of network systems.

Illustrating this point is one of the most famous misdeeds, the 1988 "Morris Worm" incident resulted in a significantly large percentage of the network systems with Internet connections being corrupted and removed from service. This was the catalyst that caused Internet users to have postmortem meetings where they decided that preventative, detective, and corrective steps had to be made active parts of their business practice.

For the past seven years, the Computer Crime and Security survey has been conducted jointly by the Computer Security Institute (www.gocsi.com) and the Federal Bureau of Investigation's San Francisco, California, office. The purpose of this survey is to raise levels of computer system awareness while measuring the magnitude and frequency of computer crimes. The 2002 survey results are based on 503 responses from computer security professionals practicing in U.S. business and government agencies. Responses to this survey confirm that computer systems threats continue to spiral upwardly with corresponding financial losses following.

Here are a few highlights from the most recent survey:

• 90 percent of the survey respondents detected computer security breaches in the last twelve months.

• 80 percent acknowledged financial losses attributable to the computer security breaches.

• Of the 503 respondents, 44 percent estimated their financial losses at more than $455 million.

• The most serious financial losses occurred through the theft of proprietary information with 26 respondents reporting more than $170 million and 25 respondents reporting more than $115 mission in financial fraud.

• Of the respondents, 74 percent reported their Internet connection as the more-frequent point of attack than their internal system.

• In 1996, only 16 percent acknowledged reporting intrusions to law enforcement, but in 2002, 34 percent reported their intrusions to law enforcement authorities.

• 40 percent detected systems' penetration from outside the organization.

• 78 percent detected employee abuse of Internet access privileges or inappropriate use of e-mail.

• 38 percent suffered unauthorized access or misuse of their Web sites in the last 12 months, while 21 percent reported they did not know if there had been unauthorized access or misuse.

Patrice Rapalus, CSI Director, remarked that the Computer Crime and Security Survey has served as a reality check for industry and government:

Over its 7 year life span, the survey has told a compelling story. It has underscored some of the verities of the information security profession; for example, that technology alone cannot thwart cyber attacks and that there is a need for greater cooperation between the private sector and the government. It has also challenged some of the profession's 'conventional wisdom;' for example, that the 'threat from inside the organization is far greater than the threat from outside the organization and that most hack attacks are perpetrated by juveniles on joy rides in cyberspace. Over the 7 year life span of the survey, a sense of the facts on the ground has emerged. There is much more illegal and unauthorized activity in cyberspace than corporations will admit to their clients, stockholders, and business partners or report to law enforcement. Incidents are widespread, costly, and commonplace. Since September 11, 2001, there seems to be a greater appreciation for how much information security means not only to each individual enterprise but also to the economy itself and to society as a whole. Hopefully, this greater appreciation will translate into increased staffing levels, more investment in training, and enhanced organizational clout for those responsible for information security.

Experience Note

The most frequent system attacks originate outside the business organization, but the most successful attacks are those committed by insiders.

Critical Incident Response

The best response to critical incidents is characterized by the "ounce of prevention is worth a pound of cure" philosophy. It is much more financially prudent to implement a sound risk management program characterized by written policies, procedures, and standards, with compliance ensured by comprehensive and unannounced audits, than it is to deal with financially devastating events after they happen.

But there are times when "bad things happen to good people" and a response must be made to a critical incident occurring despite your best efforts. It is virtually impossible to predict when someone is going to attack your system and steal your critical information except to say it is not a matter of if as much as it is a matter of when.

Firefighter Response Model

Responding to a critical incident is similar to responding to a fire. Fire departments work tirelessly to educate us about the best means to prevent fires. Safety training starts with simple programs when we are young by talking about fire-related hazards at home and school. Television and radio public service announcements tell us of the safety measures we can take to safeguard our lives at home. We see fire-safety slogans telling us "only you can prevent forest fires" and similar signs as we enter campgrounds and picnic areas. Sometimes we are visited by Fire Marshals inspecting our facilities, making certain there are marked exits and equipment to extinguish fires and save lives.

When the worst happens, a company of firefighters responds to an emergency:

• Respond to emergency contact numbers

• Trained to handle wide-ranging emergency situations

• Organized in the deployment of their tactics and equipment

• Frequently cross-trained as Emergency Medical Technicians

• Confirm that an emergency exists and the nature of it

• Take all appropriate steps to control the emergency

• Take all appropriate steps to prevent the fire from destroying priority order:

  • Lives

  • Surrounding property

  • Property where the fire is presently burning

• Take every possible step to collect and preserve evidence of criminal behavior but not at the risk of life and property

• Testify at judicial proceedings about their actions and findings

• Conduct reviews and critique improving their performance

Critical Incident Response Strategy

No one would argue that responding to critical system incidents is a complex area that is not as easy as taking a pill and waking up feeling better in the morning. Critical Incident response methodology closely follows that of the firefighters:

• Precritical incident preparation. Designated and specially trained response personnel, contact methods, equipment, and tool availability and response posture.

• Detection of critical incidents.

• Initial response evaluation. This is a preliminary step in which an initial investigation is performed and an evaluation is made quickly to determine which type of response is appropriate.

• Response. This is the step where necessary resources are deployed responding to the critical incident. The response goals are very similar to those of the firefighters: contain the damage, prevent it from further spreading, dedicate efforts in a priority manner, and pursue resumption of normal operations.

• Response posture strategy. This step is where the preliminary facts are ascertained and a "best response" plan is proposed. At this time, the proposed plan is passed to senior managers for their review and approval. It is imperative that this step be accomplished within the framework of response demands and priorities. Time is of the essence, dawdling is not acceptable here. Depending on the nature of the emergency, there will be times that an immediate hammer-to-nail response is made and there will be times when the matter may be handled the next business day.

  Experience Note

  Be careful of "crying wolf" too frequently; if every case is declared an emergency, there are no emergencies.

• Law enforcement notification. Having previously established a relationship with law enforcement authorities, responders know whether they should collect the evidence first, or secure the crime scene and wait for officers to respond.

• Legal determination. Responders must include their legal counsel in the decision process surrounding response strategy. On receiving the responder's observations and recommendations, legal counsel should be prepared to render an opinion whether the responders should collect evidence for future legal proceedings, notify law officers so they can collect relevant evidence or take immediate steps to correct damage and restore operations possibly destroying evidence. It is possible that in destroying evidence that responders are violating laws or regulations by not preserving evidence and not coordinating their efforts with law enforcement authorities. For this reason, senior managers and legal counsel must be part of the decision process.

• Evidence collection. This step collects key evidence with interviews, photographs, sketches, and physical evidence.

• Forensic duplications. This step provides bit-by-bit, forensically sound, duplications of critical media.

• Recovery. Responders take appropriate steps to isolate, contain, recover from the incident, and resume business operations.

• Reporting. Take appropriate steps to draft accurate and timely reports to stakeholders and law enforcement authorities, where applicable.

• Postmortem. This is the after-action critique and report of the actions taken during the critical incident response.

Critical Incident Planning

• If you do not plan, you're planning to fail.

Writing and implementing a critical incident plan ensures that emergencies are addressed carefully, thoroughly, and in conformity with risk management programs. As part of the response plan, draft checklists where common incidents are addressed minimizing the required time for response actions. For example, having a response checklist addressing a workstation virus will be significantly different from an employee who is discovered stealing intellectual property and e-mailing it to a competitor.

Here are some recommended elements for a critical incident response plan:

• Obtain and follow the organization's risk management plans. If your organization does not have one, today is an excellent time to start one. This plan should provide details relative to the priority of critical assets, their restoration, and the steps to be taken for resuming profitable operations.

• The critical incident response plan should outline the means of detecting emergencies, collecting preliminary information, assessing the gravity of the system attack, systems affected, spread of damage, steps necessary to stop damage, and protect personnel, data, and facilities. The recommended plan structure is simple, direct, and understandable.

• The critical incident response plan should provide a means to easily contact all relevant employees and outside resources.

• The critical incident response plan should provide specific instructions about policies, procedures, and legal requirements.

• The critical incident response plan should provide templates for any documents required during the emergency. For example, the plan should include a template for logging responder's actions and significant events during the response.

Many critical incident response plans fail because they do not include a response-owner and a senior management correspondent as part of the process. A response-owner is the employee responsible in most cases for the response the emergency receives including relevant actions from start to completion. The senior manager correspondent is the employee who will deliver information to stakeholders.

Command Post Operations

This is a sensitive topic relative to the initiation, staffing, and operation of a command post. Do not think that CPs are intended only for military or government operations because all agencies, while addressing emergency situations, should consider this response strategy. Basically, a CP is a temporary business unit assembled to address one of more crises and will remain in operation until all emergencies are stable and settled. CPs work very closely with regular business operations but have the executive "horsepower" to function independently in decision making, assigning resources, taking action, and following up.

CPs are staffed with specialists assigned particular tasks with dedicated resources at their disposal. In their most common configuration, CPs are housed in segregated facilities located within the business' headquarters. If this is not possible, plans should include relocating the CP to a secondary and equipped facility. They should be equipped with dedicated facilities such as office space, electrical generation, high-speed satellite-linked Internet connections, telephones having multiple direct lines separate from other business units, satellite-linked television for news reception, and a LAN connecting CP workstations to the business LAN and the Internet.

CP reporting structure is funnel-shaped. Information flows from telephone calls, radio, news broadcasts, and e-mail to those designated for information processing. Telephone callers may be employees, specialized response teams, members of the press, stakeholders, or the general public. Carefully trained employees are tasked to interview outside callers and collect information. They are trained relative to the information they may disclose because any comments will be attributed to the organization.

Individuals collecting information for the CP should complete a simple contact report form synopsizing the information from their call, news broadcast, or e-mail. This form may be paper-based or electronic with one copy being passed to the function-point (the single point where all collected information flows), another copy is passed to the data input unit, and the last copy is retained and archived as "work papers." If it is significant, she immediately briefs the function-point and follows the briefing with the written contact report form.

The function-point unit is the person or unit that screens incoming information and makes a determination of where the information should be routed, its priority and processing action. The function-point is a critical position requiring decisions to be based on sound business sense. The data input unit is responsible for routing the information to the unit or employee assigned to the task by the function-point. Another unit must be responsible for collecting the work papers and organizing them for future review and retrieval.

Within the CP are several critical business unit representatives. Depending on the nature of the emergency, these are suggested units that should have representatives in the CP:

• Legal

• Human Resources

• Public/Media Relations

• Senior Management

• Operations Staff

• Maintenance Staff

• Supply/Logistics Staff

• Communications Staff

• Data Input Staff

• Function-Point staff

At least in the initial stages, it will probably be required that the CP is open and staffed for 24 hours.

Experience Note

CP staff will have stages of burnout. Replace all staff members at the end of their 8-hour shifts. At the end of shifts, there should be a briefing by the outgoing shift of the events so the oncoming shift knows what has happened during the past eight hours.

There is a good reason to maintain an events log — so the oncoming employees can review it for reference purposes. Activity logs and other work papers could be made part of legal actions, so care in this area is advised. Employees should be trained that documenting facts is acceptable, while documenting opinions or editorializing are not.

Experience Note

While working in a CP, an employee made a note that was later maintained as a work paper about an event that was only hypothetical and not actual. However, when legal action was sought, the plaintiff introduced the note was as if the event actually happened. Despite the defendant's protestations and objections, the note was accepted as evidence causing significant damage to the defendant's case.

Once the emergency begins to abate, staff, duty-hours, and activities can be reduced. It is a common practice having CP unit leaders meet every half-hour during the first few hours of CP operations. At this time, they should bring important events to briefings along with any concerns. Meetings should last not more than a few minutes and are driven by the nature and treatment of the emergency.

CP employees should understand that press inquiries can have grave consequences for the organization. They should be trained to handle press calls in an appropriate manner. For example, in the face of a disaster, the CP receives a telephone inquiry from a noted news organization; the employee handling the call accepts the information and documents the inquiry by completing the contact report form. Once completed, the form is passed to the function-point where it is screened again and passed to the public relations unit at the CP for handling. One copy of the intake report is passed to the data-input unit that is creating a chronology database of events, and while making an assignment to the public relations unit with a request, they respond when the assignment is completed. In this fashion, assignments can be tracked whether they have been completed or not. Frequently, the input unit will list all uncompleted assignments and pass them to the function-point that will screen them again deciding if they need to be completed in light of the most recent events. Once the public relations unit receives the assignment from the function-point, they contact the news organization and provide appropriate information.