Performance Matters | Active Scanning Technology

During a scan, the goal of the scanner is to get as complete and accurate a scan as possible. However, the performance and behavior of that scan is also important to the vulnerability manager. Ideally, we would like to scan as much as possible during an allotted time window and get complete results. However, we also want to avoid affecting production operations. First, let’s look at the potential negative impacts to production and how we might avoid them. Then, we can look at ways to optimize scans.

In most cases, there are four ways in which a scan can adversely affect a production environment:

• By consuming bandwidth, preventing other applications from meeting service levels.
• By consuming target CPU resources in an already-busy target. Again, this can cause service levels to be missed.
• By breaking a target application or OS, causing a DoS and requiring the target to be repaired.
• By breaking a component that is facilitating the scan but is not a target. Various network components could be adversely affected by the scan process even though they are not the subject of a scan.

Bandwidth is consumed by network activity. During the scan process, parameters provided by the vulnerability manager are used to size the footprint on the network. Not only is bandwidth a factor, the number of simultaneous connections can affect intermediate devices as well. Since TCP is so commonly used, a connection is established with each target. In some cases, a connection is only attempted, leaving potentially half-open connections. Devices that track the state of connections such as firewalls, IPSs, and possibly routers, can be affected by these connections. The total number of simultaneous connections, and the rate at which these connections are made, may have an effect. Limits on both will go a long way towards maintaining good relations with network management staff.

The most efficient location in the scanner to impose these limits is the IP protocol stack and interface drivers. Bandwidth limitations are best performed at the interface driver, whereas the connections limits are better applied at the packet-creation phase where the outgoing connections table is maintained. Exceptions must be made to accommodate the critical command and control functions of the scanner. Therefore, the location of the scanner management system should be exempt from such limits.

Bandwidth consumption has the biggest impact on the network when scanning is performed outside of the local segment to which the scanner is connected. In particular, WAN links can be impacted significantly. Using today’s most cost-effective technology, most scanners are not able to produce more than 10 Mbps of bandwidth in a typical scan. However, when a T1 is the only connection from a remote office back to the corporate WAN where the scanner resides, it is easy to saturate the link event with a small scan. In most companies, it is often necessary to perform such a scan during work hours when desktop and laptop computers are powered on and connected to the network. So, the impact to business operations is significant.

To determine the appropriate amount of bandwidth to allow for a given target network, I recommend the following strategy:

1. Determine the peak utilization of the WAN link over a two-month period to coincide with the time in the business cycle during which you plan on running the scan. Alternatively, plan to run the scan during a time in the business cycle when bandwidth consumption is relatively low but the targets are still available for scanning.
2. Agree with the local business manager and network operations manager on how much of the remaining bandwidth you will be permitted to use. Set expectations when making this agreement.
3. Perform a test scan of the target network for five or more typical target hosts to gauge how much bandwidth is required. This number will have to be scaled up to derive a more accurate number. One caveat: the accuracy of this estimate will increase when the test scan can more closely reach a maximum number of connections. Sometimes, limiting the number of connections will reduce the maximum bandwidth consumed, and in other cases, it will not. It will all depend on the configuration of each target. Given the highly dynamic nature of a scan, the amount of testing should be commensurate with complexity and variability of the environment and level of criticality of the WAN link to business operations. This is both art and science.
4. Always position the scanning activity as a critical security function that will ultimately provide reports and analysis to the IT and network managers at the site. They should want this as much as you do. In a site with critical hosts, it may be worth purchasing a small amount of additional bandwidth that is dedicated to performing the scans.

Simultaneous connections, on the other hand, are fairly straightforward to manage. As previously mentioned, this parameter may affect bandwidth. The primary goal is not to overwhelm the target or the intervening network and security infrastructure. Even at low bandwidth, small packet sizes and half-open connections can generate a sizable number of simultaneous connections. Since a firewall maintains a state table for each connection, it must perform a little more CPU work for each connection. Large commercial firewalls that typically front a public Web site generally have sufficient resources to handle huge numbers of connections. However, this is not always the case. Firewalls are complex devices running many simultaneous processes. Furthermore, they are mostly single-threaded applications, which limits their performance scalability. Just as DoS attacks are performed on routers by exploiting the reliance of certain features on notoriously limited CPU resources, a scanner can do the same to a firewall. Most of these scenarios can be addressed with the following guidelines:

• Test the impact on a firewall under load for a variety of targets. Different targets can generate different connection rates. Monitor the firewall CPU reaction closely. The reaction is not linear. In some cases, packet forwarding latency may result. In other cases, it just takes longer to set up and tear down connections. Manufacturer performance specifications do not apply to traffic generated by vulnerability scanners.
• Test scanning against targets where the firewall applies additional distributed DoS and intrusion prevention capabilities. Since firewalls typically interact with the environment at OSI layer 3 and above to provide these services, the firewall may interpret a scan as a threat. It could be blocked or the results of the scan clouded. This is because the firewall features can act like a proxy for a given application. TCP connection attempts will be terminated by the firewall and then re-established from the firewall to the target only after the handshake is completed with a sanitized upper-OSI-layer result. Any probing performed with a SYN-SYN/ACK-ACK that is not fully completed by the scanner may appear to be an active host where none exists. A solution to this is to configure the firewall to allow the scanner source traffic to bypass all of the firewall inspection filters.
• Routers can also be affected. Some routers use their limited processor power to handle invalid TCP flag combinations. This is a common probing or discovery technique used by vulnerability scanners to fingerprint the OS or application. It is also a method for performing a DoS attack against a router. Although it is unlikely that a scanner will generate enough malformed traffic to have a major impact on network devices, you should be mindful of the possibility.
• Simulate the typical latency on a WAN circuit in the lab during a scan. This will help gauge scan performance as well as impact to production systems. Some vendors provide tools that can use a packet capture from a network segment and a performance profile, and then re-create the user experience with the scan traffic and any given application injected. For example, an accounting system could be placed on one side of such a device while the scanner is on the other side. The device simulates the traffic conditions at a peak time for a particular office using a previously gathered profile.
• Limit the number of TCP and UDP ports that are scanned during discovery and scanning. The tendency is to be as comprehensive as possible because you are not sure what will be found. This may work for an initial scan of a limited number of hosts. But later, you should settle on an acceptable number of ports to minimize the impact on the environment and maximize effectiveness. This is an area with diminishing returns on port numbers scanned at a logarithmic rate. You may also find that reducing the number of ports scanned will substantially lighten the bandwidth load. This is because packets can be large and numerous.

Web Application Testing

With so many competitive pressures, it was inevitable that organizations would have to find a way to distinguish themselves in the online world. So, millions of custom applications have been built to deliver customer service and application services that add value beyond original core competencies. It naturally follows that hackers would find common ways to exploit these applications, especially because they are more closely linked to valuable data. What has made these applications even more exploitable is their dependence on standard technologies and infrastructure. This is not intended as a criticism, only an observation. For example, most databases in use today are based on the SQL language. Also, many Web applications use JavaScript™. Both of these technologies are exploitable, depending on their method of implementation. No inherent security controls can be configured to prevent their exploitation at the application level.

An even greater concern is the exploitability of the open-source PHP language. It is commonly used to build Web applications and yet has many critical vulnerabilities around which the programmer must code. Since this powerful scripting language is closely integrated with the Web page and user interaction, and since it has so many powerful commands with great flexibility, exploitation is very possible.

Naturally, this Web application phenomenon presents another area of vulnerability testing. These checks have become increasingly important, as customer applications have become the primary focus of the serious hacker. There are simple methods to exploit vulnerabilities by simply manipulating the content of the URL in the Web browser. It is also quite easy to manipulate inputs on fields on the screen. So, vulnerability checks must do the same thing in many combinations using many common techniques to replicate possible attack vectors. Following is a list of some of the most common vectors:

• Input field manipulation: This involves modifying the input of one or more fields on the screen beyond what is expected by the software in normal operation. Many programmers fail to validate these inputs for size and value boundaries as well as validity. Hackers exploit this by entering characters that will cause the application to process them in a way not originally intended.
• SQL injection: This very popular attack vector is used to manipulate the database query language of the back-end database programming to reveal information or even modify the database contents. The process is started by entering the partial SQL string (‘ or 1=1—) in an input field without the parentheses will extend an underlying SQL statement to detect the presence of an unfiltered field and the fact that the SQL language is in use and accessible. This works because the first tick mark (‘) ends the current input string expected by the SQL code and then adds the logic “or 1=1.” The remaining portion tells the SQL server to ignore the remainder of the SQL statement. This is a harmless modification of the SQL query that will determine whether SQL injection is even possible. Vulnerability scanners will perform more in-depth penetration acts to reveal more details about the flaws in handling this input, including what can be discovered about the database configuration.
• Cross-site scripting (XSS): This extension of input field manipulation is used to inject JavaScript into a Web site that will appear on other users’ browsers and perform actions against other users of the system. These actions include but are not limited to directing user input to another site, capturing user data, and presenting false information. The script information can be combined with a SQL injection attack to store malicious script code in the database of the target system. When this information is retrieved by the Web application for a user, then the script is loaded into that user’s Web browser and executed to fulfill the purposes of the attacker.

So Web application checks are available in many VM systems to detect the presence of these vulnerabilities in the code of a Web site. There are other products that analyze the actual programming code, but they are no substitute for the hacker-mimicking process of attacking the application from the outside. By their very nature, these checks are brute force but usually non-destructive. The application checks are run against every field, every hyperlink, and every possible URL of a Web site. The following are more types of checks that can be performed against applications:

• Boundary check: This is a test of a range of values that are accepted by an application. It is applied to every field that is found and applies values that can be below, above, and within the range of permitted values, as well as trivial values that are outside of the expected data types. For example, an 8-bit ASCII numeric field may have 16-bit Unicode data entered. This is also called stress testing.
• Branch test: This program is used to check all of the links and possible paths to be taken in an application. The goal is to achieve as close as possible to 100 percent branch coverage, meaning that every possible program pathway is exercised. This is partly a discovery process and partly a vulnerability identification process. Some vulnerabilities in this area are the result of branches of code that are so obscure they are seldom accessed. Because of this, it is possible that some code is vulnerable because it was never tested.
• Brute force: This type of check is typically used against user ID and password fields. It differs from a boundary check in that there is a large practical range limit for what can be entered. A password field that is eight alphanumeric characters long, for example, has 8**36 possible combinations.
• Buffer overflow: This type of check is designed to challenge the target software for scale factors; that is, those inputs whose range affects the allocate memory space. Two types of scale factors can be tested: buffer definition and buffer usage. For example, if a program is designed to accept a certain size of input and store it in a memory buffer, a larger input can cause a buffer overflow. This is an example of a buffer usage scale factor, which occurs when an entered parameter directly or indirectly determines the allocated buffer space. If the parameter causes the creation of buffer smaller than the input size, then the buffer can overflow on the corresponding input.
• Code injection: In many scripting languages, it is possible to accept code as a parameter passed from one process to another. Programmers use this technique to efficiently reuse code or pass advanced parameters from one program to another. Some faults in a program can allow the inadvertent introduction of code by accepting instruction-terminating characters to be entered into a field. This is similar to SQL injection attacks.
• Session hijacking: HTTP is a stateless protocol. That means that it receives transactions over the network but does not know that the transaction is part of a particular user’s session. To keep track of this session or “state,” a file called a session cookie is implemented. There are various checks that can be performed to determine whether the session cookie can be manipulated to become a different user, and therefore gain access to that user’s data.

All of these tests, however, are typically used alone when run automatically. This is a key distinguishing factor between vulnerability testing and penetration testing. A person conducts basic reconnaissance testing using these application-vulnerability scanning tools, and then combines the results to form augmented attacks with more revealing results. For example, an SQL injection attack can be used to insert malicious code to obtain another user’s session cookie. Then, the session cookie, encrypted or not, can be used to obtain that user’s information or change his password to gain his privileges.

Performance Issues | Active Scanning Technology Detection Methods

We have discussed at some length the process of identifying ports and handling TCP connections. All of these factors have to be taken into consideration during the scan; however, the scanner cannot wait too long. At some point, the transaction attempt will “time out.” This phenomenon can be referred to as discovery tolerance. Various vendors have different levels of discovery tolerance. The amount of tolerance is loosely proportional to the accuracy of the discovery with rapidly diminishing probability of successful identification. Fortunately, we know from experience that there is no point waiting for a reply beyond a certain amount of time. Determining that point is the real skill in any fingerprinting activity. The goal is to be complete and accurate, but there is a law of diminishing returns. Two key timers affect the speed of the discovery process: the connection establishment timer and the retransmission timer.

For many TCP implementations, the connection establishment timer (TCP_KEEPINIT parameter) waits 75 seconds for response. A simple scan on a single port for 200 hosts would require over four hours to complete if none of the hosts responded. This must be adjusted to wait far less time. One effective approach is to take the maximum roundtrip time (RTT) of ICMP echo reply exchanges and add two seconds. This provides ample time for an application to respond on the required port and is likely to be far shorter than the default of 75 seconds.

With TCP connections, a discovery process can also vary the retransmission timer when additional packets are to be exchanged with the target. In normal communications, the timer begins with a value of 1.5 seconds. If no response is received, then the value is doubled to three seconds. If there is still no SYN-ACK, the timer is doubled again and we wait six seconds. This continues repeatedly until we reach a limit of 64 seconds. The process is called exponential backoff (EB). In theory, this should parallel the exponential probability that a response will ultimately be received. However, this is often impractical for host discovery purposes in vulnerability scanning. A typical OS can spend several minutes waiting for a connection to time out.

A more practical approach would be to sequentially increase the retransmission timer by smaller values for a total period of time to be some factor above the average for the target IP range. For example, let’s suppose that we are performing a discovery of network A (192.168.10.0/24) with an upper limit of 30 seconds for retransmission. If the first 16 hosts required an average of 10 seconds to respond and the mode was five seconds, we might start our retransmission timer at five seconds and increase the value by five seconds until an upper limit of 20 seconds was reached (2× average). This is a more sensible approach that will avoid a common IP stack value that can reach several minutes for a single connection. Remember that our goal is discovery of open ports and live hosts, not the reliable transmission of data to another host.

There is one other item that can be manipulated, which is not exactly a timer and can speed the discovery process considerably:

to implement all the timers TCP only requires that two functions are called periodically: (1) the fast timer is called every 200 ms and (2) the slow timer every 500 ms. TCP uses these two periodic ‘ticks’ to schedule and check all the timers described…as well as measuring round trip times.^[*]

Basically, the OS kernel must check every 200 ms to see if an acknowledgment has been received. In modern networks and operating systems, this is a very long time.

By decreasing this period, the discovery processes can recognize that the probing SYN packets it has sent have been acknowledged in a shorter time, and move on to the next probe. If the RTT from SYN to SYN-ACK is 10 ms, then under normal circumstances, the discovery process can wait for up to 190 ms to proceed with the next action. Multiply this number by hundreds of hosts and dozens of ports, and the wasted time can be tremendous.

The one caveat to modifying TCP timers is that some applications are simply slow to respond. This approach works best when probing for open ports but not necessarily for applications. There is a lot of room for creativity in scan performance optimization. This section simply illustrates some of the challenges designers can be confronted with when trying to optimize the scan process and minimize the impact on the network.

1 Black Box Testing

Once the presence of a host has been established and that presence recorded in the memory of the scanner, a series of tests or “checks” are performed to find vulnerabilities. The types of checks are dependent upon the type of host and the configuration of the scanner. Generally, two types of checks are performed. A network-based or surface check is performed, which involves the probing and analysis of what is evident with limited or no access to services on the machine other than what is offered to any other peer on the same network as that which exists between the scanner and the target. This is also known as an unauthenticated check. The other type of check is an authenticated, internal check or white box test. It is performed when the scanner is given special information and credentials to access details of the host, which are generally reserved for trusted entities.

The difference between surface and internal checks is obviously significant not only in the way they obtain information, but also in the value and quality of that information. Clearly, more detailed data can be obtained by logging into a host and perusing its configuration. Although the information tells us a lot about the host, it does not typically represent the view of an attacker who performs reconnaissance on an unknown host. Although valuable from an analysis standpoint, some attacks take place by probing the host from the view of an outsider; therefore, information that can be obtained in the same fashion is often more valuable. To summarize, a vulnerability discovered and exploitable from outside a host represents a greater exposure than if the same vulnerability could only be discovered and exploitable from a credentialed or internal check.

There is a common perception that authenticated checks are more accurate than remote checks but that’s often not true. The Windows registry is commonly used for authenticated checks but is often wrong. It’s important to consider that not all authenticated checks are created equal and that a remote check is a good method of validating authenticated information.

The black box testing process involves some straightforward testing over the network and possibly some creative use of IP and other protocols. Usually, the simple tests are harmless and efficient. The more exotic manipulation of IP protocols can cause problems on scanned hosts with applications that are ill-prepared to handle many variations. This is a vulnerability in itself. The IP stack of the host is usually capable of handling nearly any variety of traffic, but the overlying applications sometimes are not. It is another area that calls for extensive testing in order to avoid adverse effects on production systems. Most vendors are able to provide a list of known negative application interactions.

Following is a list of some common methods of reconnaissance:

• Malformed packets are sent to the host to identify the presence of a vulnerability in the response. This is similar to the discovery process and is sometimes incorporated into the same phase for efficiency. The information sent to the target can be at any one layer or multiple combinations of layers 3 through 7 in the OSI Model.
• Normal packets are sent to a known application to obtain results that will reveal vulnerability information. This is very common in the http protocol to obtain information about the Web server, application server, or back-end databases.
• Valid information is sent to the target to gather valid header response data that will reveal the version of software answering the service request. This is known as banner checking. Many software applications can obfuscate this information with simple configuration changes, so it is not the most reliable method.

These methods can be summarized conceptually in pseudo code form:

Send X to target
Listen for response Y
Match Y to possible response list
If Y is on list, note vulnerability
If Y is not on list, ignore
Get next check; Loop

Information Technology Human Resources Management Policies: Yes, Virginia, IT Employees Really Are Different

Policies governing IT human resources differ from those used in other business units because they require some stringency. Polices and procedures to hire the best-possible candidates will decrease human resource risk factors and raise the possibilities of "getting the biggest bang for the buck." IT employees are very different than others in the business world. They have very specialized skills, training, certifications, and experience that are very portable. Consequently, many IT employees feel more loyalty to their profession than to the company, requiring policies and procedures to accommodate these differences if the business unit is to function at the level that is normal for the profession and if the turnover rate is to be minimized.

Getting the Best Candidates for the Position
There are successful procedures for obtaining the best IT candidates when combined with the more-traditional methods:

- Pay a meaningful bounty to current employees for recruiting and hiring qualified people.

- The Human Resources management office should have a standardized procedure for screening employment candidates so only qualified candidates are considered. Often the creation of forms requiring basic information as to the candidate's education, training, professional experience, professional certifications, and professional recommendations provide an effective mechanism of measuring the candidates for jobs. Accepting résumés is a traditional first step in developing a qualified candidate pool but some candidates have an ability to mischaracterize their abilities, so it is recommended the standard job form is sent to candidates who appear most qualified.

- Consult with the person leaving a position that can describe the job better than anyone else. In the event of a new position, locate persons who are in the same or similar position for their input.

- After the most qualified candidates are found, confirm their work experience at their previous employer. This is a frequently ignored area. Take the time to speak with their former supervisor and ask job performance questions. Determine if this person is eligible for rehire.

- If there is a bonding or other similar requirement, obtain a waiver from the candidates and obtain a credit report.

- Do not forget to verify their professional references. It is recommended that each qualified candidate provide at least three. It is recommended the references should be thoroughly interviewed about the qualifications of the candidate.

- In the case of very-sensitive positions, require a complete background investigation to be performed by a competent and experienced firm. These background investigations should be required every few years of employees in sensitive positions. They are worth the time and expense.

- Avoid nepotism. The controls needed for IT business operations are stringent at best; hiring relatives may cause unnecessary problems. Individuals related to each other tend not to abide by the same rules as nonrelatives. Often they tend to cover up more errors and have higher rates of orchestrated thefts and embezzlements.

Job Interviews
Candidate interviews are a necessary part of the selection procedure. They usually begin after the candidates have passed preliminary screening processes.

Structured interviews are generally characterized by a list of questions suitable for interviewers with little training to ask. The questions can be divided among the interviewers or repeated using different versions looking for response consistency. This approach is very restricted and narrow.

Semistructured interviews are usually characterized by only a few major questions prepared by the interviewers in advance. Conducting this interview requires a higher degree of technical skill and preparation but allows for more flexibility than the structured interview approach. At this time, the interviewers are more able to probe the areas that seem to merit more detail from the candidate.

The unstructured interview is best conducted by senior managers and very experienced interviewers. Topical areas are covered during the interview, but the interviewer retains the freedom to adapt to the individual candidate. The reliability of this interviewing method may be questioned at some time but if it is conducted carefully and thoughtfully, it does provide the interviewer with substantial information about each person.

As soon as the candidate interview is concluded, each participating interviewer should document his objective analysis of the candidate in writing. These notes and analyses should be archived and maintained in the event of future litigation. The interviewers should make an informed recommendation to the hiring manager. If the decision is close, the hiring manager may wish to speak with the interviewers or even have a second interview with the candidates.

Performance Reviews
Most companies have employee performance review procedures in place. IT systems professionals may require specialized procedures as their performance is a function of ability as well as professional skill and personal effort. Following are several procedures that should be considered when addressing individual performance matters:

- There should be formalized criteria against which an employee's performance is measured. This criteria is not merely a job description; rather, this document sets out the performance expectation of the employee in that job. It is important that the performance criteria match the level of professional expectation for each position level. For example, if an employee is an entry-level programmer, the performance measurement criteria should be geared toward that of an entry-level programmer.

- Consider job performance measurement criteria that have unacceptable levels, acceptable levels, and truly exceptional levels of performance.

- Job performance criteria should escalate with the employee's escalating abilities.

- There should be monetary and other award incentives for employees who perform well.

- Employees should read and acknowledge their job performance criteria.

- Consider a formalized career path for employees. Can they increase their responsibilities and pay according to formalized methods?

- Performance evaluations should be face-to-face at least annually. Senior managers need to speak with their employees and let them know how they are doing. Managers should seriously consider having semiannual performance reviews so the employee is not shocked at his annual performance review. If someone is doing very well, he should be acknowledged.

- Spot awards recognize employees in a particular task. These tokens are generally something a senior manager has available to hand to an employee, recognizing his good work on the spot.

Employee Termination
To protect the organization, senior managers should use special termination procedures, as departing employees have many opportunities to vent animosity with destructive results. These are a few examples of special procedures when dealing with employee terminations:

- Employee is notified away from the work area.

- Employee's systems passwords are removed before the notice is delivered.

- Employee immediately returns company identification and any company property including tokens, smart cards, laptops, etc.

- Employee is accompanied as he packs his belongings and is escorted from the building. In some cases, the organization may want to pack the employee's personal items and send them to his residence. At no time is the employee left unaccompanied after termination notice is delivered. If transportation is required, it should be provided.

- The internal auditing unit is notified and all employee activities in the network and workstation are audited for a period of 90 days. It is a frequent occurrence that employees with a feeling they are going to be terminated will do damage in that time period.

Employee Departures on Good Terms
Employees departing on good terms, for a variety of reasons, should have an exit interview by someone other than their direct manager. This interview has the purpose of learning exactly why the employee is leaving. Data collected from departing employees may indicate undesirable situations or conditions exist and need correction.

In all cases, as soon as the employee gives notice of departure it is prudent to notify the auditors. After the person leaves the organization, his network and workstation activities should be audited for a period of at least 90 days. This action will attempt to discover if the employee is leaving with intellectual property belonging to the organization, and determine if there are any retaliatory events waiting to happen in his absence.