Performance Matters | Active Scanning Technology

During a scan, the goal of the scanner is to get as complete and accurate a scan as possible. However, the performance and behavior of that scan is also important to the vulnerability manager. Ideally, we would like to scan as much as possible during an allotted time window and get complete results. However, we also want to avoid affecting production operations. First, let’s look at the potential negative impacts to production and how we might avoid them. Then, we can look at ways to optimize scans.

In most cases, there are four ways in which a scan can adversely affect a production environment:

• By consuming bandwidth, preventing other applications from meeting service levels.
• By consuming target CPU resources in an already-busy target. Again, this can cause service levels to be missed.
• By breaking a target application or OS, causing a DoS and requiring the target to be repaired.
• By breaking a component that is facilitating the scan but is not a target. Various network components could be adversely affected by the scan process even though they are not the subject of a scan.

Bandwidth is consumed by network activity. During the scan process, parameters provided by the vulnerability manager are used to size the footprint on the network. Not only is bandwidth a factor, the number of simultaneous connections can affect intermediate devices as well. Since TCP is so commonly used, a connection is established with each target. In some cases, a connection is only attempted, leaving potentially half-open connections. Devices that track the state of connections such as firewalls, IPSs, and possibly routers, can be affected by these connections. The total number of simultaneous connections, and the rate at which these connections are made, may have an effect. Limits on both will go a long way towards maintaining good relations with network management staff.

The most efficient location in the scanner to impose these limits is the IP protocol stack and interface drivers. Bandwidth limitations are best performed at the interface driver, whereas the connections limits are better applied at the packet-creation phase where the outgoing connections table is maintained. Exceptions must be made to accommodate the critical command and control functions of the scanner. Therefore, the location of the scanner management system should be exempt from such limits.

Bandwidth consumption has the biggest impact on the network when scanning is performed outside of the local segment to which the scanner is connected. In particular, WAN links can be impacted significantly. Using today’s most cost-effective technology, most scanners are not able to produce more than 10 Mbps of bandwidth in a typical scan. However, when a T1 is the only connection from a remote office back to the corporate WAN where the scanner resides, it is easy to saturate the link event with a small scan. In most companies, it is often necessary to perform such a scan during work hours when desktop and laptop computers are powered on and connected to the network. So, the impact to business operations is significant.

To determine the appropriate amount of bandwidth to allow for a given target network, I recommend the following strategy:

1. Determine the peak utilization of the WAN link over a two-month period to coincide with the time in the business cycle during which you plan on running the scan. Alternatively, plan to run the scan during a time in the business cycle when bandwidth consumption is relatively low but the targets are still available for scanning.
2. Agree with the local business manager and network operations manager on how much of the remaining bandwidth you will be permitted to use. Set expectations when making this agreement.
3. Perform a test scan of the target network for five or more typical target hosts to gauge how much bandwidth is required. This number will have to be scaled up to derive a more accurate number. One caveat: the accuracy of this estimate will increase when the test scan can more closely reach a maximum number of connections. Sometimes, limiting the number of connections will reduce the maximum bandwidth consumed, and in other cases, it will not. It will all depend on the configuration of each target. Given the highly dynamic nature of a scan, the amount of testing should be commensurate with complexity and variability of the environment and level of criticality of the WAN link to business operations. This is both art and science.
4. Always position the scanning activity as a critical security function that will ultimately provide reports and analysis to the IT and network managers at the site. They should want this as much as you do. In a site with critical hosts, it may be worth purchasing a small amount of additional bandwidth that is dedicated to performing the scans.

Simultaneous connections, on the other hand, are fairly straightforward to manage. As previously mentioned, this parameter may affect bandwidth. The primary goal is not to overwhelm the target or the intervening network and security infrastructure. Even at low bandwidth, small packet sizes and half-open connections can generate a sizable number of simultaneous connections. Since a firewall maintains a state table for each connection, it must perform a little more CPU work for each connection. Large commercial firewalls that typically front a public Web site generally have sufficient resources to handle huge numbers of connections. However, this is not always the case. Firewalls are complex devices running many simultaneous processes. Furthermore, they are mostly single-threaded applications, which limits their performance scalability. Just as DoS attacks are performed on routers by exploiting the reliance of certain features on notoriously limited CPU resources, a scanner can do the same to a firewall. Most of these scenarios can be addressed with the following guidelines:

• Test the impact on a firewall under load for a variety of targets. Different targets can generate different connection rates. Monitor the firewall CPU reaction closely. The reaction is not linear. In some cases, packet forwarding latency may result. In other cases, it just takes longer to set up and tear down connections. Manufacturer performance specifications do not apply to traffic generated by vulnerability scanners.
• Test scanning against targets where the firewall applies additional distributed DoS and intrusion prevention capabilities. Since firewalls typically interact with the environment at OSI layer 3 and above to provide these services, the firewall may interpret a scan as a threat. It could be blocked or the results of the scan clouded. This is because the firewall features can act like a proxy for a given application. TCP connection attempts will be terminated by the firewall and then re-established from the firewall to the target only after the handshake is completed with a sanitized upper-OSI-layer result. Any probing performed with a SYN-SYN/ACK-ACK that is not fully completed by the scanner may appear to be an active host where none exists. A solution to this is to configure the firewall to allow the scanner source traffic to bypass all of the firewall inspection filters.
• Routers can also be affected. Some routers use their limited processor power to handle invalid TCP flag combinations. This is a common probing or discovery technique used by vulnerability scanners to fingerprint the OS or application. It is also a method for performing a DoS attack against a router. Although it is unlikely that a scanner will generate enough malformed traffic to have a major impact on network devices, you should be mindful of the possibility.
• Simulate the typical latency on a WAN circuit in the lab during a scan. This will help gauge scan performance as well as impact to production systems. Some vendors provide tools that can use a packet capture from a network segment and a performance profile, and then re-create the user experience with the scan traffic and any given application injected. For example, an accounting system could be placed on one side of such a device while the scanner is on the other side. The device simulates the traffic conditions at a peak time for a particular office using a previously gathered profile.
• Limit the number of TCP and UDP ports that are scanned during discovery and scanning. The tendency is to be as comprehensive as possible because you are not sure what will be found. This may work for an initial scan of a limited number of hosts. But later, you should settle on an acceptable number of ports to minimize the impact on the environment and maximize effectiveness. This is an area with diminishing returns on port numbers scanned at a logarithmic rate. You may also find that reducing the number of ports scanned will substantially lighten the bandwidth load. This is because packets can be large and numerous.

Web Application Testing

With so many competitive pressures, it was inevitable that organizations would have to find a way to distinguish themselves in the online world. So, millions of custom applications have been built to deliver customer service and application services that add value beyond original core competencies. It naturally follows that hackers would find common ways to exploit these applications, especially because they are more closely linked to valuable data. What has made these applications even more exploitable is their dependence on standard technologies and infrastructure. This is not intended as a criticism, only an observation. For example, most databases in use today are based on the SQL language. Also, many Web applications use JavaScript™. Both of these technologies are exploitable, depending on their method of implementation. No inherent security controls can be configured to prevent their exploitation at the application level.

An even greater concern is the exploitability of the open-source PHP language. It is commonly used to build Web applications and yet has many critical vulnerabilities around which the programmer must code. Since this powerful scripting language is closely integrated with the Web page and user interaction, and since it has so many powerful commands with great flexibility, exploitation is very possible.

Naturally, this Web application phenomenon presents another area of vulnerability testing. These checks have become increasingly important, as customer applications have become the primary focus of the serious hacker. There are simple methods to exploit vulnerabilities by simply manipulating the content of the URL in the Web browser. It is also quite easy to manipulate inputs on fields on the screen. So, vulnerability checks must do the same thing in many combinations using many common techniques to replicate possible attack vectors. Following is a list of some of the most common vectors:

• Input field manipulation: This involves modifying the input of one or more fields on the screen beyond what is expected by the software in normal operation. Many programmers fail to validate these inputs for size and value boundaries as well as validity. Hackers exploit this by entering characters that will cause the application to process them in a way not originally intended.
• SQL injection: This very popular attack vector is used to manipulate the database query language of the back-end database programming to reveal information or even modify the database contents. The process is started by entering the partial SQL string (‘ or 1=1—) in an input field without the parentheses will extend an underlying SQL statement to detect the presence of an unfiltered field and the fact that the SQL language is in use and accessible. This works because the first tick mark (‘) ends the current input string expected by the SQL code and then adds the logic “or 1=1.” The remaining portion tells the SQL server to ignore the remainder of the SQL statement. This is a harmless modification of the SQL query that will determine whether SQL injection is even possible. Vulnerability scanners will perform more in-depth penetration acts to reveal more details about the flaws in handling this input, including what can be discovered about the database configuration.
• Cross-site scripting (XSS): This extension of input field manipulation is used to inject JavaScript into a Web site that will appear on other users’ browsers and perform actions against other users of the system. These actions include but are not limited to directing user input to another site, capturing user data, and presenting false information. The script information can be combined with a SQL injection attack to store malicious script code in the database of the target system. When this information is retrieved by the Web application for a user, then the script is loaded into that user’s Web browser and executed to fulfill the purposes of the attacker.

So Web application checks are available in many VM systems to detect the presence of these vulnerabilities in the code of a Web site. There are other products that analyze the actual programming code, but they are no substitute for the hacker-mimicking process of attacking the application from the outside. By their very nature, these checks are brute force but usually non-destructive. The application checks are run against every field, every hyperlink, and every possible URL of a Web site. The following are more types of checks that can be performed against applications:

• Boundary check: This is a test of a range of values that are accepted by an application. It is applied to every field that is found and applies values that can be below, above, and within the range of permitted values, as well as trivial values that are outside of the expected data types. For example, an 8-bit ASCII numeric field may have 16-bit Unicode data entered. This is also called stress testing.
• Branch test: This program is used to check all of the links and possible paths to be taken in an application. The goal is to achieve as close as possible to 100 percent branch coverage, meaning that every possible program pathway is exercised. This is partly a discovery process and partly a vulnerability identification process. Some vulnerabilities in this area are the result of branches of code that are so obscure they are seldom accessed. Because of this, it is possible that some code is vulnerable because it was never tested.
• Brute force: This type of check is typically used against user ID and password fields. It differs from a boundary check in that there is a large practical range limit for what can be entered. A password field that is eight alphanumeric characters long, for example, has 8**36 possible combinations.
• Buffer overflow: This type of check is designed to challenge the target software for scale factors; that is, those inputs whose range affects the allocate memory space. Two types of scale factors can be tested: buffer definition and buffer usage. For example, if a program is designed to accept a certain size of input and store it in a memory buffer, a larger input can cause a buffer overflow. This is an example of a buffer usage scale factor, which occurs when an entered parameter directly or indirectly determines the allocated buffer space. If the parameter causes the creation of buffer smaller than the input size, then the buffer can overflow on the corresponding input.
• Code injection: In many scripting languages, it is possible to accept code as a parameter passed from one process to another. Programmers use this technique to efficiently reuse code or pass advanced parameters from one program to another. Some faults in a program can allow the inadvertent introduction of code by accepting instruction-terminating characters to be entered into a field. This is similar to SQL injection attacks.
• Session hijacking: HTTP is a stateless protocol. That means that it receives transactions over the network but does not know that the transaction is part of a particular user’s session. To keep track of this session or “state,” a file called a session cookie is implemented. There are various checks that can be performed to determine whether the session cookie can be manipulated to become a different user, and therefore gain access to that user’s data.

All of these tests, however, are typically used alone when run automatically. This is a key distinguishing factor between vulnerability testing and penetration testing. A person conducts basic reconnaissance testing using these application-vulnerability scanning tools, and then combines the results to form augmented attacks with more revealing results. For example, an SQL injection attack can be used to insert malicious code to obtain another user’s session cookie. Then, the session cookie, encrypted or not, can be used to obtain that user’s information or change his password to gain his privileges.

Application Fingerprinting: Banners | Active Scanning Technology

An important activity in a scan is to determine what applications make themselves known on the network. Similar to OS fingerprinting and IP stack fingerprinting, a vulnerability scanner can attempt to connect to a variety of possible applications on known or unknown ports, a process known as featureprinting. Among the best-known featureprinting methods is banner checking.

Some common applications on systems produce a banner whenever you attempt to connect. The contents of this banner can provide valuable information in determining the version of the OS and the software running on the host. This type of fingerprinting can often be performed by an individual using a simple program available on almost all OS platforms: Netcat.

Let’s take the example of a Web server. Figure 1 shows a typical Netcat session. When using Netcat, you can specify the TCP port to which you wish to connect. So, in the command line, we type “nc 10.1.1.10 80.” This will run the Telnet application and establish a connection to the server listening for connections on port 80.

$ nc 10.1.1.10 80 Apache Example HEAD / HTTP/1.0 HTTP/1.1 200 OK Date: Tue, 06 May 2008 23:32:00 GMT Server: Apache/2.2.8 (Unix) Last-Modified: Tue, 29 Apr 2008 21:52:29 GMT ETag: "ea9d61-48f6-44c0a0c71a140" Accept-Ranges: bytes Content-Length: 18678 Cache-Control: max-age=86400 Expires: Wed, 07 May 2008 23:32:00 GMT Vary: Accept-Encoding Connection: close Content-Type: text/html $_

(a)

$ nc 10.1.1.20 80 IIS Example HEAD / HTTP/1.0 HTTP/1.1 302 Found Cache-Control: private Content-Length: 142 Content-Type: text/html; charset=utf-8 Location: /en/us/default.aspx Server: Microsoft-IIS/7.0 X-AspNet-Version: 2.0.50727 P3P: CP="ALL IND DSP COR ADM CONo CUR CUSo IVAo IVDo PSA PSD TAI TELo OUR SAMo CNT COM INT NAV ONL PHY PRE PUR UNI" X-Powered-By: ASP.NET Date: Tue, 06 May 2008 23:58:41 GMT Connection: keep-alive $_

(b)

Figure 1: A typical Netcat session.

From here, we are able to issue a command to the remote server. Since this is port 80, it is likely a Web server; so we issue the command: “HEAD/HTTP/1.0” (Figure 1a). The server responds with some detailed header information, which tells the type of Web server software and OS. In this case, it is Apache^® running on a UNIX^® system. This eliminates the possibility of any version of Windows and makes searching for vulnerabilities much easier.

For security reasons, however, server administrators should conceal this header information, particularly that information which is in the “Server” section. But, that will not deter a good vulnerability scanner. The scanner may also be able to check for the type of Web server by making an invalid request. By using an invalid version type in the HEAD command, we can see different responses from the various Web server makers. Notice that the Apache Web server comes out with a “400 Bad Request” message (Figure 1b). The connection also gets closed; however, on IIS 7.0, the connection is not closed but the same “400 Bad Request” message is received. But, you will also notice that more server information is provided that was not found in the valid request. In earlier versions of IIS, you could distinguish it from a reaction of providing a message of “200 OK.” Similar methods are used where valid and invalid responses are captured and analyzed. In some cases, these are reported as vulnerabilities or simply information exposures.

Fingerprinting with TCP/IP

A simple method of fingerprinting is to use the well-understood ICMP. ICMP packets are used to monitor the state of an interface on a host or report the status of access to a connected device. Nine message types are available: four for making queries and five for reporting errors. Each type is defined by a number, as shown in Table 1. PING is a very popular program that sends ICMP type 8 messages. Type 8 is an echo request whereas a type 0 is an echo reply. In addition to an ICMP type, there is a code that is used to report more information about an error. By manipulating these codes into invalid values, the target’s response or failure to respond can be captured. This in itself can tell us something about the OS. Some systems do not look at the code field on an echo request. Others do and respond with an error.

Table 1: ICMP Types 

ICMP CODE	TYPE
0	Echo reply
1–2	Unassigned
3	Destination unreachable
	Code	Meaning
	0	Net unreachable
	1	Host unreachable
	2	Protocol unreachable
	3	Port unreachable
	4	Fragmentation needed and don’t fragment was set
	5	Source route failed
	6	Destination network unknown
	7	Destination host unknown
	8	Source host isolated
	9	Communication with destination network is administratively prohibited
	10	Communication with destination host is administratively prohibited
	11	Destination network unreachable for type of service
	12	Destination host unreachable for type of service
	13	Communication administratively prohibited
	14	Host precedence violation
	15	Precedence cutoff in effect
4	Source quench
5	Redirect
6	Alternate host address
7	Unassigned
8	Echo
9	Router advertisement
10	Router selection
11	Time exceeded
	Code	Meaning
	0	Time to live exceeded in transit
	16	Fragment reassembly time exceeded
12	Parameter Problem
	Code	Meaning
	0	Pointer indicates the error
	1	Missing a required option
	2	Bad length
13	Timestamp
14	Timestamp reply
15	Information request
16	Information reply
17	Address mask request
18	Address mask reply
19–29	Reserved
30	Traceroute
31	Datagram conversion error
32	Mobile host redirect
33	IPv6 Where-Are-You
34	IPv6 I-Am-Here
35	Mobile registration request
36	Mobile registration reply
39	SKIP
40–254	N/A

Another method of reconnaissance is known as IP fingerprinting. The concept is an elegant form of manipulating inputs into the protocol stack of a target and measuring the results. For a brief review, let’s look at the TCP header structure in Table 2.

Table 2: TCP Segment

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 SOURCE PORT DESTINATION PORT SEQUENCE NUMBER ACKNOWLEDGEMENT NUMBER Header Length Reserved URG ACK PSH RST SYN FIN Window Size Checksum Urgent Pointer Options (up to 40 bytes) End of Option Data

The most useful operational benefit of TCP is the fact that it guarantees delivery by acknowledging the receipt of each packet. That set of flags—SYN, ACK, and RST—are what tell the recipient the purpose of what is transmitted. Our vulnerability scanner is sending SYN packets to the target. But it is the behavior of the rest of the contents of the packet that can reveal something about the target. Sequence number is a good example. So that TCP listeners on hosts do not become confused, every packet includes a sequence number. Since the creation of the protocol, it was found that it is easily possible to “wrap” the sequence numbers because they are of limited size (32 bits). To address the potential for wrapping and having a duplicate sequence number with an old packet being mistaken for a sequence number of a new packet, a time-stamp option was introduced in RFC 1323. This is an optional field and not all operating systems’ TCP/IP implementations set the value. When the scanner sees such value sent when the time-stamp option was never used, the choice in operating systems is narrowed considerably.

Another phenomenon to measure is the incrementing of the time stamp. By first determining the RTT between the scanner and the target, you then know how much time should elapse between TCP segments. The remote OS will increment the time stamp on each segment by a certain value. The way in which the target increments the value can reveal the type of OS.

For example, we know that OS XYZ increments the time stamp by one for every 500 ms of uptime. The average RTT between the target and the scanner is 100 ms, which is 50 ms in each direction, as shown in Figure 1. We receive the first segment with a time stamp (TS1) of 100. We acknowledge this segment and start a timer. The second segment with a time stamp of 102 (TS2) arrives and we stop the clock. The elapsed time between segment 1 and segment 2 is 1100 ms. We know that the time in transit for the segments is 100 ms. So the clock value, 1100, minus the RTT, 100, gives us 1000 ms of elapsed time on the host between segments. The difference between TS2 and TS1is 2. This means that, in 1000 ms, the time-stamp value went up by two, which is 500 ms per time-stamp increment. Looking at a table of time-stamp values over time, we know that the target has incremented the time stamp by one for every 500 ms, which is OS XYZ. This technique combined with other fingerprinting methods will ultimately narrow the choice of OSs. This choice is important in determining future steps of vulnerability scanning.

 
Figure 1: The average round-trip time (RTT) between the target and the scanner is 100 ms, which is 50 ms in each direction.

Invalid flag combinations are another approach. The normal combinations, SYN, SYN-ACK, and ACK, are expected. But various host OSs react strangely to combinations such as FIN+URG+PSH, which is a combination not seen in a normal handshake. It is referred to as an Xmas or Christmas scan because it lights up the TCP flags like a Christmas tree. Another combination that can possibly fingerprint an OS is SYN+FIN. In addition to host discovery, these types of scans can determine whether a port is open on a host without establishing a TCP connection or half-open connection. That is because IP stacks that adhere to the RFC will respond with an RST packet if the port is open. If closed, there will be no response from the host.

Use of these flags can get more sophisticated as well. If it has already been established that a port is open using a harmless TCP-SYN scan, the same port can be probed with a FIN-ACK combination. It turns out that systems implementing the IP stack from Berkley Standard Distribution (BSD) will not respond according to the RFC with an RST packet. This provides more evidence as to the likely system type of the target.

By combining these and many other types of probes, a decent guess can be made as to the type of system. The work for this has been well-established by the creators of NMAP (www.nmap.org). They continue to discover new ways to scan and map targets on a network and build those techniques into their open-source tool. A little reading and experimentation with this can be very educational.

However, the topics of OS fingerprinting and IP stack fingerprinting can be tricky, unreliable, and confusing. Some OSs may share the same IP stack code but be different OS versions. For example, a variety of Linux distributions will use the same stack but this does not necessarily reveal the flavor of the OS. Virtual machine technology can further cloud the issue because the underlying hypervisor OS may respond to network traffic and proxy the connection to the actual host OS. The fingerprinting result can be quite unexpected. Firewall and virtual machines can perform network address translation (NAT) that will conceal the true nature of the target OS.

Performance Issues | Active Scanning Technology Detection Methods

We have discussed at some length the process of identifying ports and handling TCP connections. All of these factors have to be taken into consideration during the scan; however, the scanner cannot wait too long. At some point, the transaction attempt will “time out.” This phenomenon can be referred to as discovery tolerance. Various vendors have different levels of discovery tolerance. The amount of tolerance is loosely proportional to the accuracy of the discovery with rapidly diminishing probability of successful identification. Fortunately, we know from experience that there is no point waiting for a reply beyond a certain amount of time. Determining that point is the real skill in any fingerprinting activity. The goal is to be complete and accurate, but there is a law of diminishing returns. Two key timers affect the speed of the discovery process: the connection establishment timer and the retransmission timer.

For many TCP implementations, the connection establishment timer (TCP_KEEPINIT parameter) waits 75 seconds for response. A simple scan on a single port for 200 hosts would require over four hours to complete if none of the hosts responded. This must be adjusted to wait far less time. One effective approach is to take the maximum roundtrip time (RTT) of ICMP echo reply exchanges and add two seconds. This provides ample time for an application to respond on the required port and is likely to be far shorter than the default of 75 seconds.

With TCP connections, a discovery process can also vary the retransmission timer when additional packets are to be exchanged with the target. In normal communications, the timer begins with a value of 1.5 seconds. If no response is received, then the value is doubled to three seconds. If there is still no SYN-ACK, the timer is doubled again and we wait six seconds. This continues repeatedly until we reach a limit of 64 seconds. The process is called exponential backoff (EB). In theory, this should parallel the exponential probability that a response will ultimately be received. However, this is often impractical for host discovery purposes in vulnerability scanning. A typical OS can spend several minutes waiting for a connection to time out.

A more practical approach would be to sequentially increase the retransmission timer by smaller values for a total period of time to be some factor above the average for the target IP range. For example, let’s suppose that we are performing a discovery of network A (192.168.10.0/24) with an upper limit of 30 seconds for retransmission. If the first 16 hosts required an average of 10 seconds to respond and the mode was five seconds, we might start our retransmission timer at five seconds and increase the value by five seconds until an upper limit of 20 seconds was reached (2× average). This is a more sensible approach that will avoid a common IP stack value that can reach several minutes for a single connection. Remember that our goal is discovery of open ports and live hosts, not the reliable transmission of data to another host.

There is one other item that can be manipulated, which is not exactly a timer and can speed the discovery process considerably:

to implement all the timers TCP only requires that two functions are called periodically: (1) the fast timer is called every 200 ms and (2) the slow timer every 500 ms. TCP uses these two periodic ‘ticks’ to schedule and check all the timers described…as well as measuring round trip times.^[*]

Basically, the OS kernel must check every 200 ms to see if an acknowledgment has been received. In modern networks and operating systems, this is a very long time.

By decreasing this period, the discovery processes can recognize that the probing SYN packets it has sent have been acknowledged in a shorter time, and move on to the next probe. If the RTT from SYN to SYN-ACK is 10 ms, then under normal circumstances, the discovery process can wait for up to 190 ms to proceed with the next action. Multiply this number by hundreds of hosts and dozens of ports, and the wasted time can be tremendous.

The one caveat to modifying TCP timers is that some applications are simply slow to respond. This approach works best when probing for open ports but not necessarily for applications. There is a lot of room for creativity in scan performance optimization. This section simply illustrates some of the challenges designers can be confronted with when trying to optimize the scan process and minimize the impact on the network.

1 Black Box Testing

Once the presence of a host has been established and that presence recorded in the memory of the scanner, a series of tests or “checks” are performed to find vulnerabilities. The types of checks are dependent upon the type of host and the configuration of the scanner. Generally, two types of checks are performed. A network-based or surface check is performed, which involves the probing and analysis of what is evident with limited or no access to services on the machine other than what is offered to any other peer on the same network as that which exists between the scanner and the target. This is also known as an unauthenticated check. The other type of check is an authenticated, internal check or white box test. It is performed when the scanner is given special information and credentials to access details of the host, which are generally reserved for trusted entities.

The difference between surface and internal checks is obviously significant not only in the way they obtain information, but also in the value and quality of that information. Clearly, more detailed data can be obtained by logging into a host and perusing its configuration. Although the information tells us a lot about the host, it does not typically represent the view of an attacker who performs reconnaissance on an unknown host. Although valuable from an analysis standpoint, some attacks take place by probing the host from the view of an outsider; therefore, information that can be obtained in the same fashion is often more valuable. To summarize, a vulnerability discovered and exploitable from outside a host represents a greater exposure than if the same vulnerability could only be discovered and exploitable from a credentialed or internal check.

There is a common perception that authenticated checks are more accurate than remote checks but that’s often not true. The Windows registry is commonly used for authenticated checks but is often wrong. It’s important to consider that not all authenticated checks are created equal and that a remote check is a good method of validating authenticated information.

The black box testing process involves some straightforward testing over the network and possibly some creative use of IP and other protocols. Usually, the simple tests are harmless and efficient. The more exotic manipulation of IP protocols can cause problems on scanned hosts with applications that are ill-prepared to handle many variations. This is a vulnerability in itself. The IP stack of the host is usually capable of handling nearly any variety of traffic, but the overlying applications sometimes are not. It is another area that calls for extensive testing in order to avoid adverse effects on production systems. Most vendors are able to provide a list of known negative application interactions.

Following is a list of some common methods of reconnaissance:

• Malformed packets are sent to the host to identify the presence of a vulnerability in the response. This is similar to the discovery process and is sometimes incorporated into the same phase for efficiency. The information sent to the target can be at any one layer or multiple combinations of layers 3 through 7 in the OSI Model.
• Normal packets are sent to a known application to obtain results that will reveal vulnerability information. This is very common in the http protocol to obtain information about the Web server, application server, or back-end databases.
• Valid information is sent to the target to gather valid header response data that will reveal the version of software answering the service request. This is known as banner checking. Many software applications can obfuscate this information with simple configuration changes, so it is not the most reliable method.

These methods can be summarized conceptually in pseudo code form:

Send X to target
Listen for response Y
Match Y to possible response list
If Y is on list, note vulnerability
If Y is not on list, ignore
Get next check; Loop

Manipulating TCP | Active Scanning Technology Detection Methods

Some scanners do not consider ICMP to be reliable since some hosts are configured not to respond to ICMP. So, a TCP SYN packet is sometimes sent using several common ports found on a variety of network devices. Table 1 lists some common ports that are scanned to determine whether a host is present.

Table 1: Commonly Scanned Ports 

PORT	PROTOCOL
20	FTP
21	FTP
22	SSH
23	Telnet
80	HTTP
443	HTTPS
137	NETBIOS Name Service
138	NETBIOS Datagram Service
139	NETBIOS Session Service

Other ports may also be scanned, depending on the vendor and how one might configure this phase. Once the SYN packet is sent to the host, a reply of SYN-ACK is expected. Should this reply not arrive in time, the scanner will consider the port not responsive and the host not present, that is, unless the host responds on another port. The discovery of these ports is not necessarily a serial process. The scanner may “spray” the TCP SYN packets on many ports and at many hosts simultaneously. This saves time and makes more efficient use of bandwidth. Some performance issues will be discussed later.

One potential side effect of the TCP discovery method is the potential for leaving open or half-open sockets. This can have an adverse effect on a system, depending on the application listening and the integrity of the OS protocol stack. Half-open sockets occur when the scanner does not complete the connection setup with an ACK packet. The effects range from memory consumption to denial of service (DoS) for production systems. Normally, this is not a problem since only one active connection is attempted per TCP port. However, a misconfigured scan can change this. A similar phenomenon exists when scanning through a firewall. Many firewalls will function as a proxy for the connection to the host. This imposes a load on the firewall when hundreds or thousands of hosts are scanned at once. Some routers can be impacted as well when they are undersized for their operating environment.

If the connection handshake is completed with an ACK packet, an entry in a connection table is maintained on the host and/or on a firewall. The result is a further consumption of resources until the connection times out or is reset. For that reason, it is important to test the behavior of a scanner and determine whether or not a TCP reset is sent to the host and how that behavior might affect your network on a large scale. If it is possible that a large amount of scanning will be performed through a firewall, then test this scenario. The setup and breakdown of connections on a firewall are two of the more processor-intensive tasks and can degrade performance. This is not absolute, however. Proper scheduling, bandwidth shaping, and connection limits in the scan configuration can help prevent these problems. Figure 1 illustrates a scenario where a firewall may be significantly affected by multiple simultaneous scans. As you can see, the total number of connections per second can add up quickly. A scan is far more intense than regular network traffic because it is concentrated into a short time period.

 
Figure 1: A scenario where a firewall may be significantly affected by multiple simultaneous scans.

Conversely, a firewall, with all the additional security features that vendors have added, can interfere with a discovery scan. For example, some firewall vendors have added intrusion prevention capabilities as well as a SYN proxy. These features can provide false information to a vulnerability scanner, making it seem as though a host exists where there is none. This is because many scanners consider a closed port “response” to indicate the presence of a host. Otherwise, why would there be a response?

These security features can also do the reverse and obfuscate the host and its open ports. Hopefully, the firewall vendor will have the built-in ability to make an exception to the traffic by source IP address. With the correct IP address of the scanner configured in an exception list, the scan should proceed without error.

However, not all firewalls are created equal. For this reason, we will spend a little time discussing packet processing in a firewall as it is related to VM. Figure 2 shows the basic structure of how a firewall might handle traffic. Notice the stacked architecture. The reason for this is to qualify traffic for the most fundamental flaws prior to investing more processing cycles. For example, if the TCP flags are an invalid combination, the traffic should be dropped. Much of this processing can be performed in silicon and avoid burdening the firewall CPU further. On the other hand, if traffic is to be processed by some rules, more CPU cycles are required.

 
Figure 2: The basic structure of how a firewall might handle traffic.

The next step in packet processing is to save and monitor the connection state. If a SYN packet is received from a vulnerability scanner, it is compared to entries in a connection table. If the connection already exists, then this is likely a duplicate and the packet is dropped. If the connection does not exist, then an entry is made in the table. When a SYN-ACK packet is received, the same comparison is made to keep track of the state of that connection. The same is true with all related packets. If an RST packet is received, then the connection is deleted from the table. This process can take a lot of CPU if there are many thousands of connections per second and other firewall activities taking place. The constraints of this connection table data structure and the related programming code to handle the packets are the parts most crucially affected by discovery scans. If the state table is too small a data structure, then the SYN/SYN-ACK discovery process will rapidly fill this table. If that constraint shows up in testing, you will have to make sure that you limit the number of open connections during discovery.

On the other hand, if the processing code that makes comparisons to this table is inefficient, there will be a significant impact on firewall throughput. In this case, it is vital to maintain a limit on the rate of connections.

Most firewalls process rules in order. A packet is taken from a queue for comparison against the rules. Once a rule matches the traffic, the inspection process stops. The packet is then forwarded to the network interface and the next packet is extracted from the queue.

Another way to avoid a large impact on firewalls is to relocate the scanners around them. This is largely dependent on your network design. Careful placement of scanners is a primary consideration and can be crucial to scanning effectiveness.