The Standard for Vulnerability Test Data

Typically, the vulnerability information used by a particular vendor of a VM scanner or analyzer is stored in a database that is designed to work seamlessly with the software. Since standards are usually a few to several years behind industry, none of these databases are alike. The vulnerability data and method of identification vary widely from one product to another—that is, there is no standard.

However, MITRE has driven the idea of a standard for just this type of data structure. It is called the Open Vulnerability and Assessment Language or OVAL^®. As the name implies, it is a language that is structured using XML to record the details of the state of a machine (target) that has a particular vulnerability. It employs the state machine approach to identifying a vulnerability, which is a highly structured way of indicating the vulnerable state and the non-vulnerable state. So, let’s examine the OVAL process in Figure 1 to understand its benefits.

Figure 1: The OVAL process.

We start with the security researcher who discovers a new vulnerability in product X. Having firsthand knowledge of the vulnerability and how it can be identified in a vulnerable target, the researcher encodes the details of that vulnerability in an OVAL-standard-formatted XML document. This document describes exactly how to identify the vulnerability on a target by providing the information of the state of particular items that constitute a vulnerable machine. This method of vulnerability identification is called a test.

Definitions Schema

At a higher level, OVAL consists of three types of documents called schemas. The definitions schema is used to encode the state of various parts of a computer system that must exist in order to be subject to a particular vulnerability. For example, a target system may need to have a Windows 2003 server with Service Pack (SP) 2, IIS, and anonymous Web access configured to be vulnerable to vulnerability XYZ. These schemas provide the framework for encoding such conditions.

Other definitions exist to capture other important information to help round out the VM process. Patch definitions record the specific conditions that must be present in order to be eligible for a patch. The idea is to avoid haphazardly applying patches to a system inappropriately. So, the ability is provided to specify, for example, that a vulnerability that is fixed by Windows 2003 SP 2 is only applied when the target system is in fact Windows 2003 and has only SP1 installed. The patch definition capability can go far beyond this, but this understanding is sufficient for the scope of this book.

An OVAL inventory definition is just what it sounds like: a description of what defines a particular inventory item. If you want to specify that the SNMPv3 service is installed on a router, certain items must be checked to be certain it is present. The inventory definitions provide the XML schema to achieve exactly this. Since vulnerability scanning and automated configuration discovery are so closely related, it makes sense that OVAL should include this capability.

One additional area where the VM industry has very sensibly gone is compliance management. The MITRE team has understood this, and OVAL includes a schema for compliance definitions. Similar to the other schema, compliance specifies the conditions under which a system is compliant with a particular policy. Again, this is a natural extension to VM tools and processes.

System Characteristics Schema

This schema is designed to provide a standard definition of target system characteristics. These system characteristics, once collected, can be analyzed to identify vulnerabilities. When all of the target characteristics are collected, the vulnerability analyzer would compare them to the details in the definitions data to discover vulnerabilities. The design of the system characteristics schema is roughly parallel to that of the definitions schema. Although this schema is vulnerability focused, it also amounts to a list of configuration items in a configuration management database.

Results Schema

This third and final high-level schema is designed to provide a standard structure for recording the results of the vulnerability assessment. The primary benefit is that once a vulnerability is discovered, the details are captured in a format that many other security tools can interpret to properly apply patches, update configurations, initiate change processes, and take other mitigation actions as necessary. The results schema very specifically captures the details of a particular vulnerability on a specific target.

Nuts and Bolts^[*]

In the OVAL specification, tests are formatted in the OVAL definitions schema and are defined by a tag. Tests are recorded using three key XML elements: , which is the item being tested; are the values of the objects to be tested; and are the tests to be performed against those objects using the previously mentioned . An ID is assigned to each definition, object, state, and test using a notation resembling reverse DNS entries. Let’s take a look at a partial example of an OVAL definition, which is a check to see whether Windows XP^® is installed. This is an abbreviated version, so we do not spend time on details that are not instructive on this topic:

------------------------------------------------
1
2 version="3" class="inventory">
3
4

Common Vulnerabilities and Exposures | CVE

Once vulnerability information has been collected, it must be categorized and evaluated. The methods of evaluation and categorization vary by vendor. This is one key area where many products attempt to distinguish themselves. When a vulnerability is identified, the category is typically assigned according to the type of exploit required or the level of access that is granted. For the purposes of this discussion, we will avoid any vendor-specific approaches and use MITRE’s CVE methodology. According to MITRE’s Web site:

CVE is a list of information security vulnerabilities and exposures that aims to provide common names for publicly known problems. The goal of CVE is to make it easier to share data across separate vulnerability capabilities (tools, repositories, and services) with this common enumeration.

Structure

MITRE is a non-profit organization that has been making a valuable contributions to VM for years. They have been able to provide an open, standardized platform for the sharing of vulnerability knowledge. When someone discovers a new vulnerability, they frequently (but not always) report this discovery and its details to MITRE, who quickly publishes the information. Unfortunately, standards are still difficult to get adopted in products. 

Every CVE is given an identifier. In effect, this identifier allows a variety of tools from different vendors to speak the same language. A CVE provides the same description for all vendors and the same references to additional information sources. For example, “CVE-2001-0010: Buffer overflow in transaction signature (TSIG) handling code in BIND 8 allows remote attackers to gain root privileges.” This is the same understanding for everyone. It cannot be confused among various vendors.

The references in the CVE will ultimately lead a vulnerability manager to the National Vulnerability Database (NVD) run by the NIST. CVE-2001-0010, mentioned earlier, has related information in the NVD, as shown in Figure 1:

• Overview: This is a summary of the vulnerability that resembles the CVE description.
• Impact: The impact section attributes a score to the vulnerability should it be exploited. More on this later when we discuss the Common Vulnerability Scoring System (CVSS).
• References to advisories, solutions, and tools: These are typically Internet references to obtain more-detailed information about the vulnerability, how to detect it, and how to remediate. In this example, information about patches from various vendors is supplied.
• Vulnerable software and versions: A list of the version numbers that are known to possess this vulnerability. This further helps with the detection process.
• Technical details: This is information about the exact nature of the vulnerability; for example, how the software will react when exploited and why this is bad. Again, this item usually contains links to the site where the researcher has published information about his discovery.

Vulnerability Summary CVE-2001-0010 Original release date: 2/12/2001 Last revised: 5/2/2005 Source: US-CERT/NIST Overview Buffer overflow in transaction signature (TSIG) handling code in BIND 8 allows remote attackers to gain root privileges. Impact CVSS Severity (version 2.0 incomplete approximation): CVSS v2 Base score: 10.0 (High) (AV:N/AC:L/Au:N/C:C/I:C/A:C) (legend) Impact Subscore: 10.0 Exploitability Subscore: 10.0 Access Vector: Network exploitable Access Complexity: Low Authentication: Not required to exploit Impact Type: Provides administrator access, Allows complete confidentiality, integrity, and availability violation , Allows unauthorized disclosure of information , Allows disruption of service References to Advisories, Solutions, and Tools CERT/CC Advisory: CA-2001-02 Name: CA-2001-02 Type: Advisory , Patch Information Hyperlink: http://www.cert.org/advisories/CA-2001-02.html External Source: Security Focus (disclaimer) Name: bid 2302 Type: Advisory , Patch Information Hyperlink: http://www.securityfocus.com/bid/2302 External Source: PGP Security (disclaimer) Name: Vulnerabilities in BIND 4 and 8 Type: Advisory , Patch Information Hyperlink: http://www.pgp.com/research/covert/advisories/047.asp External Source: REDHAT (disclaimer) Name: RHSA-2001:007 Hyperlink: http://www.redhat.com/support/errata/RHSA-2001-007.html External Source: NAI (disclaimer) Name: 20010129 Vulnerabilities in BIND 4 and 8 Hyperlink: http://www.nai.com/research/covert/advisories/047.asp External Source: DEBIAN (disclaimer) Name: DSA-026 Hyperlink: http://www.debian.org/security/2001/dsa-026 Vulnerable software and versions Configuration 1 IS, BIND, 8.2.2 P7 IS, BIND, 8.2.2 P6 IS, BIND, 8.2.2 P5 IS, BIND, 8.2.2 P4 IS, BIND, 8.2.2 P3 IS, BIND, 8.2.2 P2 IS, BIND, 8.2.2 P1 IS, BIND, 8.2.2 IS, BIND, 8.2.1 IS, BIND, 8.2 Technical Details Vulnerability Type No vulnerability type mapping is available. CVE Standard Vulnerability Entry: http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2001-0010 Common Platform Enumeration: http://nvd.nist.gov/cpe.cfm?cvename=CVE-2001-0010

Figure 1: CVE-2001-0010.

Notice that CVEs are identifiers and not actual technical details. The main purpose of a CVE is to provide a cross-platform standard for identification of vulnerabilities. To support the quality of this identification mechanism, each vulnerability is subjected to a review process. At first, candidate status is given. This status means that the information is out there but has not been granted CVE status. A CVE editorial board discusses the merits of the candidate and votes on whether or not the vulnerability should receive full CVE entry status.

There are some caveats to the CVE database. First, it is not a vulnerability database. It is a database of vulnerability references. Second, it does not include all known vulnerabilities. It only contains those that are publicly known. So, it is possible that a vulnerability exists of which a vendor or researcher is aware but it does not appear in the CVE list. In some cases, this is because the researcher has agreed with the maker of the software that he will not reveal the vulnerability until a public patch has been released. Naturally, the researcher will want credit for the discovery.

To continue our CVE discussion, CVEs have one of two statuses: candidate or entry. An editorial board must vet the proposed vulnerability prior to it being granted entry status. Until that time, the vulnerability has candidate status. This status is provided on the CVE list when you view the details. When reading a CVE, check this status and review the reference to form your own opinion about the credibility and accuracy of information provided.

Limitations of CVE

CVE has definite limitations and is by no means an answer to all standards issues related to VM. As previously mentioned, CVE does not have a comprehensive list of all vulnerabilities in existence. Some vendors are able to identify vulnerabilities that CVE does not seem to record. Also, it does not necessarily contain all of the metadata needed to make a vulnerability system perform all of the functions that a technology vendor wishes to perform. Naturally, it shouldn’t since it is intended to provide the common-denominator information useful to everyone.

CVE is not always kept up to date. Many vulnerabilities remain in “CAN” or candidate status for years. One wonders if these vulnerabilities will ever be updated when they are known to be accurate. It is possible that some of these are configuration best practices but not necessarily to be considered vulnerabilities. Inversely, CVE does not contain all product best-practice configuration vulnerabilities since they are too numerous to review and include for the many thousands of products in use around the world.

Inference Scanning | Vulnerability Management

One final method of scanning that is seldom used exclusively for vulnerability identification is inference scanning. This method involves the analysis of data that has already been obtained for another purpose to detect the presence of a vulnerability. For example, a configuration management system may have collected detailed configuration data on targets throughout an organization. The inference scanning process would use non-intrusive methods that involve reading the configuration details from the asset database and analyzing them for vulnerabilities. Easy examples of this are discreet configuration items such as SNMP community string or vulnerability application versions.

Since inference scanning is based on factual information provided during the normal course of gathering configuration data, the reliability of an identified vulnerability is very high. Also, because the vulnerability detection process is not performed by actively probing the host on the network, there is no impact to the target. When used strictly by itself, inference scanning is not always reliable or complete since it would not involve verification by other means. It can, however, be used to augment the previously mentioned scanning processes or as an additional feature to a configuration management product. Furthermore, inference techniques can be used architecturally to make vulnerability scanning more efficient. For example, an active vulnerability scanner might collect all of the possible vulnerability information and record it for analysis; then, the inference engine would be used to analyze that data for vulnerabilities in the host. In a later phase, certain vulnerabilities would be flagged for verification by other means before being given the designation of vulnerable. Overall, inference scanning is a valuable tool but is not sufficient to deliver the most complete, reliable results on its own.

Hybrid Approach | Vulnerability Management

Combining more than one solution for VM from different vendors can be helpful in responding more quickly and thoroughly to emerging vulnerabilities. However, normalizing the output may be difficult. If you are fortunate enough to deploy more than one type of technology from the same vendor, then perhaps a unified console will eliminate this problem.

Alternative approaches are to allocate the assessment resources by organization or network. For example, it may be beneficial to use passive vulnerability scanners on a public DMZ in order to get 24-hour coverage of the security posture of the hosts. This most current assessment information can be automatically fed to a security event/incident management system (SEIM). This provides a significant advantage, for newly published vulnerabilities can be taken into account quickly when new events occur to exploit them. Active vulnerability scanners can obtain more in-depth analysis of the back-end systems and workstations where rapid response may not be as critical.

The combination of agents in DMZs and active scanners in the internal network is an excellent choice. The agents are positioned on DMZ hosts so that it is unnecessary to actively scan through the network security systems, which would otherwise require a more complex configuration. Additionally, regular audits or penetration tests of the DMZ should be conducted and agents serve as a substitute for the regular monitoring provided by active scanning.

Performance Matters | Active Scanning Technology

During a scan, the goal of the scanner is to get as complete and accurate a scan as possible. However, the performance and behavior of that scan is also important to the vulnerability manager. Ideally, we would like to scan as much as possible during an allotted time window and get complete results. However, we also want to avoid affecting production operations. First, let’s look at the potential negative impacts to production and how we might avoid them. Then, we can look at ways to optimize scans.

In most cases, there are four ways in which a scan can adversely affect a production environment:

• By consuming bandwidth, preventing other applications from meeting service levels.
• By consuming target CPU resources in an already-busy target. Again, this can cause service levels to be missed.
• By breaking a target application or OS, causing a DoS and requiring the target to be repaired.
• By breaking a component that is facilitating the scan but is not a target. Various network components could be adversely affected by the scan process even though they are not the subject of a scan.

Bandwidth is consumed by network activity. During the scan process, parameters provided by the vulnerability manager are used to size the footprint on the network. Not only is bandwidth a factor, the number of simultaneous connections can affect intermediate devices as well. Since TCP is so commonly used, a connection is established with each target. In some cases, a connection is only attempted, leaving potentially half-open connections. Devices that track the state of connections such as firewalls, IPSs, and possibly routers, can be affected by these connections. The total number of simultaneous connections, and the rate at which these connections are made, may have an effect. Limits on both will go a long way towards maintaining good relations with network management staff.

The most efficient location in the scanner to impose these limits is the IP protocol stack and interface drivers. Bandwidth limitations are best performed at the interface driver, whereas the connections limits are better applied at the packet-creation phase where the outgoing connections table is maintained. Exceptions must be made to accommodate the critical command and control functions of the scanner. Therefore, the location of the scanner management system should be exempt from such limits.

Bandwidth consumption has the biggest impact on the network when scanning is performed outside of the local segment to which the scanner is connected. In particular, WAN links can be impacted significantly. Using today’s most cost-effective technology, most scanners are not able to produce more than 10 Mbps of bandwidth in a typical scan. However, when a T1 is the only connection from a remote office back to the corporate WAN where the scanner resides, it is easy to saturate the link event with a small scan. In most companies, it is often necessary to perform such a scan during work hours when desktop and laptop computers are powered on and connected to the network. So, the impact to business operations is significant.

To determine the appropriate amount of bandwidth to allow for a given target network, I recommend the following strategy:

1. Determine the peak utilization of the WAN link over a two-month period to coincide with the time in the business cycle during which you plan on running the scan. Alternatively, plan to run the scan during a time in the business cycle when bandwidth consumption is relatively low but the targets are still available for scanning.
2. Agree with the local business manager and network operations manager on how much of the remaining bandwidth you will be permitted to use. Set expectations when making this agreement.
3. Perform a test scan of the target network for five or more typical target hosts to gauge how much bandwidth is required. This number will have to be scaled up to derive a more accurate number. One caveat: the accuracy of this estimate will increase when the test scan can more closely reach a maximum number of connections. Sometimes, limiting the number of connections will reduce the maximum bandwidth consumed, and in other cases, it will not. It will all depend on the configuration of each target. Given the highly dynamic nature of a scan, the amount of testing should be commensurate with complexity and variability of the environment and level of criticality of the WAN link to business operations. This is both art and science.
4. Always position the scanning activity as a critical security function that will ultimately provide reports and analysis to the IT and network managers at the site. They should want this as much as you do. In a site with critical hosts, it may be worth purchasing a small amount of additional bandwidth that is dedicated to performing the scans.

Simultaneous connections, on the other hand, are fairly straightforward to manage. As previously mentioned, this parameter may affect bandwidth. The primary goal is not to overwhelm the target or the intervening network and security infrastructure. Even at low bandwidth, small packet sizes and half-open connections can generate a sizable number of simultaneous connections. Since a firewall maintains a state table for each connection, it must perform a little more CPU work for each connection. Large commercial firewalls that typically front a public Web site generally have sufficient resources to handle huge numbers of connections. However, this is not always the case. Firewalls are complex devices running many simultaneous processes. Furthermore, they are mostly single-threaded applications, which limits their performance scalability. Just as DoS attacks are performed on routers by exploiting the reliance of certain features on notoriously limited CPU resources, a scanner can do the same to a firewall. Most of these scenarios can be addressed with the following guidelines:

• Test the impact on a firewall under load for a variety of targets. Different targets can generate different connection rates. Monitor the firewall CPU reaction closely. The reaction is not linear. In some cases, packet forwarding latency may result. In other cases, it just takes longer to set up and tear down connections. Manufacturer performance specifications do not apply to traffic generated by vulnerability scanners.
• Test scanning against targets where the firewall applies additional distributed DoS and intrusion prevention capabilities. Since firewalls typically interact with the environment at OSI layer 3 and above to provide these services, the firewall may interpret a scan as a threat. It could be blocked or the results of the scan clouded. This is because the firewall features can act like a proxy for a given application. TCP connection attempts will be terminated by the firewall and then re-established from the firewall to the target only after the handshake is completed with a sanitized upper-OSI-layer result. Any probing performed with a SYN-SYN/ACK-ACK that is not fully completed by the scanner may appear to be an active host where none exists. A solution to this is to configure the firewall to allow the scanner source traffic to bypass all of the firewall inspection filters.
• Routers can also be affected. Some routers use their limited processor power to handle invalid TCP flag combinations. This is a common probing or discovery technique used by vulnerability scanners to fingerprint the OS or application. It is also a method for performing a DoS attack against a router. Although it is unlikely that a scanner will generate enough malformed traffic to have a major impact on network devices, you should be mindful of the possibility.
• Simulate the typical latency on a WAN circuit in the lab during a scan. This will help gauge scan performance as well as impact to production systems. Some vendors provide tools that can use a packet capture from a network segment and a performance profile, and then re-create the user experience with the scan traffic and any given application injected. For example, an accounting system could be placed on one side of such a device while the scanner is on the other side. The device simulates the traffic conditions at a peak time for a particular office using a previously gathered profile.
• Limit the number of TCP and UDP ports that are scanned during discovery and scanning. The tendency is to be as comprehensive as possible because you are not sure what will be found. This may work for an initial scan of a limited number of hosts. But later, you should settle on an acceptable number of ports to minimize the impact on the environment and maximize effectiveness. This is an area with diminishing returns on port numbers scanned at a logarithmic rate. You may also find that reducing the number of ports scanned will substantially lighten the bandwidth load. This is because packets can be large and numerous.

Web Application Testing

With so many competitive pressures, it was inevitable that organizations would have to find a way to distinguish themselves in the online world. So, millions of custom applications have been built to deliver customer service and application services that add value beyond original core competencies. It naturally follows that hackers would find common ways to exploit these applications, especially because they are more closely linked to valuable data. What has made these applications even more exploitable is their dependence on standard technologies and infrastructure. This is not intended as a criticism, only an observation. For example, most databases in use today are based on the SQL language. Also, many Web applications use JavaScript™. Both of these technologies are exploitable, depending on their method of implementation. No inherent security controls can be configured to prevent their exploitation at the application level.

An even greater concern is the exploitability of the open-source PHP language. It is commonly used to build Web applications and yet has many critical vulnerabilities around which the programmer must code. Since this powerful scripting language is closely integrated with the Web page and user interaction, and since it has so many powerful commands with great flexibility, exploitation is very possible.

Naturally, this Web application phenomenon presents another area of vulnerability testing. These checks have become increasingly important, as customer applications have become the primary focus of the serious hacker. There are simple methods to exploit vulnerabilities by simply manipulating the content of the URL in the Web browser. It is also quite easy to manipulate inputs on fields on the screen. So, vulnerability checks must do the same thing in many combinations using many common techniques to replicate possible attack vectors. Following is a list of some of the most common vectors:

• Input field manipulation: This involves modifying the input of one or more fields on the screen beyond what is expected by the software in normal operation. Many programmers fail to validate these inputs for size and value boundaries as well as validity. Hackers exploit this by entering characters that will cause the application to process them in a way not originally intended.
• SQL injection: This very popular attack vector is used to manipulate the database query language of the back-end database programming to reveal information or even modify the database contents. The process is started by entering the partial SQL string (‘ or 1=1—) in an input field without the parentheses will extend an underlying SQL statement to detect the presence of an unfiltered field and the fact that the SQL language is in use and accessible. This works because the first tick mark (‘) ends the current input string expected by the SQL code and then adds the logic “or 1=1.” The remaining portion tells the SQL server to ignore the remainder of the SQL statement. This is a harmless modification of the SQL query that will determine whether SQL injection is even possible. Vulnerability scanners will perform more in-depth penetration acts to reveal more details about the flaws in handling this input, including what can be discovered about the database configuration.
• Cross-site scripting (XSS): This extension of input field manipulation is used to inject JavaScript into a Web site that will appear on other users’ browsers and perform actions against other users of the system. These actions include but are not limited to directing user input to another site, capturing user data, and presenting false information. The script information can be combined with a SQL injection attack to store malicious script code in the database of the target system. When this information is retrieved by the Web application for a user, then the script is loaded into that user’s Web browser and executed to fulfill the purposes of the attacker.

So Web application checks are available in many VM systems to detect the presence of these vulnerabilities in the code of a Web site. There are other products that analyze the actual programming code, but they are no substitute for the hacker-mimicking process of attacking the application from the outside. By their very nature, these checks are brute force but usually non-destructive. The application checks are run against every field, every hyperlink, and every possible URL of a Web site. The following are more types of checks that can be performed against applications:

• Boundary check: This is a test of a range of values that are accepted by an application. It is applied to every field that is found and applies values that can be below, above, and within the range of permitted values, as well as trivial values that are outside of the expected data types. For example, an 8-bit ASCII numeric field may have 16-bit Unicode data entered. This is also called stress testing.
• Branch test: This program is used to check all of the links and possible paths to be taken in an application. The goal is to achieve as close as possible to 100 percent branch coverage, meaning that every possible program pathway is exercised. This is partly a discovery process and partly a vulnerability identification process. Some vulnerabilities in this area are the result of branches of code that are so obscure they are seldom accessed. Because of this, it is possible that some code is vulnerable because it was never tested.
• Brute force: This type of check is typically used against user ID and password fields. It differs from a boundary check in that there is a large practical range limit for what can be entered. A password field that is eight alphanumeric characters long, for example, has 8**36 possible combinations.
• Buffer overflow: This type of check is designed to challenge the target software for scale factors; that is, those inputs whose range affects the allocate memory space. Two types of scale factors can be tested: buffer definition and buffer usage. For example, if a program is designed to accept a certain size of input and store it in a memory buffer, a larger input can cause a buffer overflow. This is an example of a buffer usage scale factor, which occurs when an entered parameter directly or indirectly determines the allocated buffer space. If the parameter causes the creation of buffer smaller than the input size, then the buffer can overflow on the corresponding input.
• Code injection: In many scripting languages, it is possible to accept code as a parameter passed from one process to another. Programmers use this technique to efficiently reuse code or pass advanced parameters from one program to another. Some faults in a program can allow the inadvertent introduction of code by accepting instruction-terminating characters to be entered into a field. This is similar to SQL injection attacks.
• Session hijacking: HTTP is a stateless protocol. That means that it receives transactions over the network but does not know that the transaction is part of a particular user’s session. To keep track of this session or “state,” a file called a session cookie is implemented. There are various checks that can be performed to determine whether the session cookie can be manipulated to become a different user, and therefore gain access to that user’s data.

All of these tests, however, are typically used alone when run automatically. This is a key distinguishing factor between vulnerability testing and penetration testing. A person conducts basic reconnaissance testing using these application-vulnerability scanning tools, and then combines the results to form augmented attacks with more revealing results. For example, an SQL injection attack can be used to insert malicious code to obtain another user’s session cookie. Then, the session cookie, encrypted or not, can be used to obtain that user’s information or change his password to gain his privileges.

Application Fingerprinting: Banners | Active Scanning Technology

An important activity in a scan is to determine what applications make themselves known on the network. Similar to OS fingerprinting and IP stack fingerprinting, a vulnerability scanner can attempt to connect to a variety of possible applications on known or unknown ports, a process known as featureprinting. Among the best-known featureprinting methods is banner checking.

Some common applications on systems produce a banner whenever you attempt to connect. The contents of this banner can provide valuable information in determining the version of the OS and the software running on the host. This type of fingerprinting can often be performed by an individual using a simple program available on almost all OS platforms: Netcat.

Let’s take the example of a Web server. Figure 1 shows a typical Netcat session. When using Netcat, you can specify the TCP port to which you wish to connect. So, in the command line, we type “nc 10.1.1.10 80.” This will run the Telnet application and establish a connection to the server listening for connections on port 80.

$ nc 10.1.1.10 80 Apache Example HEAD / HTTP/1.0 HTTP/1.1 200 OK Date: Tue, 06 May 2008 23:32:00 GMT Server: Apache/2.2.8 (Unix) Last-Modified: Tue, 29 Apr 2008 21:52:29 GMT ETag: "ea9d61-48f6-44c0a0c71a140" Accept-Ranges: bytes Content-Length: 18678 Cache-Control: max-age=86400 Expires: Wed, 07 May 2008 23:32:00 GMT Vary: Accept-Encoding Connection: close Content-Type: text/html $_

(a)

$ nc 10.1.1.20 80 IIS Example HEAD / HTTP/1.0 HTTP/1.1 302 Found Cache-Control: private Content-Length: 142 Content-Type: text/html; charset=utf-8 Location: /en/us/default.aspx Server: Microsoft-IIS/7.0 X-AspNet-Version: 2.0.50727 P3P: CP="ALL IND DSP COR ADM CONo CUR CUSo IVAo IVDo PSA PSD TAI TELo OUR SAMo CNT COM INT NAV ONL PHY PRE PUR UNI" X-Powered-By: ASP.NET Date: Tue, 06 May 2008 23:58:41 GMT Connection: keep-alive $_

(b)

Figure 1: A typical Netcat session.

From here, we are able to issue a command to the remote server. Since this is port 80, it is likely a Web server; so we issue the command: “HEAD/HTTP/1.0” (Figure 1a). The server responds with some detailed header information, which tells the type of Web server software and OS. In this case, it is Apache^® running on a UNIX^® system. This eliminates the possibility of any version of Windows and makes searching for vulnerabilities much easier.

For security reasons, however, server administrators should conceal this header information, particularly that information which is in the “Server” section. But, that will not deter a good vulnerability scanner. The scanner may also be able to check for the type of Web server by making an invalid request. By using an invalid version type in the HEAD command, we can see different responses from the various Web server makers. Notice that the Apache Web server comes out with a “400 Bad Request” message (Figure 1b). The connection also gets closed; however, on IIS 7.0, the connection is not closed but the same “400 Bad Request” message is received. But, you will also notice that more server information is provided that was not found in the valid request. In earlier versions of IIS, you could distinguish it from a reaction of providing a message of “200 OK.” Similar methods are used where valid and invalid responses are captured and analyzed. In some cases, these are reported as vulnerabilities or simply information exposures.

Fingerprinting with TCP/IP

A simple method of fingerprinting is to use the well-understood ICMP. ICMP packets are used to monitor the state of an interface on a host or report the status of access to a connected device. Nine message types are available: four for making queries and five for reporting errors. Each type is defined by a number, as shown in Table 1. PING is a very popular program that sends ICMP type 8 messages. Type 8 is an echo request whereas a type 0 is an echo reply. In addition to an ICMP type, there is a code that is used to report more information about an error. By manipulating these codes into invalid values, the target’s response or failure to respond can be captured. This in itself can tell us something about the OS. Some systems do not look at the code field on an echo request. Others do and respond with an error.

Table 1: ICMP Types 

ICMP CODE	TYPE
0	Echo reply
1–2	Unassigned
3	Destination unreachable
	Code	Meaning
	0	Net unreachable
	1	Host unreachable
	2	Protocol unreachable
	3	Port unreachable
	4	Fragmentation needed and don’t fragment was set
	5	Source route failed
	6	Destination network unknown
	7	Destination host unknown
	8	Source host isolated
	9	Communication with destination network is administratively prohibited
	10	Communication with destination host is administratively prohibited
	11	Destination network unreachable for type of service
	12	Destination host unreachable for type of service
	13	Communication administratively prohibited
	14	Host precedence violation
	15	Precedence cutoff in effect
4	Source quench
5	Redirect
6	Alternate host address
7	Unassigned
8	Echo
9	Router advertisement
10	Router selection
11	Time exceeded
	Code	Meaning
	0	Time to live exceeded in transit
	16	Fragment reassembly time exceeded
12	Parameter Problem
	Code	Meaning
	0	Pointer indicates the error
	1	Missing a required option
	2	Bad length
13	Timestamp
14	Timestamp reply
15	Information request
16	Information reply
17	Address mask request
18	Address mask reply
19–29	Reserved
30	Traceroute
31	Datagram conversion error
32	Mobile host redirect
33	IPv6 Where-Are-You
34	IPv6 I-Am-Here
35	Mobile registration request
36	Mobile registration reply
39	SKIP
40–254	N/A

Another method of reconnaissance is known as IP fingerprinting. The concept is an elegant form of manipulating inputs into the protocol stack of a target and measuring the results. For a brief review, let’s look at the TCP header structure in Table 2.

Table 2: TCP Segment

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 SOURCE PORT DESTINATION PORT SEQUENCE NUMBER ACKNOWLEDGEMENT NUMBER Header Length Reserved URG ACK PSH RST SYN FIN Window Size Checksum Urgent Pointer Options (up to 40 bytes) End of Option Data

The most useful operational benefit of TCP is the fact that it guarantees delivery by acknowledging the receipt of each packet. That set of flags—SYN, ACK, and RST—are what tell the recipient the purpose of what is transmitted. Our vulnerability scanner is sending SYN packets to the target. But it is the behavior of the rest of the contents of the packet that can reveal something about the target. Sequence number is a good example. So that TCP listeners on hosts do not become confused, every packet includes a sequence number. Since the creation of the protocol, it was found that it is easily possible to “wrap” the sequence numbers because they are of limited size (32 bits). To address the potential for wrapping and having a duplicate sequence number with an old packet being mistaken for a sequence number of a new packet, a time-stamp option was introduced in RFC 1323. This is an optional field and not all operating systems’ TCP/IP implementations set the value. When the scanner sees such value sent when the time-stamp option was never used, the choice in operating systems is narrowed considerably.

Another phenomenon to measure is the incrementing of the time stamp. By first determining the RTT between the scanner and the target, you then know how much time should elapse between TCP segments. The remote OS will increment the time stamp on each segment by a certain value. The way in which the target increments the value can reveal the type of OS.

For example, we know that OS XYZ increments the time stamp by one for every 500 ms of uptime. The average RTT between the target and the scanner is 100 ms, which is 50 ms in each direction, as shown in Figure 1. We receive the first segment with a time stamp (TS1) of 100. We acknowledge this segment and start a timer. The second segment with a time stamp of 102 (TS2) arrives and we stop the clock. The elapsed time between segment 1 and segment 2 is 1100 ms. We know that the time in transit for the segments is 100 ms. So the clock value, 1100, minus the RTT, 100, gives us 1000 ms of elapsed time on the host between segments. The difference between TS2 and TS1is 2. This means that, in 1000 ms, the time-stamp value went up by two, which is 500 ms per time-stamp increment. Looking at a table of time-stamp values over time, we know that the target has incremented the time stamp by one for every 500 ms, which is OS XYZ. This technique combined with other fingerprinting methods will ultimately narrow the choice of OSs. This choice is important in determining future steps of vulnerability scanning.

 
Figure 1: The average round-trip time (RTT) between the target and the scanner is 100 ms, which is 50 ms in each direction.

Invalid flag combinations are another approach. The normal combinations, SYN, SYN-ACK, and ACK, are expected. But various host OSs react strangely to combinations such as FIN+URG+PSH, which is a combination not seen in a normal handshake. It is referred to as an Xmas or Christmas scan because it lights up the TCP flags like a Christmas tree. Another combination that can possibly fingerprint an OS is SYN+FIN. In addition to host discovery, these types of scans can determine whether a port is open on a host without establishing a TCP connection or half-open connection. That is because IP stacks that adhere to the RFC will respond with an RST packet if the port is open. If closed, there will be no response from the host.

Use of these flags can get more sophisticated as well. If it has already been established that a port is open using a harmless TCP-SYN scan, the same port can be probed with a FIN-ACK combination. It turns out that systems implementing the IP stack from Berkley Standard Distribution (BSD) will not respond according to the RFC with an RST packet. This provides more evidence as to the likely system type of the target.

By combining these and many other types of probes, a decent guess can be made as to the type of system. The work for this has been well-established by the creators of NMAP (www.nmap.org). They continue to discover new ways to scan and map targets on a network and build those techniques into their open-source tool. A little reading and experimentation with this can be very educational.

However, the topics of OS fingerprinting and IP stack fingerprinting can be tricky, unreliable, and confusing. Some OSs may share the same IP stack code but be different OS versions. For example, a variety of Linux distributions will use the same stack but this does not necessarily reveal the flavor of the OS. Virtual machine technology can further cloud the issue because the underlying hypervisor OS may respond to network traffic and proxy the connection to the actual host OS. The fingerprinting result can be quite unexpected. Firewall and virtual machines can perform network address translation (NAT) that will conceal the true nature of the target OS.