Fingerprinting with TCP/IP

A simple method of fingerprinting is to use the well-understood ICMP. ICMP packets are used to monitor the state of an interface on a host or report the status of access to a connected device. Nine message types are available: four for making queries and five for reporting errors. Each type is defined by a number, as shown in Table 1. PING is a very popular program that sends ICMP type 8 messages. Type 8 is an echo request whereas a type 0 is an echo reply. In addition to an ICMP type, there is a code that is used to report more information about an error. By manipulating these codes into invalid values, the target’s response or failure to respond can be captured. This in itself can tell us something about the OS. Some systems do not look at the code field on an echo request. Others do and respond with an error.

Table 1: ICMP Types 

ICMP CODE	TYPE
0	Echo reply
1–2	Unassigned
3	Destination unreachable
	Code	Meaning
	0	Net unreachable
	1	Host unreachable
	2	Protocol unreachable
	3	Port unreachable
	4	Fragmentation needed and don’t fragment was set
	5	Source route failed
	6	Destination network unknown
	7	Destination host unknown
	8	Source host isolated
	9	Communication with destination network is administratively prohibited
	10	Communication with destination host is administratively prohibited
	11	Destination network unreachable for type of service
	12	Destination host unreachable for type of service
	13	Communication administratively prohibited
	14	Host precedence violation
	15	Precedence cutoff in effect
4	Source quench
5	Redirect
6	Alternate host address
7	Unassigned
8	Echo
9	Router advertisement
10	Router selection
11	Time exceeded
	Code	Meaning
	0	Time to live exceeded in transit
	16	Fragment reassembly time exceeded
12	Parameter Problem
	Code	Meaning
	0	Pointer indicates the error
	1	Missing a required option
	2	Bad length
13	Timestamp
14	Timestamp reply
15	Information request
16	Information reply
17	Address mask request
18	Address mask reply
19–29	Reserved
30	Traceroute
31	Datagram conversion error
32	Mobile host redirect
33	IPv6 Where-Are-You
34	IPv6 I-Am-Here
35	Mobile registration request
36	Mobile registration reply
39	SKIP
40–254	N/A

Another method of reconnaissance is known as IP fingerprinting. The concept is an elegant form of manipulating inputs into the protocol stack of a target and measuring the results. For a brief review, let’s look at the TCP header structure in Table 2.

Table 2: TCP Segment

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 SOURCE PORT DESTINATION PORT SEQUENCE NUMBER ACKNOWLEDGEMENT NUMBER Header Length Reserved URG ACK PSH RST SYN FIN Window Size Checksum Urgent Pointer Options (up to 40 bytes) End of Option Data

The most useful operational benefit of TCP is the fact that it guarantees delivery by acknowledging the receipt of each packet. That set of flags—SYN, ACK, and RST—are what tell the recipient the purpose of what is transmitted. Our vulnerability scanner is sending SYN packets to the target. But it is the behavior of the rest of the contents of the packet that can reveal something about the target. Sequence number is a good example. So that TCP listeners on hosts do not become confused, every packet includes a sequence number. Since the creation of the protocol, it was found that it is easily possible to “wrap” the sequence numbers because they are of limited size (32 bits). To address the potential for wrapping and having a duplicate sequence number with an old packet being mistaken for a sequence number of a new packet, a time-stamp option was introduced in RFC 1323. This is an optional field and not all operating systems’ TCP/IP implementations set the value. When the scanner sees such value sent when the time-stamp option was never used, the choice in operating systems is narrowed considerably.

Another phenomenon to measure is the incrementing of the time stamp. By first determining the RTT between the scanner and the target, you then know how much time should elapse between TCP segments. The remote OS will increment the time stamp on each segment by a certain value. The way in which the target increments the value can reveal the type of OS.

For example, we know that OS XYZ increments the time stamp by one for every 500 ms of uptime. The average RTT between the target and the scanner is 100 ms, which is 50 ms in each direction, as shown in Figure 1. We receive the first segment with a time stamp (TS1) of 100. We acknowledge this segment and start a timer. The second segment with a time stamp of 102 (TS2) arrives and we stop the clock. The elapsed time between segment 1 and segment 2 is 1100 ms. We know that the time in transit for the segments is 100 ms. So the clock value, 1100, minus the RTT, 100, gives us 1000 ms of elapsed time on the host between segments. The difference between TS2 and TS1is 2. This means that, in 1000 ms, the time-stamp value went up by two, which is 500 ms per time-stamp increment. Looking at a table of time-stamp values over time, we know that the target has incremented the time stamp by one for every 500 ms, which is OS XYZ. This technique combined with other fingerprinting methods will ultimately narrow the choice of OSs. This choice is important in determining future steps of vulnerability scanning.

 
Figure 1: The average round-trip time (RTT) between the target and the scanner is 100 ms, which is 50 ms in each direction.

Invalid flag combinations are another approach. The normal combinations, SYN, SYN-ACK, and ACK, are expected. But various host OSs react strangely to combinations such as FIN+URG+PSH, which is a combination not seen in a normal handshake. It is referred to as an Xmas or Christmas scan because it lights up the TCP flags like a Christmas tree. Another combination that can possibly fingerprint an OS is SYN+FIN. In addition to host discovery, these types of scans can determine whether a port is open on a host without establishing a TCP connection or half-open connection. That is because IP stacks that adhere to the RFC will respond with an RST packet if the port is open. If closed, there will be no response from the host.

Use of these flags can get more sophisticated as well. If it has already been established that a port is open using a harmless TCP-SYN scan, the same port can be probed with a FIN-ACK combination. It turns out that systems implementing the IP stack from Berkley Standard Distribution (BSD) will not respond according to the RFC with an RST packet. This provides more evidence as to the likely system type of the target.

By combining these and many other types of probes, a decent guess can be made as to the type of system. The work for this has been well-established by the creators of NMAP (www.nmap.org). They continue to discover new ways to scan and map targets on a network and build those techniques into their open-source tool. A little reading and experimentation with this can be very educational.

However, the topics of OS fingerprinting and IP stack fingerprinting can be tricky, unreliable, and confusing. Some OSs may share the same IP stack code but be different OS versions. For example, a variety of Linux distributions will use the same stack but this does not necessarily reveal the flavor of the OS. Virtual machine technology can further cloud the issue because the underlying hypervisor OS may respond to network traffic and proxy the connection to the actual host OS. The fingerprinting result can be quite unexpected. Firewall and virtual machines can perform network address translation (NAT) that will conceal the true nature of the target OS.

Performance Issues | Active Scanning Technology Detection Methods

We have discussed at some length the process of identifying ports and handling TCP connections. All of these factors have to be taken into consideration during the scan; however, the scanner cannot wait too long. At some point, the transaction attempt will “time out.” This phenomenon can be referred to as discovery tolerance. Various vendors have different levels of discovery tolerance. The amount of tolerance is loosely proportional to the accuracy of the discovery with rapidly diminishing probability of successful identification. Fortunately, we know from experience that there is no point waiting for a reply beyond a certain amount of time. Determining that point is the real skill in any fingerprinting activity. The goal is to be complete and accurate, but there is a law of diminishing returns. Two key timers affect the speed of the discovery process: the connection establishment timer and the retransmission timer.

For many TCP implementations, the connection establishment timer (TCP_KEEPINIT parameter) waits 75 seconds for response. A simple scan on a single port for 200 hosts would require over four hours to complete if none of the hosts responded. This must be adjusted to wait far less time. One effective approach is to take the maximum roundtrip time (RTT) of ICMP echo reply exchanges and add two seconds. This provides ample time for an application to respond on the required port and is likely to be far shorter than the default of 75 seconds.

With TCP connections, a discovery process can also vary the retransmission timer when additional packets are to be exchanged with the target. In normal communications, the timer begins with a value of 1.5 seconds. If no response is received, then the value is doubled to three seconds. If there is still no SYN-ACK, the timer is doubled again and we wait six seconds. This continues repeatedly until we reach a limit of 64 seconds. The process is called exponential backoff (EB). In theory, this should parallel the exponential probability that a response will ultimately be received. However, this is often impractical for host discovery purposes in vulnerability scanning. A typical OS can spend several minutes waiting for a connection to time out.

A more practical approach would be to sequentially increase the retransmission timer by smaller values for a total period of time to be some factor above the average for the target IP range. For example, let’s suppose that we are performing a discovery of network A (192.168.10.0/24) with an upper limit of 30 seconds for retransmission. If the first 16 hosts required an average of 10 seconds to respond and the mode was five seconds, we might start our retransmission timer at five seconds and increase the value by five seconds until an upper limit of 20 seconds was reached (2× average). This is a more sensible approach that will avoid a common IP stack value that can reach several minutes for a single connection. Remember that our goal is discovery of open ports and live hosts, not the reliable transmission of data to another host.

There is one other item that can be manipulated, which is not exactly a timer and can speed the discovery process considerably:

to implement all the timers TCP only requires that two functions are called periodically: (1) the fast timer is called every 200 ms and (2) the slow timer every 500 ms. TCP uses these two periodic ‘ticks’ to schedule and check all the timers described…as well as measuring round trip times.^[*]

Basically, the OS kernel must check every 200 ms to see if an acknowledgment has been received. In modern networks and operating systems, this is a very long time.

By decreasing this period, the discovery processes can recognize that the probing SYN packets it has sent have been acknowledged in a shorter time, and move on to the next probe. If the RTT from SYN to SYN-ACK is 10 ms, then under normal circumstances, the discovery process can wait for up to 190 ms to proceed with the next action. Multiply this number by hundreds of hosts and dozens of ports, and the wasted time can be tremendous.

The one caveat to modifying TCP timers is that some applications are simply slow to respond. This approach works best when probing for open ports but not necessarily for applications. There is a lot of room for creativity in scan performance optimization. This section simply illustrates some of the challenges designers can be confronted with when trying to optimize the scan process and minimize the impact on the network.

1 Black Box Testing

Once the presence of a host has been established and that presence recorded in the memory of the scanner, a series of tests or “checks” are performed to find vulnerabilities. The types of checks are dependent upon the type of host and the configuration of the scanner. Generally, two types of checks are performed. A network-based or surface check is performed, which involves the probing and analysis of what is evident with limited or no access to services on the machine other than what is offered to any other peer on the same network as that which exists between the scanner and the target. This is also known as an unauthenticated check. The other type of check is an authenticated, internal check or white box test. It is performed when the scanner is given special information and credentials to access details of the host, which are generally reserved for trusted entities.

The difference between surface and internal checks is obviously significant not only in the way they obtain information, but also in the value and quality of that information. Clearly, more detailed data can be obtained by logging into a host and perusing its configuration. Although the information tells us a lot about the host, it does not typically represent the view of an attacker who performs reconnaissance on an unknown host. Although valuable from an analysis standpoint, some attacks take place by probing the host from the view of an outsider; therefore, information that can be obtained in the same fashion is often more valuable. To summarize, a vulnerability discovered and exploitable from outside a host represents a greater exposure than if the same vulnerability could only be discovered and exploitable from a credentialed or internal check.

There is a common perception that authenticated checks are more accurate than remote checks but that’s often not true. The Windows registry is commonly used for authenticated checks but is often wrong. It’s important to consider that not all authenticated checks are created equal and that a remote check is a good method of validating authenticated information.

The black box testing process involves some straightforward testing over the network and possibly some creative use of IP and other protocols. Usually, the simple tests are harmless and efficient. The more exotic manipulation of IP protocols can cause problems on scanned hosts with applications that are ill-prepared to handle many variations. This is a vulnerability in itself. The IP stack of the host is usually capable of handling nearly any variety of traffic, but the overlying applications sometimes are not. It is another area that calls for extensive testing in order to avoid adverse effects on production systems. Most vendors are able to provide a list of known negative application interactions.

Following is a list of some common methods of reconnaissance:

• Malformed packets are sent to the host to identify the presence of a vulnerability in the response. This is similar to the discovery process and is sometimes incorporated into the same phase for efficiency. The information sent to the target can be at any one layer or multiple combinations of layers 3 through 7 in the OSI Model.
• Normal packets are sent to a known application to obtain results that will reveal vulnerability information. This is very common in the http protocol to obtain information about the Web server, application server, or back-end databases.
• Valid information is sent to the target to gather valid header response data that will reveal the version of software answering the service request. This is known as banner checking. Many software applications can obfuscate this information with simple configuration changes, so it is not the most reliable method.

These methods can be summarized conceptually in pseudo code form:

Send X to target
Listen for response Y
Match Y to possible response list
If Y is on list, note vulnerability
If Y is not on list, ignore
Get next check; Loop

Manipulating TCP | Active Scanning Technology Detection Methods

Some scanners do not consider ICMP to be reliable since some hosts are configured not to respond to ICMP. So, a TCP SYN packet is sometimes sent using several common ports found on a variety of network devices. Table 1 lists some common ports that are scanned to determine whether a host is present.

Table 1: Commonly Scanned Ports 

PORT	PROTOCOL
20	FTP
21	FTP
22	SSH
23	Telnet
80	HTTP
443	HTTPS
137	NETBIOS Name Service
138	NETBIOS Datagram Service
139	NETBIOS Session Service

Other ports may also be scanned, depending on the vendor and how one might configure this phase. Once the SYN packet is sent to the host, a reply of SYN-ACK is expected. Should this reply not arrive in time, the scanner will consider the port not responsive and the host not present, that is, unless the host responds on another port. The discovery of these ports is not necessarily a serial process. The scanner may “spray” the TCP SYN packets on many ports and at many hosts simultaneously. This saves time and makes more efficient use of bandwidth. Some performance issues will be discussed later.

One potential side effect of the TCP discovery method is the potential for leaving open or half-open sockets. This can have an adverse effect on a system, depending on the application listening and the integrity of the OS protocol stack. Half-open sockets occur when the scanner does not complete the connection setup with an ACK packet. The effects range from memory consumption to denial of service (DoS) for production systems. Normally, this is not a problem since only one active connection is attempted per TCP port. However, a misconfigured scan can change this. A similar phenomenon exists when scanning through a firewall. Many firewalls will function as a proxy for the connection to the host. This imposes a load on the firewall when hundreds or thousands of hosts are scanned at once. Some routers can be impacted as well when they are undersized for their operating environment.

If the connection handshake is completed with an ACK packet, an entry in a connection table is maintained on the host and/or on a firewall. The result is a further consumption of resources until the connection times out or is reset. For that reason, it is important to test the behavior of a scanner and determine whether or not a TCP reset is sent to the host and how that behavior might affect your network on a large scale. If it is possible that a large amount of scanning will be performed through a firewall, then test this scenario. The setup and breakdown of connections on a firewall are two of the more processor-intensive tasks and can degrade performance. This is not absolute, however. Proper scheduling, bandwidth shaping, and connection limits in the scan configuration can help prevent these problems. Figure 1 illustrates a scenario where a firewall may be significantly affected by multiple simultaneous scans. As you can see, the total number of connections per second can add up quickly. A scan is far more intense than regular network traffic because it is concentrated into a short time period.

 
Figure 1: A scenario where a firewall may be significantly affected by multiple simultaneous scans.

Conversely, a firewall, with all the additional security features that vendors have added, can interfere with a discovery scan. For example, some firewall vendors have added intrusion prevention capabilities as well as a SYN proxy. These features can provide false information to a vulnerability scanner, making it seem as though a host exists where there is none. This is because many scanners consider a closed port “response” to indicate the presence of a host. Otherwise, why would there be a response?

These security features can also do the reverse and obfuscate the host and its open ports. Hopefully, the firewall vendor will have the built-in ability to make an exception to the traffic by source IP address. With the correct IP address of the scanner configured in an exception list, the scan should proceed without error.

However, not all firewalls are created equal. For this reason, we will spend a little time discussing packet processing in a firewall as it is related to VM. Figure 2 shows the basic structure of how a firewall might handle traffic. Notice the stacked architecture. The reason for this is to qualify traffic for the most fundamental flaws prior to investing more processing cycles. For example, if the TCP flags are an invalid combination, the traffic should be dropped. Much of this processing can be performed in silicon and avoid burdening the firewall CPU further. On the other hand, if traffic is to be processed by some rules, more CPU cycles are required.

 
Figure 2: The basic structure of how a firewall might handle traffic.

The next step in packet processing is to save and monitor the connection state. If a SYN packet is received from a vulnerability scanner, it is compared to entries in a connection table. If the connection already exists, then this is likely a duplicate and the packet is dropped. If the connection does not exist, then an entry is made in the table. When a SYN-ACK packet is received, the same comparison is made to keep track of the state of that connection. The same is true with all related packets. If an RST packet is received, then the connection is deleted from the table. This process can take a lot of CPU if there are many thousands of connections per second and other firewall activities taking place. The constraints of this connection table data structure and the related programming code to handle the packets are the parts most crucially affected by discovery scans. If the state table is too small a data structure, then the SYN/SYN-ACK discovery process will rapidly fill this table. If that constraint shows up in testing, you will have to make sure that you limit the number of open connections during discovery.

On the other hand, if the processing code that makes comparisons to this table is inefficient, there will be a significant impact on firewall throughput. In this case, it is vital to maintain a limit on the rate of connections.

Most firewalls process rules in order. A packet is taken from a queue for comparison against the rules. Once a rule matches the traffic, the inspection process stops. The packet is then forwarded to the network interface and the next packet is extracted from the queue.

Another way to avoid a large impact on firewalls is to relocate the scanners around them. This is largely dependent on your network design. Careful placement of scanners is a primary consideration and can be crucial to scanning effectiveness.