Reports | Vulnerability Management

Reporting is an important part of vulnerability management and managing any process that can be measured. Before completing your evaluation of any vendor, a clear understanding among the key people involved in VM should be in place as to how the process of identifying and remediating vulnerabilities should be conducted. Then, ask critical questions about what managers, engineers, administrators, auditors, and risk managers need to know to perform their part of the process and get the most out of the system. This will lead to a definition of the reports needed. Consider distributing sample reports to stakeholders to make sure they are acceptable. Those reports have the following high-level requirements:

• Scheduling (frequency with which the reports will be needed): In some cases, reports are needed weekly to remediate the latest vulnerabilities, and in other cases, monthly to evaluate the security posture of the organization. In still other cases, reports may be needed every time a certain event occurs such as a threshold being reached on total risk to a critical target. Having an idea about the timing and frequency of all the reports will reveal what kind of scheduling flexibility is needed in automated reporting.
• Objective: Knowing the exact purpose of the report will often help identify whether the right report is present in the system or needs to be developed. Some reports produced by a system are titled differently but still may meet the objective. On the other hand, it is possible that the reverse may happen. A report may be titled “Audit Review,” which seems intended for the use of auditors; however, close observation may show that the report will not meet your specific auditor’s role definition. An ancillary point here may be that you should not be too quick to check a requirements box until you have verified the details of the reports.
• Format: Most reports are not very customizable. The order of the columns, sorting and filtering options, and header and footer content may be unsuitable for your organization. The latter requirement can be a significant problem. If a vulnerability report does not have the company-required document classification statement in the footer, and company logo or department name in the header, some concerns could arise with auditors looking for compliance in document handling. Also, the utility of a report can vary greatly with the sorting and filtering capabilities. If a scan results report can only be sorted by IP address and not by severity, perhaps the prioritization process cannot be efficiently conducted.
• Customization: When appropriate canned reports are not available, some access needs to be given to the database and schema to allow reports to be generated that simply aren’t available in the core product. This can also be provided with a report writer add-on. Any such add-on will have to be equally vetted with the requirements. Also, one must consider that additional products or consulting used will raise the overall cost of the system.
• Types: The most common critical areas for reporting are remediation, security posture, and process monitoring. The type of reports you will need will naturally depend on how your process works. Remediation reports should specifically facilitate the methods by which your process remediates. This sounds obvious but it is not. For example, a company may have a different group remediate vulnerabilities in each city or office network. So, the reports should be available by location.
  Alternatively, your organization may remediate by device type. For example, server administrators may all work for a global server group. Vulnerabilities for those servers would be assessed and prioritized, and work allocated by a central group. This may be the case even though the audit of those servers by the system took place as an audit of the overall network.
• Triggers: Some reports need to be generated when a certain set of conditions are met. These reports are typically e-mailed or copied to a specific location for retrieval. The triggering event can be when the vulnerability score or a particular target reaches a certain level or when a critical host has a certain severity of vulnerability. These reports would then be automatically e-mailed to specified persons who are necessarily authorized users of the reporting system.
• Roles: Any report should be configurable to be viewed or modified by specific roles. Those roles would be defined as system administrator, network owner, network group owner, or system (target) owner. There are two purposes for specifying which roles or combination of roles are permitted to access or modify a report:
  • Separation of duties: This assures that reports cannot be manipulated for a favorable outcome and contradict any performance penalty that may be weighed against a responsible party who fails to remediate systems.
  • Confidentiality of the data: Each report contains a potential gold mine of information to a hacker that could surely be used to assess targets and attack vectors.

• The most flexible approach to implementing roles is to allow the VM system administrator to create the roles and assign the users and user groups to those roles. More about role definitions later.

• Control: The reports should be filterable by network, IP range, system type, scanner, application, or network owner. You will quickly find that once the availability of good data is known by managers and engineers, they will ask for more. Having in-depth knowledge of the structure of local networks, network managers may ask for particular subnet reports and IP ranges even though the audit process takes place on all IP addresses that can be found. Essentially, what is audited for vulnerabilities is a superset of the reports requested by the network owners.
  Owners of applications may ask for reports showing the vulnerability state of the systems supporting their applications. This may include a combination of routers, firewalls, database servers, application servers, and Web servers. This may be especially important for SOX compliance or limited PCI audits. Large organizations with very limited PCI exposure will need to assess the vulnerability of any systems directly or indirectly related to payment processing. Being able to readily identify and audit those systems without clouding reports with vulnerability data from irrelevant systems can save time and cost in an audit. The same principle holds true for non-PCI financial systems. SOX compliance should not be seen as a reason to accept or reject functionality from the VM system; however, it can be helpful in evaluating controls related to financial systems. Public companies should carefully consider the real requirements of SOX in accordance with the 2007 SEC guidance on the topic. Vendors would be hard-pressed to provide solid advice and rationale directly related to SOX compliance and their products.

Standards | Vulnerability Management

Every vendor has a different approach to detecting vulnerabilities. In the early days of VM products, this proprietary approach was the primary means of distinguishing one product from another. The ability to be accurate yet fast was essential. However, this has led to a severe lack of sharing of information among product vendors, which threatens the effectiveness and consistency of security as a whole for customers. Since the results of vulnerability assessments may vary from one vendor to another, how can you be sure which results should be used to take remediation action? If one product detects a vulnerability by looking at a registry key, and another misses the vulnerability because it uses file detection, how can the security manager rely on either product since no single, reliable best-practice detection method has been employed? Discrepancies will exist if multiple VM systems are used that can create additional work to resolve them. So, that is not an efficient solution either.

Conversely, similar results can occur when software distribution methods used by organizations do not result in the common footprints on a system that result from a standard local installation. In some cases, applications on the Microsoft Windows^® platform can be installed with certain registry keys missing. When the vulnerability scanner looks for these registry keys and does not find them, then the vulnerability status can be misread.

A more global problem is the lack of standard vulnerability identifiers. For example, if the vulnerability management system uses proprietary identifiers for vulnerabilities and the patch management system uses Bugtraq ID numbers, how can the two be reconciled? In some cases, there is no identifier available for a particular vulnerability that was discovered by the VM system vendor, and therefore only a proprietary index is available. Some kind of standard adherence is needed to maximize the effectiveness of the overall VM process and assure consistent identification worldwide.

Solutions are available. If it were agreeable to vendors and MITRE Corporation alike that vulnerabilities they discover can be centrally identified and described, then patch management systems could rely on those identifiers.

Alternatively, some vendors have already integrated functionality such as patch management so that compatibility in vulnerability identification and corresponding remediation is not a concern. However, this limits the choices of products and functionality. The VM systems buyer is committed to the vendor for both functions. As of this writing, no vendor provides comprehensive patch support for all OSs and all vulnerabilities, nor are they likely to do so in the future. Oddly enough, at the time of this writing, there are more than 33,000 Common Vulnerabilities and Exposures (CVEs). Yet, top VM vendors cover less than 75 percent of that number. Why do so many customers tolerate this obvious deficiency? If a vendor told you that his product was no more than 75 percent effective, would that be acceptable? Customers need to recognize this key deficiency and demand more.

Recognizing that they cannot do everything, vendors will build-in the ability to interface with key management and tracking systems such as change management and incident management. The number of compatible vendors may be limited. In that case, vendors have built an API that will allow the creation of a custom connector between the VM and other systems.

We previously discussed technology and the support for standards such as the Common Vulnerability Scoring System (CVSS), CVE, and OVAL^®. There are great benefits to adopting a standard for any technology. But, given the rapid proliferation of security technologies and their competitiveness with one another, the adoption of standards should become a more critical requirement. Initially, it would seem foolish for a vendor to “give away” functionality to another vendor. However, by providing the flexibility to choose the needed products, vendors can increase their chance of selection. This creates a chicken-and-egg problem where the vendor must support the open standard and the customer must demand it. OVAL is one of the more specific and important standards that should be seriously considered on a requirements list.