Lesson 26 of 33
In Progress

Detection Optimization Process

1                 Introduction

1.1             Purpose

The purpose of this document is to define a process to identify, analyse and resolve gaps in the detection and response of security incidents. Detection optimization is focused on enhancing the detection of security incidents. This is to include the current rule set as well as detection gaps. The detection gaps can be the result of bad intelligence, missing indicators or misconfigured content..

1.2             Scope

The GSOC Detection Optimization process applies to the GSOC only. The frequency of which the supporting processes are performed is documented in Upstream (Dependent) Processes and Downstream (Affected) Processes.

1.3             Ownership

The KPMG GSOC manager owns this document. He or she is responsible for ensuring that it is updated and maintained in response to feedback from GSOC Analysts.

1.4             Audience

This document is intended for the KPMG GSOC Team.

1.5             Change Management Cycle

This document should be reviewed on an 6 month basis, or any time that the Constraints or Assumptions are believed to have changed.

1.6             Exceptions

The GSOC Deputy Manager or GSOC Manager can temporarily authorize exceptions to this process. Documentation of any process exceptions must be provided to GSOC management for process modifications.

1.7             Reporting Violations

Failure to adhere to this process must be reported directly to the Deputy GSOC Manager or GSOC Manager.

1.8             Responsibilities

The following roles have overall responsibility for elements of this process. Please note that these are not comprehensive listing of responsibilities of each of the following roles, but represent these roles specific responsibilities to support the process.

1.8.1          GSOC Manager

  • Oversight and exception approvals

1.8.2          GSOC Deputy Manager

  • Oversight and exception approvals

1.8.3          Threat Intelligence Analyst

  • Provides Feedback
  • Member of Optimization Committee

1.8.4          Tooling Engineer

  • Provides Feedback
  • Member of Optimization Committee

1.8.5          Level 2 Analyst

  • Provides Feedback
  • Member of Optimization Committee

1.8.6          Level 1 Analyst

  • Provides Feedback
  • Member of Optimization Committee

1.9             Upstream (Dependent) Processes

  • Triage Process
  • Content Management Process
  • Threat Intelligence Process
  • Incident Management Process

1.10        Downstream (Affected) Processes

  • Triage Process
  • Content Management Process
  • Threat Intelligence Process
  • Incident Management Process

2                 Detection Optimization Overview

2.1             Detection Optimization Overview

Detection optimization is focused on enhancing the detection of security incidents. This is to include the current rule set as well as detection gaps. The detection gaps can be the result of bad intelligence, missing indicators or misconfigured content.

The goals of the Detection Optimization Process are to:

  1. Discover and capture gaps or improvements in detection processes
  2. Review the gaps/improvements
  3. Determine the remediation of the identified gaps
  4. Approval and implement the remediation plan

2.1.1          Discovery and Capture of Gaps

The discovery of gaps will rely on feedback models from each GSOC team. The below sections describes what data is provided from each GSOC team.

2.1.1.1    Lessons Learned

Level 1 and Level 2 analysts will provide feedback in the form of Lessons Learned that they will complete when an incident has been resolved and as indicated in the Incident Management Process. Examining the output from these questions will help identify broken processes, gaps in incident context and detection changes or new methods.

2.1.1.2     Content Management Metrics

As outlined in the Content Management Process, the bi-weekly report from the Tooling Engineer will be sufficient information to determine the effectiveness of the content deployed. This report will include metrics such as the most effective and least effective rules and false positive. Review of this report will help to show from a content perspective the type of alerts that are successful and the types that are prone to false positives.

2.1.1.3     Threat Intelligence Metrics

As part of the Threat Intelligence Process, The Threat Intelligence team will report on metrics that rate the effectiveness of the intelligence sources and the effectiveness of the disseminated intelligence.  Review of these metrics will provide insight into specific intelligence sources that need to be optimized.

The Threat Intelligence team will also be able to report on strategic and/or Operational intelligence that will help the Committee to identify detection gaps.

2.1.1.4     RSA Archer Security Operations Metrics

These metrics will provide an overall view of how the Level 1 Analyst and Level 2 Analysts are handling incidents. These metrics will be used to identify detection possible detection gaps and to gauge the complexity of the events being detected.

The committee will look for a rise in trends of metrics such as threat vectors and cyber kill chain, this will allow the committee to assess if they have adequate detection content in place for detection.

High analyst workload can be due to a lack of context during a incident, however it can also be due to content that detects complex incidents in which the analyst has to use more time for triage, it is here that the committee will focus on.

The metrics to focus on for Detection Optimization are below.

Table 3: RSA Security Operations Metrics

MetricDescription
Incidents by TargetWhat are the most common attacked targets? Is detection sufficient for these targets?
Threat Category/VectorWhat are the most common attacked vectors and methods? Is detection sufficient?
Analyst WorkloadAre analysts working large workloads due to high false positives or missing incident context?
Incidents by Cyber Kill ChainWhat stage of the Cyber Kill change is being detected the most? The least? Is detection sufficient?

2.1.1.5     Member Firm Feedback

Feedback from Member Firms should be incorporated into the discovery process. At the moment the feedback from Member Firms will be informal and should be captured in the Lesson Learned or through emails sent to the GSOC Management or Staff.

The GSOC Management and Staff should forward any feedback received from the Member Firms to the Level 3 Analyst.

Member Firm feedback can also come in the form of how fast they are able to remediate the incident passed over to them by the GSOC. The assumption being that what was provided to the Member Firms was accurate and actionable.

2.1.2          Review the gaps/improvements

In order to review the data provided by each GSOC group and the Member Firms, a Optimization committee should be formed that is lead by the Level 3 analyst and has 1 member from each GSOC group.

On a monthly basis, the committee will review all the data from the various metric reports and lessons learned and identify any detection gaps.

Generally speaking the committee will be looking for:

  • What are the total False Positive Rates?
  • How effective are the intelligence sources? What is the most effective source? What is the least?
  • What is the threat landscape that the threat actors are currently attacking relative to KPMG?
  • Does the Level 1 or Level 2 analyst get enough information from incident detection to determine a real threat or not
  • Are the Member Firm satisfied with the timeliness and context they receive as a result of the detection?

2.1.3          Remediation and Implementation

After the review and identification of gaps, the committee will work together to put a remediation plan together in order to address the gaps. The GSOC Manager must approve the remediation plan before the committee can continue. If the GSOC Manager does not approve the plan, the committee must revise as indicated and then resubmit to the GSOC Manager.

When the committee receives approval from the GSOC Manager they can start to assign implementation tasks to the individuals. Depending on the type of tasks, additional resources may be needed.

Depending on the type of implantation tasks, different processes may be relied on to finish the tasks. For instance, if new content is being created, then the Content Management Request Process will be initiated. If the implementation tasks meet the criteria for a Change Request, then the L3 must lead this part to get the change approved.

When the tasks are completed they are reviewed and validated by the Level 3 Analyst. If the Level 3 Analyst does not think the tasks have been completely implemented he will assign additional tasks to address the incompletions.

 Once the Level 3 Analyst validates completion the changes are documented.

2.2             Detection Optimization Workflow

Figure 1: Detection Optimization Workflow