Lesson 25 of 33
In Progress

Test Management Process

1                  Introduction

1.1              Purpose

This document is a process guide and reference for the KPMG Tooling Engineer it is intended to ensure a consistent approach is followed for all testing scenarios within the Global Security Operations Centre (GSOC). This document lays out the tasks and rolls under the Tooling Engineers remit.

1.2              Scope

The GSOC Test Management Process applies to the GSOC only. The frequency of which the supporting processes are performed is documented in Upstream (Dependent) Processes and Downstream (affected) Processes.

The GSOC Test Management Process supports all testing in response to technical changes processes, testing of SA Rules and Reports (Content), onboarding of new Member Firms, User Acceptance Testing (UAT) and Operational Acceptance Testing (OAT). This document does not support performance/load testing within the GSOC as no suitable test environment is currently available.

This document does not go into elaborate detail, provide low-level technical procedures, or address all potential outcomes or failure cases. Analyst are expected to maintain and rely on technical work notes and controls to guide their execution of this process, and are expected to use their best judgement when minor adaptations are needed to execute actions.

1.3              Ownership

This document is owned by the Head of GSOC. The responsibility of ongoing management of this document, including the processes contained therein, rests with the Head of GSOC.

1.4              Audience

All members involved within the GSOC Test Management function must read and understand this process.

1.5              Exceptions

Exceptions to this process can be temporarily authorized by the Head of GSOC or GSOC Assistant Manager. Documentation of any process exceptions must be provided to GSOC management for process modifications.

1.6              Reporting Violations

Failure to adhere to this process must be reported directly to the GSOC Operations Manager or a Level 3 Analyst.

1.7              Responsibilities

The following roles have overall responsibility for elements of this process. Please note that these are not comprehensive listing of responsibilities of each of the following roles, but represent these roles specific responsibilities to support the process

1.7.1          Head of GSOC

  • Oversight and exception approvals

1.7.2          GSOC Operations Manager

  • Oversight and exception approvals
  • Reporting Violations

1.7.3          Tooling Engineer

  • Overall responsibility for the GSOC Test Management process
  • Testing support for all GSOC Technical Change Test processes
  • GSOC Content Testing
  • GSOC Onboarding of New Member Firms
  • GSOC Operational Acceptance Testing
  • GSOC User Acceptance Testing

1.7.4          Onboarding Engineer

  • GSOC Onboarding of New Member Firms

1.7.5          Level 1 Analyst

  • Provides False Positive and Rule Metrics to the Tooling Engineer
  • Provide Feedback on OAT, UAT Tickets

1.7.6          Level 2 Analyst

  • Provides False Positive and Rule Metrics to the Tooling Engineer
  • Provide Feedback on Operational Acceptance Testing (OAT) and User Acceptance Testing Tickets (UAT)

1.7.7          Level 3 Analyst

  • Provides False Positive and Rule Metrics to the Tooling Engineer
  • Reporting Violations
  • Provide Feedback on OAT, UAT Tickets for Emergency Changes

1.7.8          Threat Intelligence Analyst

  • Works with Tooling Engineer to establish stale content.
  • Identifying new content to implement and test.

1.7.9          Member Firms

  • Provides False Positive Requests to the Tooling Engineer

1.8              Upstream (Dependent) Processes

  • Continuous Improvement Process
  • Change Management Process
  • Service Management Process
  • Member Firm Onboarding Process

1.9              Downstream (Affected) Processes

  • Content Management Process
  • SECOPS Customisation Process

2                  GSOC Test Management Process

2.1              Defined Steps

This process defines how testing and its related activities should be conducted. The fundamental test process consists of the following main activities:

  • Test Strategy
  • Test Planning
  • Test Analysis and Design
  • Test Implementation and Execution
  • Evaluating Exit Criteria and Reporting
  • Test Closure Activities

Although logically sequential, the activities in this process may overlap or take place concurrently.

2.1.1          Test Strategy

Test Strategy is defines the test stages and types of testing to be completed for the system under test. The report is to contain agreed entry and exit criteria for each of the following stages. In addition the strategy report is to include a RACI Matrix for those involved in the testing process.

2.1.2          Test Planning

Test Planning is the activity of verifying the mission of testing, defining the objectives of testing, the entry exit / criteria, and the specification of test activities to meet the objectives and the mission of the GSOC. When conducting planning the Tooling engineer should consider:

  • How do I conduct the tests?
  • Why should I test?
  • What should I test?
  • When do I test?
  • Where do I test?

2.1.3          Test Analysis & Design

Test Analysis & Design is the activity where general testing objectives are transformed into tangible test objects, test condition, test cases and test data. Test Analysis & Design starts with analysing the test basis (such as requirements, content, software, system architecture, and design). The GSOC Tooling Engineer has to assess the feasibility of the test basis and define the test object. Further, necessary test data to support the test conditions and test cases must be identified.

Specific examples of testing criteria to be conducted during the Design phase are:

  • Confirm the planned Testing Environment is ready and fit for purpose
  • The test cases and/or scripts are ready
  • Downtime has been scheduled and authorized
  • Support Engineers are scheduled and available to assist in the case of system crashes
  • Tickets to ITS and any affected Member Firms are raised to notify them of the testing scenario (upon successful initial testing within a QA environment).

2.1.4          Test Implementation & Execution

Test Implementation & Execution is the activity where testing is conducted in a controlled manner. This is “When to Test”. The scope of Test Implementation & Execution covers all operational activities and aims at comparing actual results with expected results. The discrepancies observed during Test Implementation & Execution are to be analysed to establish their cause. The test system should be monitored to establish there have not been any adverse effects whilst under test. Testing will be performed in a non-production environment.

2.1.5          Evaluating Exit Criteria & Reporting

Evaluating Exit Criteria & Reporting is where the activity being tested is assessed against its defined objectives. This phase is to identify “What Happened”, it comprises checking test logs against the exit criteria specified in Test Planning so as to asses if more tests are required or if the exit criteria specified should be changed. This step concludes by writing a test summary report for stakeholders and the Head of GSOC. The test report creates a systematic overview of the capability of the system under test to fulfil give requirements. Examples of content to be included are:

  • All test scripts / cases have completed.
  • All untested scripts / cases have had their risks accepted in accordance with the GSOC Risk Management Process.
  • All Failed scripts / cases are to be logged and the Head of GSOC notified (a formal risk assessment must be taken before any failed tests can be taken into production).
  • Retest complete.

2.1.6          Test Closure Activities

Test Closure Activities collect data from completed test activities to consolidate experience, facts, and numbers. The final step in the test process includes checking which deliverables have been delivered as planned, the closure of any testing reports or the raising of change records for any which remain open. For process improvement purposes, lessons learned are analysed for future changes.

2.2              QA Environment

A Quality Assurance (QA) environment and a formal testing process are mandatory for any KPMG environment. A KPMG standard testing process in QA is strictly required to ensure changes do not introduce any vulnerabilities on the production environment. The following applies:

  • An agreed QA method must be utilized to document the test plan, procedures, and results.
  • Security features must be included in functionality, integration and regression testing.
  • IRSO must be contacted to provide input into the test plan as well as specialized security testing of the servers, application and operating system to evaluate the effectiveness of the security controls in place.

All security flaws identified during testing must be addressed prior to deployment of the application in the production environment.

2.2.1          Management of Test QA Environment

The GSOC’s QA facility is built within a segmented portion of the Atlanta Data Centre and comprises the following Equipment (as at 01/04/2015):

  • ESA
  • Log Decoder/Concentrator
  • Security Analytics Warehouse
  • Security Analytics Server
  • SecOps
  • RSA Connector Framework
  • RSA Archer/SecOps Services Centre
  • SecOps/Archer IIS Web container
  • SecOps/Archer Windows 2012 SQL Server
  • Layer 7 Reverse Proxy
  • Security Community SharePoint portal
  • Original SharePoint FileServer
  • CRITS Threat WareHouse
  • KPMG NTP Services
  • GSOC Enclave Firewall
  • CRITS Staging Server
  • RSA Authentication Manager
  • KPMG Global AAA Services

All activities within the QA build follow the same ITIL processes as that of the production environment.

It should be note that due to the location of the QA build (Atlanta, USA) careful consideration should be taken when conducting testing. Live data should not be utilised as this may breach individual Member Firm security constraints. For this reason and to achieve useful test results testing of content rules will take place on the production.

Whilst the QA build will never be able to replicate that of the production system its configuration is to be retained at the same state. Any changes which are to be implemented into the production system should first be tested in QA first.

The QA environment is to be used to test the following scenarios:

Major Upgrades: Upgrades to RSA platforms and supporting infrastructure. This is to establish that there will be no derogatory results or loss of performance as a result of undiscovered bugs/issues with new software. It is also to confirm that the current configuration and monitoring capabilities will not change or be affected in a detrimental manner.

Workflows Modifications: Changes to the configuration of SA and Archer/SecOps should be implemented within the QA environment before any changes are made to production. This is to ensure that changes will not affect live monitored data (including historical incidents).

Interface Changes:            Any proposed changes to the user interface to the GSOC systems (primarily Security Analytics, SecOps and CRITS) must first be implemented within the QA environment to establish if the change is feasible and to ensure that there are no issues that result which would alter the current configuration of the production system.

Feeds:                  When new feeds are identified and are to be incorporated into the pool of data accessible by Security Analytics and SecOps they must first be tested within QA. This is to identify any unexpected results and to establish the workload on the system as a result of the new feed/system. An example of this type of activity is CRITS where intelligence data is being imported into Security Analytics.

All testing and configuration changes within the QA environment are to be logged and recorded to enable analysis of any faults/failure identified during testing. The logged details will also form the implementation plan for the Production network upon successful completion of testing.

On completion of all testing the QA environment is to be examined to identify what changes have been made to the system with all added functionality having been identified and ratified.

Testing reports are to be addressed to the GSOC Operations Manager and are to include all successful and failed tests. In the case of failed tests a remediation plan and risk assessment are to be included.

2.3              GSOC Content Testing

Figure 1 Content Testing Workflow (Tooling Engineer)

GSOC Content is defined as the Security Analytics detection rules, reports, feeds, whitelists and blacklists. The GSOC Tooling Engineer is responsible for the upkeep and testing of this content.

All new content rules and report within RSA Security Analytics are required to undergo rigorous testing before they are deployed into a production within the GSOC. In addition when these detection capabilities have been deployed there will be an ongoing process where the Tooling Engineer reviews the initial effectiveness of the content and works with L1 and L2 Analysts to enhance the GSOC threat detection capabilities.

In response to authorised content change requests the Tooling Engineer will create new Security Analytics detection content and test each new content rule accordingly. In the case of existing rules these shall be reviewed in conjunction with the Threat Intelligence Team to establish content which requires enhancement and further testing.

Security Analytics content rules are written in Esper which is an Event Programming language; this follows strict criteria which is supported by Security Analytics system error checking. When a new rule is created it will only be saved if the syntax of the rule is supported.

When the new content has been successfully created it is to be tested using the online Esper Test facility to be found at http://esper-epl-tryout.appspot.com/epltryout/mainform.html. The Tooling Engineer is to consult with the Threat Intelligence Analyst prior to online content testing to ensure no operationally sensitive information is disclosed.

Figure 2 Example of Esper Online Testing

On successful completion of this phase of testing the content is to be uploaded onto the appropriate ESA which will conduct additional programing format check to confirm the validity of the content.

Testing of Security Analytics content will then take place on the production system, careful monitoring of the content is essential by the Tooling Engineer, the L1 and L2 Analysts to confirm there is no adverse impact to the system as a result of the content and that high false positives are not being generated.

All new Content Rules undergoing testing are to use the naming convention until they are confirmed to be functioning correctly:

  • SA-ESA-TEST-Name of rule

Where a large number of false positives are confirmed further tuning work on the content is to be conducted.

Operational Testing of the content is to be conducted by manually activating the content rule. In all cases this must be cleared by the Head of GSOC and the Member Firm/Business Unit affected. This testing can be done in a number of ways, a number of examples are included below (these are displayed as examples only and specific testing scenarios should be built for each content rule):

  • C2/Botnet Traffic:        Adding a custom created “known good”  testing site into the C2 Blacklist and manually accessing this causing an alert to trigger
  • Accessing VIP system from new Account:             This may be tested by having IT create a new test account for the GSOC and then using this account to gain access to a VIP system.
  • Deletion of System Logs:           On a test system an authorized Administrative account may login and manually delete all of the logs.
  • Malware on system:   This may be tested by downloading an EICAR Anti-Virus test pattern onto a KPMG computer system.

The Content Management Database on SecOps is to be updated with the appropriate information pertaining to the rule, its logic and it’s testing on completion.

2.4              GSOC Onboarding of New Member Firms

When new GSOC Member Firms begin the onboarding phase of their GSOC project the Tooling/Onboarding Engineer will liaise with the Member Firm to confirm the extent of logs directed to the GSOC and to assess the success and risks associated with the onboarding.

2.4.1          GSOC User Acceptance Testing

User Acceptance Testing (UAT) is the final testing performed when functional, system and regression testing is completed. UAT is conducted to ensure that logs received from a Member Firm satisfies the needs of the GSOC. For the purposes of this process UAT is defined as the onboarding process where informal communications are held between the Tooling Engineer and the Member Firm’s IT staff. At this point there has been no formal sign off on the project (OAT) and any assessment of logs or incidents detected are conducted in a less formal manner.

UAT will be conducted primarily by the GSOC Tooling Engineer, however other GSOC Analysts may be required to assist as required.

2.4.2          UAT Deliverables

At the start of the onboarding process for a new Member Firm the Tooling/Onboarding Engineer will liaise with the Member Firms IT department to confirm the setup and configuration of their logging.

Each Member firms log sources are to be tested and analysed to establish the correct information is being forwarded and that the configuration and supporting documentation has been received by the GSOC. Where this is not the case it is to be raised in the first instance to the Head of GSOC and then discussed in the next onboarding meeting with the Member Firm.

Each content rule within the GSOC is to undergo testing and evaluation against the new Member Firm (in accordance with GSOC Content Testing) with the results added to a RAG report for GSOC Management.

The RAG is to be updated by the Tooling Engineer as new log sources are added thus identifying possible gaps in detection capabilities for the Member Firm and highlighting where greater coverage has been achieved.

The log sources currently accepted by the GSOC are:

  • Active Directory Logs
  • DHCP Logs
  • DNS Logs
  • Endpoint Logs
  • Firewall Logs
  • Proxy Logs
  • SMTP Gateway Logs
  • VPN Logs

The following details identify the major points and deliverables expected from GSOC UAT:

  • GSOC Tooling Engineer coordinates with GSOC and Member Firm stakeholders and produces a UAT plan as part of the Onboarding Process which is reviewed and feedback/sign-off is established.
  • All appropriate logs from a Member Firm are being received by the GSOC Security Analytics system
  • Security Analytics Content rules function as expected (details to be added to a RAG report within Content Management Database)
  • RSA Security Analytics and SecOps operate as expected
  • Users can access the system correctly (this includes Member Firm access for reports generated)
  • Where issues are experienced during UAT defect tracking should be undertaken and tracked via a spreadsheet by the GSOC Tooling Engineer. Each entry will include detailed information about each defect experienced.

UAT Success Criteria:

  • All logs are delivered to GSOC (Where this is not the case the Head of GSOC is to be informed).
  • Logging at Member Firm give sufficient verbosity to enable effective analysis to be conducted
  • GSOC Content Rules have been confirmed to alert and detect traffic against the Member Firm’s traffic

2.4.3          Operational Acceptance Testing

For the purpose of this document OAT is focused on the Member Firm and the sign off against the logging and monitoring service. Its objective is to ensure everything is in place to operate, support and deliver the GSOC services to Member Firms. Ultimately this testing is in place to confirm that all actions have been completed which will enable the Head of GSOC to formally complete the onboarding process.

OAT Success Criteria:

  • Logging and Analysis of Member Firm data can be confirmed as sufficient for monitoring and alerting
  • The addition of a Member Firms data will not impact other pre-existing data or affect the performance of security systems and the monitoring conducted by the GSOC
  • The service can be monitored, measured and reported
  • It operates as expected
  • Users can access the system correctly
  • Known risks have been accepted by the GSOC Management team
  • The service as delivered meets Service Level Requirements as designed
  • Business and operational owners of the service have been determined and have acknowledged their ownership
  • OAT Sign off – Formal sign off by the Head of GSOC indicating the system satisfies the needs of the business as specified in the functional requirements and provides confidence in its use. Where there have been issues identified within the UAT process these are to be highlighted as risks in the formal sign off document i.e. Incomplete logs dispatched to GSOC for analysis
  • Head of GSOC has approved and accepted the documentation of operational and support policy, roles, processes, procedures and metric associated with the service