Lesson 32 of 33
In Progress

Shift Management Process

1                  Introduction

1.1              Purpose

This document is a process guide and reference for KPMG GSOC management for the process of managing shifts that support the security incident handling process.  The purpose of the shift management process is to ensure that shift schedules are managed in a fashion that optimally balances overall GSOC staffing levels, shift staffing levels, GSOC effectiveness, and employee satisfaction.

1.2              Scope

This process document does not go into elaborate detail, provide low-level technical procedures, or address all potential outcomes or failure cases.  It provides the guide for managing GSOC shifts (on-call and in-person), and how to manage shift handover.

1.3              Ownership

This document is ultimately owned by the GSOC Director.  He or she is responsible for ensuring that this is updated and maintained in response to feedback from GSOC Analysts.

1.4              Audience

This document is intended for all GSOC members on or supporting shift work (see Section 2.9 – Responsibilities).

1.5              Change Management Cycle

This document should be reviewed on at least a 6 month basis, or at any time that the Constraints or Assumptions (section 2.8) are believed to have changed.

1.6              Exceptions

Exceptions to this process can be temporarily authorized by the GSOC Director or GSOC Operations Manager.

1.7              Reporting Violations

Failure to adhere to this process must be reported to the GSOC Director.

1.8              Responsibilities

The following roles have overall responsibility for elements of this process.  Please note that these are not comprehensive listing of responsibilities of each of the following roles, but represent these roles’ specific responsibilities to support the shift management process.

1.8.1          GSOC Director

The GSOC Director has overall responsibility for and authority over the shift management process.  He is directly responsible for ensuring that HR rules are followed, and ensuring that there is adequate staffing to meet the requirements of this Shift Management Process.

1.8.2          GSOC Operations Manager

The GSOC Operations Manager has the responsibility for ensuring that shift schedules are released in a timely fashion, that Analyst vacation/training/absence requests are tracked and incorporated into shift schedules, to adjudicate any conflicts or issues in their requests, and ensure that all contractual obligations to employees and/or the GSOC are being met.  The final expectation will mean that the GSOC Operations Manager must ensure employee contracts are written in such a way that members will be contractually obligated to support the requirements of this process.

1.8.3          GSOC L1 and L2 Analysts (24×7 shift members)

GSOC L1 and L2 Analysts are responsible for requesting planned absences (due to vacation, training, or planned medical issue) in a timely fashion, for on-time shift attendance, and for immediately escalating and communicating any short-term absences (sickness, family emergency, transit issues, etc.,).  While on-shift, L2 analysts are responsible for escalating any shift-staffing issues that impact the proper functioning of the GSOC.

1.8.4          GSOC L3 Analyst

The GSOC L3 Analyst is responsible for addressing or adjudicating any on-shift staffing emergencies, escalating any issue that cannot be solved that potentially limits proper GSOC function, and for responding to on-call issues from the L2’s.  In a crisis, the GSOC L3 may also fill a L2 analyst shift position that cannot otherwise be filled.  The GSOC L3 is also responsible for analysing the impact of potential shift-schedule changes, and recommending approval for vacation and/or training events requested by analysts.  In the absence of the GSOC Operations Manager, the GSOC L3 has responsibility for ensuring that the shift schedule is appropriate updated.

1.8.5          GSOC Threat/Intel Analysts (on-call members)

GSOC Threat/Intel Analysts are responsible to fill open L2 shifts in response to planned L2 staffing limits (i.e., due to training, vacation, etc.,).  They are further responsible, when in an on-call status, to answer on-call issues, and, if necessary, to fill in a shift where a L2 cannot attend due to an unplanned issue.  GSOC Threat/Intel Analysts are expected to regularly rotate into the L2 shift role in order to ensure good synchronization between their products and GSOC processes.

1.8.6          GSOC Tooling Engineers

GSOC Tooling Engineers are expected to be available to support GSOC system issues in an on-call basis.

1.9              Upstream (Dependent) Processes

  • Triage Process

1.10          Downstream (Affected) Processes

  • None

2                  Process Overview and Considerations

2.1              Goals

The primary goals of this process are:

  • The GSOC has sufficient shift staffing at all times to conduct effective operations (as defined by the cost model).
  • During crises, the GSOC has sufficient on-call staffing availability to continue to conduct effective operations
  • During unplanned personnel availability gaps (illness, unfilled position), the GSOC has sufficient reserve staffing flexibility to continue to conduct effective operations.
  • GSOC Team members are able to meet personal obligations (vacation, training) and have the flexibility to react to unplanned events (illnesses, family emergencies) in a way that supports team morale and staffing.
  • Significant shift-staffing events such as turnover are effectively executed.

2.2              Scheduling Timelines

The purpose of this section is to identify key timing gates for employees and managers to effectively manage creating and maintain shift schedules.  Key scheduling gates include the following:

  • (Greater than 90 calendar days in advance) All GSOC shift or on-call employees request vacation time periods and/or holiday/weekend times (or on-call personnel) where they are not available and/or identify desired/required training events that they must attend.
  • (Greater than 75 calendar days in advance) GSOC Operations Manager drafts an updated GSOC shift and on-call schedule for the GSOC to all GSOC members to review and provide feedback
  • (Greater than 60 calendar days in advance) GSOC Operations Manager provides the next 60 days of GSOC shift and on-call schedule for the GSOC to all GSOC members
  • (More than 4 working days in advance) GSOC shift or on-call employees identify mutually-agreed shift swap proposals to the GSOC Operations Manager to allow a change to the GSOC shift schedule.
  • (More than 2 working days in advance) GSOC Operations Manager publishes any GSOC shift schedule swaps or changes.
  • (As soon as possible) GSOC shift or on-call employees identify emergency or unplanned events that might require them to miss a shift.

Meeting these timelines is an expectation for shift employees that must be explicitly addressed in employee contracts.

2.2.1          Internal/External Considerations

The following considerations can be used to guide shift-scheduling decisions.

Holidays.  There are two holiday considerations that affect shift planning.

  • Before the end of October, the Deputy GSOC shift manager should attempt to project A, B, C, D shift rotations for the next calendar year.  If it is clear that a supermajority (> 2/3’s) of holidays will be missed by a single shift (or two shifts), one or more 3x3x3x3 shift rotation periods should be included to attempt to address the disparity.
  • Normally, on-call personnel (for those in a rotating on-call position) will shift on-call responsibilities on Mondays.  If a holiday falls on a Monday, the on-call rotation will be extended an extra day (to Tuesday).

School/Training/Company Functions.  There will be some KPMG school or training requirements that will predictably affect some or all of the team.  These training periods should be included in the training schedule for clarity’s sake.  However, the normal shift rotation period (4 days on, 4 off, 4 nights on, 4 off) should be maintained during this timeframe.

Total Exceptions/Time-Off.  The GSOC Operations Manager should maintain a 6 month, 12 month, and lifetime list of various events that affect a member’s wellbeing, in order to ensure equitable scheduling among different team members.  This information can be used to guide decision-making for potentially conflict vacation requests.  Key items to consider:

  • Holidays missed due to work
  • Weekends missed due to work
  • Vacation taken on normally scheduled night-shift vs day-shift days
  • Total hours of on-call calls taken

2.2.2          24×7 Shift Schedule Expectations

The normal shift-schedule will appear as follows (A, B, C, D represent different shift teams).  Day represents the 0800 to 2030 shift schedule, and Night represents the 2000 to 0830 shift schedule.  There is a half-hour overlap between each shift.

12345678910111213141516
DayAAAACCCCBBBBDDDD
NightBBBBDDDDAAAACCCC
OffC,DC,DC,DC,DA,BA,BA,BA,BC,DC,DC,DC,DA,BA,BA,BA,B

2.2.3          On-call Shift Schedule Expectations

On-call shifts will be managed in a more simple fashion than 24×7 shift schedules.  On-call shifts will continue for 7 days, and will turn over at 0930 on Monday (or Tuesday if Monday is a holiday).

2.3              Process Output

The output of this process should be a 60 day schedule that looks like the following.  Note this schedule will include information about escalation or alternate presence to ensure that on-call teams are available o:

Date (Weekday)Day Shift 0800-2030Night Shift 2000-0830 (ends next day)On-call team members (Threat/Intel, Tooling Engineer)Remaining Team Presence
DD.MM.YY (Sun)Team A: L2 – NameA L1 – NameB L1 – NameCTeam D: L2 – NameD L1 – NameE L1 – NameFT/I – NameG TE – NameQL1 – NameH (Vaca) T/I – NameI (Off) TE – Name R, S, T (Avail) Team K, N (Off)
DD.MM.YY (Mon)Team A: L2 – NameA L1 – NameB L1 – NameCTeam D: L2 – NameD L1 – NameE L1 – NameFT/I – NameJ TE – NameRL1 – NameH (Vaca) T/I – NameG (Avail) TE – Name Q, S, T (Avail) Team K, N (Off)
DD.MM.YY (Tue)Team A: L2 – NameA L1 – NameB L1 – NameCTeam D: L2 – NameD L1 – NameE  T/I – NameJ TE – NameRL1 – NameH (Vaca) L1 – NameF (Vaca) T/I – NameG (Avail) TE – Name Q, S, T (Avail) Team K, N (Off)
DD.MM.YY (Wed)Team K: L2 – NameK L1 – NameL L1 – NameMTean N L2 – NameN L1 – NameO L1 – NamePT/I – NameJ TE – NameRL1 – NameH (Vaca) L1 – NameF (Vaca) T/I – NameG (Avail) TE – Name Q, S, T (Avail) Team A, D (Off)

3                  Shift Schedule Generation

The purpose of this section is to explain the process by which the 24×7 shift schedule is generated and distributed.

3.1              Team members provide input

At least 90 days prior to vacation, training event, or other significant life-event, GSOC team members must provide the following information to the GSOC Operations Manager:

  • Requests for specific vacation days off
  • Requests for days-off for specific training/professional development events, approved via the L3
  • (On-call members only) Identify weekend/holidays where the member has something personal scheduled (note:  this does not guarantee no on-call, but can be used to guide the schedule).

3.2              GSOC Operations Manager drafts schedule

At least 75 days prior to the proposed updated schedule, the GSOC Operations Manager drafts the schedule and releases it to the team for review.  This will happen every 2 weeks.

3.3              Draft schedule reviewed and feedback provided

Team members are responsible for reviewing and responding to the draft schedule within 7 days after release by the GSOC Operations Manager.  If their requests have not been accommodated in a way that they accept, this is their opportunity to escalate the issue.  Alternatively, they can use this time to identify potential shift swaps with coworkers and propose this to the GSOC Operations Manager via the L3.

3.4              Final schedule released

At least 60 days prior to any given workday schedule, the GSOC Operations Manager will release a final schedule to the team.  This updated schedule release will occur every 2 weeks.

3.5              Team members negotiate swaps

After final schedule release, the only way that a team member can change the schedule is by negotiating a swap with a fellow team member.  No more than 4 days prior to a proposal to swap a shift, team members must propose shift schedule changes to the GSOC Operations Manager via the L3.  The GSOC Operations Manager will accept or deny the proposed swap incorporating L3 recommendations and release an updated schedule no later than 2 days prior to the swapped shift.

The GSOC Operations Manager must ensure that these negotiated swaps are supported by both HR requirements and common sense.  Considerations:

  • Shift swaps should not result in members doing back-to-back shifts (i.e., day followed by night or vice versa)
  • Shift swaps should not result in members doing more than 6 shifts in a row.

3.6              Statistics Updated

This is an ongoing process executed by GSOC Operations Manager to ensure that there is an accurate collection of statistics regarding member shift staffing schedules (Section 3.3.1 refers).

4                  Shift Handover Process

Between shifts, the following guidelines apply to ensure that continuity is tracked across shifts.

4.1              24×7 Shift Hand-over

The following requirements apply to shift-hand over operations.

  • The responsibility for a successful 24×7 shift hand-over is a joint one, reflecting responsibility of both the oncoming and offgoing shift.
  • The expectation is that all shift hand-over processes will occur within a 30 minute period (between the periods of 0800 and 0830, and 2000 and 2030).
  • During shift hand-over, all open security incident tickets other than those held by the L3 will be passed to the following shift.
  • During shift hand-over, all security infrastructure-related issues or concerns (or open IT Incident tickets) will be reviewed by the off-going shift for the on-coming shift.
  • Off-going shift members are responsible for ensuring that they accurately convey key information about specific tickets and shift activity in general.
  • On-going shift members are responsible for arriving in time to begin the shift-handover, and to ask sufficient questions to ensure that they understand all information provided by the off-going shift members.

4.2              On-call Shift Hand-over

The on-call shift hand-over is necessarily less involved than the 24×7 shift handover.  The following requirements apply to on-call shift handover.

  • Any ongoing issues that the threat/intel analyst may need to track to support the coming week (i.e., questions or issues escalated by the previous week’s shift).
  • Review of any emergency or planned threat/intel coverage of L2 analyst shift gaps in the coming week.
  • Confirmation that the new on-call shift member understands that they are in an on-call status.

4.3              24×7 Handover mechanism

Shift hand-over will be conducted using the Archer SECOPS tool to facilitate handover.  Key components of this turnover:

  • L1’s and L2’s will conduct an incident-by-incident review of all security incidents that are currently assigned to an analyst for either triage or parked for investigation.  At the conclusion of discussion of every triage/parked incident, the security incident will be transferred to the coming L1 or L2.
  • The L1’s and L2’s will also complete the SECOPS shift-handover form to document completion of the shift-handover, and document discussion of key items/events during the turnover.
  • L2’s will conduct a review of all incidents in the remediation phase (i.e., which have already been passed to a member firm for remediation).  This will not necessarily require an in-depth review of every security incident, but will require security incident transfer to the oncoming L2.

5                  Exception Processes

5.1              General guidance

This exception process is intended to address any shift-absence issue that occurs with less than 96 hours of warning before a shift.  In general, there will be a best-effort to ensure that a shift-absence is covered by an on-call person.  However, it is recognized that in some cases (weather emergency, multiple absences, mid-shift emergency) this will not be possible.  In this case, it is the responsibility of the L2 (or L3, if the L2 is absent) to ensure that an appropriate decision is made which maximizes the effectiveness of the GSOC, while  minimizing risk to personnel.

5.2              L2 Absences

Ensuring that there is a qualified L2 analyst on shift is a top priority for the GSOC.  The (prioritized) options for filling a L2 Absence is as follows:

  • On-call L2 (Threat/Intel Analyst)
  • Off-call but available Threat/Intel Analyst.
  • L3 Analyst
  • Most-qualified available on-shift or otherwise available L1 Analyst

5.3              L1 Absences

L1 Absences can occasionally be allowed, but there must always be at least L1 Analyst per shift.  The (prioritized) options for filling a L1 Absence to ensure that there is always at least one L1 Analyst per shift is as follows:

  • Available L1 Analyst (assuming he/she is not in a leave status)
  • On-call L2 (Threat/Intel Analyst)
  • Off-call but available Threat/Intel Analyst

5.4              Contingencies

There will occasionally be unusual contingencies which should be addressed as follows:

  • Multiple Absence.  In the event that multiple simultaneous absences occur, there is the potential for violation of the requirements for GSOC staffing (Mininum x1 L2 and x1 L1).  In this case the GSOC Director or GSOC Operations Manager can authorize staffing below this level.
  • Business Continuity.  In the event that there is a business continuity event (such as a weather emergency) that will result in risk to members if they attempt to travel to/from the GSOC, the GSOC Director or GSOC Operations Manager can authorize decreased manning and/or execute business continuity process to authorize work-from home.
  • In-shift Emergency.  In the event that there is an in-shift emergency (i.e., shift member who feels sick or has a family emergency), the expected behaviour is for there to be an allowed gap.  With that said, the shift leader (L2) should discuss shift conditions with the L3 to confirm that an on-call doesn’t need to be brought in.
  • Work-from-home Options.  Work-from-home is not normally an approved option for meeting shift requirements.  However, the GSOC Director or GSOC Operations Manager can authorize this for emergency situations.
  • Employee Termination/Resignation.  In this case, the GSOC Operations Manager will be responsible for releasing an updated shift plan as soon as possible.  Short-notice (less than 96 working hours notice) terminations/resignation will be dealt with like any other short-notice absence.
  1. Constraints and Assumptions

The purpose of this appendix is to describe significant constraints and assumptions that are the key drivers for the design and content of this process.  The purpose of identifying these key constraints and assumptions is to ensure that when constraints change or assumptions are disproven, that the processes are examined to ensure that they still apply and are optimized for the goals of the GSOC.

Simple, Fixed Shift Schedule

The KPMG GSOC will have a fixed, simple shift schedule that requires shift personnel to work 4 days on, 4 days off, 4 nights on, and 4 nights off.  There will be very limited ability of the staff to negotiate alternative shift schedules.  The impact of this is the following:

  • Simplified shift planning
  • Less appeal to potential staff who need some flexibility in shift planning
  • Primarily fixed shift-team membership

Limited initial staffing

The initial staffing will be x9 L1 analysts, and x4 L2 analysts, to fill x2 L1 and x1 L2 24×7 positions.  This is a very tight staffing requirement, that will mean that shift analysts will work almost exclusively on shift.

Threat/Intel Analyst are considered backup L2

Due to the existence of only x4 L2 positions, this will mean that illness, training requirements, hiring gaps, etc., will need to be filled by positions other than the L2 analysts.  The Threat/Intel Analysts are expected to fill this role.

Limited L1 shift-size fluctuation allowed

In some cases, such as illness, vacation limitations, and the like, the shift schedule will be permitted to include only one L1 analyst.  L2 is expected to always have one analyst on shift.

GSOC Analysts will attend some periodic (KPMG, industry) training/events outside of GSOC schedule

There will be occasional shift-gaps that will be caused by GSOC Analysts taking some form of training, either required by KPMG or a part of their own professional development.

30 minute turnover (max 12:30 allowed of work)

The maximum shift turnover time will be 30 minutes.  This means that total normal shift time will not exceed 12 hours and 30 minutes of work.

No normal concept of “incident ownership”

At the end of a shift, all open security incidents will be turned over to the next shift.  This includes security incidents that are currently being triaged, and security incidents that are in a parked status awaiting investigation.  The only exception to this is security incidents that are currently held by the L3, which remained held by the L3.

Shift-turnover time is 8:00-8:30

Shift turnover will be begin at 8AM and 8PM, which means that the 30 minute turnover time will be complete by 8:30 AM or 8:30 PM.

On-call – 1st 15 min is free

On-call personnel are expected to be available to handle issues when not at the office.  Short calls to on-call personnel are considered “free”.  However, longer periods will be addressed by the GSOC Operations Manager with comp time to be negotiated with the employee.