1. Assessing Risk Process -Risk Impact and Risk Scales
1.1. 1
1.1.1. low
1.1.1.1. Does not bring down any services
1.2. 2
1.3. 3
1.3.1. med
1.3.1.1. Devices serve other networks
1.3.1.1.1. indirect impact
1.4. 4
1.5. 5
1.5.1. high
1.6. Criteria
1.6.1. serviceability
1.6.1.1. Robot/Website client applications Downtime
1.6.2. Business Risks
1.6.2.1. reputation
1.6.2.2. cost
1.6.2.3. how long an outage downtime is
1.6.3. Scope of Impact to Applications/Servers
1.6.3.1. Database maintenace-200(affect developers only vs external customers) servers vs 10(has med. devices on it)
1.6.3.1.1. highest to external customer impact
1.6.4. low risk for cosmetic changes
2. Notification
2.1. notification change process
2.1.1. realtime data
2.1.1.1. inventory database
2.1.1.1.1. dev system
2.1.1.1.2. Infrastructure inventory system
2.1.1.2. Configuration Management
2.1.1.2.1. CMDB
2.2. email
2.3. What gets notified
2.3.1. affects HW/SW environment
2.4. dashboard
2.5. status meetings
2.5.1. Network Status provided(included emergency)
2.6. JIra
2.7. ServiceNow
2.7.1. inventory management module
2.7.2. Stakeholder Calendar reminder invite
2.7.2.1. defines stakeholder impact activities
2.7.2.2. pre-notification prior to change approval
2.7.3. dashboard
2.7.3.1. Slack
2.7.3.1.1. logs change event
2.7.3.2. integrate Victor Ops
2.7.3.2.1. future change
2.7.4. Calendar dashboard outlook
2.8. Confluence
2.9. Centralized Location
2.9.1. Network changes
2.9.2. Database Changes
2.9.3. Development Changes
3. Non FDA regulated quick changes
3.1. Networking Shared process
3.1.1. turn off switch Port
3.1.2. Open Firewall
3.1.3. VIP Builds
3.2. removing processes- in SNow
3.3. streamlined Intake Form-Snow
3.4. Quick streamlined non approval changes
3.4.1. out of scope for QMS changes and product changes, UMA-Part 11
3.5. inscope SNowe
3.5.1. add more servers to existing server pool
3.5.1.1. DVMT supporting apps
3.5.1.1.1. Supporting applications -adding additional resources to a server pool (allready exist and tested) - exDVMT Parsers,
3.5.2. Risk Impact (1) minimal or non
3.6. out of scope-
3.6.1. emergency change
4. Reporting Process
4.1. Status tool
4.1.1. integrated to Slack
4.2. summarize notifications
4.3. Communication
4.3.1. for unforeseen needed support
4.3.2. Awareness of Ongoing/Current Issues
4.4. data for process improvements
4.5. Tracking issues closure
4.6. Data for Service Performance
4.7. Reporting for On time delivery
4.8. Snow Reporting Dashboard
4.8.1. one dashboard
4.8.1.1. different tabs
4.8.1.1.1. ongoing changes
4.9. Snow Reporting Email Notifications
5. Change Review Board Process
5.1. deny
5.2. approve
5.3. hold
5.4. request more info
5.5. Representative Qurum
5.6. Central location of upcoming change
5.7. CAB Required?
5.7.1. filter-clarify request, questions answered, ownership, make additional changes missed in UAT , fillout Intake form or elaborate the intake form
5.7.1.1. Move to CAB after approval
5.7.1.1.1. Review and Approve prior to CAB meeting if no concerns - (task(s) in Snow)
5.7.1.1.2. concerns
5.7.1.2. Time limit to prepare for CAB mtg
5.7.2. Risk Assessed, Rollback Plan required
5.7.2.1. Confluence
5.7.2.1.1. risk assessment finalized? SNow Risk Assessment approved
5.7.3. fast streamliined process
5.7.3.1. Change IP address
5.7.3.2. Update DNS- no hardcoded names or IP addresses
5.7.3.3. no cascading affect
5.7.3.4. add additional resources
5.7.3.4.1. double servers
5.7.3.5. turn on Network Switch Port
5.7.3.6. Open Firewall
5.7.3.7. Build F5 VIP
5.7.3.8. open ended addition field
6. Triage Process
6.1. prioritization
6.1.1. highest priority and emergency request
6.1.1.1. Risk Impact
6.1.1.1.1. Patient impact
6.1.1.2. Business Priority
6.1.1.2.1. CIP goals
6.1.1.3. highest cost
6.1.1.4. Management Priority
6.1.1.5. No Last minute requests
6.1.1.5.1. Zoom chat
6.1.1.5.2. notifications
6.1.1.6. Developer Access
6.1.1.6.1. issues known latter
6.1.2. Emergency Criteria
6.1.2.1. Patient Impact
6.1.2.2. SLA not met
6.1.2.3. Loosing $
6.1.2.4. Disaster Recovery risks not met
6.1.2.5. critical application
6.1.2.5.1. i.e. UMA
6.1.2.5.2. Core Services
6.1.3. Business Criteria
6.1.3.1. Cost
6.1.3.2. reputation
6.1.4. group responsibility
6.1.4.1. Marketing
6.1.4.1.1. uptime
6.1.4.1.2. ability to recover
6.1.4.2. Product team
6.1.4.3. Engineering team
6.1.4.4. Master Contacts List
6.1.4.4.1. Org/role/responsible
6.1.5. Product & Functional Requirements
6.1.5.1. monitoring requirments
6.1.5.1.1. in confluece
6.1.5.1.2. ACCS
6.1.6. Alert/Notification Tools
6.1.6.1. Victor Ops
6.1.6.1.1. Phone Alert - 24/7
6.1.6.1.2. SRE/OPS Manager Configures workflows
6.1.6.1.3. Gathers information from multiple sources to one central location and sends only 1 alert
6.1.6.1.4. Sends alerts to different roles, priorities, different times, and escalation path
6.1.6.1.5. API calls to Nagios, splunk, Network too
6.1.6.2. Nagios
6.1.6.2.1. thresholds
6.1.6.3. Splunk
6.1.6.4. Stealth Watch
6.1.6.5. Network tools
6.1.6.5.1. Netbrain
6.1.6.5.2. Orion
6.1.6.6. Cloud Tools
6.1.6.6.1. AWS Service Health Dashboard
6.1.6.6.2. Azure Tools
6.1.6.6.3. Not On-Prem- Managed Service
6.1.6.7. link to corporate emails, cell phones- sends alerts as emails, text alert, phone call
6.2. Responsibility Role (dedicated or assigned on a rotation)
6.3. Used Across OPS org
6.3.1. any team working on issues and service chagnes
6.4. Assigned Based on Expertise
6.4.1. Software
6.4.2. System
6.4.3. Change Type
6.4.3.1. Errors
6.4.3.2. Bug fixes
6.4.3.3. Correcxtions
6.4.4. Level of Outage based on level of ability and authority
6.4.4.1. Level 1-Oncall engineer
6.4.4.2. Level 2-Senior Eng to Vet
6.4.4.3. Level 3-OPS Manager to Vet
6.4.4.3.1. Pools in Outside resources
6.5. Authority to Make Changes
7. Snow Jira Workflow
7.1. Production Change?
7.1.1. fill out Requirements Form/Intake Form
7.1.1.1. Streamlined Process
7.1.1.2. Assigned to CAB for review
7.1.1.2.1. CAB Approval Task & Notification
7.1.2. Owner Manager approval task to move into CAB
7.2. Dev/UAT-Assigned to Engineer
7.2.1. existing Jira & Snow ticket
7.2.2. apply best practice guidline