1. Hardening
1.1. Hardening Test cycles needs better management/ownership/direction. There needs to be more investment in improving the tests and removing duplication. In some cases there are 4 tests that can be combined into 1.
1.1.1. Test case maint is an ongoing requirement for everyone
1.1.2. Formal test plan to be created - already being created by QA folks
1.2. Was difficult to QA new features until both back-end and front-end stories were complete.
1.3. Too reliant on manual test cases. Need to automate manual test cases to reduce what's left to do in Hardening.
1.4. We haven’t got well defined what template can go with which AC and delivery
1.5. QA team members did less test than expected as we had a lot of questions on test cases
1.6. Digital Delivery Integration issues held up hardening testing
1.6.1. Need to focus on DD and EOV integration testing in H release
1.6.1.1. Enigma team are looking to integrate with EOV early and often in H release
2. QA
2.1. Quality Owners were a help
2.2. Functional/Service strategy worked well
2.3. Test case maintenance an ongoing requirement
2.4. When CICD fails teams should be able to manually install the build so that testing isn't blocked
2.4.1. If they are issues with deployments on new builds on Team City, we can build deployment choosing build and scripts version – we have it documented on confluence.
2.5. Automation needs more attention
2.5.1. All teams have objective to do automation in H
2.6. Lost a number of QA resource during Galaxy. QA capacity further reduced when QAs are also Scrum Masters which accounts for 20-25% of their time.
2.7. Move to a new Vault instance in hardening was not picked up by all QA. Kept sending jobs to old instance and could not see them in EO Archive displays
2.8. iBeta contractors / Rebecca performed a massive amount of testing
2.8.1. Develop a plan to do continuous regression testing
3. Documentation
3.1. Process improved - Installation guide on Confluence useful
3.2. Having one place on confluence for all installation documentation was very good idea.
3.2.1. Need to build on the processes that were adopted in H to put into Confluence as our knowledge base.
3.2.2. Scrum masters need to work to make sure that we record decisions and relevant notes in Confluence.
3.2.3. Stephen to ask GES to speed up confluence
3.2.4. Stephen: Alignment call / SOS: New pages in confluence.
4. Performance Analysis
4.1. We're getting numbers / tools under our belt
4.2. ACM performance testing was pushed into hardening because of other priorities. This was risky, but lucking no changes were required.
4.2.1. Performance testing needs to be baked into the original estimation.
5. Digital Delivery (eMessaging)
5.1. Quite a lot of confusion to resolve integration
5.1.1. Waited too long to start integrating
5.1.1.1. Need to integrate early and make sure that we have good documentation.
5.1.1.2. Need to automate the deployment and integration
5.1.1.2.1. DD team is starting to work in AWS - this should help
5.2. Integration documentation was out of date and incomplete.
5.3. It wasn’t clear which team should be responsible for which integration part (for example on vendors).
6. PdM
6.1. Need to manage expectations in the field. When ACM phase 1 was delivered it got negative feedback because the implementation did not go far enough.
6.1.1. Pdm to make sure that all feedback - good and bad are passed back into the team
6.1.2. Stephen to add section to alignment call on customer feedback on the previous release features
6.2. Need to attend sprint demos more and provide feedback on features as they are being developed
6.2.1. Product to work on comms to enablement group.
6.2.2. Product to work to give more feedback in demos and to ensure that the team attends the demo.
6.2.3. Terry to start looking to have customer feedback into the sprints.
6.2.4. Stephen to reinforce to the demo attendees to give feedback
7. Release
7.1. Planning
7.1.1. Potential release content too little understood at time of release planning
7.1.2. Too many people involved
7.1.2.1. Making sure that no matter the number of people involved there is a single decision maker.
7.1.2.2. Clear roles needed in release ceremonies
7.1.3. Preparation for the next release started in Galaxy
7.1.3.1. Product backlog grooming sessions needed through the release
7.2. Stand-ups
7.2.1. Get the right people involved in triage, I'm not sure the scrum masters have the correct knowledge or are able to make the correct decisions.
7.2.2. Not enough focus on customer and usability when it comes to bugs raised at the end of the release.
7.2.2.1. Customer impact is not given high enough value in the triage process.
7.2.2.2. PdM are working to make sure that we dont focus on single customers over the needs of the market.
7.2.2.3. PdM are working to make sure that they are talking to more customers. Looking to restart user groups sessions.
7.2.3. Stand-ups could be more effective if attendees came to the meeting more prepared
7.3. Creating release deliverables
7.3.1. Still a multiple day effort
7.3.1.1. Bill Horst is looking at the process and to make sure it is a single day process. That could be via docs or changes,
7.3.2. Release artifacts in Nexus were cached. Required a reset
8. UX
8.1. UX needs to be ready prior to development starting on stories. The correct level of detail required needs to be agreed
8.1.1. Ownership TBD
8.1.2. UX is interviewing customers at the moment - updates to follow.
9. Retro next steps
9.1. Stephen to send out survey on this retro format
9.2. Stephen to collate owned action items - and we will report back on them mid release
9.3. Non-owned action items are up for people to volunteer to own.
10. Teams
10.1. Spectre (Henley) and Procodia (Poland) worked well together
10.2. Numbers dwindling in UK are not replaced, meaning that we have one less team in next release.
10.3. Enigma and Front-End teams worked well together. Ron really helped us when he joined.
10.4. New feature development would have been easier for Enigma if we could also make Front-End changes.
10.5. Good cooperation between Procodia and Spectre.
10.6. Procodia was always prepared to demo however some demo sessions was cancelled.
10.6.1. Stephen to stop cancelling the demo when there is less than 10 minutes of content.
10.6.2. Teams should always show work on completed stories even if the demo is just a set of points on a slide.
10.6.2.1. In the same way the demo should remain relevant
10.6.2.2. Teams dont need to prove that they have done the work with long demos. Talking to a single slide is ok.
10.7. Integration between vanorama and specodia very smooth
11. Backlog Grooming
11.1. Hornet Release
11.1.1. Time needs to be planned for getting next release backlog ready
11.1.2. Time needs to be allocated for spiking for I release before H ends
11.2. Galaxy Release
11.2.1. Grooming the backlog every sprint help get us ready for sprint planning
12. AWS
12.1. Excellent set of tools, well-used
12.2. Still have a mindset of holding onto VMs, resulting in unnecessary costs. Need to be more conscious of costs and delete what's not actively in use
12.2.1. Do we need more training on the use of AWS?
12.2.1.1. Terry / Bill working on training
12.3. Infrastructure made it easy to test installation of NDA
12.4. Use of AWS has made CICD reliable and valuable - no longer have various systems (UrbanCode + CloudForms) tied together in a fragile way
12.4.1. We have created a best practice system that we are starting to educate the rest of the business about.
13. Vault
13.1. Job delivery to shared Vault instance is done using RDP with cut and pasted to the Vault hot folder. This is very prone to interruption by someone else logging into the shared Vault system.
14. NDA
14.1. Dan Sullivan led NDA very well, made it easy for the teams to focus on what was required and track progress
14.2. Work was well-coordinated across multiple teams
14.3. Incremental approach to moving components into NDA worked well. Led to early merge of NDA branch being possible.
14.4. Developer install and test process much improved
14.5. We have EO installation process documented on confluence so if there are problems with deployments on CI/CD anyone can install it manually
14.6. Dan divided NDA work to consumable parts and he was always answering questions very quickly.
14.6.1. Good example of how having good backlogs / defined sprints can make a project go well
15. Sprints
15.1. There needs to be more involvement from PdM in sprint deliverables. Use the software at the end of the sprint to get agreement that what has been produced matches the requirements
15.1.1. More involvement in the demo
15.1.2. Direct customer input into the demo
15.1.2.1. Terry is investigating a way to demo to customers that does not stifle conversation in a demo.
15.1.2.2. The detail in the team demo may not be consumable for customers. Ongoing monthly enablement that can be recorded for customers.
15.1.3. More PdM involvement in the validation of the release deliverables
15.1.4. Action: Scrum masters need to be open enough to ask PdM when they feel that something is missing.
15.1.4.1. Team can make sure they close the loop on the requirements with the product owner. The demo should not be the first time that the PO sees the software.
15.1.4.2. One suggestion is to have a feature demo session for the PO as things are completed before the end of the sprint.
15.2. Enigma managed to balance release stories with Portrait support commitments.
15.2.1. In the short term the team will continue this pattern
15.3. Phoenix team achieved Galaxy commitments alongside assisting the maintenance team with escalated items
15.3.1. Keep working as a team to balance our commitments
15.3.2. We need to stop putting ourselves in the position where we need to make heroic efforts
15.4. Retros
15.4.1. Teams don't seem to have the freedom to fully go into details about what's not working. This maybe because once it's repeated a few times and nothing happens, there is no point in discussing it further. Teams don't want to blame others.
15.4.1.1. ROR: Not everyone is completely honest in ROR
15.4.1.2. We need to keep bringing the issues up. - Yaakov to try to be more responsive
15.4.1.3. Technical road bumps have been slow to be tackled - eg the merging of the source code trees in SVN
15.4.1.3.1. Bill Horst should start to work through CI/CI items
15.4.1.3.2. Tech Debt has been included in the H planning
15.4.1.4. Teams don't feel empowered to work on some issues
15.4.1.4.1. Need to help empower teams to take things on
15.4.1.4.2. Mgmt team: Need to discuss action items
15.4.1.5. Scrum Masters: Dont continue to suffer in silence.
15.4.1.5.1. Speak to your line manager
15.4.1.5.2. Email Yaakov with email titled "you are not listening"
15.4.2. Ensure scrum masters feed back information for other team retros to enable learning
15.4.2.1. Scrum masters need to feed information up and down
15.4.2.1.1. Not just forwarding on meeting notes
15.4.2.1.2. SM role is to keep the team informed as well as to communicate information from the team outwards
15.4.2.1.3. Do we need to go back to bi-monthly development team meetings?
16. Key
16.1. Start Doing
16.1.1. Action / Owner: ?
16.2. Keep Doing
16.3. Stop Doing
17. Launch enablement
17.1. Did a better job of preparing Services and Pre-sales for receipt of the release
17.1.1. Good job done by the team.
17.1.2. Roger Pilc has been talking about the great job done by team
17.1.3. Stephen: Build 4.4 enablement plan early.
17.1.4. Good work from PdM and Shirley
17.1.5. Recordings from mid release demos to build the enablement