- ChatGPT
- Prompt engineering
- Identify the cause of the problem
- 5 whys
- List potential solutions
- Investigate potential sources of similar problems
- Address the additional sources of risk
- Reduce incident duration
- Identify the cause of the problem more rapidly
- Reduce incident cost
- Reduce the number of people involved
- Define the incident owner
- Define the incident secretary/communicator
- Create and document
- Summary
- Observations (link to metrics dashboards with absolute timestamps as much as possible)
- Screenshots
- Who took the screenshot
- Link to get the graph/data
- Associated conclusions
- Links to logs
- Screenshots
- Hypotheses/theories
- Who made them
- When
- If they have been validated/invalidated
- The actions taken
- By whom
- If it had the desired effect
- etc.
- In the situation where an incident has been caused by the introduction of a code regression, revert the change and deploy as soon as possible
- Start by reducing/relieving the impact of the incident before searching for a root cause
- Use multiple data sources when data sources do not agree
- Diagram all the implicated systems and the relationship to one another in order to identify the potential locations where the problem might be
- Test your hypotheses to verify if they hold or not
- Develop a procedure over time that can be followed to diagnose similar issues
-
Write down a list of improvement suggestions in order for the incident not to reproduce itself in the future or to lessen its impact
- Once the incident is completed, have a summary of the conclusions at the top of the document with a link to the sections in the document explaining the rationale behind the conclusions
- GCP
- Traffic director
- Managing large scale services (500k+ rps)
- Imply
- Ads serving
- Ads mediation
- Grafana unified alerts
- Renovate
Every year, either at the beginning or end of the year.
A few hours spread over the course of a few days.
- Review year according to various facets
- Plan the next year according to the same facets
- I use a mind map software to do my yearly review and plan.