• Define the incident owner
• Define the incident secretary/communicator
• Create and document
  • Summary
  • Observations (link to metrics dashboards with absolute timestamps as much as possible)
    • Screenshots
      • Who took the screenshot
      • Link to get the graph/data
      • Associated conclusions
    • Links to logs
  • Hypotheses/theories
    • Who made them
    • When
    • If they have been validated/invalidated
  • The actions taken
    • By whom
    • If it had the desired effect
  • etc.
• In the situation where an incident has been caused by the introduction of a code regression, revert the change and deploy as soon as possible
• Start by reducing/relieving the impact of the incident before searching for a root cause
• Use multiple data sources when data sources do not agree
• Diagram all the implicated systems and the relationship to one another in order to identify the potential locations where the problem might be
• Test your hypotheses to verify if they hold or not
• Develop a procedure over time that can be followed to diagnose similar issues

• Write down a list of improvement suggestions in order for the incident not to reproduce itself in the future or to lessen its impact

• Once the incident is completed, have a summary of the conclusions at the top of the document with a link to the sections in the document explaining the rationale behind the conclusions

• GCP
  • Traffic director
• Managing large scale services (500k+ rps)
• Imply
• Ads serving
• Ads mediation
• Grafana unified alerts
• Renovate

Trigger

Every month, either at the beginning or end of the month.

Duration

30 minutes.

Steps

• Review what was planned for the month
• Provide feedback related to the plan
• Review yearly plan and align
• Plan next month

Notes

• I use a text editor, such as VS Code, to write my monthly reviews.

Trigger

Every year, either at the beginning or end of the year.

Duration

A few hours spread over the course of a few days.

Steps

• Review year according to various facets
• Plan the next year according to the same facets

Notes

• I use a mind map software to do my yearly review and plan.

References

• https://alexvermeer.com/8760hours/
• https://chrisguillebeau.com/how-to-conduct-your-own-annual-review
• https://www.anthonyjcampbell.com/blog/8760-hours

• Chinese Mandarin
• Scikit-learn
• Time series