Data Engineering on Azure by Vlad Riscutia

Author:Vlad Riscutia [Vlad Riscutia]
Language: eng
Format: epub
Publisher: Manning Publications
Published: 2021-08-16T16:00:00+00:00

6.3.4 Postmortems

Because we are talking about data movement at scale, it is likely that multiple pipelines will fail during any period of time due to transient issues, upstream issues, scaling issues, etc. The recommendation is to periodically look at the top offenders : ETL pipelines generating the most incidents, even if transient. Aim to understand what is causing the issues and fix them as needed to keep the overall Azure Data Factory healthy.

For example, our pipeline ingesting website telemetry might start perfectly fine, but as traffic to the website increases and logs grow in size, the pipeline might start glitching. First, we start getting a few timeouts. Over time, the pipeline more often than not times out. While still somewhat workable, each run now requires multiple retries and sometimes manual intervention. This would be a clear sign that the original implementation no longer scales, and the pipeline needs to be optimized. A core practice for an SRE is a process for incident postmortems.

Definition An incident postmortem brings teams together to take a deep look at an incident to understand what happened, why it happened, and how the issue can be prevented in the future.

We won’t talk too much about how to run postmortems. You’ll find many other resources focused on operational best practices. The only thing we want to highlight is that SRE teams usually have a recurring meeting where they do postmortems for major incidents. Rolling up top offenders into the postmortem process ensures that these issues become visible and that these don’t keep adding up until our whole ETL becomes unmanageable.

Download

Copyright Disclaimer:
This site does not store any files on its server. We only index and link to content provided by other sites. Please contact the content providers to delete copyright contents if any and email us, we'll remove relevant links or contents immediately.