Retry storms and quiet hours

We regularly see retry policies that fire every few minutes through the night because someone once lost a file to a flaky SFTP. The intent was kindness to the pipeline. The effect is a retry storm that keeps a brittle dependency warm while hiding the underlying defect.

Pattern analysis across job failures often reveals clusters at the same quiet hours — when vendor windows close, when warehouse slots are contended, or when a certificate rotates without ceremony. Treating each failure as unique burns on-call time and never retires the root cause.

A healthier posture: classify failures, cap concurrent retries per dependency, and quarantine poisoned inputs instead of hammering them. Application analytics makes those families visible; policy changes make them rarer.