A CI clone spike pulled terabytes off an internal git server and caused an outage. The cause was one URL pointing at git’s dumb HTTP protocol, which makes every fresh clone re-fetch a repo object by object.
A flaky CI failure where a freshly-created DNS record was unresolvable from inside the cluster, but resolved fine from my laptop a few minutes later. The culprit was negative DNS caching, driven by the zone’s SOA, poisoning lookups for our ephemeral per-job records.
Tracing stuck Testing Farm jobs from process list through Python stack traces to a catastrophic regex backtracking bug in tmt — caused by a 1M-character line in test output.