A DNS Record That Existed Everywhere Except CI

A flaky CI failure where a freshly-created DNS record was unresolvable from inside the cluster, but resolved fine from my laptop a few minutes later. The culprit was negative DNS caching, driven by the zone’s SOA, poisoning lookups for our ephemeral per-job records.

June 30, 2026 · 7 min · thrix

How a Single Regex Stalled 30 Testing Farm Jobs for Hours

Tracing stuck Testing Farm jobs from process list through Python stack traces to a catastrophic regex backtracking bug in tmt — caused by a 1M-character line in test output.

March 6, 2026 · 5 min · thrix