The Bug That Only Showed Up When Nobody Was Watching
2026-07-09A two-second dashboard refresh, a virtual machine going idle, and a decade of accumulated scheduler code conspired to freeze a dual-Xeon server at random. The hardest part wasn't finding the bug. It was building an instrument that could see it happen at all.
00The trace
This is the artifact the entire investigation turned on: a kernel stall dump caught live, on the third recurrence, after two failed attempts to catch it any other way.
Two CPUs, two stack traces, one lock. CPU 0 is a VM's virtual processor going idle. CPU 25 is the kernel's own memory-reclamation housekeeping thread. Neither ever gets the lock back.
01The symptom
Tower is a dual-socket Xeon NAS running Unraid, hosting a few dozen Docker containers and three KVM virtual machines. It would, every so often, simply stop. Not crash: stop. The web GUI died. SSH died. Ping stopped returning. The one thing that kept answering was the motherboard's remote management console. That meant the host OS was locking up solid, not a power event or a hardware fault.
There was no pattern anyone could see. It happened with containers running and without them, with the array under heavy load and while mostly idle, once during a routine parity check. Diagnostics captured after each incident showed nothing. Every diagnostic snapshot only ever captured the tail end of a clean shutdown sequence, never the freeze itself.
02Instrument zero
Before any of this could be diagnosed, the diagnostics themselves needed fixing. Three separate blind spots had been quietly swallowing evidence, in some cases for months.
Blind spot — logging
Unraid's "mirror syslog to flash" feature was silently broken. Its config
generator referenced a template that was never actually written into
rsyslog.conf, so every debug-level log line that was supposed to
survive a reboot got quietly discarded instead. Every previous freeze had been
logged into a void.
Blind spot — hardware events
The BMC's event log was completely full: 512 of 512 entries, static since the prior year. Any new thermal, power, or watchdog event was being dropped on the floor rather than recorded.
Blind spot — memory
This board has ECC RAM, but the kernel module that reports correctable and uncorrectable memory errors had never been loaded. Zero visibility into a classic cause of exactly this symptom.
All three got fixed before the investigation could really begin. Persistent logging restored, event log cleared, ECC reporting loaded. If a system can't produce evidence of its own failure, it's not ready to be debugged, regardless of how sharp the hypothesis is.
03The isolation, and its red herrings
With real instruments in place, the plan was methodical: strip the system down to bare metal, then add layers back one at a time, watching for a recurrence at each stage.
A full parity check ran end to end under this configuration with zero anomalies. This turned out, much later, to be the single most important negative result in the whole investigation. Nobody knew it yet.
VMs, Docker, roughly sixty containers including GPU-accelerated ones, all running together. It froze. The persistent logs, working for the first time, showed something concrete just before it: a fan had flapped, and a watchdog script's own recovery routine hammered the motherboard's management chip with reset commands four times in seven minutes, visibly confusing the host's hardware management channel each time.
A real bug, and it got fixed: the watchdog got rate-limited so a flapping fan can't loop forever. But it was a coincidence, not the cause.
Same rough timing, but with the GPU-accelerated containers deliberately left off. It froze anyway. This time it left no trace: no fan event, no hardware log entry, no kernel warning. Just silence, then a reboot.
The leading theory at this point was Docker's networking layer. The container network mode in use has a well-documented history of kernel-level hangs under heavy churn. It was a reasonable theory. It was also wrong, and chasing it burned real time. The rabbit hole did turn up something worth knowing: the container network's driver setting is silently unenforceable for this specific network type, a genuine platform quirk.
This time there was a live console session open. Network connectivity died first. The freeze followed a few seconds later. That ordering, combined with a remote console that stayed responsive throughout, pointed hard at a kernel-internal lockup rather than anything at the network card or hardware level. And this time, the persistent logs caught the actual kernel stall dump shown above.
04Reading the trace
Stack traces read backwards. The bottom is where things started, the top is where they got stuck. Read that way, the two backtraces tell a precise story.
kvm_vcpu_halt()
A VM's CPU has nothing to do and goes idle. This happens constantly, on every VM, all the time.
i915_pmu_event_stop()
The scheduler's routine context-switch housekeeping stops a performance-counter event registered against the GPU driver.
guc_engine_busyness()
Stopping that counter calls the GPU driver's internal engine-busy accounting code.
__queue_work()
That accounting code tries to schedule a small piece of follow-up work on a kernel worker thread.
raw_spin_rq_lock_nested()
Waking the worker requires the scheduler's run-queue lock. Practically everything in the kernel eventually needs this lock.
native_queued_spin_lock_slowpath()
The lock never comes free. The CPU spins on it forever.
Once one CPU is stuck spinning on that lock indefinitely, it's only a matter of time before something else needs it too. In this case the kernel's own memory-reclamation housekeeping thread, on a second CPU, is what tripped the stall detector and produced the diagnostic dump. From there it cascades: scheduling breaks down for anything sharing that lock, including the softirqs that handle network packets. That's why the network died first and the rest of the system followed seconds later.
No panic. No oops. No error message anywhere. Nothing actually crashed. The kernel simply stopped making forward progress on one lock, and eventually on everything downstream of it. That's precisely why two earlier freezes left no trace at all. This class of failure is silent by construction.
05The trigger
The trace explains the mechanism, but a VM going idle happens thousands of times a day without incident. The missing piece was why a GPU performance-counter event was active at that exact moment.
The answer was a dashboard widget. An Unraid plugin I installed just to show GPU utilization on the web dashboard. It refreshes every two seconds while the page is open, and it does so by opening a low-level performance-monitoring handle directly into the GPU driver each cycle. That handle is the second half of the collision: a VM idling while the handle happened to be open was the whole trigger. Neither alone was a problem. Unraid's own built-in stats plugin never touches the GPU at all. I checked.
Small monitoring tools have a long history of triggering large failures. Not because the tool is badly written, but because it's often the only thing in the entire system regularly exercising some obscure, rarely-touched corner of a driver or kernel subsystem. Everyone else uses the GPU to decode video. Almost nobody opens a raw performance counter into it every two seconds while a hypervisor is also running. That narrow intersection is exactly where a latent race condition survives undetected. Upstream test suites don't reliably cover it either. This general area of the driver has a real, documented history of similar races, with fixes landing as recently as a few kernel versions before the one running here. This specific interaction didn't appear to be one of the already-known ones.
06The fix, and the proof
Resolved. Both GPU-monitoring plugins were removed outright. Everything else went back: all three VMs, the full container set, actual real-world use including a live video transcode. Nothing was held back this time. The mechanism no longer depended on container count or GPU workload, only on whether that one perf-counter handle was ever open.
Eight-plus hours in, with real usage rather than synthetic waiting: uptime holding, zero memory errors, zero hardware events, zero kernel warnings. Every earlier "clean" run is retroactively explained rather than contradicted. The 24-hour bare-metal test especially: no VMs running means the lock chain above never had a first domino to knock over, regardless of anything else happening on the box.
07What it actually took
- Fix your instruments before trusting your hypotheses. Every theory formed before persistent logging actually worked was built on incomplete evidence, no matter how sound the reasoning looked at the time.
- A clean negative result is still data. The 24-hour bare-metal run looked, for most of the investigation, like it had ruled nothing in or out. It had actually already pointed at the answer. It just took two more freezes and a lucky live observation to read it correctly.
- Silence is a symptom. A lockup that never panics will never explain itself in a log file. The only way to see it is to catch the kernel's stall detector mid-fire. That needs a system watched closely enough, for long enough, in the right place.
- The smallest tool in the room is not automatically the safest. A two-second dashboard refresh has no business bringing down a server. It only does so by being the one thing regularly walking into a room nobody else visits.
Kernel trace preserved for reference and possible upstream report · hardware: SuperCloud CS13000 / X11DPH-i, BIOS 4.7 · GPU: Intel DG2/G11 [Arc A380]
</post>