Skywire Development Update — September 29, 2026

The twenty-ninth was almost entirely about load that had been quietly accumulating on the deployment services as the fleet grew — transport discovery rewriting data in Redis it already held, a CXO feed loop that could deadlock under enough concurrent subscribers, and autoconnect downloading the entire network’s transport list on every single cycle just to pick a few peers. Profiling on live production hosts turned up roughly a dozen separate places where the same fact was being recomputed, re-read or re-written far more often than it could possibly have changed, and each one got its own targeted fix. The address resolver gained a real per-transport-type reachability feed to replace a confirmation-only binding that never actually told autoconnect who could be reached. None of this changes what the network does; all of it changes how much work proving that costs.

Skywire: A Deadlock in the CXO Feeds Loop

5256 found the most serious of the day’s bugs directly from a production incident: deleting a connection from a feed’s head sent on a channel the same event loop could already be blocked waiting to drain, from inside its own broadcast path, so the two sides could freeze each other permanently — observed on one host as thirty thousand parked goroutines, over a gigabyte of heap against a 256-megabyte limit, and eighty percent of CPU spent in garbage collection trying to keep up with all of it. The fix drains the same queue the broadcast path already drains while it waits, closing the exact cycle a new regression test reproduces directly.

5272 fixed a narrower race in the same subscription path: a publisher answers a subscribe request and immediately pushes its last known root, but the subscriber only registered itself to actually receive that push after the acknowledgment came back, so the push could arrive, find nobody listening yet, and simply be dropped — leaving that subscriber waiting for the next heartbeat, up to forty-five seconds later, for data it should have had immediately. 5265 found a subscriber’s entire first fill being thrown away and restarted from nothing whenever it took longer than its own first-sync timeout to complete, which is exactly backwards for a large feed that legitimately needs more than forty-five seconds the first time; it now simply keeps waiting on the fill already in progress; instead of discarding it.

5259 found a reconciliation pass checking whether a connection was already subscribed to a feed by walking every feed on the entire node to find that one connection’s entry, which on a node holding thousands of visor feeds was costing roughly a sixth of one production host’s total CPU, all of it serialized through the single event loop every other connect and subscribe has to wait on; a direct two-lookup check answers the identical question in a fraction of the time.

5257 found a cleanup routine that walked every sequence number from a feed’s newest down to zero on every single pass, regardless of how many of those sequences the feed’s storage actually still held, so a feed republishing every forty-five seconds accumulated roughly nineteen hundred pointless lookups a day — measured as a third of one production host’s CPU spent on deletes that found nothing to delete.

Skywire: Transport Discovery Stops Rewriting What It Already Knows

5269 found transport discovery applying a visor’s routine forty-five-second heartbeat as if it were brand new data every single time, even though the actual tree it describes is content-addressed and a heartbeat that changed nothing produces the identical hash as the last one; a root that repeats its reporter’s last tree is now skipped entirely except once every ninety seconds, just often enough to keep registrations and uptime heartbeats from expiring.

5268 found the deeper version of the same problem: every heartbeat carries every one of a visor’s telemetry shards regardless of which ones actually changed, so with both ends of roughly ninety thousand transports reporting, the service was reapplying the same unchanged numbers roughly four thousand times a second — now an unchanged shard applies once, with the uptime heartbeat it still needs to keep alive capped at once every ninety seconds per transport rather than once per ingest, which had previously been inflating a live transport’s uptime count by two to four times over.

5275 found that fix still left both edges of roughly eighty thousand transports reporting their byte counters on every forty-five-second keepalive, which the earlier dedup couldn’t skip since those counters genuinely do change constantly; since the counters are cumulative, applying fewer of the identical reports loses no data at all, so latency now keeps only the latest reading and throughput the peak over whatever reports were held between applies. 5274 found a sixty-second metrics tick re-reading every single transport’s stored figures regardless of whether they had moved, measured on one production host at over a hundred thousand redis reads per tick; now only the transports actually marked as changed get re-read. The same fix found every service restart re-aggregating a full thirty-day window from scratch, costing nearly five million reads in forty-five seconds after a single deploy — a day’s totals are now saved once that day settles and simply loaded back on the next startup.

5260 found the metrics feed re-reading its entire thirty-day window every half hour and the daily stats re-reading every five minutes, both scanning days whose figures could not possibly have changed anymore; only genuinely open days — today, and briefly yesterday right after UTC midnight — are read repeatedly now, with the rest read once at startup and then left alone.

5261 replaced a network-wide daily total that had been recomputed by reading every single transport’s daily record back out of storage with one atomic script that updates the running total directly as each report arrives, so the total is exact without ever having to be read back and summed. 5264 found latency and throughput records for a transport being kept for thirty-five days regardless of whether that transport still existed, which accounted for roughly one in five of production’s 5.4 million Redis keys describing transports that had been gone for weeks; those records now expire thirty minutes after a transport’s last report instead.

5271 found three separate code paths each independently writing a transport’s heartbeat on their own schedule, together costing a write roughly every twenty seconds per live transport; a day’s uptime is now simply the fraction of its 288 five-minute slots that got at least one heartbeat, exact regardless of how many paths happen to report in any given slot.

Skywire: A Routing Feed That Ships Only What Changed

5262 found the feed visors actually route on built from a version of the transport list that carried no latency or throughput figures at all, even though route calculation reads both — and separately found it quietly publishing two nineteen-megabyte variants every single minute that differed only in whether they included self-loops, each fetched in full by every subscriber.

5263 replaced that whole feed for routing purposes with one that ships one leaf per visor and sends a tick that touches only the shards that actually changed, rather than the entire multi-megabyte snapshot regardless of how much of it moved. 5266 then returned the original all-transports feed to carrying topology alone now that metrics live in the new routing feed, keeping it only for subscribers that predate the split. 5267 found autoconnect downloading that entire multi-megabyte transport list on every single cycle merely to derive two small sets of candidate peers, when one of those sets is already exactly what the address resolver’s own bindings describe and the other fits in a small, rarely-changing summary leaf; autoconnect now reads the small, purpose-built data instead of the whole network graph every time.

Skywire: Knowing Who Can Actually Be Reached

5270 gave autoconnect something more useful than a binding that only says a visor is listening for a transport type: a new address-resolver feed that actually dials each bound stcpr, squicr or stcp address back and separately checks whether a sudph-capable visor holds the UDP control connection a hole-punch needs, ranking peers by when they last confirmed a real inbound connection of each type without ever gating on it. 5273 found that reachability probing taking roughly half an hour per round, almost entirely spent waiting out a five-second timeout against NAT’d visors that were never going to answer in the first place, with every service restart discarding the in-memory bookkeeping and starting that half hour over; widening the worker pool from eight to sixty-four brought a full round down to about three and a half minutes.

Skywire: Shared Plumbing Across the Deployment Services

5258 cleared out years of drift across the five deployment services’ own command-line flags, which had quietly accumulated two different pprof flag spellings, inconsistent shorthand letters, and a config-file merge that silently dropped several settings for one of the five services specifically; all five now bind the same shared flag set and merge their config files through the same shared code, with every old spelling still accepted so no existing deployment breaks. 5276 fixed a Redis connection string that names a Unix socket path being mangled into an invalid URL by code that assumed every connection string needed a network scheme prefixed onto it, which mattered specifically because connecting over a local socket rather than the network stack cuts out real, measured traffic between two services running on the same host.