Skywire Development Update — September 16, 2026

The sixteenth was the day the route-multiplexing campaign went fully live on a frozen rig, measuring a real proxy session end to end after every change rather than reasoning about one in the abstract, and the result reads as a single long investigation rather than a list of features: a multi-tunnel dial that kept landing every tunnel on the same first hop, a retransmit storm that turned out to be the send-window math punishing a slow leg for being slow, and a window finally built from what a peer’s acknowledgements actually prove rather than a guess sized on the wrong delay. Underneath that campaign, a separate run of transport-layer bugs surfaced on live and benchmark traffic alike — a dmsg entry seed hiding half of a folded visor’s identity, a re-dialed peer’s fresh connection torn down by its predecessor’s death throes, a QUIC session culled every two minutes for want of a ping branch, and a receive window sized for a fraction of the bandwidth a 150ms path can actually carry. CI and lint housekeeping, an android build break and a long-running compaction that was OOM-killing an exit visor closed out the day.

Skywire: Diversify Dials Learn to Avoid Each Other

The campaign’s opening problem was the simplest to state and the hardest to see: asking for several tunnels kept producing several route groups over the one same first hop, which defeats the point of having more than one. 4945 found the first cause — only --routed on the skysocks client actually requested a route group, so a --tunnels 2 session gave two streams over one direct transport and zero route groups at all; N>1 now implies the routed dial for every tunnel. 4946 found the second: the parallel candidate-route path every multi-tunnel dial takes raced the finder’s top candidates without applying the exclusions a diversify dial seeds, so each extra tunnel was still handed the same preferred first hop regardless. Candidates are now filtered to a free first hop first, falling through to the sequential path’s own disjoint fallback when none remains.

Even with both of those in place, the rig kept putting every tunnel on one transport, and 4947 gave the investigation something to read instead of a debug log the rig cannot access — a dial_decision mux event recording exactly how a diversify dial’s exclusions, candidate race, finder and local-calculation fallback each behaved, visible in visor state --select diag. That trail is what let 4949 and 4952 find the remaining leaks precisely: the RSN-oracle’s two-hop lookup returned an answer before the finder’s disjoint filter could run and ignored the exclusions outright, and separately the local-route memo keyed a cached path without accounting for which exclusions the current dial carried, so an earlier dial’s path could be served back over a transport this one was supposed to avoid.

4955 closed the last of them — a cold-start race where the transport-creation hook finishes before the finder does, simply because the direct transport already exists, taking the direct route and abandoning the finder’s filtered result entirely. Only once all four were fixed did --tunnels 3 actually land on three different first hops.

Skywire: A Retransmit Storm, Traced to the Window Itself

With tunnels actually diversifying, the rig surfaced a worse problem: a three-leg group re-sending roughly one and a half frames for every frame it sent. 4951 gave the investigation a way to tell which mechanism was asking for each retransmit — the reactive SACK path, the proactive head-of-line nudge, or a demote-time flush — and the counters pointed almost entirely at the reactive path. 4954 found why: that path floors its retransmit threshold on one group-wide send-to-ack delay, which every fast leg’s SACK sample keeps dragging back down, so a frame merely in flight on a slower leg reads as lost within a second of the next fast-leg sample arriving. The estimate is now kept per leg, fed by that leg’s own acknowledged and DSACK’d frames, and a hole is judged against the delay of the leg it actually rode.

4948 found a second, self-sustaining version of the same failure one layer down: once the threshold undershoots a loaded leg’s real delay, every frame on it gets re-sent before its acknowledgement ever arrives, which empties the very sample set that could have corrected the estimate. A DSACK proves the original frame arrived late rather than being lost, so it is now folded into the delay estimate as an unambiguous sample before the threshold can widen.

Two smaller but load-bearing fixes rode alongside that chain. 4950 found that an operator-pinned second leg sat in standby indefinitely with zero bytes carried, because aux legs only promote out of standby when a policy engine is wired, and the visor runs with none by default — a pinned leg is promoted immediately on add now, with the promotion itself recorded as a leg_promoted event. 4953 then found that promotion could still be silently dropped: the standby table only grew as legs were selected, so a leg promoted immediately after being appended had no slot yet to write the promotion into. The table now grows to cover the index being written, and a pinned leg is promoted unconditionally rather than only when it happens to already read as standby.

Skywire: A Real Per-Leg Send Window

The retransmit storm turned out to have a second, independent cause once the delay estimation was fixed: the predictive schedulers bounded each leg by an in-flight estimate modeled on the visor’s own send rate rather than on what the peer had actually acknowledged, so a slow leg was fed seconds deep and every one of those frames read as lost. 4956 replaced that model: the retransmit buffer now attributes held and acknowledged bytes per transport, and each leg’s window becomes what the peer’s SACKs prove it delivers per round trip, with a writer parking once every ready leg is at its window rather than racing ahead of what any of them can carry.

The first run surfaced two gaps immediately. 4958 found the window had only ever applied to the predictive scheduler modes, leaving the capacity mode the exit actually runs during a download un-bounded and re-sending thousands of frames per transfer regardless; the window now refreshes for every mode on its own 250ms cadence rather than only at the slower rebuild interval, which had also been pinning an upload’s growth to doubling every five seconds.

4960 found the window was still sized over the ping RTT rather than the leg’s actual measured feedback delay — stretched well past that RTT by delayed acknowledgements and SACK cadence — so a binding window shrank on every refresh instead of growing.

4961 made a related piece of wire overhead optional rather than default: every current visor had been negotiating forward-error-correction repair frames on any multi-leg group regardless of need, a flat overhead upward of twenty-five percent on legs whose gaps SACK recovery already refills for free: it is now gated behind routing.mux_fec, off unless both ends opt in. 4962 found and fixed the deadlock hiding underneath all of this measurement: the window refresh and the SACK handler took the leg-table lock and the retransmit-buffer lock in opposite orders, which froze a live three-leg upload at 34MB with every RPC touching the group hung behind it for eleven minutes. Fixing the lock order also let two further improvements land safely — a group with only one ready leg no longer parks its writer at all, since there is nothing to shed load to, and the window now refreshes every 100ms and keeps its last proven value through a quiet interval, so a transfer’s ramp from the floor no longer costs its first second.

Two tooling fixes made all of this measurable in the first place. 4957 gave the load-test sink the ability to answer HTTP range requests, since the proxy’s transparent range-splitter needs a range-capable origin to actually split a download across tunnels, and without it a multi-tunnel session had been carrying every byte over one tunnel regardless of how many disjoint first hops existed underneath it. 4959 found the splitter would still never fire on the bench rig even with a range-capable sink, because it only ever treats port 80 as plaintext HTTP and the rig’s real port 80 is occupied by something that cannot serve the benchmark; --range-port lets the splitter and the client agree on a different plaintext port for exactly this case, with production unaffected at the default.

Skywire: Picking the Right Leg

Two fixes closed out the campaign’s day by improving which leg actually carries a given byte. 4963 replaced skysocks’s fewest-streams tunnel-selection rule, which had been sending equal-sized chunks to tunnels regardless of how different their proven throughput was — a ten-megabyte upload split evenly between a 1.6 and a 6.5 MB/s tunnel, a download split evenly between a 2.2 and a 3.3 MB/s one — with a meter that remembers each tunnel’s peak proven rate and weighs a new stream by how much of that capacity its direction is already using.

4964 fixed the mirror-image mistake in the receive-side stall detector, which compared every leg’s received bytes against whichever leg had received the most — reasonable in general, except that while downloading the only inbound traffic on the other legs is their own SACKs, so the detector read a perfectly healthy leg as stalled and parked it, collapsing a three-leg download onto the one leg actually carrying payload forty-five seconds in. It now compares payload bytes only, and every park it triggers is recorded as an event rather than only a log line.

Skywire: Transport-Layer Fixes Beneath the Campaign

Several of the day’s fixes were not about the mux window at all, found on live traffic rather than the rig. 4923 traced route-setup failures through both setup nodes — one at 42% success, the other at 29% — to a dmsg client’s bootstrap seed, which records every dmsg server as server-only and answers a lookup from that seed before ever asking live discovery; a server folded into a visor registers both halves under one key, so a lookup for its client half stopped at the seed and failed outright, tripping circuit breakers for every route through it. A seed hit now falls through to live discovery first, with the seed kept only as a fallback when discovery is unreachable.

4925 found that a peer reconnecting after a restart could have its brand-new connection torn down by the read error from the connection it was replacing: transport ids are deterministic, so a re-dial lands on the same ManagedTransport and swaps the underlying conn, but the read loop’s error handling did not know a swap had happened and treated the old conn’s death as fatal for the transport as a whole, closing the fresh one a second after its own handshake completed.

4926 found every dmsg-over-QUIC session being culled roughly every two minutes, independent of the ALPN collision fixed the day before: the liveness ping dispatched on the two mux implementations dmsg actually uses but had no branch for QUIC, whose native streams serve as the dmsg streams directly, so every ping attempt failed outright and two consecutive failures is what the loop needs to declare a session dead. 4927 found a timer leak in the UDP demultiplexer’s read loop, which built a fresh deadline timer inside its loop but only stopped it with a deferred call that a connection sitting idle — which the demux creates eagerly for QUIC, sudph and WebRTC whether or not any of them are in use — would never reach, leaking one timer and one defer record per deadline change for the life of the process.

4929 found the actual throughput ceiling behind several of the campaign’s own measurements: every QUIC configuration in the tree left flow control at quic-go’s defaults, a 768KB initial connection window that caps a single-stream transport at roughly 5MB/s on a 150ms path and that quic-go only grows when the receiver drains faster than this codebase’s receive chain does — raised to a 12MiB initial, 96MiB maximum connection window across every QUIC-based carrier. 4936 found the diagnostic gap that made that ceiling hard to see in the first place, and a second bug alongside it: a transport that closes without a FIN from either side — because the peer cannot send one over a dead link and the local reader is parked waiting — left every stream that had been riding it registered forever, which is why a visor with 395 live transports was still reporting 27 “live” streams on an app with nothing running.

4928 kept the transport event ring usable on a busy visor, after discovering that a public exit’s own autoconnect traffic could flood its 256-entry ring in under three minutes, rolling the very close event someone was chasing out of existence; the ring grows eightfold and every transport additionally gets a dedicated last-close slot keyed by its id, so a flapping transport cannot push its own history out from under itself.

4930 gave the router the same kind of event history for mux legs that 4919 had given transports, after a pinned two-hop session lost its only leg during a live measurement with nothing in visor state recording who removed it or why. 4931 stopped a pinned proxy’s exit from being silently rotated onto a random service-discovery pick whenever a routine health probe failed once, which is how a configured exit kept drifting away from what the operator had actually set; the same fix stopped every start from overwriting the app’s other flags — --reconnect, --direct, --tunnels — back to a canonical six-token form.

4932 found a lock held across an entire packet write that serialized every read behind any in-flight write on the same transport, so a slow or half-open peer stalled traffic in both directions at once; the write now happens outside that lock, with a dedicated lock kept only to stop two writers’ frames from interleaving on the wire. 4933 fixed proxy start reporting a newly started client’s own errors as if they belonged to whichever client was currently in the state variable the code never actually switched, and 4934 gave each in-process skysocks-client launch its own config struct after discovering that several clients under one visor had been silently sharing one set of package-level flag variables, so the most recently started client could take over the dial settings of ones already running.

4935 closed a half-open-probe leak in the route setup node’s circuit breaker, where an intermediate hop admitted for a health probe was never released unless the specific outcome the code checked for happened to occur, leaving both production setup nodes refusing most routes through a reachable peer for over twenty minutes until a half-hour force-reset cleared it. 4924 fixed proxy start --route printing a reconcile error and silently falling back to an unpinned route whenever a direct transport to the exit already existed, since the direct shortcut it took builds no route group for the pins to land on.

Skywire: CI, Lint and Build Housekeeping

4937 and 4939 cleared the lint lane of findings that had been failing every open PR regardless of what it touched — discarded test returns needing the same //nolint:errcheck comment their neighbors already carried, an intentional truncation that needed to be marked as such, and two more of the same shape that reappeared the moment two other PRs merged alongside the first fix. 4940 stopped ten merges in fifteen minutes from spawning roughly eighty queued workflow runs and saturating the organization’s runner concurrency, by giving every push-triggered workflow a concurrency group that cancels a superseded run in favor of the newest commit’s.

4941 fixed the android build, broken since the desk grew its own listener: the mobile stub never gained the two functions hypervisor.go calls to decide whether to start one, and fixing that made a second problem obvious — the phone was linking the entire 38MB wasm command module it can never serve, pushing the mobile binary 28MB over its budget. 4942 fixed proxy test --json emitting plain English restore messages into the middle of a caller’s JSON output, the same class of bug the day’s lint fixes were chasing but in application code rather than a test. 4943 found a long-running startup compaction OOM-killing an exit visor seven times in one day, because it copied an entire eighteen-gigabyte object store inside a single database transaction that keeps every dirtied page in memory until commit; it now commits in 32MiB batches, resumable from the last key.

Skywire: The Bench, and the Changelog

4938 closed the day with the campaign’s reference measurements made repeatable rather than one-off: seven reference sets run on the frozen rig, direct and single-pinned-route transfers through six intermediates at both sizes and both directions, scripted to run unattended and summarized into one table, alongside a changelog pass covering everything merged since 4910.