Skywire Development Update — October 9, 2026

The ninth of October was mostly about memory, and about finding out where it was going on the deployment hosts. A browser visor that ran out of memory while searching for a route got a local route search that no longer queues the same node over and over, a dmsg server’s stream census stopped growing forever when nothing reads it, and two content-addressed stores that had swollen to gigabytes were brought back down, one by sweeping unreachable objects at startup and one by no longer holding a month of settled metrics in the heap. Two diagnostic endpoints went in to pin down a growing pile of KCP sessions on the address resolver. Around that, v1.3.100 was released, the browser module learned to build itself alongside the native binary on a toolchain that supports it, the network graph view moved into its own small wasm module, and the visor’s log level finally became one setting that can be changed while it runs.

Skywire: A Route Search That Stops Repeating Itself

5533 started from a browser visor running out of memory inside the local route search while dialing. The breadth-first search queued a node once for every transport that reached it, even though only the first copy at a given depth is ever expanded, so on a network with many parallel transports a search that found no path at all could hold over a million queued nodes, and several of those searches run at once. Later copies are now dropped before they are queued, while copies of the destination are kept so the candidate paths can still be ranked, and the route the search chooses is unchanged. On a synthetic network of three thousand visors with thirty transports each, a seven-hop search with no path went from 372 MB and 392 ms to 8.4 MB and 127 ms. The existing test that compares the search against the older implementation now also runs at a higher transport count.

Skywire: Stores That Had Grown Without Bound

5534 found the dmsg stream census only dropping finished streams when something actually asked for the census, which nothing on a dmsg server ever does, so the map kept an entry for every stream the process had opened since it started. One deployment visor was holding 120 MB of them. The census now prunes finished entries whenever the map has doubled in size since the last prune, and a new test fails without that change.

5539 looked at the visor stats stores on both deployment hosts, which had reached 2.4 GB against roughly 2 MB on ordinary visors, with the visor keeping the file mapped. On one host the stored roots reached only 66 objects, while some six million others all still carried a reference count above zero, so neither the runtime cleanup nor the startup compaction would ever free them. A disk-backed publisher now walks every stored root before its node starts and deletes whatever none of them reaches, in batches of one hundred thousand, and removes nothing at all if any stored root cannot be walked completely. On a copy of that store it removed a little over six million objects in 99 seconds, and the next open compacted the file from 2,309 MB to 1 MB. Where the reference counts leak in the first place is still open, so for now this bounds the store to whatever builds up between restarts.

5540 dealt with transport discovery’s metrics feed, which kept thirty days of metrics as leaves of a single node. Every five-minute tick re-encoded that node with all thirty days inlined, and the publisher held them all in memory as well as on disk, about 160 MB on one host. Each day is now its own sub-tree, so a settled day is published once and afterwards only referenced. Readers already stitched parts together by prefix, so nothing changes for them, and a day an older publisher wrote as one leaf is removed before its parts are written. The tree store gained an option, off by default and turned on by the metrics publisher, to drop the leaf bytes of a sub-tree once it is published, refilling them from the store when needed.

Skywire: Chasing KCP Sessions on the Address Resolver

One deployment visor was holding about twenty thousand KCP sessions from the address resolver’s SUDPH listener, 1.6 GB and growing by roughly 130 MB an hour, while only about 450 visors were actually bound to it. Every accept path in the resolver closes its session, so it was not clear which side was keeping them alive. 5535 added a /sudph-sessions endpoint that reports how many sessions were accepted and closed, how many are still in memory, whether the resolver still references them, and the KCP library’s own process-wide counters. Sessions the resolver had already closed turned out to be staying in memory, about four hundred more each hour, with nothing in the resolver pointing at them, so 5546 extended the same endpoint to report the size of the KCP listener’s own session map and how many closed sessions are still in it, read by reflection under the listener’s lock since the map is unexported. Both are diagnostic only.

Skywire: v1.3.100 and a Module That Builds With the Binary

5537 released v1.3.100, with its changelog and the embedded browser module rebuilt from that tree. The main change for operators is the KCP rework that landed over the previous days, which ends the per-session updater goroutines that had made v1.3.99 visors use a lot of CPU.

5538 taught the browser module to build itself during the native build when the Go toolchain supports a //go:embedbuild directive, which an experimental Go fork provides. On that toolchain, installing skywire straight from the development branch now produces a current desk rather than whatever module was last committed, and stock Go simply ignores the directive and embeds the committed module as before. The module’s revision file now carries a second line with the module’s own stamp, so a module built together with the binary no longer triggers the stale-module warning. 5542 followed a change in that toolchain, which now requires a committed file of the same name to be explicitly marked as the fallback; the committed module is that file, and both older toolchains and a build with the directive turned off embed it.

5548 stopped the network graph pages loading the entire skywire browser module, 37 MB compressed, just to run the WebGL graph view. The view now builds from its own command into a module of about 1 MB, embedded in every build except mobile, and served by the transport visualizer, transport discovery’s graph page and the skysocks status page. A make target builds it reproducibly with the toolchain named in go.mod, and CI fails if the committed copy is stale. In a browser the module boots in about 30 ms and draws a graph.

Skywire: One Log Level, Changeable at Runtime

5545 found a deployment visor logging at debug, about 170 lines a second, even though every service block asked for info. The setup-node block asked for debug, and the setup node set the process-wide level that every embedded service and library logs through. Setting the visor’s log level then changed only the visor’s own logger, and transport discovery’s aggregator had decided on debug once, when it was built. The setup node now uses a block-scoped logger like the other services, so its level stays its own. The visor sets the process-wide level together with its own at startup and whenever the level is changed, and the aggregator’s debug output now checks the current level on every line, so it follows a change made at runtime.

Skywire: Reaching a pty Host, and Not Hanging on One

5541 let the visor reach a standalone pty host over plain TCP using the visor’s own key. Before this, pty exec over a TCP address dialed from the CLI with a random key unless it could read the visor’s key from its config, which needs root, so a host whose whitelist held the visor’s key could not be reached as the visor while the visor was running. The visor’s pty exec now accepts a TCP scheme that dials the address directly with its own key. This is a plain TCP connection, not a transport. The CLI uses it when no identity is given, and the existing key options still dial from the CLI itself, which works while the visor is down. Interactive shells are unchanged, and a test checks that a key not on the whitelist is refused.

5544 found that after one deployment visor restarted, every pty exec to it over the default transport hung for its full timeout, while forcing skynet or dmsg answered in under a second. The pooled session’s stream had died without an error, so the call blocked forever, and since the blocked call still held the session, the pool kept handing that same dead session to every later exec. An exec now waits at most its command timeout plus fifteen seconds, after which the session is retired and closed and the next exec dials afresh.

Skywire: An Autoconfig Form

5547 added an interactive front end to skywire autoconfig. It lists every autoconfig setting with its current value from skywire.conf and its help text, grouped, in a terminal form, and with a port flag it serves the same form as a local web page guarded by a one-time token. Either can print the equivalent command with only the changed flags instead of applying it. Saving runs the binary’s own autoconfig with those flags, so skywire.conf is still written by only one path, and secret values are never shown. The form logic and the web handlers are tested; the terminal form has not yet been run in a real terminal, and no real apply was run.

Skywire: Tests, Dependencies and a Build That Broke

5536 fixed the red tests on the development branch. The chart store read its backend without holding its lock, which the race detector caught when a page render overlapped the sampler, and two tests built paths that only worked with forward slashes and so failed on Windows. 5543 updated thirty Go modules across the host, js/wasm, Windows and macOS builds, among them golang.org/x/net, x/crypto, x/sys and the Prometheus client, while keeping gvisor pinned because its newest version does not build. One of those updates, go-fuse 2.12.0, turned out not to compile on 32-bit ARM and x86, and the binary publishing job had failed ever since, so nothing new had reached the fleet; 5549 put go-fuse back at 2.11.0, with full builds passing on all four Linux architectures.