Back to list
Skywire Development Update — October 8, 2026
The eighth of October laid the groundwork for moving the deployment services into the host visors that already run the dmsg servers. A visor can now run an embedded service under that service’s own key, with its own dmsg client and identity, restart it on failure without going down itself, and stop, start or restart it from the CLI; setup node, transport setup, STUN and the config service joined the services it can run this way, and the end-to-end environment was rebuilt to run every deployment service inside one visor as production does. Along the way a deadlock in the first such move was found and fixed, and a memory limit that was starving a combined host of heap stopped overriding the one set in its service unit. Transport discovery’s Redis was trimmed from weeks of per-transport history down to days, with settled days archived on disk, and several allocation hot spots on the deployment hosts were removed. Visors also began serving signed reach cards so peers can find each other without asking the address resolver, and a standalone dmsg web proxy learned to reach visors over a transport to a visor beside it.
Skywire: Deployment Services Inside the Visor
5513 let a block in embedded_services that names a key of its own, inline or in a key file, run the way skywire svc run runs it: with its own dmsg client, sessions and port 80, inside the visor process. The deployment services are known by their own keys, baked into every visor’s config, so until now they could not move into a visor, where embedded services answered under the visor’s key behind a path prefix. Such a service is rebuilt and restarted after any stop with a backoff, so a failing service never takes the visor down, and visor state reports it with its key and restart count. A config regen now keeps embedded_services, since autoconfig regenerates on every package update and would have dropped a host visor’s services. A service discovery block run this way answered /health under its own key over dmsg, locally and from another host.
5514 added skywire cli visor services to list those services and stop, start or restart one over the visor RPC, and so over the hypervisor link, since restarting one had meant restarting the visor and, on a deployment host, its dmsg server too. 5517 extended running on its own to any block that cannot be mounted under the visor’s key, even one that names no key, which covers STUN and blocks that keep their key in their own config file; transport setup can now serve only its dmsg RPC with no plain-HTTP admin API, and the config service gained a services package so a visor can run it.
5515 fixed the first such move deadlocking. Moving the address resolver into a production host visor hung because the visor’s own address resolver client retries until a resolver answers, and the embedded services waited on modules that wait on that client, so the one it needed never started; it was rolled back after about four and a half minutes. A service under its own key needs nothing from the visor, so it now starts in a module that waits on none. 5516 let GOMEMLIMIT from the environment win over memory_limit, whose automatic setting of sixty percent of free RAM at start had replaced the limit in the service unit and left a host running three embedded services about 2 GB of heap and a garbage collection every 0.75 seconds. 5518 made a regen keep an existing memory_limit instead of resetting it to automatic.
5523 rebuilt the end-to-end environment to match: one test visor is now the deployment host, folding in the dmsg server and running all eight deployment services in its process, each under its own test key, and the separate services container is gone. Docker was not available locally, so the PR’s own e2e job was the test.
Skywire: Transport Discovery Keeps Days, Not Weeks, in Redis
5509 archived each settled per-day metrics leaf, about 21 MB each, on disk under the store’s data path, kept the last fourteen days in Redis, and had the hourly cleanup drop older leaves once they are on disk, with reads falling back to the archive so the thirty-day CXO feed keeps its window. It also stopped every start re-saving each loaded day and resetting its TTL. 5510 ran that cleanup two minutes after start as well as hourly, since every merge to develop restarts transport discovery, often within the hour.
5511 cut the edge pairs that label per-transport bandwidth from thirty-five days to fourteen, about 615 thousand of 5.35 million keys on one host. 5529 then kept each transport’s daily bandwidth hash for three days instead of two weeks, about 1.1 GB of a 2.6 GB Redis, reading older days from the archived leaf, and fixed the cleanup deleting only the hashes of the day exactly eight days back, so a missed day had lived out its full TTL. 5530 kept a transport’s day uptime hash and online set for 36 hours instead of eight days, since only today’s are ever read, about 1.2 million keys, and 5531 kept the edge pair one day past the bandwidth it labels. 5528 moved the metrics feed’s object store to disk when a data path is set, where it had been held in the heap alongside the publisher’s own tree, about 330 MB, so it is now page cache the kernel can drop.
Skywire: Allocations on the Deployment Hosts
5508 found half of a host visor’s full core spent in the CXO publisher’s cleanup sweep over a 2.4 GB store, nearly all of it in a cursor seek that looked up every object from the root of the tree whether it deleted it or not; the sweep now advances with Next and seeks only after a delete, taking a 100-thousand-object sweep from 33 ms to 10. 5519 compared public keys as their 33 bytes rather than building two big integers per call or comparing hex strings, which had been about 500 MB of 2.8 GB allocated in thirty seconds, taking the edge sort from 214 ns and two allocations to 27 ns and none. 5520 had the routing publisher encode only the shards that changed instead of every visor’s shard each minute, so an unchanged cycle with 100 thousand transports drops from 260 ms and 75 MB to 43 ms and 34 MB. 5521 made transport IDs without allocating, six allocations and 422 bytes per call before, with a test checking the IDs against the original function for every transport type.
5527 started each relayed dmsg stream on a 4 KiB buffer, growing to 32 KiB after four reads in a row fill it, where every stream had taken a pooled 32 KiB buffer per direction, 117 MB for about 1,800 mostly small streams on one host; a stream whose sides can copy directly takes no buffer at all.
Skywire: Two Leaks and a Five-Second Cut
5503 found every HTTP connection a visor took in over skynet leaving a goroutine behind for good, because the single-connection listener it was served through waited on a close nothing made once the connection ended; one operator’s visors held 177 of them, and twenty requests now leave none. 5504 found any dmsg request whose response took more than five seconds cut off when the requester reached its dmsg server over QUIC. The server’s bridge refreshed the handshake read deadline before each read only for a net.Conn, which a QUIC stream is not, so the five-second deadline stayed; it now refreshes any stream with deadline methods, and a test answering after seven seconds fails at 5.4 without the fix.
Skywire: Reach Cards, and Proxies That Reach Visors Over a Transport
5512 had each visor serve its own address resolver records, signed, at GET /reach over dmsg and skynet, so a dialer can learn how to reach a peer from the peer itself. A dialer answers from a held card first, otherwise from the bindings feed while fetching the card in the background, and only then asks the resolver; sudph still goes through the resolver, which also asks the peer to dial back. The card follows ar_transport_limit, so a visor that stays out of the resolver serves none, and it is rebuilt when the visor moves networks. On a live visor its card verified, though no other visor served one yet. 5526 added tp --skynet to fetch a visor’s own signed transport list through the local visor, check its owner and signature, and print it like tp --remote.
5505 let a standalone dmsg web proxy, which keeps its own key, reach visors over one stcp transport to a visor beside it, tried before dmsg, reaching that visor directly and any visor it holds a transport to as a one-hop relay. A relay that will not carry a stream now answers with a FIN so the origin falls back to dmsg at once rather than after eight seconds. Two visors with no dmsg server answered in 0.7 seconds over the relay, and visors without a transport fell back to dmsg in 1.8 to 3.2 seconds, down from nine. 5506 added a choice of transport type and carried the proxy’s dmsg session over that same transport to the visor’s dmsg relay, so one transport carries both.
Skywire: Three-Hop List Routes, Counted and Filtered
5522 added counters for every route source in visor state, after three-hop routes from transport lists were found in 3 of 79 attempts on one visor and none of 6 on another with no way to tell why; each reason a list route misses is now counted. 5525 passed those routes through the same first-hop filter the route finder’s and the oracle’s routes use, since the list path always took its best candidate and kept handing back a route that had just died.
Skywire: Status Pages Mark Each Start
5507 recorded each service’s start time, version and commit in its chart store and drew a dashed line at each start on every chart, since every merge to develop redeploys the services and a change in a chart often lines up with one. 5524 made a charts page that fails to render log the error and the range, where it had answered “charts unavailable” and logged nothing.