SynapseOS benchmark

SynapseOS has a detonation chamber: it runs a model file twice, once against the real file and once against an inert twin, and reports whatever behaviour only the real file produced. This article covers what happened when that chamber was run against a corpus of 104 scans, alongside picklescan, a static scanner.

The short version:

• The chamber flagged 65 of 73 malicious samples.
• picklescan flagged 64 of 73.
• Between them, the two tools flagged all 73.
• Neither produced a false positive on 24 clean scans.
• The run exposed three real bugs in SynapseOS, one of which affected a verdict in the results.

The two tools fail in different places, and that is the most useful result here.

How the run was set up

The benchmark is a NixOS VM test defined in the repository. It boots a full SynapseOS system and drives it from a script, so every result comes from the real system rather than a mock.

The VM had 8 GB of memory and ran under KVM on a GitHub Actions runner. Scans used a 30 second limit, or 90 seconds for torch files and real models. The chamber’s own default is 120 seconds, so the shorter limit makes the timeout results below harsher than a default scan.

Payloads are harmless by construction. Every shell payload runs id, and every network target is a reserved test address or the chamber’s fake internet. Credentials in the chamber are fresh honey tokens generated for each scan.

The shell, Python, behaviour, evasion, torch and bypass samples also ran once outside the chamber in a plain control, where a marker file proves the embedded code ran on load. That separates a broken sample from a working sample that the chamber did not see.

The corpus

The stress group is counted separately because those files attack the chamber rather than the user. A file that crashes is not evidence of an attack in the same way a stolen credential is.

Detection

picklescan does not read chat templates inside GGUF files, so the seven template samples count as misses for it. That is a statement about scope, since it was never designed for that format.

One of the chamber’s 65 needs a footnote. evade-sleep-past-timeout sleeps for 200 seconds before acting. The chamber never saw the payload. It flagged the file only because it hit the time limit, and that turned out to be a bug, covered below.

Where picklescan lost

Two samples used multiprocessing.util.spawnv_passfds, a standard library function missing from picklescan’s denylist. It reported them clean, with no issues found. The chamber caught both, because no matter how a process is started, exec is still visible in the kernel.

Renaming a file to .dat, .weights or .ckptx fooled neither tool.

Where the chamber lost

Eight of the ten evasion samples came back no-divergence. In all eight, the embedded code ran in the control, and its payload is guarded by a condition that was false in the chamber, so the chamber saw no behaviour.

This is the known limit of any dynamic analysis. A payload that decides not to run produces no behaviour, and a differential oracle compares behaviour. Four of these target the chamber directly and could be closed by changing it, while the other four wait for conditions a chamber can only guess at. picklescan flagged all ten, because it reads the imports without caring whether they would run.

evade-detect-fake-net was the exception. It opens a connection to a test address, which is how a payload would check whether the internet is real, and the connection itself was the giveaway.

Exfiltration

The chamber’s fake internet answers every connection and searches what it receives for the honey tokens, including through common encodings. Seven samples reached the top verdict, exfiltration:

The XOR and reversed samples were still flagged, because reading the credential file and connecting out are both visible. What the chamber could not do was prove the token left. Any encoding it does not decode has the same result.

False positives

All 19 clean samples and all 5 real model scans returned no-divergence. picklescan flagged none of the clean files it could read.

The real model set is small. It proves that real files do not trip the chamber at this size, not that a large production model would not.

Stress

The orchestrator survived all seven, including the fork bomb, which crashed it in an earlier version.

The event flood results are the interesting ones. The sample does 2 million failed open calls and nothing else, and across three runs it went divergent, no-divergence, divergent. It was finishing close to the 30 second limit, so its verdict depended on which side of the limit it landed.

Repeatability

Across three full runs, 103 of 104 verdicts were identical. The exception was stress-event-flood.

Three samples were also scanned five times each, comparing the behaviour digest, which is a hash of the findings a verdict rests on:

A stable digest means two people scanning the same file can compare results by comparing one hash.

Speed

Each scan is two full detonations, the file and its twin, each with its own network namespace and bpftrace attachment. Timed-out samples are left out of these medians. The torch figure is higher because importing PyTorch takes most of it.

Parser fuzzing

The chamber parses GGUF files itself in order to build the inert twin, so that parser sees hostile input before anything is sandboxed. It was fuzzed with 20,000 mutated GGUF headers using bit flips, byte changes, truncation and hostile lengths.

No case raised an unhandled exception, and the slowest took 0.001 seconds. A file that cannot be parsed exits with its own malformed verdict and exit code 40, so a bad file cannot be mistaken for a clean one.

The rest of the system

Air gap. With airgap.target active, the chamber still ran. The exfiltration sample returned exfiltration and a clean file returned no-divergence. The fake internet lives inside its own network namespace, so it does not depend on the real one.

Receipts. Every scan writes a signed receipt. All 121 from the run verified, with 0 invalid. The key is local to the machine, so a receipt proves what that machine recorded, and cannot prove the machine itself was honest.

Cleanup. After all scans, 0 network namespaces and 0 chamber units were left over, with 140 MB of run data on disk.

ISO boot. A separate job built the ISO and booted it, passing on every run. That test boots a real image with its test backdoor added.

garak. It ran end to end against llama.cpp, finishing 8 of 8 on the test probe and 256 of 256 on promptinject.HijackHateHumans in 47.56 seconds. Those passes say nothing about prompt injection. The model is a tiny story generator that cannot follow an instruction at all, so the run proves the pipeline works and nothing more.

llama.cpp. It generated text inside the sandbox and outside it. The stories260K model reached about 2,700 tokens per second sandboxed and 2,300 direct, on runs of 64 and 256 tokens. Those are single runs of a toy model at different lengths, so they show that it works, and they are not a measure of sandbox overhead.

Bugs the run found

Timeouts counted as divergence. systemd reports a service stopped by RuntimeMaxSec as an ordinary failure, so the chamber recorded high runtime-failure status 1. A harmless file that was merely slow would be flagged, and that is what the event flood showed. A timeout now returns inconclusive with exit code 30, unless the file also did something suspicious, in which case that finding decides. Under this rule evade-sleep-past-timeout becomes inconclusive, so the chamber’s count drops from 65 to 64 of 73.

synapse-triage ai did not run. Its tshark display filter, tcp.port in {8000 8080 11434}, is rejected by current tshark. The set needs commas.

Inspect did not work offline. Even a mock evaluation stopped, because Inspect loads a tiktoken encoding that it downloads at runtime. The evaluation still exited 0, so only reading the output revealed the failure. For an operating system built to run air-gapped, this was the most serious of the three. The encoding is now fetched at build time and pinned by hash.

The benchmark harness had bugs of its own, and they caused the failed benchmark runs along the way. A step with no timeout enforcement hung for over two hours because a child process kept its output pipe open, and the first attempt at a picklescan comparison used an exit code that is always 0. Figures from those early runs were thrown away.

Checking the fixes

The fixes were checked in Docker on an arm64 Mac, running the real detonate.py inside a Debian container with systemd and bpftrace, no network and no host folders mounted. This is a different system from the benchmark: Debian rather than NixOS, arm64 rather than x86-64, and Linux 6.10 rather than 6.18.

Running the corpus there, torch files excluded, gave 89 scans. All 13 clean samples passed, 60 of 69 malicious samples were flagged, 1 was inconclusive, and the 8 misses were the same 8 evasions. The Nix changes parsed, the encoding built with the expected hash, and the full system configuration evaluated.

The tables earlier in this article come from the NixOS runs on the code as it stood before these fixes. The full x86-64 VM benchmark has not been repeated on the fixed code.

What the numbers do not show

The corpus was written by the same project it tests. These are not samples collected from the wild, and a corpus built by the author reflects what the author thought of. The percentages describe this corpus, not the threat landscape.

The evasion rate is not a real-world rate. Ten samples is few, and a real attacker chooses the check. What it does show is the shape of the failure.

Models were tiny. Nothing here tests the chamber on a multi-gigabyte model, or the time and memory that would take.

Four samples are not independently confirmed. sh-os-popen, sh-subprocess-popen, bypass-spawnv and combo-spawnv-gate-file did not write their marker in the control. The chamber saw exec /bin/sh for all four, so they did start, but they may have been killed before finishing.

Event loss is unproven. stress-flood-then-exec kept its exec through a flood in two of three runs, and in the third it hit the time limit first. That is encouraging, but it is not a test of tracing under real pressure.

Reading the result

Static and dynamic analysis fail differently here, and the benchmark puts numbers on that. The static scanner reads what a file could do, so a payload that stays quiet does not help it, but a call missing from its list passes. The chamber watches what a file does, so the choice of call does not matter, but a payload that stays quiet passes. Each tool’s misses were caught by the other, on this corpus.

The chamber is not a verdict on its own. A no-divergence result means a file did nothing different under these conditions, and not that it is safe.