Fabric, rebuilt and measured
- Per-link throughput109–110 Gbit/s
- Retransmits0
- Two hops via the hub31.9 Gbit/s
- Links live4
- Measured2026-08-05
the two-hop drop is the hub's kernel forwarding, not the cable
The cluster Live
Four compact single-board AI computers rather than a rack. Each carries a unified memory pool shared between CPU and GPU. The desktop is a terminal and a cockpit; every training job lives here.
What the cluster has actually done
- A three-node distributed training smoke test passed — real sharded training across three boards, not a benchmark of a single box.
- A text training run reached step 151,760 of 200,000 at roughly 9,770 tokens per second, holding around 95% GPU utilisation at 66–70°C.
- The audio cortex run that produced the lab's first sound completed here.
- A large mixture-of-experts model was requantised locally and served across nodes as a one-off "we ran a 314-billion-parameter model at home" demonstration — honestly slow, and not a daily driver.
Published gotcha: unified memory is one pool, not two
The monitoring tool reports GPU memory as unavailable on these boards, and it is tempting to read the inference runtime's large "VRAM" figure and the operating system's separate "RAM" figure as two budgets that add up. They are the same physical memory. One node holds roughly one node's worth, full stop.
Concrete consequence: a 179 GB quantised model cannot be made to fit on a single ~128 GB unified node no matter how the numbers are arranged. It needs a genuine multi-node split. Anything that computes a layer-offload count from reported free GPU memory on this platform is computing from a null.
Negative result: a broadcast storm we caused ourselves
Bridging two fabric ports to make one flat network closed a hidden physical loop in the cabling. Multicast receives climbed past 900 million packets, a node's CPU pegged, and management access was lost — recoverable only because a separate wired path existed.
It then happened again on a different node weeks later. Root cause: the bridge was defined in the declarative network configuration, so disabling it at the connection-manager level did nothing — it was recreated on every boot. Removing the configuration files fixed it permanently.
Standing rules adopted: never bridge fabric ports; use routed point-to-point links; trace every cable endpoint before assuming the topology; and never apply a network change remotely on a machine whose only management path is the network you are changing.
Negative result: weeks lost to a fabric that was never broken
The 200G links looked dead for weeks. The cables were fine the whole time. Two network profiles had been bound to interfaces that had no cable in them, so they could never activate — while the interfaces that did have cables sat unconfigured.
Diagnostic rule adopted: read carrier state and negotiated speed, not ping. A port reporting carrier up at 200,000 Mb/s means the physical layer is good and every remaining failure is software. Ping tells you nothing about which of the two you are looking at.
Root-cause: the reboot loop that was not thermal
One node hard-reset roughly every ten minutes for an hour — while running cool, with no out-of-memory event and no kernel panic. Logs cut off mid-line, which is the signature of an abrupt platform reset rather than a software crash.
The mechanism turned out to be the platform watchdog, armed with a ten-second timeout and configured to reboot. The trigger was a firmware-management service repeatedly probing a network card that was reporting insufficient slot power — while the cables were being physically handled. Any stall longer than ten seconds and the watchdog fires. Masking that one service produced immediate multi-hour stability.
The lesson that generalises: "it runs cool, therefore it is not a hardware problem" is false. Ruled out with evidence rather than assumption: thermal (it died cold, with zero guard events), correctable bus errors (present even during a seventeen-hour stable run), memory, and machine-check exceptions.
Thermal reality, stated plainly Live
These boards are dense and quiet, and under sustained full load they will hard-reset in the region of 90°C. That is a real ceiling, not a tuning problem.
- A guard daemon polls temperatures and sheds load before the ceiling, pausing trainers rather than letting the box die mid-run — pause-to-survive beats crash-mid-render.
- Early guard thresholds were set too high to help: the hardware reset below the threshold that was supposed to protect it. Fixed by measuring the actual failure point instead of trusting a datasheet number.
- Active cooling and a UPS were added. The boards have no internal battery, so a brief brown-out is a hard reset — an ordinary mains flicker was silently in the crash statistics.
Interconnect discipline
- Never tensor-parallel across the network. Pipeline-parallel only — the per-token traffic is on the order of kilobytes and crosses even a modest link comfortably, while tensor parallelism does not.
- Memory bandwidth on-board outruns any network here by more than two orders of magnitude. The link is for shipping work, not for pretending four boards are one big board.
- A four-node ring is not viable with two ports per node: the diagonal pairs are never cabled and direct-memory transport cannot be routed through a middle host without collapsing to a slow path. Two or three nodes without a switch; four needs a switch. This was checked against vendor documentation rather than assumed.
- Never conclude hardware is absent from an empty device listing. The network cards on these boards power down when no cable link is present and vanish from the bus — they were declared missing once, wrongly, and the correction is on the record.
Fleet health guard Live
A small watchdog on every node, on a timer, in alert-only mode by default — it reports storms, memory exhaustion and GPU faults rather than taking action on its own.
One command from the desktop returns fleet-wide health, with the exit code carrying the worst severity so it composes into other scripts. The failure class that cost us the two storms above can no longer recur silently, which is the entire point of writing it.
The rule that governs all of it
No model training on the desktop machine. Ever. Not training, not inference, not "just a small chat server".
The rule exists because it was broken once: a service described as a chat front-end was found to be running local inference on the workstation. If the cluster is unavailable the answer is to wait, not to fall back to the desktop. Remote clients that call the cluster over HTTP are fine — they are not the thing being ruled out.
The federation panel Live
A public, sanitized status page for the fleet: every node and service shown honestly green, red or grey, refreshed on a short interval, with a staleness banner that tells you when the data itself is old rather than quietly showing you yesterday.
Honest scale
- Subscribers0
- Accounts in existence0
- Compute workers1 (dry-run)
Those numbers are published rather than hidden because a status panel that inflates its own scale is the one thing a status panel must never do.
Two hard lines held during design
- No mass sign-up across free tiers. The request existed; it was refused. Automated registration across hundreds of services gets an account banned and sprays personal data across services nobody vetted. A handful of legitimate, deliberate free-tier accounts instead.
- Free tiers cannot train. They are ephemeral and their terms forbid sustained compute. They serve; the cluster trains. Selling it any other way would have been the easy story and a false one.
the experiment of actually measuring cloud-vs-cluster training remains worth running
ALMA-OS for Jetson Prototype
A lean edge-AI operating system built from the kernel up with Yocto and the Tegra layer — deliberately not a stock vendor image with the desktop stripped off afterwards.
What exists on disk
- Custom kernel5.15.185
- Kernel image41 MB
- Flashable bundle1.6 GB
- Build completed2026-06-18
- Flashed to a boardnot yet
real artefacts, verified on disk — and an unflashed image is not an OS
The honest gap, and a documentation lesson
Everything is ready except the last forty-five minutes: putting a board into recovery mode and writing the image to it. Until that happens this is a build, not an operating system, and it is labelled Prototype for exactly that reason.
Worth recording: a gate document in the repository claimed the image could not be built on this setup — and had been true when written, then silently went stale after the build succeeded. A stale document that contradicts the disk is worse than no document. Trust the artefacts.
Borrowed compute חישוב מושאל
Two experiments in using capacity that already exists — one harvesting a machine's idle cycles, one a small distributed-training client.
ERAN-GRID — idle-cycle harvesting Prototype
An agent that runs work only while a machine is genuinely idle and yields the instant its owner returns. Proven end to end: owner active means hold, idle means claim, run, return a result — and a yielded job is requeued rather than lost.
Safety and privacy are structural, not promised:
- Opt-in and default off. It never auto-starts.
- No microphone, no camera, by construction — the sensor set is asserted empty in code. Anything else would make it spyware, and calling it something friendlier would not change that.
- Refuses to run on battery. Hard-pauses on heat.
honest: the workload unit is still a neutral stand-in, not real training
ERANCLOUD — the Local-SGD client Prototype
A small client that performs a register, pull, train, delta, push cycle. What is verified is a local toy dry-run on a toy model, and nothing beyond that.
- The public coordinator is LIVE as of 2026-08-24 (federation.iddoperez.ai/join) with a working volunteer loop, verified the same night. What it trains is the cloud student brain, not the sovereign organism. No fake connection command is displayed on the page.
- Live mode fails closed: a coordinator that cannot be reached never silently degrades into a simulation that looks like success.
- An installer stays unreleased until install, launch, uninstall and platform application-control checks all pass. The source archive is published separately in the meantime.
What a four-node edge cluster cannot do מה שאי אפשר
- It cannot train a frontier-scale model, and nothing on this site suggests otherwise. It can train small models properly, and it can serve large ones slowly.
- Splitting a very large model across nodes is a demonstration, not a workflow — a model that fits one node runs an order of magnitude faster than the same class of model split across two.
- Sustained full-load operation is limited by cooling before it is limited by compute.
- Adding a fourth node to a switchless fabric is not a cabling problem, it is a topology impossibility. Four nodes want a switch.