gpupool

How it works

Your GPU boxes can't be found — they sit behind NAT and get switched off. So they don't wait to be reached. Each one dials out to a broker and holds the connection open. "Find another computer" then stops being a search: the broker already knows which machines are alive, because the dead ones stopped checking in.

No port forwarding, no static IP, no router configuration. The GPU machine never listens on a public port.

Download

A single self-contained executable — no Node install needed on the GPU machine.

npx gpupool --help

Setting up a GPU machine

1. Run something on a local port

   Anything that speaks HTTP. gpupool neither knows nor cares what it is — Ollama, vLLM, ComfyUI, a FastAPI service, your own code.

   ollama serve            # listens on 127.0.0.1:11434

2. Pair the machine, once

   The token comes from whoever runs the broker. Credentials are written to ~/.gpupool/credentials.json and reused from then on.

   gpupool login --broker https://your-broker.fly.dev \
                 --token at_REPLACE_WITH_YOUR_OWN \
                 --label office-3090

3. Say what to expose

   Create gpupool.yaml next to wherever you'll run the agent. The health path is polled, so a crashed app drops out of the pool while the machine stays in it.

   environments:
     chatapp:
       port: 11434
       health: /api/tags
   
   maxConcurrency: 4

   Omit health and the agent falls back to a TCP connect check, which catches the app having exited but not the app being broken.

4. Install it as a service

   This is the step that matters. Without it the agent dies when you close the terminal and never comes back after a reboot — so the machine only serves when someone remembered to start it.

   gpupool service install              # starts at logon, no admin needed
   gpupool service install --system     # starts at boot, needs admin/root
   gpupool service install --dry-run    # print the definition, change nothing

   Platform	Mechanism	Behaviour
   Windows	Task Scheduler	restarts every minute on failure; keeps running on battery
   Linux	systemd	Restart=always, RestartSec=5
   macOS	launchd	KeepAlive, RunAtLoad

   On Linux, a user service still stops at logout unless you enable lingering:

   sudo loginctl enable-linger $USER

5. Check it joined

   $ gpupool status --admin-key <key>
   office-3090  (0/4 busy)
     chatapp  :11434  ready
   home-4070  (2/4 busy)
     chatapp  :11434  ready
     imagegen  :9999  DOWN - connect ECONNREFUSED 127.0.0.1:9999

   Health is per environment, not per machine — a crashed app is pulled while everything else on that box keeps serving.

Using it from your app

The broker is a transparent HTTP proxy, so whatever your local app speaks, your app speaks. If it's Ollama, every OpenAI-compatible client works with a one-line change and no SDK at all.

import OpenAI from 'openai';

const openai = new OpenAI({
  baseURL: 'https://your-broker.fly.dev/v1',
  apiKey: process.env.GPUPOOL_KEY,       // pk_chatapp_...
});

const stream = await openai.chat.completions.create({
  model: 'gpt-oss-64k:latest',
  messages: [{ role: 'user', content: 'hello' }],
  stream: true,
});

Nothing in your app knows which machine answered, or that a machine went away between two requests.

App keys isolate apps from each other

Each key resolves to exactly one environment, and that binding is the only thing deciding where a request lands. The chatapp key cannot reach imagegen's port down any path. Repointing or revoking an app means editing the broker's key map — the app itself never changes.

WebSockets and sticky sessions

WebSocket upgrades tunnel too, both directions, with binary frames and server-initiated pushes intact — ComfyUI and notebook kernels report progress that way. Browsers can't set headers on a WebSocket, so the key may travel as a query parameter, which is stripped before the path reaches your app:

new WebSocket('wss://your-broker.fly.dev/ws?key=pk_wsapp_...');

Send x-gpupool-session and the broker prefers the machine that served that session last, so a warm cache or loaded model isn't thrown away. It's a preference, not a pin: if that machine is gone the request goes elsewhere, because falling back beats failing.

When nothing is available

You get an immediate 503 rather than a hang:

{ "error": { "message": "No machine available for environment \"chatapp\" right now. Try again later.",
             "type": "no_capacity" } }

Two variants, because they need different fixes. "No machine available … try again later" means a machine declares this environment but none is ready — a temporary condition. "No machine is serving …" means nothing in the pool declares it at all — a config mistake.

Live demo

Point this at a running broker. It sends a streaming chat completion and renders tokens as they arrive, so you can see the whole path working: browser → broker → agent → your local model.

Running the broker

It's the only piece that needs hosting: a small stateful service that relays bytes and never computes, so the smallest instance your platform sells is enough. A Dockerfile and fly.toml are in the repo.

Tokens and app keys are opaque strings you generate yourself — there's no registration step. Make them unguessable:

node -e "console.log('at_'+require('crypto').randomBytes(16).toString('hex'))"   # agent token
node -e "console.log('pk_'+require('crypto').randomBytes(16).toString('hex'))"   # app key

fly launch --no-deploy
fly secrets set \
  GPUPOOL_AGENT_TOKENS=at_REPLACE_WITH_YOUR_OWN,at_REPLACE_FOR_SECOND_MACHINE \
  GPUPOOL_APP_KEYS='{"pk_chatapp_REPLACE_ME":"chatapp"}' \
  GPUPOOL_ADMIN_KEY=pick_something_long
fly deploy

What happens when things break

Current limits

Worth knowing before you build on it.

• Agent tokens are issued by hand — there's no account system, and anyone holding one can join the pool and serve traffic. Treat them as trusted.
• One broker, no multi-region, and its registry lives in memory.
• Raw TCP isn't forwarded — HTTP and WebSocket only.
• No metering or quotas.
• The Windows binary is unsigned, so Smart App Control blocks it.
• Consumer GPUs have modest VRAM, so a pool like this serves small and mid-size models well and large ones not at all.