How much runs at once

A self-hosted TestVibe admits work against four independent budgets. This page is all of it: what each budget caps, what happens when one fills up, where to change them, and how big the machine underneath has to be.

Appliance maintenance. The appliance provides a restricted recovery menu, with no general-purpose operating-system login. Use your hypervisor for disk backups and resizing. Appliance shell commands below are reference for support-assisted maintenance; they cannot be entered in the recovery menu. Compose and Kubernetes commands run on a host you manage.

The shape of the install

TestVibe ships one runner image that runs in two modes, and a self-hosted install runs one of each:

Container	Runs	Network
runner	Test runs, suite runs, coverage scans, load-test machines	Restricted to the site under test
browser	AI generation, the assistant	Reaches the URLs you point it at

The split is by network access, not by workload size. Keeping them apart means the interactive path cannot quietly become the execution path.

On Compose they are the runner and browser services; on Kubernetes the -runner and -browser Deployments; on the appliance testvibe-runner.service (port 8081) and testvibe-browser.service (port 8082), both bound to loopback.

Both stay at one replica. Run state lives in the process that accepted the job, and TestVibe polls that same endpoint for progress. A second replica does not share the load — it loses runs. The Helm chart refuses to render with runner.replicaCount or browser.replicaCount above 1, and Compose --scale is not supported. Capacity comes from giving the one container more CPU.

The four budgets

Every one of these is a whole number of concurrent slots. Work that does not fit queues; it is not rejected.

Setting	Caps	Default	Where it runs
TV_SANDBOX_LOCKED_MAX	Concurrent test runs — single features and whole suites share this one budget	10	runner
TV_SANDBOX_GENERATION_MAX	Concurrent AI generations	10	browser
TV_SANDBOX_LOAD_MAX	Concurrent load-test machines — a budget of machines, not of runs	20	runner
TV_SANDBOX_ROAMING_MAX	Concurrent assistant sessions — the chat/explore browser, and nothing else	10	browser
TV_SANDBOX_ACQUIRE_TIMEOUT_SECONDS	How long queued work waits for a free slot before it is told no	1800 (30 min)	—

Coverage scans get a fifth budget with no setting of its own. They are sized by the same number as concurrent test runs but counted in their own queue, so a full coverage budget can never block a test run, or the reverse.

“Locked” and “roaming” name network posture, not workload. Locked is the egress-restricted pool that untrusted generated test code runs in; roaming is the pool that reaches the open internet. That is why the test-run cap is the locked one and the assistant cap is the roaming one — a naming that has caused real confusion, including in our own Settings → Administration. If you take one thing from this page: TV_SANDBOX_ROAMING_MAX does not control test runs.

How the budgets interact

• They do not compete. Each workload admits against its own budget. A queue of twenty generations never delays a test run, and a saturated load test never stops someone using the assistant.
• First in, first out. Waiters are queued rather than raced, so a job that has waited a while cannot keep losing freed slots to jobs that arrived after it.
• Load is all-or-nothing. A sharded load run waits until its whole set of machines is free, then takes them together. It never starts at reduced scale.
• Your licence can lower one of them. A standard licence caps concurrent test runs at 2 whatever you configure, and excludes load testing entirely. An enterprise licence caps neither. A licence only ever lowers a number — it never raises one you set.

What happens when a budget is full

1. The work queues. A run sits in queued / allocating; a generation shows as queued in the app. Nothing has started and nothing is charged.
2. It keeps waiting for up to TV_SANDBOX_ACQUIRE_TIMEOUT_SECONDS — thirty minutes by default. This is why a busy install feels slow rather than broken.
3. Then it is told, honestly. The run finishes with errorClass: capacity_unavailable and retryable: true in its stats — classified as an environment outcome, not a test failure and not a bug. Over the ops API and the CLI the same condition is an HTTP 503 carrying "error": "capacity_unavailable", "creditsCharged": false and a Retry-After: 30 header, so automation can retry without guessing.

How to tell you are at the ceiling

Three signals, cheapest first:

• Runs sit in “queued” for minutes while the box is not obviously busy. That is the queue doing its job, and the cue to raise a budget or add CPU.
• The container log names the pool: [run 1234] FAILED: SandboxPoolBusyException: The 'locked' sandbox pool is at capacity; no session became available in time. The pool name in that message tells you which budget you hit.
• The database tells you exactly. Against your TestVibe database:

-- what currently holds a slot, by workload
SELECT pool, purpose, count(*) AS active
  FROM tv_sandbox_lease
 WHERE status = 'active'
 GROUP BY pool, purpose
 ORDER BY 1, 2;

-- what is waiting behind it
SELECT kind, status, count(*)
  FROM tv_job
 WHERE status IN ('queued','dispatching','running')
 GROUP BY kind, status
 ORDER BY 1, 2;

purpose is run, generation, load, assistant or coverage. An active count sitting at the configured maximum with a non-empty queued row behind it is the definition of “at the ceiling”.

One more limit sits above all four: a self-hosted install allows any single account 25 in-flight jobs of a kind. In practice the four budgets bind long before that does.

Generations — and your model endpoint

A generation is one AI agent driving one real browser while a model writes and verifies the test. It is the most expensive thing this product does per unit, and usually the setting that decides how much CPU the box needs.

Your model endpoint is the other ceiling

A self-hosted install brings its own model, connected on the administration screen's AI providers page. That endpoint has a concurrency limit of its own, and TestVibe does not know what it is:

• TestVibe does not rate-limit your endpoint — the generation budget is what protects it. If your gateway comfortably serves four concurrent conversations, set TV_SANDBOX_GENERATION_MAX=4. Setting it to 10 against a four-way endpoint converts a capacity limit into timeouts and failed generations.
• The model must support tool calling. Generation is a tool loop; a model without it stalls rather than failing cleanly.
• Vision is optional but load-bearing for some features. Tell TestVibe the models accept images only if they actually do — visual comparison and screenshot-grounded healing depend on it.

Set the generation budget to the smaller of what your CPU can drive and what your model endpoint can serve. Those two numbers are rarely the same, and the smaller one is your real limit.

How long one generation holds a slot

• A generation gets a 90-minute budget by default, adjustable per project between 5 and 120 minutes. Most finish in a small fraction of that; the budget is a ceiling, not a target.
• A generation whose host restarted mid-flight is resumed in place when its browser session is still alive, rather than re-run from the beginning. If it cannot be resumed it is retried at most twice — lower than other work, because re-driving a generation from zero spends real model time.
• So a generation occupies its slot for minutes, not seconds. Three concurrent generations on a small box will keep that box busy for as long as they run.

Test runs — the licence cap and the isolation

A standard licence limits concurrent test runs to 2, whatever TV_SANDBOX_LOCKED_MAX says. An enterprise licence does not limit them. The clamp is applied at each run admission, so activating a new licence changes the effective limit without a restart. Only test runs are clamped this way — generations, assistant sessions and coverage are deliberately untouched.

How concurrent runs are kept apart

Every admitted run takes an exclusive lease, enforced by a unique index in the database, so two runs can never be handed the same slot. What that slot is differs between the hosted service and your install:

	Hosted TestVibe	Your install
A slot is	A freshly created, single-use micro-VM, destroyed after the run	A concurrency slot on your one always-on runner container
Runs are separated by	Hypervisor isolation	A separate OS process and its own temporary working directory per run, each driving its own browser
Network	Egress restricted to the site under test in both

What this means for you. Concurrent runs on a self-hosted install share a container, so they share its kernel, its CPU and its filesystem outside the per-run working directory. That is a deliberate trade: you own the hardware, and the code being run is your own test suite. If you need runs isolated at machine level, talk to us.

Load tests — the arithmetic

Load runs consume capacity differently from everything else. Their budget counts machines, one run can take several at once, and it takes them all or none. Two more settings size a machine:

Setting	What it controls	Default
TV_LOAD_VUS_PER_SANDBOX	Browser virtual users packed onto one machine	20
TV_LOAD_PROTOCOL_VUS_PER_SANDBOX	Protocol (k6) virtual users one machine is sized for	1000

“Machines” are concurrency slots, not hosts. On a self-hosted install every load machine is a slot on your single runner container. A 100-user browser load run is five slots — and 100 Chromium instances — on that one container. Set TV_SANDBOX_LOAD_MAX from what the hardware can survive, not as a fleet size.

	Simple (browser) mode	Advanced (protocol / k6) mode
A virtual user is	A full Chromium browser replaying a journey	A protocol-level client inside one k6 process
Machines used	ceil(users / 20) — so 100 users take 5	Always exactly 1
Ceiling	20 × the machine budget, capped at the product limit of 100 users	1000 users in that one machine

Virtual users are split as evenly as the shards allow — 90 over 5 machines becomes 18 each — and each machine reports its own metrics, which TestVibe merges into one result. Lowering TV_LOAD_VUS_PER_SANDBOX spreads the same user count over more machines rather than making it cheaper, and lowers the browser-mode ceiling with it.

What happens when a run does not fit

• Above the ceiling — refused before it starts, with the arithmetic. Asking for more users than perMachine × budget (capped at 100 for browser mode) fails immediately rather than being silently reduced. The message spells out the calculation and names the settings to raise: “…requests 120 virtual users but the current capacity is 100 (20 browser VUs/sandbox × 20 load sandboxes, capped at the product limit of 100).”
• Within the ceiling but nothing free — it queues, for the length of its own schedule plus 15 minutes. Admission is all-or-nothing: a 5-machine run waits until all five slots are free and takes them together, holding its place at the front of the queue. Half a load test is worse than none — the numbers look real and are wrong. Smaller runs do not jump it.
• Not licensed at all. Load testing is not included in the standard edition. There a load run is refused before capacity is even considered, with a message naming the edition — so you can tell “not purchased” from “broken”.

Prefer protocol mode for volume. One machine drives up to 1000 protocol users; the same number in browser mode is impossible on any single host. Reach for browser mode when you specifically need real rendering, and keep the user counts modest.

Where to change them

Two places, and one always wins. The environment is the floor: a value pinned in .env, systemd or your Helm values cannot be overridden from Administration — the screen shows such a key as Managed by host rather than storing something inert.

In Settings → Administration

1. Sign in to TestVibe and open Settings → Administration.
2. Go to Capacity. You should see one field per budget, labelled by workload: Concurrent test runs, Concurrent AI generations, Load-test machines, Concurrent assistant sessions.
3. Change a number and save. It reaches the running app within the settings refresh interval (TV_INSTANCE_CONFIG_REFRESH_SECONDS, 30 seconds by default) — no restart. It applies to the next piece of work that asks for a slot; anything already running is untouched.

In the environment

Docker Compose — add these variables to the web service environment, for example in docker-compose.override.yml, then recreate the service. Adding an unreferenced variable to .env alone does not pass it into a container.

services:
  web:
    environment:
      TV_SANDBOX_LOCKED_MAX: "6"
      TV_SANDBOX_GENERATION_MAX: "3"
      TV_SANDBOX_LOAD_MAX: "10"
      TV_SANDBOX_ROAMING_MAX: "6"

docker compose up -d web

Kubernetes — edit the complete web.extraEnv list in your protected values file, preserving other entries and secrets. Apply it with the installed chart version and namespace:

helm upgrade testvibe oci://testvibe.azurecr.io/charts/testvibe \
  --version <installed-version> -n testvibe -f installation-values.yaml

On the appliance, use Settings → Administration for capacity changes. Host environment edits are support-assisted maintenance; the recovery menu is not a general-purpose shell.

Sizing the machine

The budgets throttle; they do not create capacity. Raising one above what the hardware can drive only moves the queue inside the container, where you cannot see it. The symptom is runs that take longer and time out, not more throughput.

The arithmetic that matters: every concurrent unit of work is a real Chromium browser. Budget roughly 1 vCPU and 1–2 GB of RAM per concurrent slot you actually intend to use, on top of the app, database and object storage. The shipped Kubernetes defaults reflect this — each of the two containers requests 1 vCPU / 2 GiB with a 6 GiB memory limit and a 1 GiB /dev/shm.

Usage	Machine	TV_SANDBOX_LOCKED_MAX test runs	TV_SANDBOX_GENERATION_MAX generations	TV_SANDBOX_LOAD_MAX load machines
Evaluation — one or two people trying it	4 vCPU / 8 GB	2	1	2
A team — 10–25 users, runs through the working day	8 vCPU / 16 GB	4	2	5
Continuous runs, CI-driven, or regular load testing	16+ vCPU / 32 GB+	8	4	10

Set TV_SANDBOX_ROAMING_MAX (assistant sessions) to about the same number as test runs. These are starting points derived from the per-slot cost above, not benchmark results. Raise one budget at a time and watch two things: runs queueing steadily means you are at the ceiling and should add CPU; runs failing on timeouts means a budget set well above what the box can drive. A queue is a much better failure than a timeout — queued work still succeeds.

Do not size for N concurrent users

Concurrent users is the wrong unit. Most of a user's session is reading results, which costs nothing. Size for concurrent work: decide how many runs, generations and load machines you want in flight at the same moment, give the box a vCPU and 1–2 GB for each, and set the budgets to that number. CI changes the shape more than user count does — a suite occupies exactly one slot for its whole duration, so if your peak is “four pipelines land at once”, size for four.

When one machine is not enough

There is no supported way to spread TestVibe's execution tier across several machines today. The ceiling is one runner and one browser container on one host, scaled vertically. If you reach it, talk to us — the fix is per-replica addressing in the application, not a setting you can change.

Related

• License · AI providers · Reference