Skip to main content

Roadmap & changelog

This handbook documents current reality. A few adjacent capabilities are planned but not built — listed here so you don't go looking for them in the reference.

Roadmap​

PlannedStatus
Remote MCP with OAuth (hosted MCP endpoint, no local install)Planned. Today the MCP server runs locally over stdio with an API key.
Billing and admission gatesShipped: cloud plan access, request budgets, credits, storage, and capacity are enforced. See Conventions & errors.
Browser/exploration tools over the APIThe in-app assistant's sandboxed browser tools stay app-only for now.
info

Additive changes (new fields, new routes) land inside v1 without notice — write clients that ignore unknown fields. Breaking changes get a new version path.

Changelog​

2026-08-20 — the CLI wait contract: bounded waits, and four exit codes​

  • --timeout <seconds> on run, verify, load run, coverage scan|run and generate --watch. It is one budget for the whole command — a 19-run suite waits at most this long in total, not 19×. Precedence: --timeout → TESTVIBE_WAIT_TIMEOUT_SECONDS → 5400 s; --timeout 0 waits indefinitely. Previously --wait/--watch had no ceiling at all, so a wedged run could hang a CI job forever.
  • Exit codes are now a four-value vocabulary: 0 finished and passed · 1 finished and something actually failed · 2 a generation parked to ask a question · 3 no verdict — the timeout ceiling elapsed, or contact with the server was lost. 3 is explicitly not a test failure; keep it distinct from 1 in your pipeline. See Run TestVibe in CI.
  • generate --watch exits 1 on a failed or cancelled generation. It exited 0, so a pipeline carried on with a feature that has no tests.
  • verify --wait reports an unfinished run as “no verdict”, never “did NOT pass”, and the summary exits 1 only if something actually failed — 3 if nothing failed but a run was still unfinished. run --all --wait behaves the same way.
  • Full contract: the testvibe CLI. No REST or MCP surface changed.

2026-07-17 — npm publish​

  • The CLI/MCP package is live on the public npm registry as testvibe — npm install -g testvibe (unscoped; @testvibe/cli was never published). The server-hosted curl | sh / iwr | iex installers remain as the npm-free path.

2026-07-06 — telemetry read surface​

  • Telemetry: get_live_servers, list_error_groups, list_server_metrics — read the app's live server vitals (CPU/memory/sessions), fingerprinted application-error groups, and a load run's persisted server samples from CI or an agent; testvibe telemetry live|errors|metrics. The ingest side stays the drop-in telemetry agent / TestVibe.Telemetry package.
  • 3 new operations across the API, CLI, and MCP server.

2026-07-06 — load test authoring​

  • Load testing: create_load_test, update_load_test, delete_load_test — author load tests from CI or an agent (simple mode replays functional journeys, advanced drives k6 against paths), edit their configuration, and delete them; testvibe load create|update|delete …. Listing and dispatching (list_load_tests / run_load_test) already shipped.
  • 3 new operations across the API, CLI, and MCP server.

2026-07-06 — plugin management, variables & secrets​

2026-06-11 — suite runs, run diagnosis, project detail​

  • run_suite (POST /projects/{id}/runs): dispatch several features — or every generated feature — in one call; testvibe run --all --wait is now the one-line CI gate.
  • diagnose_run (GET /runs/{id}/diagnosis): a failed run's failing tests, Gherkin, executed spec sources, artifact links, and log tail in one call.
  • get_project (GET /projects/{id}): project detail incl. test site URLs; testvibe project.
  • delete_file (DELETE /files/content?path=): remove stored files under the write allowlist; testvibe files delete.
  • status filters on list_features and list_runs.
  • MCP tools now carry readOnlyHint/destructiveHint annotations; run --wait (single and --all) exits non-zero on failure.

2026-06-10 — v1 initial release​

  • /api/v1/ops API: projects, features CRUD, generation (dispatch/status/log/cancel), runs (dispatch/list/digest/artifacts), files, configs.
  • API keys (tvb_…) in Settings → CLI & API keys, replacing the old CLI pairing flow.
  • testvibe CLI and MCP stdio server (19 tools), sharing one manifest with the API.

Current contract notes​

Generation and runs now expose explicit terminal outcomes and per-operation credits. Generation supports model selection and answering paused questions. Load configuration PATCH merges only supplied fields, and load-run measurements have their own endpoint. The reference describes those current contracts; dated changelog entries above describe historical releases rather than current tool counts.