Skip to content

UI tests that run in isolation

Test Automation · End-to-end testing and visual regression

The Playwright suite in your repository runs on an isolated runner, in parallel shards and on the browsers you choose, against an environment you proved is yours, and every failure comes back classified. An agent proposes new tests from a plain-language description, validated in a real run, and screenshots are compared with the approved baseline. Declarative plans support conditions, cleanup, database checks and steps interpreted by AI, and results reach GitHub Actions and Jira.

The problem

What this product solves

UI tests break for reasons that are not bugs: a dirty environment, the network, a different machine, a brittle selector. Telling them apart eats the time of the people who should be fixing things.

Without visual comparison, a layout change slips through review and only shows up for the customer.

Writing and maintaining tests is the work that always gets postponed, and one changed selector takes down the whole suite.

updated

Who uses it, and for what

Typical profiles, no customer names. Each use carries the status of the feature behind it.

  • QA team at an e-commerce companyRun the checkout suite on every release, across more than one browser, and get every failure classified.E2E tests
  • Front-end teamSee which screen changed from the approved version before a customer notices.Visual regression
  • Small team without dedicated QADescribe in plain language what to test and accept only the test that passed a real run.AI test authoring

Features

What the product does

Self-service: add it from your account.

  • E2E tests

    Runs the Playwright suite in your repository on an isolated runner, in parallel shards and on the chosen browsers (Chromium, Firefox, WebKit), against an environment with proven ownership; each failure comes back classified by rule (likely bug, flaky, infrastructure, broken test), with trace, screenshot and video, and gates CI (CLI, API and pipeline). Optional AI summary, as a suggestion.

  • AI test authoring

    You describe what to test in plain language and point to an environment with proven ownership; an agent reads the page in read-only mode on the browser runner and proposes a declarative test using role and accessible-label selectors. A proposal is only shown as ready after it passes a real run (up to 3 attempts) and only becomes a saved test once a person accepts it; the equivalent .spec.ts file can go to review through Code Automation. When a saved test breaks on a selector, the agent suggests the new selector with before and after screenshots. AI cost goes through the Gateway, capped per request and per day.

  • Visual regression

    Each scenario screenshot is compared pixel by pixel with a baseline approved by a person, with per-screenshot tolerance, a diff image, desktop and mobile viewports and flakiness history; a difference above the threshold or changed dimensions fail the run.

  • Composable test plans

    Declarative test plans (a closed list of actions, no code) written with reusable blocks, parameters, data rows and per-scenario setup, plus one-time suite setup and teardown. The platform expands the source into a regular plan, which goes through the same rules as any plan, and the screen shows each run step by step, with errors and screenshots. Editing a block does not change saved plans: the screen flags it and a person creates the new version. No vault for sensitive parameters.

  • Conditional, data and AI steps

    Steps that let declarative plans cover real flows: conditions (if) with then and else branches, cleanup that runs even after a failure (finally), a value captured from a same-origin JSON response, a database check through the data connector (read-only, pass or fail only), actions and checks described in plain language and interpreted by AI on every run, and a resilient mode that swaps a broken selector within the run. AI requests are capped in count and cost per run.

  • GitHub Actions, Jira and recording

    A GitHub Actions action that runs the Playwright suite through API v1 and uses the result as the workflow gate; a workspace link to Jira Cloud to open an issue from a failed run, one per run; and conversion of text recorded with npx playwright codegen into a declarative plan, without running anything.

How it works

From start to audit record

  1. Target

    You register the origin of the environment under test and prove it is yours by publishing a file with a code on it.

  2. Test

    Your repository Playwright suite runs as is. For a new test, you describe the intent and an agent proposes the scenario, validated in a real run.

  3. Isolated run

    Parallel shards on the browser runner, on the browsers you chose, with ownership checked again before each one.

  4. Triage

    Each failure comes back classified by rule (likely bug, flaky, infrastructure or broken test), with trace, screenshot and video.

  5. Visual comparison

    Scenario screenshots are compared pixel by pixel with the approved baseline, and any change produces a diff image.

  6. Gate

    The result decides CI through the CLI, the API or as a pipeline step: a failed test fails the job.

  7. Beyond AIUBY

    The GitHub Actions action brings the gate into your workflow, and a failed run becomes a Jira Cloud issue in one click.

Features

Feature by feature

What each one does, how it works, an example from the documentation and what it does not do. Each badge comes from the catalog.

E2E tests

Your repository's Playwright suite runs isolated and comes back triaged.

Test Automation runs the Playwright suite that already lives in your repository, as is: a shallow clone of the requested ref, dependency install and npx playwright test, one shard per isolated executor run, in the browsers you choose (Chromium, Firefox and WebKit). The target is always an environment whose ownership you have proven, and ownership is checked again before each shard.

Each failure comes back classified by rule, with the reason alongside: likely bug, flaky, infrastructure or broken test. The run result works as a CI gate, through the CLI, the API or as a pipeline step. For teams without a suite, the product also runs declarative scenarios: a closed list of actions, with no code.

How it works

  • Parallel shards and retries per run. A run passes only if no test failed; a flaky test counts as passed, as in Playwright.
  • Triage by fixed rules, in order: passed on a retry (flaky), DNS, network or crashed browser (infrastructure), a selector that did not find the element or an error in the test code (broken test), history that alternates over the last 10 runs (flaky), and everything else (likely bug).
  • Failure artifacts according to your playwright.config: screenshot, trace and video, plus one JUnit file per shard. Only admins can download traces and videos.
  • CI gate: aiuby tests run --wait exits with 0 when it passed and 6 when any test failed. The browser_tests pipeline step runs the suite from the same repository and commit and fails the job if any test fails.
  • Declarative scenarios: goto, click, fill, press, select, waitFor, checks for visible element, text, URL and title, and screenshot, with selectors by role, text, test-id or limited CSS, on desktop and mobile viewports. Navigation stays locked to the verified origin.
  • Optional AI failure summary, for admins only: it sees only the failed tests, with the message redacted, and returns a suggestion that cites the tests. AI reclassifies and runs nothing.
Through the CLI, in CI Triggers the repository suite against the staging environment, in two browsers and two shards, and waits for the result: the exit code is the gate.
export AIUBY_TOKEN=...   # key with the "Run Playwright suites in CI" scope (tests:run)
aiuby tests run --suite acme/loja-e2e --target https://staging.acme.com.br \
  --ref "$GITHUB_SHA" --browser chromium --browser firefox --shards 2 --wait

What it does not do

  • It only tests origins with ownership proven by the /.well-known/aiuby-test-automation.txt file, and the environment must be reachable from the internet.
  • npm only: pnpm and yarn are not accepted. Projects in playwright.config must be named chromium, firefox or webkit.
  • Caps per run: 4 shards by default (8 at most), 600 s per shard by default, up to 20 artifacts and 3.5 MB per shard; anything that does not fit is listed as discarded. Artifacts are kept for 14 days by default.
  • Triage is a rule, not a verdict: a selector that did not find the element may be an interface that changed on purpose or an element that disappeared because of a bug.

AI test authoring

Describe what to test and get a test that actually passed.

You describe in Portuguese what you want to test and point to a target with proven ownership. An agent reads the page in the browser executor, in read-only mode (accessibility tree, data-testid and a screenshot, with no click, keypress or submit), and proposes a declarative test with selectors by role and accessible label.

The proposal only shows as ready after it passes a real run, and only becomes a saved test when a workspace admin accepts it. When a saved test breaks because of a selector, the same agent suggests the new selector, with before and after screenshots.

How it works

  • The model response is treated as data: it goes through the closed plan schema, needs at least one check per scenario and gets a final evidence screenshot.
  • Up to 3 proposals per request: if validation fails, the error and the page at the point of failure go back to the model. After the third, the request ends as "did not pass", with the reason for each attempt.
  • Read-only by default: a proposal with fill, select, keypress, or a click on a button or link with an effect name (sign out, delete, buy, submit) is refused before it runs. Write actions only on targets an admin has marked as a test environment.
  • An equivalent Playwright .spec.ts file, generated by deterministic translation of the validated plan, to download or to open a ticket in Code Automation, which follows the full review flow.
  • Selector fix for failures classified as broken test: only the selector changes, the corrected scenario runs again, and accepting creates a new plan version.
  • AI cost through the Gateway, metered and shown on each request, with a per-request cap and a daily per-organization cap, on both amount and number of requests.
Through the session API Request a test, follow the proposal, accept it, send the .spec.ts for review in Code Automation and request a new selector for a step that broke.
POST /api/platform/test-automation/authoring                        { targetId, intent, startPath?, viewport? }   (202)
GET  /api/platform/test-automation/authoring/{id}                   proposal, attempts, sources, screenshots, .spec.ts
POST /api/platform/test-automation/authoring/{id}/accept            { name, description? }
POST /api/platform/test-automation/authoring/{id}/code-automation   { repositoryId, path, image, testCommand }
POST /api/platform/test-automation/authoring/fixes                  { runId, scenarioKey, viewport? }   (202)

What it does not do

  • The agent does not browse on its own: it sees the starting page and the pages the proposal went through. A long flow may not close within 3 proposals.
  • In read-only mode it does not test login, cart or forms; those need a test environment and a fictitious account. There is no credential vault for tests.
  • "Passed" means the proposed steps and checks passed once; it does not guarantee the test checks what you intended. Read the steps before accepting.
  • It does not write to repositories and has no API v1, CLI or MCP: acceptance is by a person, in the UI.

Visual regression

A screen that changed shows up before your customer sees it.

Each screenshot step in a scenario produces a PNG that is compared, pixel by pixel, with the approved baseline for that plan, scenario, viewport and screenshot name. If the fraction of differing pixels exceeds the threshold, or the dimensions changed, the run fails and the platform produces a diff image: the change in red over the screenshot in light gray.

The baseline is a human decision: an admin approves the screenshot that serves as reference, and approving another one replaces the previous one. A screenshot without a baseline does not fail the run; it is marked as having no reference, waiting for approval.

How it works

  • Per-screenshot tolerance: per-pixel difference threshold (0.1 by default) and maximum fraction of differing pixels (0.001 by default).
  • Four outcomes per screenshot: no baseline, identical, changed and different dimensions. The last two fail the run.
  • Stable screenshots by design: animations off, reduced motion, hidden cursor, fixed time zone and locale, and scale 1.
  • Desktop (1280 by 720) and mobile (390 by 844, with touch) viewports, each with its own baseline.
  • Flakiness history per scenario and viewport over the last 20 runs, with the rate of flips between passed and failed.
  • An approved baseline does not expire; other screenshots follow artifact retention.
Through the session API Create the plan with screenshot steps, request the run, read the artifacts and approve the screenshot that becomes the baseline.
POST /api/platform/test-automation/plans        {targetId, name, description?, definition, note?}
POST /api/platform/test-automation/runs         {planId, version?}   (202)
GET  /api/platform/test-automation/runs/[id]    run with scenarios, steps and artifacts
POST /api/platform/test-automation/baselines    {artifactId}

What it does not do

  • It only compares screenshots from declarative scenarios; failure screenshots from the repository's Playwright suite are not compared.
  • Up to 10 screenshots per run (screenshots times viewports), PNG of up to 512 KB each.
  • Comparison is per pixel, with no semantic layout analysis: any difference above the tolerance counts as a change.
  • Only admins approve baselines; other members read the result.

Composable test plans

Write the login once and reuse it in every plan.

A declarative plan is a closed list of actions, with no code. Composition lets you write that plan without repeating steps: a block stores a sequence with parameters (a login, for example), the scenario references values such as {{termo}}, a table of data rows runs the same scenario with several values, and a shared setup goes at the start of every scenario.

The platform expands the source into a regular plan before saving it, and that plan goes through the same rules as any other: allowed selectors, navigation only on the origin with proven ownership, and caps on scenarios, steps and screenshots. The browser executor receives the expanded plan, with no knowledge of blocks or parameters.

How it works

  • Reusable workspace block, with declared parameters and a version that increments on every edit.
  • Parameters in any step text and in the scenario name; a referenced parameter that is not defined is an error, with the step path.
  • Data rows: one scenario generated per row, with the row overriding the plan's default values.
  • Suite setup and teardown, run once: they become the first and last scenario, and teardown runs even when an earlier scenario failed, unless the deadline has passed.
  • Preview before saving, without storing anything. An edited block does not change a saved plan: the screen flags which blocks changed and a person generates the new version.
  • Per-run results on screen: each scenario and viewport, step by step, with time, error, screenshots and flakiness over the last 20 runs.
Through the session API and API v1 Create a block, preview or save a plan source (mode preview, create or version), read the source and list plans with an API key.
POST /api/platform/test-automation/tasks                 { key, name, description?, params?, steps }
POST /api/platform/test-automation/compose               { mode, targetId?, planId?, name?, note?, source }
GET  /api/platform/test-automation/plans/{id}/source     source of the current version and blocks that changed
GET  /api/v1/engineering/test-plans                      plans, composition and latest run (engineering:read)

What it does not do

  • A failing step ends the scenario; to clean up inside the scenario anyway, the plan uses the finally step.
  • Parameters are not secrets: they are stored as plain text in the source and the plan and may appear in screenshots and traces. Use test accounts and test data.
  • Values are fixed when the version is created; there are no per-environment values at run time and no AI-generated parameters.
  • Up to 50 blocks per workspace and 20 scenarios per plan, counting data rows. A block cannot use another block.

Conditional, data and AI steps

Conditions, cleanup, database and AI inside the declarative plan.

The declarative plan is still a closed list of actions, with no code, but it gains the steps a real flow needs. if checks whether an element is visible or hidden, or whether the path contains a fragment, and runs the then branch or the else branch, which can nest. finally marks cleanup: the steps after it always run, even after a failure. captureResponse stores a field from the most recent JSON response from the origin itself, such as an order number, for later steps to check. dbCheck verifies in the database what the screen promised.

Two steps use AI on every run. aiAct takes an instruction in Portuguese and, each time, AI chooses a step from the same closed list (click, fill, press, select, wait or navigate within the same origin), checked on the platform and again in the executor. aiAssert judges the page it read against a criterion written in the plan and stores the reason in the step result. In resilient mode, when a selector does not find the element, the executor asks AI for a replacement in the same run and records the swap in the step message.

How it works

  • Per-step results stay aligned with the plan: if branches are steps from the same list, and a skipped step shows as skipped.
  • Cleanup with its own 15 s timeout, separate from the scenario deadline.
  • A captured value becomes a variable, such as [[pedido]], usable in fill, text check and URL check; only JSON responses of up to 64 KB from the verified origin.
  • dbCheck: a SELECT written in the plan runs on the platform, through the workspace data connector, read-only and only on allowed tables. The check is has rows, empty or equals a value, and the executor receives only passed or failed, with the reason.
  • The AI instruction comes from the saved plan, never from the executor request. Outside a target marked as a test environment, AI only navigates and checks.
  • Up to 8 actions per aiAct, 40 AI requests per run and a cost cap per run, with every request recorded.

Example use case

An e-commerce team tests checkout in staging, marked as a test environment. The plan signs in with the test account, uses the aiAct "adicione ao carrinho o primeiro produto em promoção" (add the first sale item to the cart), captures the order number from the checkout API response, uses dbCheck to confirm the order exists in the database with the right status and, in finally, cancels the order to leave the environment clean, even when an earlier check fails.

What it does not do

  • There are no loops and no conditions on variables: if only looks at elements and paths.
  • dbCheck works with Postgres-compatible databases, through the data connector, and the SQL does not accept captured values.
  • aiAssert judges the text of the page it read, without looking at the screenshot.
  • Resilient mode does not change the saved plan: a permanent selector fix requires a person to accept it.

GitHub Actions, Jira and recording

Test results reach the workflow and the ticket.

The GitHub Actions action triggers the repository's Playwright suite through API v1, waits for the gate and fails the job when the suite fails. It writes the run id, the gate (passed, failed or pending) and the link as outputs, and puts the classified failures in the job summary.

With Jira Cloud connected to the workspace, an admin can open a ticket from a failed run, with the scenarios and steps that failed and the run link. And those who would rather show the path than describe it record by browsing with npx playwright codegen on their own machine and paste the text: the platform converts each line, as text, into a declarative step and returns what did not fit, with the line and the reason.

How it works

  • Composite action with no dependencies, with inputs for the API key (repository secret), target, repository, ref, browsers, shards, wait and maximum time.
  • The key uses the tests:run scope, the same as the CLI and the browser_tests pipeline step.
  • Jira Cloud per workspace, with email and API token stored encrypted; at most one ticket per run.
  • Converted recordings: goto, clicks, fill, keypress, select and common assertions become steps, and the plan goes through the same validation as any other.
  • Nothing in the recorded text is executed or evaluated as code.

Example use case

On a pull request, the team workflow calls the AIUBY action with the PR commit and the staging target. When checkout breaks, the job fails with the summary of classified failures right in GitHub. The QA lead opens the run in AIUBY and creates the Jira ticket with the failed steps, without copying anything by hand.

What it does not do

  • The Jira ticket is opened by a person: there is no automatic creation, and only Jira Cloud is supported.
  • There is no recorder inside AIUBY: recording happens on your machine, with Playwright codegen.
  • The action is for GitHub Actions; other CIs use the CLI or API v1.

AI and limits

What AI does here, and what it never does on its own

Does

  • Reads the page in read-only mode (roles, accessible names, data-testid and a screenshot) and proposes a declarative test.
  • Fixes its own proposal from the run error, in up to 3 attempts per request.
  • Suggests a new selector for a step classified as a broken test, with before and after screenshots.
  • Summarizes a run's failures as a suggestion that cites the tests.
  • Interprets, on every run, the plain-language steps in the plan: it picks the next action from the closed list or judges the page against the criterion, with caps on requests and cost per run.
  • In resilient mode, swaps a selector that did not find the element within the same run, and records the swap.

Never does

  • Never saves a test, creates a plan version or writes to a repository without an admin's acceptance.
  • Never proposes write actions (filling, clicking a button, submitting) outside a target marked as a test environment.
  • Never reclassifies a failure or runs anything based on the summary.
  • Never uses a value that looks like a key, token, CPF, CNPJ or card number in a fill step.

Governance and security

Rules the code enforces

  • Runs only against an origin with proven ownership. Ownership is checked again before every run, and navigation stays locked to that origin.
  • Customer code runs on the isolated runner under its own user, with no capabilities and no access to the orchestrator secret. The callback is signed with a single-use token.
  • The billing gate and the test.run policy, scoped per site, are evaluated before running.
  • Caps per organization (runs per day and minutes per month) and per run (shards and time per shard).
  • An AI test proposal only becomes a saved test with an admin's acceptance, and only after passing a real run.
  • The visual baseline is approved by an admin. Requests, acceptances, runs and approvals are audited.
  • Page text is redacted before it goes to the model: keys, tokens, ID documents and emails are removed.

Integrations

What it connects to

Only integrations that exist in the code. Each one carries the status of the feature that uses it.

  • PlaywrightRuns the repository @playwright/test suite, with chromium, firefox and webkit projects and JSON and JUnit reports.E2E tests
  • GitHubShallow clone of the requested ref. Private repositories use the workspace read-only token.E2E tests
  • PipelinesThe browser_tests step runs the suite from the same repository and commit as the pipeline and fails the job if any test fails.E2E tests
  • CLI and API v1aiuby tests run and the run endpoints, with a key scoped to tests:run, for use in any CI.API, CLI & keys
  • Code AutomationThe .spec.ts file of an accepted test opens a ticket to land in the repository with review.AI test authoring
  • Model gatewayAuthoring and selector repair call the model through the Gateway, capped per request and per day.AI test authoring
  • PoliciesThe test.run policy decides, per site, whether a run may proceed.Governance & policies
  • GitHub ActionsA composite action that runs the suite through API v1 and fails the job when the suite fails, with classified failures in the summary.GitHub Actions, Jira and recording
  • Jira CloudAn issue opened by an admin from a failed run, with the steps and the link.GitHub Actions, Jira and recording
  • Data connectorThe dbCheck step checks a Postgres-compatible database, read-only and only on allowed tables.Conditional, data and AI steps

FAQ

Straight answers

What can I use today?

Available: E2E tests, AI test authoring, Visual regression, Composable test plans, Conditional, data and AI steps, GitHub Actions, Jira and recording. Self-service: add it from your account.

Will I need to rewrite my tests?

Available: E2E tests. No. Your repository Playwright suite runs as is, as long as it uses npm, has chromium, firefox or webkit projects in playwright.config and reads the environment URL from BASE_URL. Teams without a suite can describe scenarios with a closed list of actions, no code required.

Can I test any website?

Available: E2E tests. No. Only an origin whose ownership you proved by publishing a file with a code on it, and navigation stays locked to it. Other origins can only load resources, such as a CDN an admin allows.

How do I know whether a failure is a bug or a flaky test?

Available: E2E tests. Each failure comes back classified by fixed rules, with the reason: flaky (passed on a retry or alternates in history), infrastructure (DNS, network, a crashed browser), broken test (a selector that did not find the element, an error in the test code) or likely bug (the element exists, but the value, URL or state is not what was expected). It is a rule, not a verdict: the screen shows the reason so you can check.

Does AI write and save tests on its own?

Available: AI test authoring, Ticket to PR. No. It proposes. A proposal is only ready after passing a real run, in up to 3 attempts, and only becomes a saved test when an admin accepts it. Passing once does not guarantee the test checks what you wanted, so read the steps before accepting. For the .spec.ts file to reach the repository, it goes as a ticket through Code Automation, with review.

Can I use real passwords or personal data in tests?

Available: E2E tests, AI test authoring. Do not. The typed value is stored as plain text in the scenario and may appear in screenshots and run records, and there is no credential vault for tests. Use a test environment with fictitious accounts and data.

Can I use GitHub Actions or open Jira issues?

Available: GitHub Actions, Jira and recording. Yes. The GitHub Actions action triggers the suite through API v1 with a key scoped to tests:run and fails the job when the suite fails. With Jira Cloud connected to the workspace, an admin opens an issue from a failed run, one per run. There is no automatic issue creation and no Jira other than Cloud.

How much does it cost?

Available: E2E tests, AI test authoring, Visual regression. This product has no published price. Runs are capped per day and in minutes per month, per organization, and AI usage for authoring is metered by the Gateway, capped per request and per day. When the product has a price, it will appear on the Pricing page.

Works together with

  • AI EngineeringTest authoring and failure summaries call the model through the Gateway, with the workspace budget, policy and masking.
  • Cost ManagementThe AI cost of every authoring request is metered and recorded, and counts toward the workspace AI usage.

Test Automation

Self-service: add it from your account.