E2E tests
Your repository's Playwright suite runs isolated and comes back triaged.
Test Automation runs the Playwright suite that already lives in your repository, as is: a shallow clone of the requested ref, dependency install and npx playwright test, one shard per isolated executor run, in the browsers you choose (Chromium, Firefox and WebKit). The target is always an environment whose ownership you have proven, and ownership is checked again before each shard.
Each failure comes back classified by rule, with the reason alongside: likely bug, flaky, infrastructure or broken test. The run result works as a CI gate, through the CLI, the API or as a pipeline step. For teams without a suite, the product also runs declarative scenarios: a closed list of actions, with no code.
How it works
- Parallel shards and retries per run. A run passes only if no test failed; a flaky test counts as passed, as in Playwright.
- Triage by fixed rules, in order: passed on a retry (flaky), DNS, network or crashed browser (infrastructure), a selector that did not find the element or an error in the test code (broken test), history that alternates over the last 10 runs (flaky), and everything else (likely bug).
- Failure artifacts according to your playwright.config: screenshot, trace and video, plus one JUnit file per shard. Only admins can download traces and videos.
- CI gate: aiuby tests run --wait exits with 0 when it passed and 6 when any test failed. The browser_tests pipeline step runs the suite from the same repository and commit and fails the job if any test fails.
- Declarative scenarios: goto, click, fill, press, select, waitFor, checks for visible element, text, URL and title, and screenshot, with selectors by role, text, test-id or limited CSS, on desktop and mobile viewports. Navigation stays locked to the verified origin.
- Optional AI failure summary, for admins only: it sees only the failed tests, with the message redacted, and returns a suggestion that cites the tests. AI reclassifies and runs nothing.
export AIUBY_TOKEN=... # key with the "Run Playwright suites in CI" scope (tests:run)
aiuby tests run --suite acme/loja-e2e --target https://staging.acme.com.br \
--ref "$GITHUB_SHA" --browser chromium --browser firefox --shards 2 --waitWhat it does not do
- It only tests origins with ownership proven by the /.well-known/aiuby-test-automation.txt file, and the environment must be reachable from the internet.
- npm only: pnpm and yarn are not accepted. Projects in playwright.config must be named chromium, firefox or webkit.
- Caps per run: 4 shards by default (8 at most), 600 s per shard by default, up to 20 artifacts and 3.5 MB per shard; anything that does not fit is listed as discarded. Artifacts are kept for 14 days by default.
- Triage is a rule, not a verdict: a selector that did not find the element may be an interface that changed on purpose or an element that disappeared because of a bug.