Skip to content

Know what breaks before it breaks

Resilience Testing · Risk analysis, chaos, load and recovery

It starts with a read that changes nothing: replicas, limits, probes and single points of failure for each service, with a score and evidence, and the same lens on every infrastructure PR, in CI. Then come chaos and load generated by the agent in your cluster, with probes, rollback and approval, and recovery drills with measured RTO and RPO.

The problem

What this product solves

A recovery plan that was never tested is an assumption.

Availability risk usually slips in through a small PR: one replica fewer, a removed probe, a database without backups.

Chaos tools are scary for good reason: an experiment with no limit, no abort or no rollback becomes the incident itself.

updated

Who uses it, and for what

Typical profiles, no customer names. Each use carries the status of the feature behind it.

  • Platform team at a digital bankBlock at the PR a service with a single replica, no probe or a database without backups.Resilience risk scan
  • SRE teamGive every service a risk score with the evidence for each point.Resilience risk scan
  • Engineering leadershipProve with a measured drill that the service comes back within the agreed RTO and RPO.Disaster recovery drills

Features

What the product does

Self-service: add it from your account.

  • Resilience risk scan

    Review of replicas, limits, probes and single points of failure, plus a PR check: the new risks a PR introduces in Dockerfile, compose, Kubernetes and simple Terraform, with a score, expiring suppressions and a CI gate through the CLI. Text-based reading: Helm, Kustomize and Terraform modules are not resolved.

  • Chaos experiments

    Chaos in your Kubernetes cluster through the delivery pull agent: deleting pods, scaling replicas, isolating the Deployment network and squeezing CPU or memory, with probes that abort and give the verdict, guaranteed rollback, approval by another person in production, ChaosHub, scheduling, GameDays and a resilience report. Only on delivery-owned Deployments and with the faults the cluster allows.

  • Load testing

    Load generated by the agent in your cluster against a target with verified ownership, with phases, up to five concurrent scenarios, a verdict by p95 and error rate, per-second series and comparison with the previous run; triggered from the UI, the v1 API, a pipeline step or a GitHub Action. GET and HEAD only, with throughput and duration caps on the agent.

  • Disaster recovery drills

    Recovery plan with RTO and RPO on a target with verified ownership and a drill measured by the health probe: restoration only counts after the outage and three consecutive healthy responses, and data age is checked against the RPO. Your team performs the disaster and the restore.

  • Resilience insights

    AI-proposed chaos experiment from the service risks and dependents, approved by an admin before it becomes an experiment, analysis of the result against the hypothesis and a risk lifecycle (open, accepted with a deadline, resolved, reopened) with the owner notified when the risk reopens.

How it works

From start to audit record

  1. Read

    What the platform records about the service is read without changing anything: Compute, dependencies, SLOs, incidents and findings.

  2. Score

    Each risk with evidence, criticality and blast radius. A rule with no data shows up in coverage.

  3. PR check

    The new risks a PR introduces in Dockerfile, compose, Kubernetes and simple Terraform, with a CI gate.

  4. Suppression

    An accepted risk gets a justification and a deadline. It leaves the score and stays listed.

  5. Experiment

    The agent in your cluster applies the allowed fault, probes watch and rollback is guaranteed. In production, someone else approves.

  6. Load

    Scenarios with phases and p95 and error targets, triggered from the app, API v1 or the pipeline.

  7. Proof

    The recovery drill measures time to recover and data age against the RTO and RPO.

Features

Feature by feature

What each one does, how it works, an example from the documentation and what it does not do. Each badge comes from the catalog.

Resilience risk scan

Availability risk shows up in the PR, with a score and evidence

The risk scan assesses each service with what the platform records about it: Compute applications, the dependency and ownership graph, SLOs, incidents from the last 30 days and configuration findings. Each rule ends in one of four states (failed, passed, no data, not applicable), and the result is a score, the coverage and the blast radius across the services that depend on it.

The PR check asks the same question before merge: it reads the PR's Dockerfile, docker compose, Kubernetes manifests and simple Terraform and flags only the new risks it introduces. Below the workspace minimum score, the check blocks the pipeline.

How it works

  • Risks read in the PR: a single replica, missing probes, missing memory limit, unpinned image, database without Multi-AZ, without backup or without deletion protection.
  • Score = max(0, 100 − sum of weights), with critical 25, high 15, medium 8 and low 3. In the per-service scan, the weight grows with criticality and with dependents.
  • Only new risks count: what was already in the base (same rule, same resource) is listed separately and does not lower the score.
  • Suppression by rule and path, with a justification and an expiry of up to 90 days. A suppressed risk leaves the score and stays listed.
  • Workspace minimum score defaults to 60. Zero never blocks, and only admins can change it.
  • The response returns the rule, file and line, resource and recommendation for each new risk.
In the pipeline, through the CLI Exit code 0 when cleared and 6 when blocked by the minimum score.
export AIUBY_TOKEN=...   # key with resilience:check
aiuby resilience check --repo acme/checkout --ref "$GITHUB_HEAD_REF" --base main
aiuby resilience check Dockerfile k8s/app.yaml --base origin/main   # local files; base via git show

What it does not do

  • The scan is text-based: Helm, Kustomize and Terraform modules are not resolved.
  • An unrecognized file type becomes a note in the response, never "approved".
  • A rule with no data neither lowers the score nor counts as passed: a high score with low coverage means little was checked.
  • It does not call your cloud, inject faults or generate load. The check accepts up to 50 files per call.

Chaos experiments

Kill pods in your cluster, with guaranteed rollback and probes watching

Chaos experiments run in your Kubernetes cluster through the same pull agent used for delivery, with no new channel and no third-party binary. The platform sends the agent only the fault for the current stage of an approved run, signed and bound to the namespace and the digest. The agent only applies faults the cluster's local list allows, with its own Role, and only on Deployments that delivery marked as its own.

The faults are deleting pods, scaling replicas (including to zero), isolating the Deployment network with a NetworkPolicy and tightening the CPU or memory limit. The original value is annotated on the resource, and rollback happens on the agent's next cycle when the stage ends, when someone aborts, when the deadline passes or when the agent loses contact with the platform.

How it works

  • Deployment-ready and HTTP probes in the namespace track the run, abort when they fail, and produce the verdict and the experiment score.
  • Production only runs with approval from another workspace admin, bound to the digest and valid for 24 hours. Each environment's policy can refuse.
  • A deployment freeze refuses the run and aborts it midway; a rollout in progress refuses it; one button stops everything at once.
  • ChaosHub with ready-made experiments (half the pods, service to zero, network isolation, tight CPU and memory) and workspace action templates.
  • Scheduling every N hours, and GameDays that trigger several experiments and open a drill incident, outside DORA and without paging on-call.
  • Resilience report with runs, probe pass rate and the workspace score, the average of each experiment's latest score.

Example use case

The SRE team wants to know whether checkout recovers on its own after losing half its pods. They import the ready-made experiment from ChaosHub, pick the Deployment and add an HTTP probe on the health path. In staging, an admin triggers it directly; for production, another person approves the same experiment version. The agent deletes the pods, the probe fails for too long, the experiment aborts, the agent restores the annotated state and the report records the verdict and the score.

What it does not do

  • It only acts on Deployments with the delivery ownership label, in the environment namespace. It does not touch nodes or the cloud network, and it does not inject latency.
  • Without the local fault list and the chaos Role applied in the cluster, the agent runs nothing.
  • Each stage lasts 30 to 900 seconds, never above the agent's local cap (600 by default). An experiment totals at most 30 minutes and 5 stages.
  • Scheduled runs in production still wait for another person's approval.

Load testing

Load generated in your cluster, with a verdict on p95 and errors

Load tests come from the pull agent you run in your cluster: the platform sends a signed action with the scenarios, and the agent generates requests against a target whose ownership the organization has proven, within the caps configured locally. Each scenario has a method, path, throughput, duration and targets for p95 and error rate, and can have warm-up, plateau, spike and ramp-down phases.

A run can combine up to five scenarios at once. At the end, each scenario passes if it stayed within p95 and error rate and delivered at least 90% of the planned requests; the run passes if all of them pass. The per-second series, the per-phase summary and the comparison with the previous run are shown on screen.

How it works

  • The same run is available from the UI and from API v1, with its own key scope, so it can become a pipeline stage.
  • A load step in the pipeline and a GitHub Action (aiuby-load-test) trigger the scenarios and use the verdict as the job gate.
  • Per-scenario comparison with the previous run: difference in p95, error rate and throughput.
  • Caps on the agent that the platform cannot raise: by default 20 requests per second and 60 seconds per scenario, configurable up to 50 and 300.
  • Combined runs with total throughput capped at 50 requests per second.

Example use case

Before releasing a version of the product catalog, the pipeline calls the Action with two scenarios: the listing with a spike phase and search on a plateau. The agent in the cluster generates load against staging, search exceeds the agreed p95, the job fails with the verdict, and the screen shows the phase where latency rose and the difference from the previous week's run.

What it does not do

  • It only generates GET and HEAD requests, with no body or free-form script.
  • It only targets endpoints with verified ownership; third-party addresses are refused.
  • It is not a high-volume stress testing tool: the caps live on the agent, on your side.

Disaster recovery drills

RTO and RPO measured in a drill, not declared in a spreadsheet

A recovery plan links a target with verified ownership to a health path, the expected status, the RTO and, optionally, the RPO, with the procedure in text. In a drill, the clock starts when someone begins it. The platform probes health every 15 seconds and only marks the restore as verified after it sees the outage and three consecutive healthy responses.

Time to recovery is compared to the RTO. When the health response includes a date field, the age of the restored data is compared to the RPO. Each probe is kept as drill evidence, and the result goes into the audit log.

How it works

  • A drill passes only if it recovered within the RTO and, with an RPO set, with data fresh enough.
  • Without seeing the outage, the drill does not count as a recovery: three good responses from the start prove nothing.
  • Once the RTO plus one hour of slack passes without recovery, the drill fails on its own.
  • One drill per plan at a time, abortable at any moment.

Example use case

The database team needs to show it can restore the orders service within 30 minutes with at most 15 minutes of data lost. They register the plan with the health path that returns the date of the latest order, start the drill and run the restore procedure. The platform records the outage, the three healthy responses and the data age, and the drill closes as passed with the measured time.

What it does not do

  • It does not take the service down or perform the restore: your team causes the disaster and the recovery, following the procedure.
  • It only measures through the target's HTTP health path; the RPO depends on the response including the date field.

Resilience insights

From risk found to proposed experiment to risk with an owner

Starting from the service's latest assessment, with the failed rules and the services that depend on it, AI proposes a chaos experiment with a hypothesis and a plan. The suggestion stays pending until a workspace admin approves it, choosing environment and Deployment; it then becomes a regular experiment, which also goes through run approval. After the experiment, the probe verdict decides whether the hypothesis held, and AI only writes the summary.

Each assessment risk (service and rule) has a lifecycle: open, accepted with a reason and deadline, resolved or reopened. A resolved risk that fails again in a later assessment, or an expired acceptance, comes back as reopened, and the risk owner is notified.

How it works

  • The proposed plan lists the dependent services to watch during the experiment.
  • Without AI available, a fixed rule-to-fault map applies; nothing runs on its own.
  • Reopening is checked on every new assessment of the service, with one notification per assessment.
  • The risk owner is a person in the organization, chosen by an admin.

Example use case

The payments service assessment flags a single replica and three dependent services. AI proposes scaling to zero with the hypothesis that the fallback keeps checkout up, citing the dependents. The team lead approves the suggestion for staging, the experiment runs, the probe holds the verdict and the summary explains the result. The risk is marked as accepted for 30 days, with the lead as owner; when it expires with the rule still failing, it reopens and she is notified.

What it does not do

  • AI does not approve, choose the target or run experiments.
  • The verdict comes from the probes, not from AI text.
  • The suggestion starts only from the rules that failed in the service's latest assessment.

AI and limits

What AI does here, and what it never does on its own

Does

  • Proposes a chaos experiment with a hypothesis, based on the rules that failed in the service assessment and on the services that depend on it.
  • Writes the summary of the experiment result, against the proposed hypothesis.

Never does

  • Never approves or runs an experiment: the suggestion waits for a workspace admin, and execution goes through chaos approval.
  • Never decides the verdict: the experiment probes decide.
  • Never picks the environment or target Deployment.

Governance and security

Rules the code enforces

  • The risk read writes nothing, does not call the customer cloud and injects no faults.
  • A rule with no data never counts as passed: it shows up in coverage.
  • The minimum score that blocks a PR belongs to the workspace, and only admins change it.
  • Suppression requires a rule, a path, a justification and a deadline of up to 90 days. It is revocable and recorded in the audit trail.
  • The PR check uses a key with its own scope, separate from engineering read access.
  • Chaos and load only run through the agent you install, with the fault list, the Role and the caps defined in your cluster. The platform cannot raise those limits.
  • Every action sent to the agent is signed and bound to the namespace and digest. Chaos only acts on Deployments carrying the delivery ownership label.
  • Chaos in production requires approval from another admin, valid for 24 hours. A deployment freeze refuses and aborts.
  • An AI experiment suggestion only becomes an experiment after a workspace admin approves it.

Integrations

What it connects to

Only integrations that exist in the code. Each one carries the status of the feature that uses it.

  • GitHubReads PR files from a repository in the engineering catalog.Resilience risk scan
  • Dockerfile and docker composeUnpinned images, missing healthcheck, restart policy and memory limit.Resilience risk scan
  • KubernetesManifests with one replica, no probes, no memory limit or a latest image.Resilience risk scan
  • TerraformSimple files: databases without Multi-AZ, without backups or without deletion protection.Resilience risk scan
  • aiuby CLI in CIThe check command returns an exit code the pipeline uses as a gate.API, CLI & keys
  • Delivery pull agent (Kubernetes)Applies and rolls back chaos faults and generates load inside your cluster.Chaos experiments
  • GitHub ActionsThe aiuby-load-test action triggers load scenarios and uses the verdict as the job gate.Load testing

FAQ

Straight answers

What can I use today?

Available: Resilience risk scan, Chaos experiments, Load testing, Disaster recovery drills, Resilience insights. Self-service: add it from your account.

Will it run chaos in production?

Available: Chaos experiments, Resilience risk scan. Only if you want it to and someone else approves. In production, every experiment needs approval from another workspace admin, bound to that experiment version and valid for 24 hours, and the environment policy can refuse it. The agent only applies the faults your cluster allows and rolls back when the stage ends, when someone aborts or when it loses contact with the platform. The risk read and the PR check execute nothing: they only read.

Which faults does chaos inject?

Available: Chaos experiments. Deleting pods, scaling replicas (down to zero), isolating the Deployment network with a NetworkPolicy and tightening the CPU or memory limit, always on Deployments labeled by delivery. It does not take down nodes, inject latency or touch cloud networking or services.

How does load testing work?

Available: Load testing. The agent in your cluster generates GET and HEAD requests against a target with verified ownership, within the caps you define on it. Each scenario has p95 and error rate targets, and the run passes or fails against them. You can trigger it from the app, API v1, a pipeline step or the GitHub Action, and use the verdict as a gate.

How is the score calculated?

Available: Resilience risk scan. With a published formula: score = max(0, round(100 − Σ weight(severity) × exposure)) over the rules with risk; weights: critical 25, high 15, medium 8, low 3; exposure = (critical service ? 1.5 : 1) × (1 + 0.05 × min(transitive dependents, 10)); coverage = evaluated ÷ (evaluated + no data); a rule with no data neither deducts nor counts as passed; below 60: at risk. There are 16 rules in the per-service read. The PR check has its own rule set for infrastructure files.

Does a high score mean the service is resilient?

Available: Resilience risk scan. Not necessarily. A rule with no data neither deducts nor passes: it shows up in coverage. A high score with low coverage means little was verified.

Does the PR check block the merge?

Available: Resilience risk scan. It blocks the pipeline when the PR score falls below the workspace minimum, which defaults to 60 and never blocks when set to zero. Only new risks count. What was already in the base is listed separately. Through the CLI, the block comes out as exit code 6, and your CI decides what to do with it.

How much does it cost?

Available: Resilience risk scan, Chaos experiments, Load testing, Disaster recovery drills, Resilience insights. This product has no price of its own published. Its features depend on the Cortex contract, the same one as Engineering Intelligence, and are enabled per organization by the AIUBY team.

Works together with

  • Security TestingConfiguration findings from the service scanners count toward the risk score.
  • Software DeliveryCompute applications are read to check replicas, probes and limits, and the check runs in the PR pipeline.
  • Developer PortalThe graph of services, owners and dependencies provides the criticality and blast radius of each risk.

Resilience Testing

Self-service: add it from your account.