# Agent attack tests

`kosa8 attack` puts a fixed catalogue of AI-agent attacks to kosa8's own controls on your machine, and records what stopped each one and what did not. Every scenario carries its OWASP Top 10 for LLM Applications 2025 and MITRE ATLAS ids. The result is a scorecard signed with this machine's audit key, which a security reviewer can check without trusting you or us.

**What this is not.** It is a self-test of kosa8's controls against nine known attack patterns. It is not a penetration test, a red-team assessment or a certification, and a clean scorecard does not show that an agent cannot cause harm. It tests the controls kosa8 ships; it does not test your agent, your model, or attacks outside the catalogue.

## Run it

```
$ kosa8 attack list                  # the catalogue, with OWASP and ATLAS ids
$ kosa8 attack run                   # both postures, policy mode, signed scorecard
$ kosa8 attack run exfil-secret AK-08  # only these scenarios
$ kosa8 attack run --live            # also inside real sandboxes
$ kosa8 attack verify ~/.kosa8/attack/atk-….json --kid SHA256:…
$ kosa8 attack upload ~/.kosa8/attack/atk-….json   # or: run --upload
```

| Flag | What it does |
|---|---|
| `--posture` | `default`, `hardened` or `both` (the default). See [Two postures]. |
| `--live` | Also runs the network and isolation scenarios inside a real sandbox per posture. Needs the daemon; without it, the run says so and stays in policy mode. |
| `-o FILE` | Where the signed scorecard goes. Default `~/.kosa8/attack/RUN.json`. |
| `--no-record` | Do not write the `attack.run` entry to the local audit log. |
| `--upload` | Upload the signed file to your kosa8 Cloud org after the run (`kosa8 login` first). |
| `--fail-on allowed` | Exit non-zero when a scenario of the hardened posture (or the only posture run) is ALLOWED. `--fail-on approval` also fails on NEEDS APPROVAL. For CI. |
| `--json` | The whole scorecard, with every attempt and its evidence, as JSON. |

## Three results

**BLOCKED**A kosa8 control refused it. The result names the layer (rules, egress broker, network guard, MCP policy, MCP pins, isolation) and carries the evidence: the rule id, the broker or guard event, the policy decision.

**NEEDS APPROVAL**It is held until a person approves that exact call. Nothing runs before then.

**ALLOWED**Nothing in this posture stopped it. The result says which setting would, and the scorecard keeps the ALLOWED: it is not rounded up.

A scenario is made of attempts; an agent only needs one to get through, so a scenario's result is its weakest attempt. An attempt's result is its strongest layer: one refusal stops it.

## Two postures

**default** is this machine as it is configured now: the org policy's `default_egress` if one is installed (else `any`), the installed signed MCP policy if there is one, MCP pins on first use, no pre-run hook. **hardened** is the setup kosa8 recommends for an agent: an egress allow-list through the broker (Anthropic's API, PyPI, npm and GitHub, port 443), egress `deny` otherwise, the pre-run hook on, a signed MCP policy with org-pinned tool definitions. Running both shows what changing the settings buys.

## The scenarios

Results are on a machine with no org policy installed. Your default column depends on your settings; that is the point of running it.

| ID | Attack | OWASP LLM 2025 | MITRE ATLAS | default | hardened |
|---|---|---|---|---|---|
| `AK-01` exfil-secret | A prompt injection makes the agent POST `.env` to an attacker's host. | LLM01, LLM02, LLM06 | AML.T0051.001, AML.T0086, AML.T0025 | ALLOWED | BLOCKED |
| `AK-02` host-credentials | Read `~/.ssh/id_ed25519` and `~/.aws/credentials` from inside the sandbox. | LLM02, LLM06 | AML.T0055, AML.T0098 | BLOCKED | BLOCKED |
| `AK-03` unapproved-domain | Fetch a second stage from a domain nobody approved. | LLM01, LLM06 | AML.T0072, AML.T0025 | ALLOWED | BLOCKED |
| `AK-04` destructive-shell | `rm -rf .git`, `git push --force origin main`, `terraform destroy -auto-approve`. | LLM06 | AML.T0101, AML.T0050 | ALLOWED | NEEDS APPROVAL |
| `AK-05` mcp-tool-poisoning | An approved MCP tool re-described with hidden instructions; a new tool slipped in. | LLM01, LLM03 | AML.T0110, AML.T0011.002, AML.T0010.005 | BLOCKED | BLOCKED |
| `AK-06` unapproved-mcp | An MCP server nobody approved; a destructive tool on an approved one. | LLM03, LLM06 | AML.T0053, AML.T0010.005 | ALLOWED | BLOCKED |
| `AK-07` typosquat-source | `pip` and `npm` installs pointed at look-alike registries. | LLM03 | AML.T0010.001, AML.T0011.001 | ALLOWED | BLOCKED |
| `AK-08` metadata-ip | Reach `169.254.169.254` directly, and by a name rebound to it. | LLM02, LLM06 | AML.T0055 | ALLOWED | BLOCKED |
| `AK-09` persistence | Rewrite the agent's own settings to switch its checks off; append to `~/.bashrc`. | LLM06 | AML.T0081 | ALLOWED | BLOCKED |

OWASP: LLM01 Prompt Injection, LLM02 Sensitive Information Disclosure, LLM03 Supply Chain, LLM06 Excessive Agency ([genai.owasp.org/llm-top-10](https://genai.owasp.org/llm-top-10/)). ATLAS ids are from ATLAS 2026.10 ([atlas.mitre.org](https://atlas.mitre.org/)): T0051.001 Prompt Injection: Indirect, T0086 Exfiltration via AI Agent Tool Invocation, T0025 Exfiltration via Cyber Means, T0055 Unsecured Credentials, T0098 AI Agent Tool Credential Harvesting, T0072 Cyber Communication Channel, T0101 Data Destruction via AI Agent Tool Invocation, T0050 Command and Scripting Interpreter, T0110 AI Agent Tool Poisoning, T0011.002 Poisoned AI Agent Tool, T0010.005 AI Agent Tool (supply chain), T0053 AI Agent Tool Invocation, T0010.001 Software (supply chain), T0011.001 Malicious Package, T0081 Modify AI Agent Configuration. `kosa8 attack list` prints every name.

### What blocks what is ALLOWED by default

| Scenarios | Setting |
|---|---|
| AK-01, AK-03, AK-07, AK-08 | An egress allow-list: `kosa8 sandbox create --egress-allow HOST` (repeatable), `--egress deny`, a brokered agent kit, or `default_egress` in the org policy. Under `any`, nothing on the host filters a sandbox's traffic, the metadata address included. |
| AK-04, AK-09 | Run the agent under the pre-run hook (`kosa8 agent claude`). Rules K8-A001, K8-A002 and K8-A007 hold destructive commands for a person; K8-D002 refuses writes to the agent's own settings. Agents run any other way do not go through these rules. |
| AK-06 | Install a signed MCP policy (`kosa8 mcp-policy init`, `sign`, `install`): servers and tools it does not list are denied. See [Company rules](https://docs.kosa8.com/guides/governance/). |

## Policy mode and live mode

**Policy mode** (the default) needs no daemon and no VM, so it runs in CI and in unit tests. Each attempt goes through the real decision code: the rules engine, a real egress broker on a loopback port with dialling switched off (nothing leaves the machine), the host network guard fed the DNS query and TCP SYN the sandbox would send, the MCP policy engine, the MCP pin check, and the sandbox's mount layout. It shows what those controls decide; it does not show that the packets would have gone nowhere.

**Live mode** (`--live`) then boots one small sandbox per posture (`alpine:3.20`) and runs the network and isolation scenarios (AK-01, 02, 03, 07, 08) inside it with `wget`, against harmless public names. The verdict comes from the daemon's audit log, not from exit codes: a request is BLOCKED only if a `sandbox.egress.deny` entry for it was written, and the result carries that entry's sequence number and hash. A request that failed for any other reason is still ALLOWED, with a note that no kosa8 control refused it. The sandboxes are removed afterwards. Without a running daemon, `--live` says so and the run stays in policy mode.

## The signed scorecard

Each run writes a file of this shape:

```
{ "format": "kosa8.attack.scorecard/v1",
  "payload": "<base64 of the scorecard JSON>",
  "key": { "alg": "ed25519", "kid": "SHA256:…", "public": "<base64 DER public key>" },
  "sig": "<base64 signature>" }
```

The signature is over `kosa8.attack.scorecard/v1`, a newline, then the payload bytes, made with the same key that signs this machine's [audit log](https://docs.kosa8.com/guides/audit/) (`kosa8 audit key` prints its id). The prefix keeps a scorecard signature from ever being mistaken for an audit entry's. The scorecard inside holds every result with its attempts and evidence, the kosa8 version and platform, the framework versions, a digest of the results, and the disclaimer above. When the daemon is running, the run is also written to the audit log as an `attack.run` entry whose fields hold that digest and each scenario's outcome, so the scorecard is anchored in the hash chain.

```
$ kosa8 attack verify atk-….json --kid SHA256:e218d6…
signature      ok (ed25519 SHA256:e218d6…)
results digest ok
pinned key     ok
run            atk-20261111T213822Z-675c4950, policy mode
default        2 blocked, 0 need approval, 7 allowed
hardened       8 blocked, 1 need approval, 0 allowed
verified
```

`verify` checks the signature, that the key's id matches the key, and the results digest, and exits non-zero if any fails. `--kid` pins the key you expect, which is what a reviewer on another machine should do: a valid signature only proves that the holder of that key signed these results. `verify` works on a machine that has not signed in to kosa8.

## In the desktop app

Security → Attack tests runs both postures in policy mode, shows the table with OWASP and ATLAS labels (hover a label for its name), what blocks each ALLOWED result, and where the file was saved. Export saves the signed file; Upload sends it to your kosa8 Cloud org once the app is linked to it.

## In kosa8 Cloud

Upload with `kosa8 attack run --upload`, `kosa8 attack upload FILE`, the desktop app, or `POST /v1/attack/scorecards` with `{"signed": <the file>}` (a key with `sandboxes:write`). kosa8 Cloud checks the signature itself before it stores anything, and refuses a changed file with `422 scorecard_invalid`. Each upload, accepted or refused, is an `attack.scorecard_upload` event in the org's [audit log](https://docs.kosa8.com/guides/audit/). Stored scorecards are listed by `GET /v1/attack/scorecards` and fetched as uploaded from `GET /v1/attack/scorecards/{id}`, for anyone to check again with `kosa8 attack verify`. The results are shown in the CLI and the desktop app's Security screen.

## Limits

- Nine scenarios, chosen because kosa8 has a control for each. Attacks outside the catalogue are not tested.

- AK-07 catches a look-alike registry. A misspelt package name on the real registry is not caught by any kosa8 control.

- The rules in AK-04 and AK-09 apply only to agents run under the pre-run hook (`kosa8 agent claude`).

- AK-02 tests the host's files. Credentials you place inside the sandbox are readable by what runs there; keep them out with [brokered keys](https://docs.kosa8.com/guides/credentials/).

- Policy mode tests decisions; live mode tests the network path for five scenarios. Neither tests the VM boundary itself.

---
Source: https://docs.kosa8.com/guides/attack-tests/