# Metrics, logs and labels

Four things for running many sandboxes at once: what each one is using, what ran in it, a way to find it again, and a webhook when one ends without you deleting it or a command in it runs out of memory.

## Metrics

```
curl -s https://api.kosa8.com/v1/sandboxes/sbx_…/metrics -H "Authorization: Bearer $KOSA8_API_KEY"
# {"metrics":{"id":"sbx_…","state":"running","cpus":2,"mem_mb":4096,"disk_gb":32,
#   "cpu_pct":48.7,"cpu_seconds":312.4,"mem_bytes":1342177280,"mem_pct":31.3,
#   "disk_bytes":2147483648,"net":{"out_frames":9120,"out_bytes":1203391,"in_frames":15003,"in_bytes":20311822},
#   "egress_denied":0,"at":"2026-11-14T12:00:00Z"}}
```

- **Read from outside the sandbox.** The host reads the VM's own process and disk files; nothing runs inside the sandbox to answer. So asking never wakes a sleeping sandbox and never keeps one awake: a sleeping sandbox answers `{"state":"sleeping","sleep_bytes":…}`.

- **`cpu_pct`** is CPU used over a quarter second, as a share of the sandbox's vCPUs (100 is every vCPU busy). `cpu_seconds` is the total since the VM booted or woke, so two readings give a rate over any period.

- **`mem_bytes`** is host memory the VM holds. Memory the guest has touched stays held, so it rises with use and seldom falls; it is not the guest's "free" figure.

- **`disk_bytes`** is the space the sandbox's disks take on the host. The disks are sparse: a 32 GB disk that is mostly empty takes far less.

- **`net`** counts Ethernet frames and bytes (headers included) the guest sent that its egress policy passed, and those delivered to it. `egress_denied` is packets the policy refused.

- CPU and memory are `null` where the host cannot read them (a host whose VMs run inside the daemon, such as macOS). The call takes about a quarter second.

## Logs

```
curl -s 'https://api.kosa8.com/v1/sandboxes/sbx_…/logs?limit=20' -H "Authorization: Bearer $KOSA8_API_KEY"
# {"id":"sbx_…","entries":[
#   {"at":"…","cmd":["pip","install","-r","requirements.txt"],"exit":0,"ms":8312,"stdout_tail":"…Successfully installed …"},
#   {"at":"…","cmd":["python","train.py"],"exit":137,"ms":40210,"stream":true,"stderr_tail":"Killed","oom":true}]}
```

Every command run in the sandbox, through the API or a direct exec, is logged on its host: when, the command (long arguments clipped), the exit code, how long it took, and the last 2 KiB of stdout and stderr. The newest 200 are kept, oldest first in the answer. The log lives on the sandbox's host, not in kosa8's database, and is removed with the sandbox. Commands inside a long-running process you started (a server's own requests) are not logged; only the commands you ran.

## Labels

```
curl -s -X POST https://api.kosa8.com/v1/sandboxes \
  -H "Authorization: Bearer $KOSA8_API_KEY" -H 'Content-Type: application/json' \
  -d '{"labels":{"customer":"acme","run":"eval-42"}}'

curl -s 'https://api.kosa8.com/v1/sandboxes?label=run=eval-42&label=customer' -H "Authorization: Bearer $KOSA8_API_KEY"

curl -s -X PUT https://api.kosa8.com/v1/sandboxes/sbx_…/labels \
  -H "Authorization: Bearer $KOSA8_API_KEY" -H 'Content-Type: application/json' -d '{"labels":{"run":"eval-43"}}'
```

- Up to 16 labels. Keys are 1–63 letters, digits, `.`, `_`, `/` or `-`, starting with a letter or digit; values are strings up to 256 characters.

- `?label=key=value` matches that value; `?label=key` any sandbox with the key. Repeat it and every one must match. It works with `limit`, `cursor` and `all=1`.

- `PUT …/labels` replaces them all; `{"labels": null}` clears them. Labels are for your bookkeeping and are not secret: do not put credentials in them.

## Webhooks

- **`sandbox.killed`** `{id, reason}`: a sandbox ended without you deleting it. Reasons: `ttl` (its `ttl_s` or `delete_after`), `egress` (its egress policy could not be applied at create), `not_on_host` (its host no longer has it, for example after a host reboot), `host_lost` (its host was silent for 24 hours) and `host_removed` (your own host was removed). It comes beside the `sandbox.deleted` or `sandbox.error` event you already get.

- **`sandbox.oom`** `{id, exit, cmd, at}`: a command ended with exit 137 and the guest's out-of-memory killer had just killed a process. The sandbox keeps running; the command did not. Give the sandbox more memory (a larger `size`), or use less. It reaches you with the host's next heartbeat, within about a minute. A kill with exit 137 that was not for memory (`kill -9`) is not reported, and an image without `cat` leaves `oom` unknown in the log rather than guessed.

Subscribe as for any other event: `POST /v1/webhooks {"url": …, "events": ["sandbox.killed", "sandbox.oom"]}`. A new endpoint that names no events gets every event, these two included; an existing endpoint keeps the events it chose. See [Webhook recipes](https://docs.kosa8.com/guides/cloud-webhooks/) for checking signatures.

---
Source: https://docs.kosa8.com/guides/cloud-observability/
