Guides / Python / Errors and troubleshooting

Python errors and troubleshooting

Every exception the kosa8 package raises, with a real message from kosa8 0.9.36 and what to do about it, then a table of problems you can meet and how to fix them.

What this is, in plain words

When something goes wrong, the SDK raises an exception: Python's way of stopping and saying what happened. kosa8's exceptions are sorted into kinds, so your program can react to each kind differently: "the daemon is not running" calls for a different response than "that sandbox does not exist" or "the company policy forbids this".

Most messages come straight from the kosa8 daemon and end with advice in brackets: what to do next. The SDK keeps the problem and the advice in separate attributes, so a program can show the advice to a person, or hand it to an AI agent as the reason its step failed.

One thing is not an exception: a command in the sandbox that fails. If make test exits with code 2, run or exec returns an ExecResult with exit_code == 2. An exception means kosa8 itself could not do what you asked.

The hierarchy

Exception
└─ Kosa8Error                      anything this SDK raises; also client-side checks
   ├─ Kosa8ConnectionError         the daemon could not be reached
   ├─ Kosa8TimeoutError            a socket timeout passed
   ├─ Kosa8StreamError             run, exec or attach ended with an error frame (output kept)
   ├─ Kosa8ProtocolError           a streamed frame was not valid NDJSON (0.1.1)
   ├─ Kosa8CheckpointError         checkpoints: none to rewind to, bad name, lock timeout, failed rewind
   └─ Kosa8ApiError                the daemon answered with an HTTP error (any status, e.g. 500)
      ├─ Kosa8BadRequestError      400
      ├─ Kosa8AuthError            401
      ├─ Kosa8PolicyError          403
      │  └─ Kosa8ApprovalRequiredError   403 with a signed-approval refusal
      ├─ Kosa8NotFoundError        404
      ├─ Kosa8ConflictError        409
      └─ Kosa8UnsupportedError     501

All of them are importable from kosa8. Catch the specific class first and Kosa8Error last; catch Kosa8ApprovalRequiredError before Kosa8PolicyError, since it is a subclass. Plain Kosa8Error (not a subclass) is raised for checks the SDK makes before sending anything, and for a second write_stdin (0.1.1). On 0.1.0 a failed rewind was also a plain Kosa8Error; from 0.1.1 it is a Kosa8CheckpointError, which existing except Kosa8Error handlers still catch.

Attributes

from kosa8 import *

def show(label, fn):
    try:
        fn(); print(label, "-> no error")
    except Kosa8Error as e:
        print(f"{label} -> {type(e).__name__} status={e.status}\n   message: {e}\n"
              f"   daemon_message: {e.daemon_message}\n   remediation: {e.remediation}\n   {e.method} {e.path}")

k = Kosa8Client()
show("not found delete", lambda: k.delete_sandbox("doc-py-nope"))

not found delete -> Kosa8NotFoundError status=404
   message: no such sandbox: doc-py-nope (list sandboxes with `kosa8 sandbox ls`)
   daemon_message: no such sandbox: doc-py-nope
   remediation: list sandboxes with `kosa8 sandbox ls`
   DELETE /kosa8/v1/sandboxes/doc-py-nope

Its body was {'error': 'no such sandbox: doc-py-nope', 'remediation': 'list sandboxes with `kosa8 sandbox ls`'}. The other messages below come from the same show helper.

Each class

Kosa8ConnectionError

When: the request never reached a daemon. Any OSError while connecting or sending becomes this: no socket file, a file that is not a socket, a path too long, nothing listening on a TCP port.

kosa8 daemon unreachable at unix:///tmp/no-daemon-here.sock: [Errno 2] No such file or directory (is kosa8 installed and its daemon running? start it with `kosa8 daemon start`)
kosa8 daemon unreachable at http://127.0.0.1:1: [Errno 61] Connection refused (is kosa8 installed and its daemon running? start it with `kosa8 daemon start`)
kosa8 daemon unreachable at unix:///etc/hosts: [Errno 38] Socket operation on non-socket
kosa8 daemon unreachable at unix:///tmp/xxxx…xxxx/kosa8.sock: AF_UNIX path too long

What to do: start the daemon (kosa8 daemon start, then kosa8 daemon status) or fix the address. e.__cause__ is the original OSError; e.__cause__.errno is 2 (no file) or 61 (refused) on macOS. Nothing was done on the daemon side, so the call is safe to retry once it is up. The "is kosa8 installed" hint is added only for "no such file" and "refused".

Kosa8TimeoutError

When: you set a timeout (on the client or the call) and no bytes arrived for that many seconds, while connecting, waiting for the answer to start, or between two pieces of a streamed answer.

POST /kosa8/v1/sandboxes/doc-py-g1/exec timed out talking to unix:///Users/dhruvit/.kosa8/run/kosa8.sock
POST /kosa8/v1/sandboxes/doc-py-g4/exec timed out mid-stream

What to do: know that the work was not cancelled. In testing, a command that timed out on the client side went on to finish in the sandbox. Before retrying something that is not idempotent (creating a sandbox, taking a snapshot with a fixed id), check whether it happened. To limit how long a command may run, limit it inside the sandbox (["timeout", "30", …]). Details: Timeouts.

Kosa8StreamError

When: a run, exec or attach started streaming and then ended with an error frame instead of an exit code, or the stream stopped with no final frame at all. A container id that does not exist is reported this way, not as a 404:

Kosa8StreamError 'no such container: nope'          # status None, daemon_message 'no such container: nope'
stream ended unexpectedly (no final frame)          # the connection dropped mid-run (from the source)

What to do: e.stdout and e.stderr hold what the command printed before the error; log them. A dropped stream usually means the daemon restarted or the sandbox was deleted or rewound from somewhere else mid-command; check get_sandbox(name). exec_frames never raises this: it yields the error frame for you to inspect.

Kosa8CheckpointError

When: the client-side checkpoint bookkeeping refuses, before any request is sent.

no checkpoints for "doc-py-never" — take one with checkpoint() or `kosa8 checkpoint doc-py-never`
no checkpoint "nope" for sandbox "doc-rb-db"
invalid sandbox name "../etc"

What to do: for the first two, list what exists with list_checkpoints(name) or kosa8 checkpoint ls NAME; to= takes a checkpoint id or a label (case-insensitive). A rewind that raises this has changed nothing: the target is looked up before the sandbox is touched. The name check refuses empty names, ., .., names containing / or \, and names starting with a dot, because the name is a key in the shared checkpoint file.

Two more cases from 0.1.1. The checkpoint file's lock is waited for at most lock_timeout seconds (FileCheckpointStore(path=None, *, lock_timeout=10.0)); tested by holding the lock from another process with lock_timeout=1:

timed out after 1s waiting for …/ck/checkpoints.json.lock; another kosa8 process is holding it

Something else (the CLI, the Mac app, another program) held the lock that long; nothing was written, so retry, or raise lock_timeout. On 0.1.0 the call waited forever. And a rewind that failed halfway is this class too.

Kosa8ProtocolError (0.1.1)

When: a line of a streamed answer (run, exec, exec_frames, attach) is not valid JSON, or its data is not valid base64. (From the source; a working daemon does not send such frames.) The message has the form POST /kosa8/v1/sandboxes/X/exec: the daemon sent a frame that is not valid NDJSON (<reason>): <first 200 bytes of the line>, with method and path set and the decoding error as __cause__. On 0.1.0 a raw json.JSONDecodeError escaped instead.

What to do: check that KOSA8_HOST points at a kosa8 worker and not at a proxy that rewrites or buffers responses, and that client and daemon versions match (kosa8 version). The command may have run; check before retrying.

Plain Kosa8Error: checks before sending

When: the SDK refuses an argument itself. Nothing reaches the daemon.

refusing to talk to 10.0.0.5:7070 over plain http: the token and every request would cross the network unencrypted. Use https://10.0.0.5:7070 (a worker behind a TLS-terminating tunnel or proxy), or set insecure=True / KOSA8_INSECURE=1 if this link is already private and trusted
KOSA8_HOST scheme "ftp" is not supported; use http://, https://, tcp:// or unix://
actor must be human, agent or tool, not 'robot'
an egress allow-list needs at least one host; use egress="deny" for no network
egress_allow already limits the network to the hosts it lists; do not also set egress
give at least two variants; with one there is nothing to compare

The first three come from Kosa8Client(...) itself, so a bad KOSA8_HOST fails at construction, not at the first call. Also from the source: KOSA8_HOST '…' names no host, and from explore, forked 1 branches for 2 variants; see list_sandboxes() for what was created and keep X is not one of the branches (…); all of them were kept. What to do: fix the argument; retrying cannot help.

Kosa8CheckpointError: a rewind that failed halfway

When: rewind deleted the sandbox (that has to happen first, because the restored copy takes its name) and then the restore was refused. (From the source; not provoked for this page.) On 0.1.1 it is a Kosa8CheckpointError with this message, and __cause__ is the daemon's error:

restore failed and doc-py-demo no longer exists (it was deleted to free its name for the restore): <the daemon's error> — the checkpoint is intact; recover with restore_snapshot('doc-py-demo-ckpt-1', name='doc-py-demo')

(0.1.0 raised a plain Kosa8Error saying restore failed and doc-py-demo is now stopped: …, which was wrong: the sandbox has been deleted, not stopped.)

What to do: fix the cause, then run exactly the call the message gives. The snapshot is intact and the checkpoint list was not changed. On a Mac the usual cause is a locked screen: macOS will not restore a saved VM while the screen is locked (see the table below). The sandbox is gone from list_sandboxes() until you restore it; its name is free again but its disk files may remain under ~/.kosa8/sandboxes/, and a create_sandbox with that name then fails (below).

Kosa8BadRequestError (400)

When: the daemon refused a value. Most often an egress rule.

egress allow "*.com": com is a public suffix, under which anyone can register a name, so the wildcard would admit strangers' hosts; allow a domain below it, as in *.example.com
egress policy "sometimes": want any, deny, fakenet, or allow:HOST[,HOST...]

What to do: fix the input; the same request fails the same way. These carry no remediation; the message is the explanation.

Kosa8AuthError (401)

When: an agent token was sent and is not valid, or a remote worker's bearer token is missing or wrong. A bad agent token is never treated as "no token": every call is refused, even health().

agent token is not recognised; mint one with `kosa8 agent token create NAME`

What to do: check KOSA8_AGENT_TOKEN and agent_token= (the environment variable is used when the argument is not given), mint a new token, or for a remote worker set token= / KOSA8_TOKEN. Remember that KOSA8_TOKEN is ignored when you pass host= in code. See Agent identity.

Kosa8PolicyError (403)

When: the org policy forbids the action outright: a registry not on its allow-list, a label rule, a network rule wider than policy allows. (Not provoked for this page, to leave the test machine's policy alone; the JavaScript page shows a real one, policy "…" forbids registry "docker.io" (allowed: ghcr.io), from the same daemon code.)

What to do: do something the policy allows, or ask whoever manages it. kosa8 policy show prints the active policy. Retrying does not help. For an agent, pass the message back as the step's result: it says exactly what is not allowed.

Kosa8ApprovalRequiredError (403)

When: the policy wants named people to approve this exact action, and the call brought no usable approval. The SDK raises this subclass when a 403 body has an approval dict with a string why. (From the source; not provoked for this page.)

from kosa8 import Kosa8ApprovalRequiredError, Kosa8PolicyError

try:
    k.delete_sandbox("runner")
except Kosa8ApprovalRequiredError as e:
    print("why:", e.approval["why"])
    print("ask for it with:", e.approval.get("request_command"))
    print("signers:", e.approval.get("signers"))
    approval_id = input("approval id: ")              # from a person, out of band
    k.delete_sandbox("runner", approval=approval_id)
except Kosa8PolicyError as e:
    print("not allowed:", e)

What to do: show e.approval to a person, wait for the approval id, then repeat the same call with approval=id; it is sent as the X-Kosa8-Approval header. An approval covers one call. Never let an agent invent or fetch its own approval. Note that checkpoint, rewind and delete_sandbox make several requests and pass the same approval to each. See Company rules.

Kosa8NotFoundError (404)

When: no such sandbox.

no such sandbox: doc-py-nope (list sandboxes with `kosa8 sandbox ls`)

What to do: on clean-up paths, ignore it: the thing is already gone. Elsewhere, check the name. get_sandbox returns None instead of raising. Two things that sound like 404s are not: a missing container is a stream error, and a missing snapshot is a 500.

Kosa8ConflictError (409)

When: the request conflicts with the current state.

sandbox doc-py-e2 already exists (choose another name, or remove the existing sandbox with `kosa8 sandbox rm NAME` (`kosa8 sandbox ls` lists them))
sandbox data disk already exists: /Users/dhruvit/.kosa8/sandboxes/doc-py-guide-data.img (name in use?) (choose another name, or remove the existing sandbox with `kosa8 sandbox rm NAME` (`kosa8 sandbox ls` lists them))
snapshot already exists: doc-py-g2-manual (choose another id, or delete the existing snapshot first)
sandbox is not running: doc-py-g2-copy (start it with `kosa8 sandbox up doc-py-g2-copy`)

What to do: for a name in use, pick another name (a random suffix avoids it) or delete the old sandbox. The second message is the awkward one: no sandbox of that name is listed, but its disk files are still on disk, typically after a rewind that failed halfway. If a snapshot of it exists (kosa8 snapshot ls), restore it under that name; otherwise use another name. For "not running", start_sandbox(name); that is a cold boot, so containers do not come back. From 0.1.1, checkpoint handles a snapshot id taken by a concurrent checkpoint itself (it steps to the next number); on 0.1.0 that surfaced as this error (see the table).

Kosa8UnsupportedError (501)

When: the daemon's platform cannot run sandboxes: they need macOS on Apple silicon, or Linux with KVM. (From the source; the test machine supports sandboxes.)

What to do: run on supported hardware, or point the client at a worker that is (KOSA8_HOST). On Linux, check that /dev/kvm exists and that the daemon's user can open it, and run kosa8 doctor.

Kosa8ApiError (any other status)

When: the daemon answered with a status that has no class of its own, in practice 500. Two met while testing:

no such snapshot: doc-py-nope-snap (list snapshots with `kosa8 snapshot ls`)          # restore_snapshot, status 500
guest not ready on vsock port 1024 after 25s (run `kosa8 doctor`; check ~/.kosa8/log/)  # create_sandbox, status 500

What to do: read daemon_message; the class alone says little. The second one happened while the machine was booting many sandboxes for other jobs at once; the same create succeeded when run again a minute later. It is the one error on this page worth retrying automatically, a couple of times with a pause, after deleting any half-made sandbox of that name. If it keeps happening, run kosa8 doctor and read ~/.kosa8/log/kosa8d.log.

A sensible way to catch them

from kosa8 import (Kosa8Error, Kosa8ConnectionError, Kosa8TimeoutError, Kosa8StreamError,
                   Kosa8ApprovalRequiredError, Kosa8PolicyError, Kosa8NotFoundError)

def agent_step(ws, cmd):
    """Run one model-proposed command; return text for the model, raise only if kosa8 is down."""
    try:
        code, out = ws.sh(cmd)
        return f"exit {code}\n{out}"
    except Kosa8TimeoutError:
        return "the command produced no output for 60 s and was abandoned (it may still be running)"
    except Kosa8StreamError as e:
        return f"the command could not finish: {e.daemon_message}\n{e.stdout}{e.stderr}"
    except Kosa8ApprovalRequiredError as e:
        return f"needs a human approval: {e.approval['why']}"
    except Kosa8PolicyError as e:
        return f"refused by policy: {e}"
    # Kosa8ConnectionError, Kosa8NotFoundError (sandbox gone) and the rest propagate:
    # the loop cannot continue, so stop it and clean up.

(A pattern; ws is the Workspace from the developer guide.)

Troubleshooting