Guides / Ruby / Errors and troubleshooting

Ruby errors and troubleshooting

Every exception the gem raises is a Kosa8::Error, and one that comes from the daemon keeps the daemon's own words: what went wrong, and what to do about it. This page lists each class with a real message, then the problems you are likely to meet and how to fix them.

What this is, in plain words

When something goes wrong, Ruby raises an exception: an object that says what happened. Your program can rescue it and decide what to do, or let it stop the program with a message. The gem's exceptions come in families, so you can rescue one precise problem ("that sandbox does not exist") or a whole family ("anything from kosa8") with one line.

There is one rule worth learning first: a command that fails is not an exception. If your command runs and exits with code 1, you get a normal result with exit_code 1. Exceptions are for when kosa8 could not do what you asked at all: the daemon is not running, the sandbox does not exist, the image is missing, a rule forbids it.

The hierarchy

StandardError
└─ Kosa8::Error                       anything this gem raises; also checks made in the gem
   ├─ Kosa8::ConnectionError          the daemon could not be reached
   ├─ Kosa8::TimeoutError             a connect or read took longer than the timeout
   ├─ Kosa8::StreamError              a run, exec or attach ended with an error, not an exit code
   ├─ Kosa8::CheckpointError          checkpoint bookkeeping: none to rewind to, bad name, lock, failed rewind
   ├─ Kosa8::ProtocolError            the daemon's output stream was not valid NDJSON (since 0.1.1)
   └─ Kosa8::ApiError                 the daemon answered with an HTTP error (500, 503, … land here)
      ├─ Kosa8::BadRequestError       400
      ├─ Kosa8::AuthError             401
      ├─ Kosa8::PolicyError           403
      │  └─ Kosa8::ApprovalRequiredError   403 that a signed approval would lift
      ├─ Kosa8::NotFoundError         404
      ├─ Kosa8::ConflictError         409
      └─ Kosa8::UnsupportedError      501

All inherit from StandardError, so a bare rescue => e catches them. Rescue the more specific class first: Ruby uses the first rescue clause that matches, and ApprovalRequiredError is also a PolicyError, which is also an ApiError. Kosa8::TimeoutError is the gem's own class and is not Ruby's Timeout::Error.

Fields

In 0.1.0 the HTTP method was read with e.method, which hid Ruby's Object#method (so e.method(:message) raised ArgumentError). Since 0.1.1 it is e.http_method, and e.method(:message) works again:

http_method="GET" method(:message)=#<Method: Kosa8::NotFoundError(Exception)#message>

See them for yourself

This script provokes most of the errors without needing a sandbox. The AuthError line sends one harmless request with a made-up agent token; nothing is changed.

require "kosa8"
k = Kosa8::Client.new
def show(label)
  yield
  puts "#{label}: no error"
rescue Kosa8::Error => e
  puts "#{label}: #{e.class} status=#{e.status.inspect}"
  puts "  message:     #{e.message}"
  puts "  daemon_message: #{e.daemon_message.inspect}" if e.daemon_message
  puts "  remediation: #{e.remediation.inspect}" if e.remediation
  puts "  request:     #{e.http_method} #{e.path}" if e.http_method
end

show("ConnectionError") { Kosa8::Client.new(socket_path: "/tmp/k8rb-none.sock").health }
show("NotFoundError") { k.sandbox_egress("doc-rb-missing") }
show("BadRequestError") { k.create_sandbox(name: "doc-rb-bad", egress_allow: ["*.com"]) }
show("AuthError") { Kosa8::Client.new(agent_token: "not-a-real-token").list_sandboxes }
show("Error (client check)") { k.create_sandbox(egress: "any", egress_allow: ["x.io"]) }
show("CheckpointError none") { k.rewind("doc-rb-never") }
show("CheckpointError name") { k.checkpoint("../etc") }
show("TimeoutError remote") { Kosa8::Client.new(host: "https://10.255.255.1:7440", timeout: 2).health }

The run this output comes from had a few more cases; their messages are under each class below, and they are cut from this listing.

$ ruby errors.rb
ConnectionError: Kosa8::ConnectionError status=nil
  message:     kosa8 daemon unreachable at unix:///tmp/k8rb-none.sock: No such file or directory - connect(2) for /tmp/k8rb-none.sock (is kosa8 installed and its daemon running? start it with `kosa8 daemon start`)
  request:     GET /kosa8/v1/health
NotFoundError: Kosa8::NotFoundError status=404
  message:     no such sandbox: doc-rb-missing (list sandboxes with `kosa8 sandbox ls`)
  daemon_message: "no such sandbox: doc-rb-missing"
  remediation: "list sandboxes with `kosa8 sandbox ls`"
  request:     GET /kosa8/v1/sandboxes/doc-rb-missing/egress
BadRequestError: Kosa8::BadRequestError status=400
  message:     egress allow "*.com": com is a public suffix, under which anyone can register a name, so the wildcard would admit strangers' hosts; allow a domain below it, as in *.example.com
  daemon_message: "egress allow \"*.com\": com is a public suffix, under which anyone can register a name, so the wildcard would admit strangers' hosts; allow a domain below it, as in *.example.com"
  request:     POST /kosa8/v1/sandboxes
AuthError: Kosa8::AuthError status=401
  message:     agent token is not recognised; mint one with `kosa8 agent token create NAME`
  daemon_message: "agent token is not recognised; mint one with `kosa8 agent token create NAME`"
  request:     GET /kosa8/v1/sandboxes
Error (client check): Kosa8::Error status=nil
  message:     egress_allow already limits the network to the hosts it lists; do not also set egress
CheckpointError none: Kosa8::CheckpointError status=nil
  message:     no checkpoints for "doc-rb-never" — take one with checkpoint or `kosa8 checkpoint doc-rb-never`
CheckpointError name: Kosa8::CheckpointError status=nil
  message:     invalid sandbox name "../etc"
TimeoutError remote: Kosa8::TimeoutError status=nil
  message:     GET /kosa8/v1/health timed out talking to https://10.255.255.1:7440: execution expired
  request:     GET /kosa8/v1/health

Each class

Kosa8::ConnectionError

In plain words: the gem could not reach kosa8 at all.

When: the request never got to a daemon: no socket file, a socket with nothing listening, a socket path too long for the operating system, a remote host that refuses or cannot be resolved, a TLS failure. Technically: any SystemCallError, IOError, SocketError or OpenSSL::SSL::SSLError while sending or reading. The hint about kosa8 daemon start is added for "no such file" and "connection refused". See the real message above.

Handle: start the daemon (kosa8 daemon start) or fix the address (KOSA8_HOST, socket_path:). If it happened before the request was sent, nothing was done and a retry is safe. Since 0.1.1, an exception raised inside your own on_stdout/on_stderr/on_status callback comes out as itself, not as a ConnectionError. In our test a callback that raised IOError, "closed stream (mine)" made run raise exactly that: IOError: closed stream (mine).

Kosa8::TimeoutError

In plain words: kosa8 took longer to answer than the gem was told to wait.

When: Net::HTTP's connect or read timeout passed (Net::OpenTimeout, Net::ReadTimeout). Which timeout applies:

• You passed timeout: (on the call or the client): it is the read timeout, and the connect timeout is the smaller of it and 30 s. During a run the clock restarts each time the command prints, so it fires when the command is silent that long. run(…, ["sleep", "30"], timeout: 3) printed:

  POST /kosa8/v1/sandboxes/doc-rb-v11/run timed out talking to unix:///Users/dhruvit/.kosa8/run/kosa8.sock: Net::ReadTimeout with #<UNIXSocket:(closed)>

• No timeout, streaming call (run, exec, exec_frames, attach): no read timeout at all since 0.1.1, so it waits as long as the command runs. A 70-second silent sleep 70; echo done returned exit 0 "done\n" after 71s. (In 0.1.0 the same call raised this error at 60 s.)
• No timeout, any other call: Net::HTTP's defaults, 60 s to connect and 60 s per read. A remote worker that never answers gives GET /kosa8/v1/health timed out talking to https://10.255.255.1:7440: execution expired (here with timeout: 2).

Handle: pass a timeout: only when you want the client to give up (Timeouts). The command in the sandbox is not stopped by a timeout: in our test the sleep 30 container was still listed as running afterwards. Do not blindly retry a run after a timeout, or you will have two copies running.

Kosa8::StreamError

In plain words: the command could not be started, or the stream of its output broke.

When: run, exec or attach began streaming and the daemon ended the stream with an error frame instead of an exit code, or the stream stopped with no final frame. Real messages:

image "docker.io/library/python:3.12-alpine" not found in engine: image "docker.io/library/python:3.12-alpine": not found
failed to create shim task: OCI runtime create failed: executable file `python3` not found in $PATH: No such file or directory
no such container: k99-nope

The truncated case reads stream ended unexpectedly (no final frame). status is nil (the HTTP answer itself was a success); daemon_message is the frame's error text.

Handle: pull the image into the sandbox first (sb.pull), check the container id (sb.containers), or check the program exists in the image, or run it through sh -c, which turns a missing program into an ordinary exit code 127. e.stdout and e.stderr hold any output produced before the failure. exec_frames never raises this: it hands you the error frame, {"done"=>true, "error"=>"no such container: k99-nope"}.

Kosa8::ProtocolError

In plain words: the answer that came back was not in the format kosa8 speaks, so the gem cannot read it. You should not see this from a real kosa8 daemon.

When (since 0.1.1): a line of a run, exec, exec_frames or attach stream is not valid JSON, or its data is not valid base64. Usually something other than kosa8 is answering at that address: a proxy's error page, a different service on the port. Provoked with a fake server that answered a run with the line this is not json:

POST /kosa8/v1/sandboxes/sb/run: the daemon sent a frame that is not valid NDJSON (767: unexpected token at 'this is not json'): "this is not json"

Handle: check what is listening at KOSA8_HOST or the socket; the message quotes the first 200 characters of the bad line. In 0.1.0 this surfaced as Ruby's JSON::ParserError.

Kosa8::CheckpointError

In plain words: a problem with the undo list, found in your program before or after talking to kosa8.

When:

• nothing to rewind to: no checkpoints for "doc-rb-never" — take one with checkpoint or `kosa8 checkpoint doc-rb-never`
• a to: that matches no id or label: no checkpoint "nope" for sandbox "doc-rb-err"
• a sandbox name that could escape the store (empty, starting with a dot, containing / or \): invalid sandbox name "../etc"
• the checkpoint file's lock held by another process for longer than lock_timeout (10 s by default; since 0.1.1, before which it waited for ever). Provoked by holding the lock in the test script, with lock_timeout: 2: timed out after 2s waiting for /tmp/demo/checkpoints.json.lock; another kosa8 process is holding it
• a rewind whose restore failed after the sandbox was deleted: restore failed and NAME no longer exists (it was deleted to free its name for the restore): <daemon error> — the checkpoint is intact; recover with restore_snapshot("NAME-ckpt-N", name: "NAME"). (From the source; not reproduced.) Here e.cause is the daemon's error.

Handle: take a checkpoint first, or list them with sb.checkpoints. For the lock: find the other kosa8 process (a CLI command, another program) that is stuck, or retry. For the failed rewind: fix the cause (on a Mac, unlock the screen) and run the restore_snapshot call from the message.

Kosa8::Error (itself)

In plain words: the gem refused your request before sending it, because it could never work or would be unsafe.

When: a check in the gem. Real messages:

egress_allow already limits the network to the hosts it lists; do not also set egress
an egress allow-list needs at least one host; use egress: "deny" for no network
actor must be human, agent or tool, not "root"
refusing to talk to 10.0.0.5:7440 over plain http: the token and every request would cross the network unencrypted. Use https://10.0.0.5:7440 (a worker behind a TLS-terminating tunnel or proxy), or pass insecure: true / KOSA8_INSECURE=1 if this link is already private and trusted
KOSA8_HOST scheme "ftp" is not supported; use http://, https://, tcp:// or unix://
give at least two variants; with one there is nothing to compare
stdin of container k1-b998c4e0 in doc-rb-v11 was already sent and is closed: the daemon treats one write_stdin body as the whole of stdin. Pass an IO or an Enumerable to stream input

The last one (since 0.1.1) comes from a second write_stdin to the same container through the same client. In 0.1.0 the daemon answered {"ok"=>true} and the input was silently lost.

Others in the source: KOSA8_HOST "…" names no host, and from explore, forked N branches for M variants; see list_sandboxes for what was created and keep X is not one of the branches (…); all of them were kept.

Handle: fix the call; the message says what is wrong. Nothing was sent, except for the two explore messages, which come after the fork: the branches exist, and list_sandboxes shows them.

Kosa8::ApiError (other statuses)

In plain words: kosa8 tried and something went wrong on its side.

When: any HTTP error without a closer class, usually 500, or 503 for a locked Mac. Real messages, both 500:

guest not ready on vsock port 1024 after 25s (run `kosa8 doctor`; check ~/.kosa8/log/)
pull : manifest: unauthorized after token refresh (check the image reference and network)

The first came from create_sandbox on a heavily loaded Mac: the new machine did not answer in time. A retry a minute later succeeded. The second is what pulling an image that does not exist looked like (docker.io/library/does-not-exist-kosa8:nope). On a Mac with the screen locked, every restore (rewind, fork, explore, restore_snapshot) fails with a 503 whose message begins macOS blocks restoring a VM while the screen is locked; unlock and retry.

Handle: read the message and its remediation. kosa8 doctor checks the machine, and ~/.kosa8/log/kosa8d.log has the daemon's side. A 500 from a busy machine is often worth one retry; a 500 from a bad image reference is not.

Kosa8::BadRequestError (400)

When: the daemon refused a value, most often an egress rule (real message in the script output above). Handle: fix the input; the same request fails the same way every time.

Kosa8::AuthError (401)

When: an agent token was sent and is not valid (real message above), or a remote worker's bearer token is missing or wrong. A bad agent token is never treated as anonymous. Handle: mint a token with kosa8 agent token create NAME, or set KOSA8_TOKEN / token: (remember that KOSA8_TOKEN is ignored when you pass host: in code). See Agent identity.

Kosa8::PolicyError (403)

When: the org policy forbids the action outright: a registry not on the allow-list, a label rule, open network where the policy forbids it. Not reproduced for this page. Handle: do something the policy allows, or ask whoever manages it; kosa8 policy show prints the active policy. See Company rules.

Kosa8::ApprovalRequiredError (403)

When: the policy wants named people to approve this exact action, and the call carried no usable approval. The gem chooses this class when a 403's body has an approval hash with a String why. A subclass of PolicyError, so rescue it first. Not reproduced for this page.

Handle: show e.approval to a person, wait for the approval id, then repeat the same call with approval: id. An approval is spent by the one call it covers.

begin
  sb.delete
rescue Kosa8::ApprovalRequiredError => e
  warn "needs approval: #{e.approval['why']}"
  id = wait_for_approval_id(e.approval)   # yours to write: ask a person
  sb.delete(approval: id)
end

Kosa8::NotFoundError (404)

When: no such sandbox or snapshot (real message above). Handle: on clean-up paths, usually ignore it: the thing is already gone. get_sandbox returns nil instead of raising. A container that does not exist is reported differently, as a StreamError. If delete_sandbox raises this, the checkpoint records for that name are left in the store; a later delete_sandbox of a sandbox with the same name removes them.

Kosa8::ConflictError (409)

When: the request conflicts with what exists. Real messages:

sandbox doc-rb-err already exists (choose another name, or remove the existing sandbox with `kosa8 sandbox rm NAME` (`kosa8 sandbox ls` lists them))
snapshot already exists: doc-rb-g2-snap (choose another id, or delete the existing snapshot first)

Handle: choose another name, or delete the old one first. If you see the snapshot message from checkpoint, two callers checkpointed the same sandbox at once and the gem ran out of retries. Since 0.1.2 it steps to the next id by itself, up to 10 times; in our test three threads checkpointing one sandbox at once, twice over, all succeeded (see Threads). On 0.1.0 and 0.1.1 one of them could fail with snapshot already exists: doc-rb-v11-ckpt-1 … (that one is a real 0.1.1 message); retrying the call works.

Kosa8::UnsupportedError (501)

When: the machine cannot do this: no microVM support (sandboxes need macOS on Apple silicon or Linux with KVM), or a guest image too old for a feature. Not reproduced for this page. Handle: kosa8 doctor says what is missing.

A handler that covers the cases

begin
  # ... kosa8 calls ...
rescue Kosa8::ApprovalRequiredError => e
  ask_a_person(e.approval)            # then retry with approval: id
rescue Kosa8::PolicyError => e
  warn "refused by policy: #{e.daemon_message}"
rescue Kosa8::NotFoundError
  # already gone
rescue Kosa8::ConnectionError => e
  abort "kosa8 is not running: #{e.message}"
rescue Kosa8::TimeoutError => e
  warn "gave up waiting (the command may still be running): #{e.message}"
rescue Kosa8::StreamError => e
  warn "could not run: #{e.message}\npartial output: #{e.stdout}#{e.stderr}"
rescue Kosa8::Error => e
  warn e.message                      # includes the daemon's remediation
end

That block is a pattern, not a program: ask_a_person is yours to write. Order matters: subclasses before their parents, Kosa8::Error last.

Troubleshooting