Guides / Local models

Local models

Download a model, serve it, and point your app at it. Your code and prompts never leave the machine, and there is nothing to pay per word.

What runs the model

kosa8 serves models with llamay, AZMX's inference engine, when it is installed, and with a smaller engine built into kosa8 when it is not. Either way the command is the same and so is what your app talks to — a model you have pulled can always be served.

$ kosa8 doctor
✓ local model runtime            llamay at /usr/local/bin/llamay

$ kosa8 model pull Qwen/Qwen2.5-0.5B-Instruct-GGUF:q4_k_m
$ kosa8 model ls
NAME                           SIZE     SOURCE                                  PULLED
qwen2.5-0.5b-instruct-q4_k_m   468MiB   hf.co/Qwen/Qwen2.5-0.5B-Instruct-GGUF   2026-07-29

$ kosa8 model serve qwen2.5-0.5b-instruct-q4_k_m
loading qwen2.5-0.5b-instruct-q4_k_m…
kosa8 model gateway on http://127.0.0.1:7434
  OpenAI:    POST /v1/chat/completions
  Anthropic: POST /v1/messages
  running on llamay on http://127.0.0.1:58965
  logs:      ~/.kosa8/log/model-qwen2.5-0.5b-instruct-q4_k_m.log

/v1/chat/completions   what OpenAI libraries call
/v1/completions        answered like /v1/chat/completions: send messages, not prompt
/v1/models             what is available
/v1/messages           what Anthropic libraries call

$ export OPENAI_BASE_URL=http://127.0.0.1:7434/v1
$ export ANTHROPIC_BASE_URL=http://127.0.0.1:7434

$ curl -s http://127.0.0.1:7434/v1/chat/completions \
    -H 'Content-Type: application/json' \
    -d '{"model":"anything","messages":[{"role":"user","content":"Reply with the single word: ready"}],"max_tokens":8}'
{"choices":[{"finish_reason":"stop","index":0,"message":{"role":"assistant","content":"ready"}}], …}

Choosing the engine

--backend decides which engine serves. The default, auto, uses llamay when it is on your PATH and the built-in engine otherwise, and tells you when the built-in one was chosen.

What you get from each:

Token probabilities

Ask for logprobs and the gateway passes the request to llamay and the answer back: for each token, how likely the model thought it and its nearest alternatives were. This is what lets a program treat an answer as a measurement rather than a string.

$ curl -s http://127.0.0.1:7434/v1/chat/completions -H 'Content-Type: application/json' -d '{
    "messages":[{"role":"user","content":"Answer with exactly one letter, A or B. …"}],
    "max_tokens":1, "temperature":0, "logprobs":true, "top_logprobs":5 }'
# choices[0].logprobs.content[0].top_logprobs, as probabilities:
   "B"   0.7968
   "A"   0.2030
   "No"  0.0001

kosa8 uses exactly this for the optional classifier that decides whether a sensitive tool call needs a person: see Rules for tool calls. Token probabilities are returned for ordinary requests, not streamed ones.

Tool calling

Send tools on either endpoint and the model can call them. The gateway answers with OpenAI tool_calls or Anthropic tool_use blocks, streamed or not, and takes the results back as tool messages or tool_result blocks, so agent frameworks and coding agents that work through tools can run on a local model. tool_choice is honoured: auto, none, required (Anthropic: any), or one named function.

$ curl -s http://127.0.0.1:7434/v1/chat/completions -H 'Content-Type: application/json' -d '{
    "messages":[{"role":"user","content":"What is the weather in Paris?"}],
    "tools":[{"type":"function","function":{"name":"get_weather",
      "parameters":{"type":"object","properties":{"city":{"type":"string"}}}}}],
    "tool_choice":"required" }'
{"choices":[{"finish_reason":"tool_calls","message":{"content":null,"tool_calls":[
  {"id":"call_…","type":"function","function":{"name":"get_weather","arguments":"{\"city\":\"Paris\"}"}}]}}], …}

The gateway offers the tools to the model in the Hermes format, the one Qwen2.5, Qwen3 and most open tool-calling models were trained on, and reads calls back out of the answer. A model that was not trained on that format will answer in text, and you get the text: a call is only reported when the model wrote one that parses. Small models are reluctant to call on their own; a 7B model or larger, or tool_choice, works better. Images in messages are dropped.

Sharing one with your team

A model is several gigabytes. Having everyone download it separately is slow, and on a locked-down network may not be allowed at all. kosa8 can publish a model into the same registry your company already uses for container images.

$ kosa8 model push qwen2.5-0.5b-instruct-q4_k_m registry.company.com/models/qwen:v1
$ kosa8 model pull-oci registry.company.com/models/qwen:v1 --name qwen
$ kosa8 model serve qwen

It is stored as a standard OCI artifact, so a registry that holds container images can hold this too — no special model server, nothing new for anyone to approve. The fingerprint is checked as it downloads; a damaged or altered file is refused rather than quietly kept.

Which models may be served

An organisation can list the models it has approved, by the fingerprint of the weights, in its signed policy. kosa8 then refuses to serve anything else — a matching name is not enough. kosa8 model check tells you where a model stands before you try.

$ kosa8 model check qwen2.5-0.5b-instruct-q4_k_m
qwen2.5-0.5b-instruct-q4_k_m
  digest     sha256:74a4da8c9fdbcd15bd1f6d01d621410d31c6fc00986f5eb687824e7b93d7a9db
  signature  unsigned
  policy     corp has no approved model list: any model may be served
  recording  off

The policy can also ask for every request to be recorded, as fingerprints or in full. See Policy reference for allowed_models and model_recording.

Using it with the assistant

$ kosa8 ai --provider local -m "what is using the most memory?"

With no ANTHROPIC_API_KEY set, kosa8 ai uses the local model gateway by default, so nothing is sent anywhere.

Where to go next