CLI Reference

mt is the operator-facing entry point to Mentat. It talks to mentat-server over HTTP and is designed to cover the common lifecycle of a private workload platform: deploy, inspect, scale, update, and recover.

global options
mt [--server <URL>] <command>

Options:
  --server <SERVER>  Mentat server address [default: http://127.0.0.1:4646]
  -h, --help         Print help

Core Commands

mt up <up.yaml>

Bring a Hull service up from a single YAML — bootstrap or update. Synthesises the container manifest, registers the service spec, generates the Condor ingress block, runs the build, repacks the rootfs, pushes, extracts on host, and scales. Idempotent. No SSH required. A preflight adjusts the deploy to the project before anything converges: it mirrors runtime.port into PORT/HOSTNAME for node-launched services (an operator-supplied value always wins), refuses to converge when another service already holds the port — naming it — and refuses to package macOS-native artifacts (Mach-O .node addons) that would abort on the Linux host. After submit it verifies the cluster actually stored the port from the YAML, so a stale ServiceSpec can no longer keep routing the old one while the deploy reports success.

mt up up.yaml
mt --server https://www.getmentat.run up up.yaml

# up.yaml shape
name: stil-site
domain: stil.gitweave.run
aliases: [gitweave.run]
replicas: 2
build:
  cwd: ./
  cmd: stil run build
  output: dist
runtime:
  serve: static    # preset: static | node | argv
  port: 3007
  memory: 64M
  cpu: 0.5
  pids: 16
health: /

mt deploy <deploy.yaml>

Update content of an existing Hull service. Skips manifest write / spec register; just builds, repacks, pushes, extracts, bounces. Use when the service is already registered (mt up handled bootstrap).

mt deploy deploy.yaml

# deploy.yaml shape
service: stil-site
build:
  cwd: /path/to/landing
  cmd: stil run build
  output_dir: dist
target:
  www_path: /www
  preserve: [/www/releases]
replicas: 2

mt run <file.yaml>

Submit a workload definition file to the control plane. Low-level entry point for non-Hull drivers (Firecracker, Sandbox, Docker, Exec). Hull services usually go through mt up instead.

mt run services.yaml
mt run --server http://10.0.0.1:4646 production.yaml

mt status [job-id]

Inspect jobs and allocations. Use a job ID to drill into one deployment.

mt status
mt status abc123-def456

mt deploy-image <service> --image <tag>

Roll out a new image using the service's deployment strategy (legacy single-flag flow, kept for non-Hull services and CI scripts).

mt deploy-image api-gateway --image registry.example/api-gateway:v2
mt deploy-image streamforge --image streamforge:v2 --strategy canary

mt ps

Show all running workloads with live cgroups v2 metrics. Displays service name, driver, endpoints, RAM usage, CPU time, PIDs, and health status. Color-coded by driver type.

mt ps

SERVICE                DRIVER        EP   ADDRESS          RAM     CPU     PIDS   STATUS
streamforge-node1      firecracker   1    172.16.0.2:8080  256M    —       —      healthy
oxidestore-node1       firecracker   1    172.16.0.5:9000  256M    —       —      healthy
mentat-web             sandbox       1    127.0.0.1:3001   2/32M   12ms    3      healthy
nextjs-ssr             sandbox       1    127.0.0.1:3002   45/256M 890ms   8      healthy
api-jarvis             docker        1    172.17.0.3:5001  512M    —       —      healthy

mt scale <service> <count>

Change the target replica count for a service.

mt scale api-gateway 5
mt scale api-service 3

mt rollback <service>

Return a service to the previous deployable version.

mt rollback api-service

mt canary promote <service>

Promote an in-flight canary to the full rollout.

mt canary promote streamforge

mt canary abort <service>

Abort an in-flight canary and revert the change.

mt canary abort streamforge

mt diagnose <service>

Why is this service in the state it is in. Probes every endpoint now instead of waiting for the monitor's next tick, shows the failure counters behind each status, pulls the workload's recent output through the agent, and lists findings — each one saying what to do about it. Read-only: it changes nothing. This is the work an operator otherwise does over SSH with journalctl and the container runtime's log directory.

mt diagnose mentat-web

mentat-web  0/1 healthy  ·  1 desired
  probe: http /
  last auto-restart: 40s

  ✖ 10.88.0.2:3001  mentat-web-e7585c4c  unhealthy  (2/3 failed probes)
      probe now: fail http://10.88.0.2:3001/ (12ms) — connection refused

  critical  www.getmentat.run has no healthy upstream — ingress will 502
            → ingress only emits a route for healthy endpoints; fix the
              endpoint and the route comes back on its own.
  info      auto-restarted 40s ago (cooldown 60s)
            → the workload may still be booting.

mt diagnose mentat-web --json   # same report, machine-readable

mt get <mt://resource>

Read cluster state as JSON — the machine-facing counterpart to the colour-coded tables mt ps and mt nodes print for humans. Nothing here writes, so it is safe to hand to a script or an agent. The mt:// scheme is optional.

mt get mt://services                 # every ServiceSpec
mt get mt://services/titan           # one service and its endpoints
mt get mt://services/titan/health    # status plus the counters behind it
mt get mt://nodes                    # nodes with lifecycle conditions
mt get mt://registry                 # the whole endpoint registry
mt get mt://sandboxes                # live ephemeral sandboxes

# mt://services/{name}/health, abridged
{
  "endpoints_registered": 1,
  "endpoints_healthy": 1,
  "health_check": { "probe": "http", "path": "/", "max_failures": 3 },
  "last_restart_secs_ago": null,
  "endpoints": [
    { "ip": "10.88.0.2", "port": 3001, "status": "healthy",
      "consecutive_failures": 0, "restart_in_flight": false,
      "node_condition": "ready" }
  ]
}

mt snapshot <service>

Snapshot a running Firecracker microVM: pause it, capture the full memory and disk state, resume it in place. The service keeps serving — the pause is the length of the capture, not a restart. The resulting snapshot is what fork and sandbox resume from.

mt snapshot api-janus
mt snapshot api-janus --name before-migration

mt fork <service> --count <n>

Fork N replicas of a microVM from a snapshot. Each replica resumes from the exact frozen memory image in well under a second — no kernel boot, no application start-up, no cache to warm. Captures a fresh snapshot first unless --snapshot names an existing one. Every fork is addressable at its own IP.

mt fork api-janus --count 8
mt fork api-janus --count 4 --snapshot before-migration

mt sandbox <create|list|exec|fork|rm|pool>

Ephemeral agent-sandboxes: fork a throwaway microVM from a template service, reach it at its own clone IP, run commands inside the guest over vsock, and let it auto-reap on its TTL. This is the sovereign counterpart to E2B / Modal / Daytona — the same shape of primitive, on hardware you own, with the isolation of a real VM rather than a shared container. A warm pool keeps sandboxes pre-forked per template so create returns instantly.

mt sandbox create python-base --ttl 900
mt sandbox list
mt sandbox exec sbx-4f21 python3 -c "print(1+1)"
mt sandbox fork sbx-4f21 --count 5     # branch the same live state
mt sandbox rm sbx-4f21

# keep 4 sandboxes pre-forked so create is instant
mt sandbox pool python-base 4
mt sandbox pool                         # list pools and how full they are

mt nodes

List registered nodes with their lifecycle condition, reason, age and running endpoints. Alias of mt host list.

mt nodes

NODE          ADDRESS            CONDITION            SINCE   ENDPTS
mentat-gcp    10.128.0.4:4647    Ready                14d     23
mentat-falp   10.0.1.9:4647      MaintenancePlanned   6m      7

mt host cordon|drain|uncordon <node>

Node lifecycle with the Kubernetes v1.37 condition vocabulary (Ready, MaintenancePlanned, DrainInProgress, Drained, GracefulNodeShutdownInProgress). cordon stops new placement without touching what runs; drain stops every workload in order — no kill -9 — deregisters them and keeps the node closed, recording exactly which services it stopped; uncordon returns the node to Ready and brings back precisely those replicas, never a global reschedule. The scheduler skips unschedulable nodes and the health monitor no longer restarts on them. The agent announces graceful shutdown on SIGTERM. Node is an id, an id prefix, or an address.

mt host cordon mentat-falp -r KernelPatch -m "apt upgrade + reboot"
mt host drain mentat-falp -r KernelPatch
mt host list          # watch DrainInProgress -> Drained
# ... maintenance ...
mt host uncordon mentat-falp

mt logs <service>

Inspect logs from a service. Use -f to follow.

mt logs api-gateway
mt logs api-gateway -f

Typical Operator Flow

# Start control plane and host agent
mentat-server --bind 0.0.0.0:4646
mentat-agent --kvm --server http://localhost:4646

# Submit a stack definition
mt run services.yaml

# Check what is running
mt status
mt nodes

# Change one service safely
mt deploy-image api-gateway --image registry.example/api-gateway:v2
mt rollback api-gateway

Observed Production Output

mt run services.yaml
$ mt run services.yaml
  scheduling 4 services...
  driver: firecracker  x2 services
  driver: exec         x1 services
  driver: docker       x1 services

  streamforge-broker x2  8ms   <- production traffic
  oxidestore        x1  6ms   <- S3 API live
  mentat-agent      x1  2ms
  api-jarvis        x1  380ms

  all services healthy
  firecracker + docker + exec  ·  private control plane

Interfaces

Terminal UI

mt lens --server your-host:4646

Desktop App

Mentat Desktop connects to the API over HTTPS for graphical monitoring and day-to-day operations. Use it when a team wants visibility without living entirely in the terminal.

Safe Change Management

# Canary first
mt deploy-image streamforge --image streamforge:v2 --strategy canary

# Evaluate, then decide
mt canary promote streamforge
# or: mt canary abort streamforge

Snapshots, Forks & Sandboxes

A Firecracker microVM can be frozen and resumed, which turns a running service into something you can branch. Snapshot captures the state, fork resumes N copies of it in well under a second each, and sandbox wraps the same primitive as a throwaway environment with a TTL — the shape of thing agent platforms rent out, running on your own hardware.

# Branch a running service
mt snapshot api-janus --name before-migration
mt fork api-janus --count 8 --snapshot before-migration

# Throwaway environment for an agent
mt sandbox pool python-base 4            # keep 4 warm, create is instant
id=$(mt sandbox create python-base --ttl 900)
mt sandbox exec $id python3 -c "print(1+1)"
mt sandbox fork $id --count 5            # branch the same live state
mt sandbox rm $id                        # or let the TTL reap it

Node Maintenance

Patching a host is a first-class operation, not an improvised sequence of mt scale 0. Close the node, drain it in order, do the work, open it again — Mentat remembers what it stopped and restores exactly that.

mt host cordon mentat-falp -r KernelPatch   # no new placement
mt host drain  mentat-falp -r KernelPatch   # stop + deregister, in order
mt host list                                # Drained
# reboot, upgrade, replace the disk
mt host uncordon mentat-falp                # Ready + the same replicas back