Skip to content

AbstractGateway — Troubleshooting

Each entry starts from a symptom you can see, then gives the likely causes, how to confirm them, the fix, and where the full explanation lives. For conceptual questions, see faq.md.

Two commands answer most questions:

abstractgateway-config status          # data dir, auth mode, login service, running gateway
curl -sS http://127.0.0.1:8080/api/health

Starting and signing in

serve says "Refusing to start: no sign-in would protect this gateway"

Cause. --host points beyond this computer (for example 0.0.0.0) and the gateway was started with neither user accounts nor a token.

Fix. Choose the exposure with the network setting instead of --host; user accounts are turned on for you:

abstractgateway network set lan        # or: internet --acknowledge-internet
abstractgateway serve

Or keep the gateway on this computer: abstractgateway serve (loopback) or --host 127.0.0.1. See configuration.md.

serve refuses a weak token

Cause. A shared ABSTRACTGATEWAY_AUTH_TOKEN shorter than 15 characters or easy to guess, on a non-loopback bind or with public wildcard origins.

Fix. Use a long random token, or use user accounts (ABSTRACTGATEWAY_USER_AUTH=1). See security.md.

The gateway does not start: "this gateway needs abstractruntime>=…"

The installed AbstractRuntime lacks live token deltas or the built-in tool deny rules the gateway relies on; the message names what is missing. Upgrade it in the gateway's Python, then start again:

pip install -U "abstractruntime>=0.9.0"

Checks and fixes.

  • A link works once and for 10 minutes. Mint a new one with abstractgateway claim --open.
  • It only works from a browser on the gateway machine (a loopback connection without proxy headers). From another computer, sign in with a user id and token instead.
  • abstractgateway claim exits with code 2 when the running gateway uses a shared token without user accounts: that gateway cannot redeem links. Sign in with the token, or start the gateway with user accounts.

See first-run.md.

I lost the admin token

It is kept in <data dir>/auth/bootstrap-admin-token (mode 0600). abstractgateway-config status prints the data dir. For a loopback gateway you can also sign in with abstractgateway claim --open and rotate tokens from the console's Accounts tab (Rotate on the account's row).

401, 403, 429 or 413 from /api/gateway/*

Status Likely cause Fix
401 missing or invalid Authorization: Bearer <token> or session sign in again; check the token file
401 with user_accounts_off_admin_only a non-admin account while user accounts are off turn user accounts on, or sign in as an admin (security.md)
403 (origin not allowed) the browser page's origin is not in the allowlist add it with abstractgateway network set --allowed-origins https://…
403 on an admin route the signed-in principal is not an admin use an admin account
429 repeated failed sign-ins from the same client address (lockout) wait for the backoff; check trust_proxy behind a proxy
413 the body exceeds ABSTRACTGATEWAY_MAX_BODY_BYTES (or the attachment / bundle limit) send less, or raise the limit (security.md)

The gateway does not start: an AbstractRuntime module is missing

serve or runner stops with No module named 'abstractruntime.automation_queries' (or abstractruntime.automations): the installed AbstractRuntime predates automations. Install AbstractRuntime 0.7.0 or later in the gateway's Python, then start again. See automations.md.

The log shows [FATAL] gateway watchdog and the gateway restarted (exit code 75)

The event loop was blocked for longer than serve --watchdog-seconds (default 30): the gateway could answer nothing, /api/health included. The watchdog wrote the stack of the blocked event-loop thread and of every other thread to the gateway log, then exited with code 75 so the service manager started a fresh gateway. The stack under "the event-loop thread is blocked here" names the code that blocked; report it as a bug with that excerpt.

The same facts are kept next to the data: <data dir>/incidents/watchdog-<UTC stamp>.json (when, how long, the innermost frame of the event-loop thread, the innermost gateway frame, the requests being served) and watchdog-<stamp>.threads.txt (every thread's stack). The restarted gateway reads the newest one: an admin sees "Gateway restarted at

A Read aloud that was speaking when the gateway restarted cannot resume (the audio was being produced by the process that died): press Read aloud again. Gateways up to 0.13.0 left such a reply waiting ("Waiting for an event › Streaming voice synthesis is running."); the current gateway closes those at startup with the sentence "Read aloud was interrupted because the gateway restarted; the audio was not finished. Press Read aloud again." and the conversation continues.

  • Logs: macOS ~/Library/Logs/AbstractGateway/gateway.err.log; Linux journalctl --user -u abstractgateway; the installer's background mode and scripts/start-local.sh write gateway.log in their log folder.
  • Who restarts it: the macOS LaunchAgent (KeepAlive with SuccessfulExit: false: any non-zero exit), the systemd user unit (Restart=on-failure), the installer's background loop and the start-local.sh supervisor. A gateway started by hand in a terminal, or the Windows Run entry, is not restarted.
  • A gateway whose stall comes from native code that keeps the Python lock is stopped by the backstop, a small separate process that watches a heartbeat from the event loop: 15 s after the limit it has every thread's stack dumped to the log, writes a backstop incident file and kills the gateway (same restart).
  • Sleep is not a hang. Waking a computer can make the clock jump by minutes; the watchdog and the backstop both check once more whether the event loop is still running before acting; a live loop at most leaves the log line [WARN] gateway watchdog: the event loop resumed after …s; not restarted. On Windows the backstop is still faulthandler's timer, which cannot make that check, so a long sleep can restart the gateway there. To confirm after a sleep/wake, the incident folder stays empty and the gateway's process start time is unchanged: ls ~/Library/Application\ Support/AbstractGateway/incidents 2>/dev/null; ps -o lstart= -p "$(pgrep -f 'abstractgateway serve' | head -1)" (use your data folder if it is elsewhere).
  • GET /api/health reports watchdog: {enabled, limit_s, last_tick_age_s}.
  • --watchdog-seconds 0 turns the watchdog off (a debugging session paused on a breakpoint); it is never active with --reload.

Network access

A change of network mode "needs a restart"

A listening socket cannot move. After abstractgateway network set …, the status reads restart_required: true until the gateway restarts: abstractgateway network restart, the tray's Restart AbstractGateway…, or stop and start serve.

If network status says the restart cannot apply the setting, the running gateway was started with --host/--port (they win over the setting), or its login item pins them: run abstractgateway service enable once, then restart.

lan or internet is refused (HTTP 409)

The reason_code says why:

  • user_auth_required: the gateway runs without user accounts (a shared token only, or ABSTRACTGATEWAY_USER_AUTH=0).
  • auth_disabled: it was started with authentication or read protection off (ABSTRACTGATEWAY_SECURITY=0, ABSTRACTGATEWAY_PROTECT_WRITE=0, ABSTRACTGATEWAY_PROTECT_READ=0).
  • acknowledgement_required: internet needs --acknowledge-internet (or the confirmation in the console or tray).

Start the gateway without those variables (a plain abstractgateway serve) and set the mode again. See security.md.

Another computer cannot open the console

  • Check abstractgateway network status: the mode must be lan or internet, applied (no restart pending).
  • Use an address from abstractgateway network addresses; the machine's firewall must allow the port.
  • The console accepts the gateway's own addresses (LAN, Bonjour, Tailscale) automatically; one that appears later is accepted within a minute.
  • Behind a reverse proxy or tunnel, add its public origin with abstractgateway network set --allowed-origins https://your.host.

Workflows and runs

GET /api/gateway/bundles returns no bundles

  • ABSTRACTGATEWAY_FLOWS_DIR points at an empty directory. Unset it to serve the shipped bundles (shipped-workflows.md), or upload a bundle with POST /api/gateway/bundles/upload.
  • A bundle can be present but not served: the skipped array names it with the reason (for example a min_runtime floor or a compile error).

A run stays RUNNING and nothing happens

  • GET /api/health reports the runner (runner.runners[].status). With serve --no-runner, start abstractgateway runner on the same data dir.
  • StartRunResponse.runner_warning is set when no runner is ticking the data dir.
  • The gateway may be paused ("paused": true on /api/health, a banner in the console): resume it from the tray, the console, or POST /api/gateway/host/resume.

A run start answers 409 naming agents.default_workflow.<interface>

The run asked for the gateway default (flow_id: "@default"), and the saved default for that interface cannot run: its workflow was removed or deprecated, or no longer declares the interface. The message names the setting and where its value comes from. Choose another workflow in the console (Workflows → Default workflow per app), or run abstractgateway config unset agents.default_workflow.<interface> to return to the built-in default. A 400 means the request itself is wrong: interface is missing, or bundle_id/bundle_version were sent with @default. See configuration.md.

Replies arrive only at the end (no live text)

  • The run did not ask for streaming: send input_data._runtime.stream: true, or turn on agents.streaming_default (it applies to interactive POST /runs/start only, never to schedules, bridges or the entity loop).
  • The call could not stream: its llm.delta_end says reason: "unavailable" with a detail (for example structured_output, node_stream_off, provider_cannot_stream); the answer is complete either way.
  • GET /api/gateway/discovery/capabilities must show capabilities.streaming.deltas: true.

See api.md.

The skills list is empty

GET /api/gateway/skills says why in warnings and which shelf it read in shelf_source. A saved skills.shelf that does not exist or holds no skills/ folder is reported as unavailable; unset it to use the gateway's own copy, or refresh that copy with Refresh the curated shelf (console, Apps) or POST /api/gateway/admin/skills/reseed. See configuration.md.

"LLM nodes but no default provider/model is configured"

Configure the text route, for example:

abstractgateway-config set-default input.text \
  --provider lmstudio --model qwen/qwen3.5-9b --base-url http://127.0.0.1:1234/v1

Or pick a default model in the console (Multimodal, or Use as default on a downloaded model in Models). See configuration.md.

A run fails because a model's weights are missing

The error names the capability route that selected the model. Download it from the console's Models tab or with abstractgateway models download <provider> <artifact>, or choose another default. See model-downloads.md.

"LLM/tool execution requires AbstractCore integration", "Visual Agent nodes require AbstractAgent", or memory_kg_* nodes ask for AbstractMemory

These packages are part of the base install. Check the environment the gateway runs in:

pip show abstractgateway AbstractRuntime abstractcore abstractagent AbstractMemory

For KG memory, keep the default lancedb backend; sqlite works only when the installed AbstractMemory exposes SQLiteTripleStore.

/voice/tts or /audio/transcribe answer "capability unavailable"

Configure the voice routes (output.voice, input.voice) in the console's Multimodal tab, or the Gateway-scoped voice variables for a remote backend (ABSTRACTGATEWAY_VOICE_TTS_ENGINE, ABSTRACTGATEWAY_VOICE_REMOTE_BASE_URL, …). Local voice engines need the apple or gpu extra. See configuration.md.

Catalog routes return only gateway_static defaults

The request reached a gateway without the capability packages you expected, often another abstractgateway serve still running from another environment on the same port. Stop it and start the one from your current environment.

Automations

An automation does not fire

  • No runner is ticking the data folder: check runner.runners[].status on GET /api/health; with serve --no-runner, start abstractgateway runner on the same data folder.
  • The gateway is paused ("paused": true on /api/health).
  • The automation is paused, archived or has no ticks left: read status and next_fire_at in its summary (GET /api/gateway/automations/{automation_id}).
  • The schedule starts later (start_at) or ended (until, count).

See automations.md.

An occurrence reads waiting

It waits for a person: a question from the workflow (ask_user), a tool batch to approve (tool_approval, under policy.tool_approval: "ask" or for a tool that always asks), or an event. Its waits say which; answer with the resume command and the payload shape for that kind (automations.md). A 422 invalid_request on payload means the answer does not fit the wait: a tool approval takes {"approved": true}, not {"response": "…"}.

A command answers 409

  • automation_busy: an occurrence is already running or waiting to run; wait for it, or send automation.stop_current.
  • invalid_state: the state rules the command out (already paused, not paused, nothing running, archived), or a run command such as pause or cancel was sent to an automation id; use the automation.* commands.
  • revision_conflict: someone revised the automation; read it again and resend with the new revision.
  • identity_conflict: the command_id or request_id was already used for a different request; generate a new one.

See automations.md.

A command was accepted but nothing changed

The receipt means queued. AbstractRuntime records whether the command was applied or rejected in the automation's ledger (GET /api/gateway/runs/{automation_id}/ledger, automation.command_result records); a rejected command names its reason there. Check that a runner is ticking the data folder.

A discussion turn is refused

  • 400 about context.messages: a discussion's history is provided by the gateway; send only the prompt.
  • 409 session_attribution_failed or history_unavailable: the discussion or the automation's history cannot be read from the store; start a new discussion with POST /api/gateway/automations/{automation_id}/discuss.

See automations.md.

Engines, models and apps

Install buttons are disabled or answer 403

Installs run on the gateway machine, so they follow the allow_engine_install setting: on by default for a loopback gateway and for someone at the gateway machine; off by default for a browser on another computer. An admin can turn it on. Installs also require an admin account.

An engine install stops in needs_admin or needs_tools

This is expected when a step needs an administrator password or the Apple command-line tools. Use Continue with administrator password or Install tools in the console, or abstractgateway engines continue <job-id>. On a headless machine, run the command the job shows and press Re-check. See engines.md.

A download says "Stalled" or ends "failed"

  • stalled: no bytes for 15 seconds; the job keeps trying and resumes by itself.
  • failed: ended_reason says what happened (a dropped connection, a Hub error, a full disk, a gateway restart) and what a new download reuses. Start the download again.
  • A parent job id (grp_…) answers 404 after a gateway restart; its children stay readable with GET /api/gateway/jobs.

See model-downloads.md.

An app does not install or start

Reason in the card or API Fix
network_unavailable the npm registry (or PyPI, for Node.js) is unreachable; installed apps keep working offline
no_free_port free a port in the app's usual range or set apps.ports
crash_loop open Show log (Technical details) or abstractgateway apps logs <app>
installs_not_allowed see "Install buttons are disabled" above
app_loopback_only the app listens on 127.0.0.1; open it from the gateway machine
started_outside_gateway the app was started elsewhere (dev stack, npx); stop it there
foreign_binary another program holds the terminal app's name in the install folder (often the older Python abstractcode from PyPI); run uv tool uninstall abstractcode or remove the named file, then install again

See apps.md.

The backlog folder is "not available on this gateway"

The saved backlog folder no longer exists (a deleted or unmounted checkout). Choose Use the gateway's own folder in Continuum or the console, or run abstractgateway config set triage_repo_root /path/to/checkout. See configuration.md.

Desktop

There is no tray icon

serve prints Desktop tray: started (pid …) or the reason it did not:

Reason Fix
missing_dependency pip install "abstractgateway[tray]" (Linux also needs the GTK/AppIndicator bindings)
headless no display (SSH, container, service); expected
dev_reload start without --reload
runner_only the tray belongs to the process that serves the console
no_tray_flag the gateway was started with serve --no-tray; start it without the flag

GNOME needs the AppIndicator extension. If the helper started and then disappeared, read <data dir>/logs/tray.log; GET /api/gateway/host/tray reports its exit code, and POST /api/gateway/host/tray/show (admin) starts it again. See tray.md.

The Assistant opened from the console is not signed in

  • It was already running: a running Assistant receives no sign-in code. Quit it and open it again from the console or the tray.
  • More than two minutes passed before it started, or the code was already used: open it again from the console.

See apps.md.

"Start at login" reads "needs repair" (service status: broken)

The registration points at a program that no longer exists (a moved or reinstalled gateway), is unreadable or disabled, or pins --host/--port so the Network setting cannot apply. Run:

abstractgateway service enable     # rewrite the registration for this gateway
abstractgateway service status

other means the login item belongs to another data folder. See first-run.md.

Starting the login service fails with 5: Input/output error (macOS)

abstractgateway service install (and service enable --start-now, and the installer, which runs service install) registers the login item and starts it now: it writes the LaunchAgent, then runs launchctl bootout (which stops a gateway already running as that login item) and launchctl bootstrap. launchd stops the old job in the background after bootout answers, and a bootstrap of the same login item before it is gone fails with 5: Input/output error (or 37). The gateway waits up to 15 seconds for launchd to finish and retries the bootstrap a few times, so this usually resolves by itself.

abstractgateway service enable without --start-now, and the Start at login switch in the consoles and the tray, only write the LaunchAgent for the next login; they run no launchctl command, so they never hit this.

If service install still fails (or launchd refuses the login item with another code, which is not retried), the message says how many attempts were made and why, and start at login is off: the LaunchAgent file is removed, so nothing starts at the next login. A gateway that was running as this login item was stopped by bootout and is not running now. Fix the cause the message shows, then:

abstractgateway service install    # registers the login item and starts it now
abstractgateway service status     # reads "on"

An update says "already up to date", or "didn't finish"

  • Already up to date means the update ran and changed nothing: an AbstractFramework installer install already has the newest release (the check compares releases, not the newest gateway on PyPI), or the package manager found nothing newer. No restart is needed.
  • The update didn't finish shows the exit code and the last lines of the log (web console: Update log under the version line). For an installer install, the installer's full log is the newest install-*.log in <data dir>/logs/; fix the cause it names and update again, or run the one-line install in a terminal, which does the same thing. The running gateway keeps working.
  • The installer changed since it was checked: a newer check fetched a different install.sh. Check again, review the confirmation, and update.
  • Check again first: an installer update runs only the install.sh whose sha256 the confirmation showed, and the request named none (a client older than the gateway, or an API call without installer_sha256). Check again and update from the confirmation.
  • An update is already running: one update runs at a time; wait for it to finish (its log is under the version line).
  • Couldn't check for updates (cannot compare the versions …): the release or PyPI named a version the gateway cannot read. Nothing is offered; check again later.
  • An installer install on Windows is updated by pasting the PowerShell line the hint shows (the gateway cannot replace its own files while it runs).

See tray.md.