AbstractGateway deployment

AbstractGateway can run as a Python process or as a containerized server. The container path is the recommended baseline for a single self-contained Gateway deployment because it packages the HTTP API, durable runner, AbstractRuntime, and the Runtime-owned provider/tool stack together.

Published image

Release images are published to GHCR. The default image is the light, portable server image:

docker pull ghcr.io/lpalbou/abstractgateway:0.2.28

NVIDIA hosts can try the experimental full GPU image when local vLLM/HuggingFace/Diffusers engines are wanted. This image is published best-effort until it has a real CUDA build and smoke gate:

docker pull ghcr.io/lpalbou/abstractgateway:0.2.28-gpu

Legacy aliases ghcr.io/lpalbou/abstractgateway-server:* and ghcr.io/lpalbou/abstractgateway-server-nvidia:* are still published for a transition period. New deployments should use abstractgateway.

The default image installs the base abstractgateway package, which includes:

• AbstractRuntime
• AbstractMemory[lancedb]>=0.2.6
• abstractagent
• FastAPI/Uvicorn

This profile supports hosted/commercial providers, OpenAI-compatible text and multimodal provider routing, Runtime-owned tool execution, KG memory, and provider/session prompt-cache controls. Remote embeddings are included through the embedding.text capability route for hosted providers, LM Studio, vLLM, other OpenAI-compatible endpoints, or a remote AbstractCore server. Local sentence-transformer embeddings and hardware-local model runtimes remain explicit opt-ins, so the base Linux image does not pull PyTorch/CUDA runtime packages. MLX, vLLM, HuggingFace Transformers, local Diffusers/sdcpp, AbstractVoice local engines, and local AbstractMusic engines belong in native abstractgateway[apple] or abstractgateway[gpu] installs.

The NVIDIA image installs abstractgateway[gpu] and uses a CUDA/PyTorch base. It is experimental and release automation publishes it as best-effort for linux/amd64; the default image remains the release-grade portable linux/amd64 and linux/arm64 image. Treat the NVIDIA image as production-ready only after a CUDA host build/smoke gate is added and passes.

Apple Silicon / MLX

There is no Apple/MLX Gateway Docker image target. MLX uses Apple's Metal stack, while Docker Desktop runs Linux containers without Metal/MPS device access. The supported Docker shape is a lightweight Gateway container calling a host-native OpenAI-compatible inference endpoint:

docker run --rm --name abstractgateway \
  -p 8080:8080 \
  -e ABSTRACTGATEWAY_DATA_DIR=/data \
  -e ABSTRACTGATEWAY_USER_AUTH=1 \
  -e OPENAI_BASE_URL="http://model-runner.docker.internal/engines/v1" \
  -v "$PWD/runtime:/data" \
  ghcr.io/lpalbou/abstractgateway:latest

Set the execution-host text route separately:

docker exec abstractgateway abstractgateway-config set-default input.text \
  --provider openai-compatible \
  --model your-model \
  --base-url http://model-runner.docker.internal/engines/v1

Other host-native endpoints are also valid: LM Studio at http://host.docker.internal:1234/v1 with LMSTUDIO_BASE_URL, Ollama at http://host.docker.internal:11434 with OLLAMA_BASE_URL, or mlx_lm.server exposed on a host port. For fully native non-Docker installs with local engines, use pip install "abstractgateway[apple]" on Apple Silicon, and pip install "abstractgateway[gpu]" on GPU workstations or NVIDIA Docker builds.

Compose quickstart

Create an env file from the template, adjust provider keys/defaults, then start the server. The default env keeps user auth enabled and bootstraps default/admin if missing:

cp docker/abstractgateway-server/.env.example docker/abstractgateway-server/.env
docker compose --env-file docker/abstractgateway-server/.env \
  -f docker/abstractgateway-server/compose.yml up -d

For the experimental NVIDIA image on a GPU host with the NVIDIA Container Toolkit:

docker compose --env-file docker/abstractgateway-server/.env \
  -f docker/abstractgateway-server/compose.yml \
  -f docker/abstractgateway-server/compose.nvidia.yml up -d

The default compose profile binds to 127.0.0.1:8080, mounts a durable Gateway data volume at /data, mounts bundles from flows/bundles at /data/flows, and exposes a container workspace at /workspace.

Smoke checks:

curl http://127.0.0.1:8080/api/health

ADMIN_TOKEN="$(docker compose -f docker/abstractgateway-server/compose.yml exec -T abstractgateway cat /data/auth/bootstrap-admin-token)"
curl -H "Authorization: Bearer $ADMIN_TOKEN" \
  http://127.0.0.1:8080/api/gateway/me

Core configuration

Required for hosted/container user-auth mode:

• ABSTRACTGATEWAY_USER_AUTH=1: enables Gateway user tokens and per-user routing
• ABSTRACTGATEWAY_BOOTSTRAP_ADMIN=1: creates default/admin if missing

Optional:

• ABSTRACTGATEWAY_AUTH_TOKEN: legacy shared admin bearer token for compatibility/bootstrap APIs; browser apps should use Gateway user tokens

Common:

• ABSTRACTGATEWAY_ALLOWED_ORIGINS: browser origin allowlist
• input.text capability route: default for LLM/agent nodes
• ABSTRACTGATEWAY_TOOL_MODE: approval, passthrough, delegated, or local dev modes
• ABSTRACTGATEWAY_STORE_BACKEND: file or sqlite
• ABSTRACTGATEWAY_DB_PATH: SQLite file, when using sqlite
• ABSTRACTGATEWAY_RUNNER: 1 for combined API+runner, 0 for API-only
• ABSTRACTGATEWAY_MEMORY_STORE_BACKEND: lancedb or memory for KG workflows and /kg/query; sqlite works when the installed AbstractMemory build exposes SQLiteTripleStore

Provider keys and endpoints:

• OPENAI_API_KEY
• ANTHROPIC_API_KEY
• OPENROUTER_API_KEY
• PORTKEY_API_KEY / PORTKEY_CONFIG
• OPENAI_BASE_URL / OPENAI_API_KEY for generic OpenAI-compatible endpoints
• OPENAI_COMPATIBLE_BASE_URL / OPENAI_COMPATIBLE_API_KEY as legacy operator aliases; prefer OPENAI_BASE_URL for AbstractCore discovery
• LMSTUDIO_BASE_URL
• OLLAMA_BASE_URL
• VLLM_BASE_URL

Image/voice plugin endpoints:

• ABSTRACTVISION_BACKEND: openai, openai-compatible, diffusers, or sdcpp
• ABSTRACTGATEWAY_VISION_BACKEND / ABSTRACTGATEWAY_VISION_BASE_URL / ABSTRACTGATEWAY_VISION_API_KEY / ABSTRACTGATEWAY_VISION_MODEL_ID (legacy ABSTRACTVISION_* names still work)
• ABSTRACTGATEWAY_VOICE_TTS_ENGINE / ABSTRACTGATEWAY_VOICE_STT_ENGINE (openai by default in the server image; legacy ABSTRACTVOICE_* names still work)
• ABSTRACTGATEWAY_VOICE_REMOTE_BASE_URL / ABSTRACTGATEWAY_VOICE_REMOTE_API_KEY
• ABSTRACTGATEWAY_VOICE_TTS_MODEL / ABSTRACTGATEWAY_VOICE_STT_MODEL

Core catalog proxying:

• ABSTRACTCORE_SERVER_BASE_URL: explicit standalone Core server URL for voice, TTS/STT, and vision catalog routes
• ABSTRACTGATEWAY_ABSTRACTCORE_SERVER_AUTH_TOKEN: Core server auth token, separate from Gateway auth
• ABSTRACTGATEWAY_CORE_CATALOG_TIMEOUT_S: timeout for catalog routes

Filesystem/media controls from AbstractCore remain available:

• ABSTRACTCORE_SERVER_BASE_URL_ALLOWLIST
• ABSTRACTCORE_SERVER_URL_FETCH_ALLOWLIST
• ABSTRACTCORE_SERVER_MEDIA_ROOT
• ABSTRACTCORE_SERVER_ALLOW_LOCAL_FILES

Cache and auth notes

Gateway auth is controlled by ABSTRACTGATEWAY_* variables and protects /api/gateway/*. AbstractCore provider/server auth variables control upstream provider access inside AbstractCore integrations. Keep those two layers separate: clients receive only the Gateway token, while provider keys stay in the server environment.

Prompt-cache control endpoints are exposed under /api/gateway/prompt_cache/* where supported by the active provider/model. Session lifecycle routes under /api/gateway/sessions/{session_id}/prompt_cache/* provide Gateway-owned naming/status/prepare/clear/rebuild orchestration on top of those provider controls. They are not a provider-independent local KV cache or full CachedSession persistence system.

Local-source image

Before a version is published to PyPI, build from the checkout:

ABSTRACTGATEWAY_INSTALL_MODE=local \
ABSTRACTGATEWAY_IMAGE_TAG=0.2.28-local \
docker compose -f docker/abstractgateway-server/compose.yml up -d --build

Release automation builds the published image from the PyPI package after the PyPI release is available, matching the AbstractCore server image pattern.