AbstractCore integration¶

AbstractVision offers two integration surfaces for AbstractCore:

1) Capability plugin (so abstractcore can discover a vision backend) 2) Tool helpers (so you can expose vision tasks as tools with artifact-ref outputs)

Code pointers: - Plugin: ../../src/abstractvision/integrations/abstractcore_plugin.py - Tools: ../../src/abstractvision/integrations/abstractcore.py - Entry point registration: ../../pyproject.toml ([project.entry-points."abstractcore.capabilities_plugins"])

See also: - Artifacts: docs/reference/artifacts.md - Backends: docs/reference/backends.md

1) Capability plugin (AbstractCore → VisionCapability)¶

The plugin registers these backend ids:

abstractvision:openai (default backend id) and abstractvision:openai-compatible (legacy compatibility backend id) are registered by ../../src/abstractvision/integrations/abstractcore_plugin.py. The implementation also supports local backends.

Current behavior: - Default abstractvision:openai: OpenAI HTTP (https://api.openai.com/v1). Set OPENAI_API_KEY. - OpenAI model ids are configured, not discovered dynamically. Providers may expose an OpenAI-compatible GET /models catalog; AbstractVision exposes it through abstractvision provider-models, VisionManager.list_provider_models(...), and the plugin method llm.vision.list_provider_models(...), but the plugin does not call it automatically or use it to select a model. The static plugin default is gpt-image-1; set OPENAI_IMAGE_MODEL_ID, OPENAI_IMAGE_MODEL, ABSTRACTVISION_MODEL_ID, or vision_model_id for newer provider models. - Compatible HTTP: set OPENAI_BASE_URL to a local/remote compatible /v1 server. Set ABSTRACTVISION_BACKEND=openai-compatible when you want to force compatible-endpoint semantics. - Legacy abstractvision:openai-compatible: keeps compatible-endpoint defaults when that backend id is selected directly. - Local Diffusers: install abstractvision[diffusers], then set ABSTRACTVISION_BACKEND=diffusers with runwayml/stable-diffusion-v1-5 or another supported Diffusers model. It is cache-only/offline unless ABSTRACTVISION_DIFFUSERS_ALLOW_DOWNLOAD=1 is set. Local text_to_video groundwork exists but is currently experimental and disabled from the normal local plugin surfaces. - Local MLX-Gen (Apple-first): install abstractvision[mlx-gen] (or abstractvision[all-apple]), then set ABSTRACTVISION_BACKEND=mlx-gen. The current AbstractVision release is validated on Apple Silicon first; the install extra also exposes Linux support when upstream mlx-gen / mlx markers are available. Download exact model repos such as abstractvision download AbstractFramework/flux.2-klein-4b-4bit --provider mlx-gen, abstractvision download AbstractFramework/ernie-image-turbo-8bit --provider mlx-gen, abstractvision download briaai/FIBO --provider mlx-gen, abstractvision download prism-ml/bonsai-image-ternary-4B-mlx-2bit --provider mlx-gen, abstractvision download AbstractFramework/seedvr2-3b-8bit --provider mlx-gen, abstractvision download AbstractFramework/seedvr2-7b-8bit --provider mlx-gen, abstractvision download Wan-AI/Wan2.2-TI2V-5B-Diffusers --provider mlx-gen, abstractvision download AbstractFramework/wan2.2-t2v-a14b-diffusers-8bit --provider mlx-gen, or abstractvision download AbstractFramework/wan2.2-i2v-a14b-diffusers-8bit --provider mlx-gen (stored in the Hugging Face cache; ABSTRACTVISION_MODEL_DIR is only a legacy import root). Use routed exact model ids such as mlx-gen/AbstractFramework/flux.2-klein-4b-4bit, mlx-gen/prism-ml/bonsai-image-ternary-4B-mlx-2bit, mlx-gen/AbstractFramework/seedvr2-3b-8bit, mlx-gen/AbstractFramework/seedvr2-7b-8bit, mlx-gen/AbstractFramework/wan2.2-t2v-a14b-diffusers-8bit, or mlx-gen/AbstractFramework/wan2.2-i2v-a14b-diffusers-8bit. q4 repos are the memory-efficient default recommendation for most generation/edit models; SeedVR2 defaults to canonical q8 packages and offers q4 packages for tighter memory. Bonsai ternary is already pre-packed at 2-bit and is selected by its exact repo id. Legacy mflux provider values remain accepted as aliases. - stable-diffusion.cpp: set ABSTRACTVISION_BACKEND=sdcpp and configure either a model path or a curated model key such as flux2-klein-base-4b. Use an external sd-cli, or install abstractvision[sdcpp] for the python binding fallback. - The plugin reads AbstractCore owner config keys when present, then falls back to ABSTRACTVISION_* env vars. - Gateway/Core should pass process-level config or owner.config and report readiness; they should not mutate AbstractVision environment variables per request.

Key config keys (owner.config): - vision_backend_instance / vision_backend_factory (advanced injection hooks; bypass env-driven backend creation) - vision_backend (openai, openai-compatible, diffusers, mlx-gen, or sdcpp; default openai; mflux is accepted as a compatibility alias) - vision_model_id (Diffusers/OpenAI-compatible model id; default gpt-image-1 only for the official OpenAI profile and runwayml/stable-diffusion-v1-5 for Diffusers) - vision_device / vision_torch_dtype / vision_allow_download / vision_auto_retry_fp32 (Diffusers) - vision_base_url / vision_api_key (OpenAI or compatible HTTP) - vision_mflux_model / vision_mflux_base_model / vision_mflux_allow_download (MLX-Gen; env/config names keep MFLUX compatibility) - vision_mlx_gen_lora_paths / vision_mflux_lora_paths, vision_mlx_gen_lora_scales / vision_mflux_lora_scales, and vision_mlx_gen_lora_target_roles / vision_mflux_lora_target_roles for backend-default MLX-Gen LoRA adapters - vision_model_dir (legacy preset import root used by MLX-Gen compatibility/migration helpers; new downloads land in the Hugging Face cache) - vision_sdcpp_model / vision_sdcpp_diffusion_model / vision_sdcpp_bin (stable-diffusion.cpp; vision_sdcpp_model may be a curated model key such as flux2-klein-base-4b) - vision_sdcpp_vae / vision_sdcpp_llm / vision_sdcpp_llm_vision / vision_sdcpp_clip_l / vision_sdcpp_clip_g / vision_sdcpp_t5xxl / vision_sdcpp_extra_args (stable-diffusion.cpp component mode) - vision_timeout_s (optional; applies to OpenAI-compatible catalog/control behavior, not image/video generation deadlines) - vision_models_path (optional provider catalog path; default /models) - Optional video endpoint keys: - vision_text_to_video_path - vision_image_to_video_path - vision_image_to_video_mode

Env-only aliases: - ABSTRACTVISION_DIFFUSERS_MODEL_ID is accepted for the Diffusers plugin backend before falling back to ABSTRACTVISION_MODEL_ID. - OPENAI_BASE_URL is accepted by the OpenAI-shaped HTTP backend. - Compatible OpenAI-shaped endpoints can use OPENAI_API_KEY when they require bearer auth. - OPENAI_IMAGE_MODEL_ID and OPENAI_IMAGE_MODEL are accepted when vision_model_id / ABSTRACTVISION_MODEL_ID are unset. - ABSTRACTVISION_SDCPP_CLIP_L, ABSTRACTVISION_SDCPP_CLIP_G, and ABSTRACTVISION_SDCPP_T5XXL are accepted for stable-diffusion.cpp component mode.

Examples:

# Local Diffusers. Pre-download weights first, or explicitly allow runtime downloads.
export ABSTRACTVISION_BACKEND=diffusers
export ABSTRACTVISION_MODEL_ID=runwayml/stable-diffusion-v1-5
export ABSTRACTVISION_DIFFUSERS_DEVICE=auto

from abstractcore import create_llm

llm = create_llm("openai", model="gpt-4o-mini")
png_bytes = llm.vision.t2i("a red square", width=512, height=512, steps=20)

Per-call routing can select the same backend/model ids used by the abstractvision shell commands. For MLX-Gen, pass the exact published model repo id; there is no separate quantization parameter.

from pathlib import Path

from abstractcore import create_llm

llm = create_llm("openai", model="gpt-4o-mini")

png_bytes = llm.vision.t2i(
    "a studio product photo of a white ceramic mug with the AbstractFramework logo",
    provider="mlx-gen",
    model="AbstractFramework/qwen-image-2512-4bit",
    steps=20,
    guidance_scale=1.0,
    on_progress=lambda event: print(f"{event.phase}: step {event.step}/{event.total_steps}"),
)

edited_png = llm.vision.i2i(
    "replace the background with a clean white studio setup",
    image="./input.png",
    provider="mlx-gen",
    model="AbstractFramework/qwen-image-edit-2511-4bit",
    steps=20,
    guidance_scale=2.5,
    strength=0.75,
    on_progress=lambda event: print(f"{event.phase}: step {event.step}/{event.total_steps}"),
)

multi_ref_png = llm.vision.i2i(
    "compose the subject using the second image as style and layout reference",
    image="./subject.png",
    provider="mlx-gen",
    model="AbstractFramework/qwen-image-edit-2511-4bit",
    reference_images=[Path("./style-reference.png").read_bytes()],
    steps=20,
    guidance_scale=2.5,
)

def on_video_progress(event):
    print(
        f"{event.phase}: step {event.step}/{event.total_steps}, "
        f"frame {event.frame}/{event.total_frames}"
    )

mp4_bytes = llm.vision.t2v(
    "a red fox walking through a snowy forest, cinematic",
    provider="mlx-gen",
    model="AbstractFramework/wan2.2-t2v-a14b-diffusers-8bit",
    width=432,
    height=240,
    num_frames=41,
    fps=10,
    steps=20,
    guidance_scale=4.0,
    guidance_2=3.0,
    max_sequence_length=256,
    on_progress=on_video_progress,
)

first_frame_mp4 = llm.vision.i2v(
    "./first-frame.png",
    prompt="slow camera push-in",
    provider="mlx-gen",
    model="AbstractFramework/wan2.2-i2v-a14b-diffusers-8bit",
    width=432,
    height=240,
    num_frames=41,
    fps=10,
    steps=20,
    guidance_scale=3.5,
    guidance_2=3.5,
    max_sequence_length=256,
    on_progress=on_video_progress,
)

llm.vision.i2i(...) accepts image as bytes, a local file path, or an artifact ref dict when an artifact_store is supplied. Without an artifact store the plugin returns bytes; with artifact_store=... it returns the stored artifact ref, matching the runtime/gateway artifact path. Typed per-call keyword arguments such as control_image, control_strength, lora_adapters, guidance_2, and on_progress survive the plugin boundary. Unknown per-call keyword arguments such as reference_images and max_sequence_length are preserved in the AbstractVision request extra dict for local backends. llm.vision.t2i(...), llm.vision.i2i(...), llm.vision.t2v(...), and llm.vision.i2v(...) forward on_progress to MLX-Gen. Image/edit events include diffusion-step progress; Wan video events use diffusion-step progress and also include frame counters and frame_progress.

For task-specific Wan A14B video models, guidance_2 is a typed request parameter for the second-stage/low-noise guidance path. Pass it directly to t2v(...) / i2v(...); do not wrap it in extra.

For MLX-Gen base-Qwen structured control, pass control_image=<bytes> and optional control_strength=<float> directly to t2i(...). Unsupported backends reject those fields explicitly instead of silently ignoring them.

The same applies to LoRA adapters. Prefer the shared typed contract:

from abstractvision import LoRAAdapterSpec

png = llm.vision.t2i(
    "Pixel art astronaut floating above Earth",
    provider="mlx-gen",
    model="AbstractFramework/qwen-image-2512-8bit",
    lora_adapters=[
        LoRAAdapterSpec(
            source="prithivMLmods/Qwen-Image-2512-Pixel-Art-LoRA:Qwen-Image-2512-Master-Pixel-Art-LoRA.safetensors",
            scale=1.0,
        )
    ],
)

Wan LoRA calls may also need target_role in each adapter spec. TI2V-5B uses transformer. A14B routes require explicit high_noise_transformer / low_noise_transformer roles.

# OpenAI API.
export OPENAI_API_KEY=...
export OPENAI_IMAGE_MODEL=gpt-image-1

# Local OpenAI-compatible HTTP server, for example AbstractCore Server.
export ABSTRACTVISION_BACKEND=openai-compatible
export OPENAI_BASE_URL=http://localhost:8000/v1
export ABSTRACTVISION_MODEL_ID=server/default

Provider Catalog Discovery¶

Core/Gateway hosts can inspect provider-advertised model catalogs through the same capability object used for generation:

models = llm.vision.list_provider_models(task="text_to_image")
for model in models:
    print(model["id"])

The return value is a JSON-safe list of dictionaries serialized from ProviderModelInfo. Raw provider metadata is retained in a bounded raw field for diagnostics. This method is explicit inspection only: it does not mutate the configured backend or select a generation model.

Backends that do not implement provider catalog listing raise a clear AbstractVision error instead of returning a misleading empty catalog. When multiple backends are enabled, the plugin probes their catalogs concurrently so discovery waits for the slowest backend instead of the sum of serial scans. Local Diffusers and stable-diffusion.cpp model discovery remain separate local-backend concerns, while MLX-Gen and Diffusers provider listings reflect cache-backed snapshots rather than a separate ~/models download tree.

The same capability object also exposes installed adapter discovery for routes that support LoRAs:

adapters = llm.vision.list_provider_adapters(
    provider="mlx-gen",
    model="AbstractFramework/wan2.2-ti2v-5b-diffusers-8bit",
    task="text_to_video",
)
for adapter in adapters:
    print(adapter["id"], adapter.get("suggested_target_roles"))

Adapter truth stays backend-owned. The plugin serializes ProviderAdapterInfo into JSON-safe dictionaries, but it does not invent compatibility metadata on its own.

Batch generation through Core¶

The plugin exposes explicit batch helpers in addition to the singular task methods:

llm.vision.t2i_batch(...)
llm.vision.i2i_batch(...)
llm.vision.t2v_batch(...)
llm.vision.i2v_batch(...)

These methods use the same typed request contract as the singular calls: stacked lora_adapters=[...], explicit seeds=[...], optional base seed expansion, and typed video controls such as guidance_2 / flow_shift.

videos = llm.vision.t2v_batch(
    "A compact silver spaceship lifts off from an icy canyon at dawn.",
    provider="mlx-gen",
    model="AbstractFramework/wan2.2-ti2v-5b-diffusers-8bit",
    count=2,
    seeds=[6251, 6252],
    flow_shift=3.0,
    lora_adapters=[
        LoRAAdapterSpec(
            source="owner/wan-style:video.safetensors",
            scale=0.9,
            target_role="transformer",
        )
    ],
)

Local Model Residency Control¶

Core/Gateway hosts can also control in-process local model residency through the same capability object:

llm.vision.load_resident_model(
    {"task": "text_to_image", "provider": "mlx-gen", "model": "AbstractFramework/flux.2-klein-4b-4bit"}
)
llm.vision.load_resident_model(
    {"task": "text_to_video", "provider": "mlx-gen", "model": "AbstractFramework/wan2.2-t2v-a14b-diffusers-8bit"}
)

loaded = llm.vision.list_loaded_models()
resident = llm.vision.list_resident_models()

llm.vision.unload_resident_model(
    {"provider": "mlx-gen", "model": "AbstractFramework/flux.2-klein-4b-4bit"}
)

Notes:

this surface is local-only and process-local;
it controls only AbstractVision-managed in-process backends (diffusers, mlx-gen, sdcpp);
OpenAI/OpenAI-compatible HTTP backends are intentionally rejected here, even on localhost, because the plugin cannot honestly control another process's loaded-state.

list_loaded_models() reports both explicitly preloaded resident models and transient currently loaded local models from recent generation requests. list_resident_models() returns the explicit pinned subset.

Each loaded-model entry includes stable routing metadata such as load_id, backend_kind, resident, and a tasks list of observed task aliases using that loaded backend.

For deterministic unload behavior, prefer:

load_id, or
both provider/backend and model.

Broad unload filters that match multiple loaded models are rejected as ambiguous.

Shared request normalization¶

llm.vision calls go through VisionManager, which applies backend normalization hooks before execution. That means model-specific constraints are shared with the CLI/REPL and playground API instead of being reimplemented in the plugin layer.

Current examples: - Distilled MLX-Gen/FLUX-family models can clamp guidance or minimum steps in the backend. - Diffusers-backed image requests pick up registry-driven defaults such as recommended step counts, guidance defaults, and dimension constraints. - Unsupported parameters, such as negative prompts on constrained model families, are dropped in the backend rather than surfacing as per-surface validation bugs.

2) Tool helpers (`make_vision_tools`)¶

make_vision_tools(...) builds AbstractCore @tool callables for: - adapter discovery for the selected model - text→image - batch text→image - image→image - batch image→image - image upscaling - multi-view image - text→video - batch text→video - image→video - batch image→video

The helper surface now stays aligned with the main AbstractCore capability plugin surface:

lora_adapters=[...] is accepted on text→image, image→image, text→video, and image→video tool calls.
batch tools accept count plus optional explicit seeds=[...].
image-edit tools accept reference_images=[...], where each item can be an artifact ref dict or a {"b64": "..."} payload.
vision_list_adapters(task=...) returns the currently discovered compatible adapter inventory for the selected model.

Important: - Tool outputs are designed to be artifact refs, so VisionManager.store must be set (../../src/abstractvision/integrations/abstractcore.py). - This module requires AbstractCore to be installed by the host application. AbstractVision does not install AbstractCore as a dependency.

Tip (framework mode): - If your runtime provides an artifact store (e.g. AbstractRuntime), use RuntimeArtifactStoreAdapter so tool outputs can be stored and referenced across processes (see docs/reference/artifacts.md).