Skip to content

Completed: Capability-level residency hooks for warm image models

Metadata

  • Created: 2026-05-19
  • Status: Completed
  • Completed: 2026-05-19
  • Priority: P1

Current Code Reality

AbstractVision already has most of the low-level primitives needed for model residency:

  • VisionBackend.preload() and VisionBackend.unload() exist on the shared backend contract;
  • MFluxVisionBackend, HuggingFaceDiffusersVisionBackend, and StableDiffusionCppVisionBackend implement best-effort preload/unload behavior;
  • the AbstractCore plugin caches routed backends in _routed_backends;
  • repeated requests for the same provider/model within one long-lived _AbstractVisionCapability instance already reuse the same backend object, so same-model warm reuse exists implicitly today;
  • the plugin already exposes list_provider_models() / list_available_models(), but that is provider catalog discovery, not resident-model listing;
  • PlaygroundState already has explicit load_model(), active_snapshot(), and unload_active() for a single active model in the local playground surface.

What is still missing is a stable capability-level control surface that AbstractCore can use to preload, inspect, and unload image models intentionally.

Evaluation

This proposal is useful, especially for:

  • long-lived Core worker processes that want deterministic warm local image models;
  • local backends with expensive first-load cost (mflux, diffusers, sdcpp);
  • operational control and debugging (loaded_at, last_used_at, explicit unload).

The current gap is narrower than “AbstractVision cannot keep models warm”:

  • warm reuse for the same model already happens implicitly inside a single capability instance;
  • the real missing piece is explicit control and honest observability across requests and model switches.

The original proposal is directionally correct, but it needs tightening:

  • _routed_backends alone is not enough to answer “what is resident?” because a cached backend may already have been unload()ed;
  • the plugin currently unloads the previous backend when generation switches to a different backend/model, so a preload-only API would be undone unless the switch behavior also changes;
  • the example task and model ids do not match current AbstractVision terminology (text_to_image / image_to_image, and real image model ids such as flux2-klein-4b);
  • exposing an internal cache_key as part of the public API would be brittle;
  • allowing arbitrary per-request backend options such as device, dtype, or quantize in the first version would conflict with current backend-construction and cache-key semantics.

Problem

Core wants generic /acore/models/load, /acore/models/loaded, and /acore/models/unload behavior across modalities. For image generation, Core should not:

  • reach into AbstractVision private plugin fields;
  • guess whether a cached backend object is still loaded;
  • depend on provider catalog listing as a proxy for warm model state.

Without an explicit plugin-owned residency surface:

  • loaded state remains implicit and process-private;
  • Core cannot reliably list which local image models are intentionally kept warm;
  • a model preloaded for one request may be unloaded on the next model switch;
  • long-lived worker processes cannot expose a clean, serializable residency contract.

Add explicit residency hooks to the AbstractVision AbstractCore capability object:

def load_resident_model(request: Mapping[str, Any]) -> Mapping[str, Any]: ...
def list_resident_models(filters: Mapping[str, Any] | None = None) -> list[Mapping[str, Any]]: ...
def unload_resident_model(request: Mapping[str, Any]) -> Mapping[str, Any]: ...

These methods should be public plugin methods that Core calls through the same capability object it already uses for generation and provider-catalog discovery.

Request normalization

The first pass should accept only the fields that AbstractVision can key reliably today:

{
  "task": "text_to_image",
  "provider": "mflux",
  "model": "flux2-klein-4b"
}

Notes:

  • task may be accepted for Core protocol alignment and validation, but it should not be part of the residency identity in v1 because current local backends use the same loaded weights for both text_to_image and image_to_image;
  • routed ids such as mflux/flux2-klein-4b or raw Diffusers ids such as runwayml/stable-diffusion-v1-5 may still be accepted as convenience input, but the plugin should normalize them to one canonical load_id;
  • do not add per-request backend-config overrides such as device, dtype, or quantize in v1. Those are currently process/config driven, and supporting them correctly would require changing backend construction and cache-key semantics. That can be a follow-up only if Core proves it needs multiple resident variants of the same model in one process.

Public state shape

Return stable API fields, not plugin-internal tuple keys:

{
  "task": "text_to_image",
  "provider": "mflux",
  "model": "flux2-klein-4b",
  "load_id": "mflux/flux2-klein-4b",
  "backend_kind": "mflux",
  "scope": "process",
  "state": "resident",
  "resident": true,
  "source": "explicit_preload",
  "loaded_at": 0,
  "last_used_at": 0,
  "error": null
}

Keep any backend cache key internal to the plugin.

Internal design

The plugin should keep using _routed_backends as the backend object cache. It should not add a second backend-object cache.

It should add a separate residency metadata sidecar, for example:

  • _resident_models: dict[backend_key, ResidentModelRecord]
  • _active_request_backend_key: Optional[backend_key]
  • a small lock around residency state
  • if concurrent generation/unload is expected, the same refcount/retire pattern already used by PlaygroundState

The key distinction is:

  • _routed_backends answers “which backend objects have been constructed?”
  • _resident_models answers “which models are intentionally kept warm right now?”

If both the playground and plugin continue to grow residency behavior, extract a small package-owned helper after the first plugin implementation rather than duplicating the state machine twice.

Generation interaction

load_resident_model() should:

  • resolve the same backend key used by generation;
  • call backend-native preload();
  • record loaded_at and last_used_at;
  • not generate an image.

t2i() / i2i() should:

  • preserve today’s same-model reuse behavior;
  • update last_used_at for resident entries;
  • stop auto-unloading a backend on model switch if that backend was explicitly loaded as resident;
  • continue auto-unloading non-resident backends on switch so current memory behavior remains conservative by default.

This is the main adjustment missing from the original proposal. Without it, a successful preload would be undone as soon as the next request activates a different backend.

Listing semantics

Expose two views:

  • list_loaded_models() for all plugin-visible local loaded models in the current process;
  • list_resident_models() for the explicit pinned subset.

That distinction is useful because AbstractVision already has opportunistic same-model warm reuse for the active backend, even when a model was not explicitly preloaded.

list_loaded_models() / list_resident_models() should report plugin-owned, process-local state only.

It should not inspect backend private fields such as _model, _pipelines, or _py_model to guess whether a backend is loaded. A later optional backend hook such as residency_info() can be added only if plugin-owned metadata proves insufficient.

HTTP backends

OpenAI/OpenAI-compatible HTTP backends should be out of scope for this surface, even when the endpoint is on localhost.

  • load_resident_model() / unload_resident_model() should fail clearly for HTTP backends;
  • list_loaded_models() / list_resident_models() should report only in-process local backends controlled by this plugin.

This keeps the contract honest. AbstractVision can control its own local backend objects; it cannot prove or orchestrate model residency inside another server process over a generic OpenAI-shaped API.

Worker Compatibility

The residency hooks should work inside a long-lived worker process, but the contract should be explicitly scoped:

  • residency is process-local, not cluster-global;
  • responses must be deterministic and JSON-serializable;
  • AbstractVision owns model-residency reporting, not worker lifecycle or orchestration policy.

Non-Goals

  • No cross-process or cluster-wide residency registry in v1.
  • No attempt to pin models inside remote providers that expose only OpenAI-compatible inference.
  • No automatic model selection from provider catalogs.
  • No VisionManager public API expansion in the first pass.

Success Criteria

  • AbstractCore can warm a local AbstractVision model without private-field reach-through.
  • AbstractCore can inspect both explicitly resident models and transient currently loaded local models.
  • AbstractCore can list explicitly resident image models with stable provider/model/load metadata.
  • Explicitly resident models survive generation requests that switch to other models until they are unloaded.
  • Non-resident models keep the current unload-on-switch behavior.
  • OpenAI-compatible HTTP backends are rejected for residency control rather than being reported as falsely resident.
  • The implementation does not depend on backend private fields to report residency.

Validation Ideas

  • Unit tests for residency request normalization and canonical load_id generation.
  • Plugin tests that repeated same-model generation still reuses the same backend object.
  • Plugin tests that request-driven warm local backends appear in list_loaded_models().
  • Plugin tests that an explicitly resident backend is not unloaded when another model is generated.
  • Plugin tests that unload_resident_model() clears metadata and calls unload().
  • Plugin tests that OpenAI-compatible HTTP backends are rejected for residency control.
  • If concurrency is supported, tests that in-flight generation is not unloaded mid-request.

Completion Report

Date: 2026-05-19

Summary

  • Added explicit process-local residency control to the AbstractCore plugin through load_resident_model(), list_loaded_models(), list_resident_models(), and unload_resident_model().
  • Preserved conservative unload-on-switch behavior for non-resident local backends while keeping explicitly resident backends loaded across model switches.
  • Reported stable plugin-owned residency metadata (load_id, backend_kind, resident, state, timestamps, observed tasks) without depending on backend private fields.
  • Rejected OpenAI/OpenAI-compatible HTTP backends for residency control so the contract stays honest about what AbstractVision can actually keep loaded in-process.
  • Kept the public residency surface process-local and JSON-safe for Core/Gateway-style route adapters.

Files And Symbols Touched

  • src/abstractvision/integrations/abstractcore_plugin.py
  • _AbstractVisionCapability.load_resident_model
  • _AbstractVisionCapability.list_loaded_models
  • _AbstractVisionCapability.list_resident_models
  • _AbstractVisionCapability.unload_resident_model
  • _AbstractVisionCapability.unload_model
  • _AbstractVisionCapability._record_loaded_model
  • _AbstractVisionCapability._activate_request_backend
  • _AbstractVisionCapability._find_loaded_backends
  • _AbstractVisionCapability._normalize_loaded_filters
  • tests/test_abstractcore_plugin.py
  • docs/reference/abstractcore-integration.md
  • CHANGELOG.md

Behavior Changes

  • Core/plugin callers can now preload, inspect, and unload local resident image backends intentionally instead of relying only on implicit same-model reuse.
  • list_loaded_models() now distinguishes explicitly resident entries from transient request-loaded local backends.
  • Explicitly resident local backends survive switches to other models until they are explicitly unloaded.
  • OpenAI/OpenAI-compatible backends are refused for residency control, even when pointed at localhost.

Validation

  • Historical shipped validation is reflected in the 0.3.7 changelog entry and the plugin test coverage added with that release.
  • PYTHONPATH=src python -m unittest tests.test_abstractcore_plugin -q passed on 2026-05-20 during backlog verification, 33 tests.
  • git diff --check passed on 2026-05-20 during backlog verification.

Docs

  • Public integration docs describe the residency control surface and its process-local scope.
  • Changelog records the 0.3.7 residency feature and its semantics.

Residual Risks

  • The current residency surface is honest about loaded state, but it does not prove that a first matching inference is fully warmed. That follow-up was later captured, then deprecated, in docs/backlog/deprecated/2026-05-20_true_image_warmup_semantics.md.
  • Residency is process-local only; cluster-wide orchestration and worker lifecycle policy remain downstream concerns.

Backlog / Code Drift Found

  • This item remained in docs/backlog/proposed/ even though the code, tests, docs, and changelog already showed it was complete.
  • The item has now been moved to docs/backlog/completed/ and its completion report recorded.

Follow-ups

  • A narrower true-warmup follow-up was proposed separately at the time and later deprecated after docs/backlog/completed/019_best_effort_preload_warmup_for_local_backends.md addressed the operational warmup need without adding a public warmness contract.
  • If the repo later adds backlog overview or recurrent hygiene files, this completion should be reflected there as well.

Priority Impact

  • This no longer competes for implementation priority.
  • Remaining priority is on whether true warmup semantics and stronger engine-specific warmup are worth adding beyond current residency.