Skip to content

Planned: Model family, component, overlay, and adapter inventory graph for catalog and curated local flows

Metadata

  • Created: 2026-05-20
  • Status: Planned
  • Priority: P1
  • Completed: N/A

ADR status

Context

AbstractVision now has a useful capability registry, a useful curated downloader, and a clearer package policy baseline. What it still lacks is a clean taxonomy for the artifacts it surfaces.

The current planning problem is not “support adapters somehow.” It is more specific:

  • external users should not need to understand internal side artifacts such as VAE or encoder files when a curated local flow exists;
  • adapter-like repos must not be presented as standalone runnable models when the runtime cannot actually execute them that way;
  • catalog, downloader, CLI, playground, and AbstractCore surfaces should agree on what is a full model, what is a required component, and what is an optional overlay.

The 2026-05-20 review surfaced the exact ambiguity to fix:

  • stable-diffusion.cpp component stacks such as FLUX/Qwen require side artifacts like VAE and LLM/encoder files, but those are required components, not adapters;
  • Diffusers LoRA and MFLUX LoRA inputs are optional overlays, not full model families;
  • some Hugging Face repos look like “models” in an inventory pass but are really overlays or base-model-dependent attachments, for example:
  • lightx2v/Qwen-Image-Lightning
  • fal/Qwen-Image-Edit-2511-Multiple-Angles-LoRA
  • black-forest-labs/FLUX.1-Redux-dev

Those entries were removed from the curated registry the same day because the current product surface does not yet model them honestly enough.

Current code reality

Files and symbols re-checked before revising this item:

  • src/abstractvision/assets/vision_model_capabilities.json
  • src/abstractvision/model_capabilities.py
  • src/abstractvision/model_downloads.py
  • src/abstractvision/vision_manager.py
  • src/abstractvision/types.py
  • src/abstractvision/backends/huggingface_diffusers.py
  • src/abstractvision/backends/mflux.py
  • src/abstractvision/cli.py
  • src/abstractvision/integrations/abstractcore.py
  • docs/getting-started.md

What is already implemented:

  • The registry schema supports models -> downloads[] -> tasks.
  • download-model, model-presets, and model-catalog already expose a package-owned curated surface on top of the registry plus _PRESETS.
  • stable-diffusion.cpp now has a package-owned curated bundle path via resolve_sdcpp_model_selection(...), which resolves required cached companion artifacts for keys such as flux2-klein-base-4b and qwen-image.
  • Diffusers already has real overlay behavior:
  • _parse_loras()
  • _apply_loras()
  • documented loras_json and rapid_aio_repo request-extra flows
  • MLX-Gen now also has a package-owned typed LoRA request contract through LoRAAdapterSpec / lora_adapters=[...] across text_to_image, image_to_image, text_to_video, and image_to_video.
  • VisionManager now exposes public batch orchestration methods:
  • generate_image_batch(...)
  • edit_image_batch(...)
  • generate_video_batch(...)
  • image_to_video_batch(...)
  • VisionManager.list_provider_adapters(...) and abstractvision adapters ... now expose backend-owned MLX-Gen installed-adapter discovery by exact model/task route.
  • Wan video requests now have typed flow_shift.
  • The bundled MLX-Gen proof page and proof assets now use a corrected TI2V-5B visual validation at 832x480.
  • ImageGenerationRequest and ImageEditRequest still include extra, so the package retains a low-level carrier for backend-owned overlay metadata where a typed surface is not appropriate yet.

What is missing or brittle:

  • The registry has no first-class distinction between:
  • parent model family
  • runtime variant
  • required component
  • optional overlay
  • VisionManager gates only on task names. It does not interpret task.requires, and current shipped JSON does not yet exercise base_model_id as a real runtime contract.
  • _PRESETS and curated bundle logic still encode important catalog truth in code rather than in a clearer package data model.
  • Installed-adapter discovery is currently MLX-Gen-specific. There is still no cross-backend overlay inventory story for Diffusers or stable-diffusion.cpp.
  • The current docs show working Diffusers overlay examples, but the catalog cannot yet tell a user whether a repo is a standalone model, a required component, or an overlay attachment.
  • multi_view_image exists as a task key, but no shipped backend currently implements it, so LoRA repos that imply multi-view behavior must not be cataloged as if they were runnable tasks.
  • The catalog still lacks an explicit artifact-role taxonomy, so overlays, required components, and runtime-ready variants are not yet modeled as a first-class graph in package data.

Problem

The current registry and download surfaces are still too flat. They are good at listing model ids and repo ids, but they are not yet a clean graph of:

  • parent model families;
  • runtime-specific variants;
  • required components that must be assembled for one runtime path;
  • optional overlays that only make sense with a compatible parent backend/model;
  • runtime-ready versus download-only versus unsupported states.

That makes the curated catalog less honest than it should be and pushes too much repo knowledge back onto the user.

What we want to do

Introduce an explicit package-owned artifact taxonomy and use it to improve:

  • vision_model_capabilities.json
  • model-catalog
  • model-presets
  • download-model
  • curated local runtime resolution
  • overlay-aware generation surfaces where the backend already supports them

The first goal is still not “generic adapter orchestration.” The first goal is to stop confusing model families, required components, and optional overlays. The second goal is to expose locally installed overlays honestly enough that users can discover, select, and apply them without leaving AbstractVision.

Why

Users are not asking for a raw list of Hugging Face repos. They want reliable answers to:

  • what model family this is;
  • what I can run now;
  • what I can only download;
  • what extra components are silently required for this runtime;
  • whether a repo is a full model, a required component, or an optional overlay.

Without a clearer graph, the registry keeps drifting toward an inventory dump instead of a product surface for discovery, download, and execution.

Decision boundaries

1. Required components are not overlays

VAE, text encoders, vision encoders, and similar side artifacts that are mandatory for a runtime path belong in a required-component category. They should not be modeled as adapters.

2. Optional overlays are not standalone model families

LoRA, lightning/distilled transformer overrides, redux-style priors, and similar attachments should not be promoted as first-class runnable models unless a shipped backend can really execute them as such.

3. Package-owned local composition is allowed when the package can keep it honest

The existing sdcpp bundle resolver proves that AbstractVision can own required-component resolution for curated flows. That pattern should be extended deliberately, not hidden.

4. Backend-owned overlay semantics must stay backend-owned

This task must not invent a fake generic cross-backend adapter runtime if Diffusers and MFLUX need different semantics. A shared package surface is acceptable only if it preserves backend truth.

4a. Backend-owned installed-adapter compatibility is allowed

The package may expose a typed adapter inventory surface, but exact compatibility classification must stay backend-owned. For MLX-Gen, that means using cached model-card metadata, file layout knowledge, and route truth from mlxgen capabilities, not duplicating adapter compatibility tables in a static AbstractVision asset unless a dynamic path proves insufficient.

4b. Multi-output is orchestration, not a new single-request contract

Generating N images or N videos is a useful public feature, but it should not distort the core single-request generation dataclasses. Batch generation belongs above exact single-run requests and can reuse the same backend truth with repeated seeds.

5. Tool exposure is conditional, not automatic

AbstractCore tool surfaces should expose overlay-aware inputs only after the package has a clean, typed, package-owned contract for them. The current fixed-parameter tools are safer than exposing aspirational generic adapter knobs.

Requirements

  • Keep one clear parent entry per real model family.
  • Represent engine/format-specific runnable downloads as runtime variants, not as independent model families.
  • Represent required side artifacts explicitly and separately from optional overlays.
  • Expose installed adapter inventory for backends that can classify cached adapters honestly enough for local discovery.
  • Preserve route-specific adapter facts such as target roles and validation status.
  • Preserve a simple public way to request multiple outputs without pretending every backend has a native batched generation contract.
  • Keep TI2V-5B route constraints honest in both normalization and proof artifacts.
  • Preserve direct HF repo metadata where useful, but stop presenting overlays as if they were standalone runnable models.
  • Surface runtime state clearly:
  • runnable now
  • curated local flow with package-owned required components
  • download-only
  • remote-only
  • overlay-only
  • requires unsupported backend work
  • Keep curated local flows seamless where the package already owns the mapping.
  • Expose overlay-aware generation only where a shipped backend already supports it, and reject it clearly elsewhere.
  • Keep offline usage after download.
  • Preserve backward compatibility where practical, or add an explicit compatibility layer if the registry schema evolves.

Suggested implementation

Phase 1: make the taxonomy explicit

  • Add an additive artifact-role model that can distinguish:
  • parent model family
  • runtime variant
  • required component
  • optional overlay
  • Prefer an additive schema evolution or compatibility translation layer over a risky big-bang rewrite.
  • Document source/provenance per artifact role:
  • official
  • runtime-native community
  • fallback community

Phase 2: align curated local flows with the taxonomy

  • Move current special-case bundle knowledge toward package data where practical.
  • Treat sdcpp FLUX/Qwen bundle resolution as the first concrete example of package-owned required component assembly.
  • Keep errors actionable when required cached components are missing.

Phase 3: expose overlays only where runtime support already exists

  • Diffusers:
  • formalize the currently documented LoRA and Rapid-AIO request-extra paths
  • decide whether to keep them as explicit extra conventions or introduce a typed package-owned overlay field
  • MFLUX:
  • keep request-level typed LoRA adapters as the primary path;
  • add installed-adapter discovery that resolves cached LoRAs into model/task-aware inventory rows using backend-owned compatibility logic;
  • surface exact route facts such as supports_lora, lora_status, lora_target_roles, and lora_validation_profile alongside installed adapters.
  • Do not promise cross-backend overlay parity if the runtimes are materially different.

Phase 4: evaluate higher-level surfaces

  • Add a package-owned adapter inventory DTO and manager/backend hooks such as list_provider_adapters(...).
  • Add a package-owned batch orchestration surface above single exact requests, for example:
  • generate_image_batch(...)
  • edit_image_batch(...)
  • generate_video_batch(...)
  • image_to_video_batch(...)
  • Keep seed planning explicit and reproducible. Batch mode should accept either explicit seed lists or a count plus deterministic/random expansion policy.
  • Update CLI and catalog output so overlays are not mistaken for runnable models.
  • Add CLI discovery for locally installed adapters and simple batch generation entry points.
  • Only extend integrations/abstractcore.py after the package-owned overlay surface is explicit enough to expose safely.
  • Add early rejection paths when a user selects an overlay without a compatible parent/runtime.

Phase 5: codify Wan TI2V route policy

  • Add typed flow_shift request fields for video routes that support it.
  • Keep model-default flow_shift backend-owned, but document and normalize the current practical TI2V-5B 480p-class policy (832x480 / 480x832 with flow_shift=3) without hiding the native 720p-class defaults.
  • Replace stale undersized TI2V proof runs with real validation assets.

Scope

Included:

  • capability/download taxonomy work for model families versus variants versus required components versus overlays;
  • downloader and catalog alignment with that taxonomy;
  • curated local-flow improvement where the package can already own required-component resolution;
  • overlay-aware generation surface design for backends that already support overlays;
  • installed-adapter discovery for MLX-Gen when cached metadata is sufficient;
  • typed flow_shift surfacing for Wan routes;
  • multi-output orchestration above single exact generation calls;
  • clear rejection behavior for unsupported overlay/runtime combinations.

Non-goals

  • Implement every overlay or adapter family immediately.
  • Treat VAE/encoders as user-managed details in curated flows that the package can own.
  • Build a generic “adapter engine” abstraction that hides runtime-specific semantics.
  • Expose overlay parameters through AbstractCore tools before the package-facing contract is clean.
  • Reclassify every historical repo in one risky pass without a compatibility strategy.
  • Ship local video runtime work as part of this item.
  • Invent a static adapter-capabilities asset first if dynamic backend-owned discovery is already sufficient.

Expected outcomes

  • The curated registry no longer confuses parent model families, required components, and overlays.
  • Curated local component-based flows such as sdcpp are modeled as package-owned required component assembly rather than as hidden repo trivia.
  • model-catalog and download-model can tell a user whether an artifact is runnable, download- only, a required component, or an optional overlay.
  • Overlay-aware generation is either clearly supported end to end for a backend/model path or clearly rejected with an actionable error.
  • Locally cached LoRAs can be discovered and filtered by compatible model/task route through a package-owned adapter inventory surface without pretending overlays are standalone models.
  • TI2V proof assets and docs no longer use undersized smoke-check runs as visual validation.
  • Users can request multiple outputs through a simple public surface while the core single-request generation contract stays stable.
  • AbstractCore integration does not advertise overlay composition beyond what the package can really honor.

Validation

  • PYTHONPATH=src python - <<'PY' from abstractvision.model_capabilities import VisionModelCapabilitiesRegistry VisionModelCapabilitiesRegistry() print("ok") PY
  • PYTHONPATH=src python -m abstractvision.cli model-catalog --json
  • PYTHONPATH=src python -m abstractvision.cli model-presets --all-targets --all --json
  • PYTHONPATH=src python -m abstractvision.cli show-model <model_id>
  • Add tests for:
  • additive artifact-role validation
  • catalog/download rendering for parent models versus components versus overlays
  • curated required-component resolution for at least one component-based local flow
  • overlay-aware request acceptance and rejection for Diffusers and MFLUX where applicable
  • AbstractCore tool exposure staying aligned with the supported overlay contract
  • Manual smoke checks:
  • one Diffusers LoRA path
  • one curated sdcpp component bundle path
  • one MFLUX overlay path only if request-level overlay support is actually added

Progress checklist

  • [ ] Define the minimal additive taxonomy for parent families, runtime variants, required components, and overlays.
  • [ ] Decide whether the schema evolves directly or through a compatibility translation layer.
  • [ ] Move at least one existing component-based curated flow onto the clearer taxonomy.
  • [ ] Update catalog/download output so overlays are not shown as standalone runnable models.
  • [x] Decide the package-owned overlay input contract for backends that already support overlays.
  • [ ] Add tests and docs for runnable versus download-only versus component versus overlay states.

Guidance for the implementing agent

Re-check the current code before implementing anything. Favor a small explicit taxonomy over a clever graph rewrite. Keep required components and optional overlays separate. Do not preserve adapter entries as fake standalone models just because the old JSON format was permissive. Treat seamless curated local flows as a UX feature the package should own when it has enough knowledge to do so honestly.