Skip to content

003 hf model landscape and capability registry

Task 003: HF model landscape + capability registry (single source of truth)

Date: 2026-01-23
Status: Completed
Priority: P0

Main goals

  • Establish a repeatable process to discover/track HuggingFace generative vision models (image/video) and capture their capabilities in:
  • src/abstractvision/assets/vision_model_capabilities.json
  • Define a small, stable capability schema that can describe what a model can do (tasks + parameters + constraints) without embedding backend-specific code.

Secondary goals

  • Make it easy to add new models later with a clear “how to update capabilities” checklist.
  • Keep licensing information informational (metadata), not enforcement logic.

Context / problem

AbstractVision’s job is to provide a clean abstraction and to describe model capabilities accurately. This requires a single source of truth that is: - human-editable and reviewable (JSON) - machine-consumable (registry + validation) - stable over time (versioned schema)

We also need to support multiple families of models for the same task (T2I/I2I/T2V/I2V/multi-view), because best models change quickly and users choose their own.

Constraints

  • Capabilities are descriptive: we do not enforce whether a user should use a model.
  • Schema stability: evolve schema via schema_version and additive changes.
  • No hardcoding: capability logic must come from JSON, not scattered in code.

Research, options, and references

Model references (seed set provided by user): - T2I: https://huggingface.co/Qwen/Qwen-Image-2512 - I2I: https://huggingface.co/Qwen/Qwen-Image-Edit-2511 - Multi-view (LoRA): https://huggingface.co/fal/Qwen-Image-Edit-2511-Multiple-Angles-LoRA - T2V: https://huggingface.co/Wan-AI/Wan2.2-T2V-A14B - T2V: https://huggingface.co/tencent/HunyuanVideo-1.5 - T2V: https://huggingface.co/genmo/mochi-1-preview - T2I/I2I candidate: https://huggingface.co/zai-org/GLM-Image - T2I turbo candidate: https://huggingface.co/Tongyi-MAI/Z-Image-Turbo - I2V: https://huggingface.co/Lightricks/LTX-2

Decision

Chosen approach: - Maintain a versioned JSON file that enumerates: - model ids - task support (t2i, i2i, multi_view_image, t2v, i2v) - parameters and whether they are required/optional - requirements (e.g., “LoRA requires base model id”) - notes/constraints as structured metadata where possible

Why: - Avoids coupling the public API to any one model or backend. - Enables deterministic capability checks for routing and user UX.

Dependencies

  • Backlog tasks:
  • Planned: docs/backlog/planned/004_capability_schema_and_validation.md

Implementation plan

  • Define the capability schema rules in a short doc section (inside this task or an ADR later).
  • Ensure vision_model_capabilities.json includes:
  • All user-provided models
  • Clear supported tasks per model
  • Parameter surface for each task
  • Add/extend a code validator that checks:
  • schema shape
  • all tasks referenced exist in tasks
  • required fields present
  • referenced base_model_id exists in the same file

Success criteria

  • vision_model_capabilities.json is complete for the seed set.
  • A registry can answer:
  • “does model X support task Y?”
  • “which models support task Y?”

Test plan

  • Add automated tests that assert:
  • every seed model id exists in the JSON
  • expected core tasks are present and mapped correctly

Report (fill only when completed)

Summary

  • Implemented the capability “single source of truth” at src/abstractvision/assets/vision_model_capabilities.json (seed models + tasks).
  • Added a small query registry (VisionModelCapabilitiesRegistry) at src/abstractvision/model_capabilities.py.
  • Added basic unit tests to prevent drift at tests/test_vision_model_capabilities.py.

Validation

  • Tests: python -m unittest discover -s tests -p "test_*.py" -q