Skip to content

017 mlx mflux backend strategy

Task 017: Native MLX engine beyond MFLUX

Date: 2026-05-20 Status: Deprecated
Priority: P1
Deprecated: 2026-05-25

Main goals

  • Create a true first-class MLX image backend for AbstractVision that can run MLX-packaged vision models on Apple Silicon beyond the subset supported by MFLUX today.
  • Treat MFLUX as the short-term compatibility bridge, not the long-term Apple runtime abstraction.
  • Make MLX a real engine choice in AbstractVision, CLI, playground, and AbstractCore instead of a catalog/download target that is mostly routed through MFLUX-specific rules.

Secondary goals

  • Keep base abstractvision import-light and local runtimes behind optional extras.
  • Preserve the VisionManager, artifact output, and AbstractCore capability-plugin contracts.
  • Keep current MFLUX users working while the native MLX path is introduced incrementally.

Context / problem

The product goal is broader than “make current MFLUX presets work.” AbstractVision is supposed to help users discover, download, and run compatible vision models across machines and runtimes with as little manual assembly as possible.

For Apple Silicon, the repo currently has many MLX-flavored download variants and an implemented MFLUX backend, but it does not have a generic/native MLX backend. That means:

  • MLX is not a true engine in the product today.
  • Apple-local MLX support is limited to what the MFLUX runtime happens to support.
  • Some curated MLX/MFLUX download paths are broader than what the MFLUX backend can actually run.

The correct long-term shape is:

  • mlx becomes a first-class runtime/backend category in AbstractVision;
  • mflux remains an adapter/bridge for the subset of models where it is the practical runtime;
  • model discovery and download surfaces distinguish:
  • MLX download available;
  • runnable through MFLUX today;
  • runnable through native MLX engine;
  • download-only pending runtime support.

This task exists because Apple-local MLX support should be a real platform capability, not just a set of MFLUX-specific exceptions.

Current code reality

Files and symbols inspected before rewriting this item:

  • src/abstractvision/backends/mflux.py
  • src/abstractvision/model_downloads.py
  • src/abstractvision/playground_server.py
  • src/abstractvision/integrations/abstractcore_plugin.py
  • src/abstractvision/assets/vision_model_capabilities.json
  • docs/reference/backends.md
  • README.md

What exists today:

  • A shipped optional MFluxVisionBackend in src/abstractvision/backends/mflux.py.
  • Apple-oriented curated download variants that use target="mlx" and engine="mflux".
  • AbstractCore routing that explicitly rejects generic mlx/... and tells callers to use mflux/<preset>.
  • Playground and CLI catalog surfaces that already expose MLX-targeted downloads.

What is still wrong or incomplete:

  • Generic MLX is not a real backend or provider today; it is still rejected in the plugin.
  • The playground does not fail fast consistently on mlx/... model ids and can still default that shape to Diffusers routing.
  • The MFLUX backend supports only a small hard-coded family table, while the curated MLX/MFLUX catalog is broader.
  • There are multiple competing truths for Apple-local support:
  • registry/download metadata;
  • hard-coded preset table;
  • runtime family table in the MFLUX backend;
  • route-selection logic in CLI/playground/AbstractCore.

Constraints

  • Do not make MLX, MFLUX, Torch, Diffusers, or stable-diffusion.cpp mandatory base dependencies.
  • Preserve offline generation after models are downloaded.
  • Keep generated outputs as AbstractVision artifacts.
  • Keep local runtime imports lazy and avoid model loads at import time.
  • Prefer permissive licensing for new runtime dependencies.
  • Preserve current MFLUX behavior as a compatibility bridge while the native MLX path is built.
  • Follow ADR 0005 and ADR 0006: curated download variants should be runnable or clearly marked as download-only / backend-not-supported, and runtime selection must stay explicit.

Research, options, and references

  • Current runtime implementation:
  • MFLUX exists today as the only shipped Apple-local MLX-adjacent runtime path.
  • References:
    • src/abstractvision/backends/mflux.py
    • src/abstractvision/backends/__init__.py
  • Current routing reality:
  • CLI, playground, and AbstractCore now fail fast on generic mlx/...; only explicit mflux/... routing is runnable today on Apple-local MLX-adjacent paths.
  • References:
    • src/abstractvision/integrations/abstractcore_plugin.py
    • src/abstractvision/playground_server.py
    • src/abstractvision/cli.py
  • Current catalog/download reality:
  • Apple auto-targeting prefers mlx, and current curated Apple-local downloads are represented through engine="mflux" presets rather than a true native mlx engine.
  • References:
    • src/abstractvision/model_downloads.py
    • src/abstractvision/assets/vision_model_capabilities.json

Option A: Keep MFLUX as the permanent Apple-local answer

Pros:

  • Lowest short-term implementation cost.
  • Already integrated into CLI, playground, and AbstractCore.
  • Keeps current Apple 8-bit flows working where supported.

Cons:

  • MLX remains a fake engine category rather than a real runtime.
  • AbstractVision stays bound to MFLUX-supported model families.
  • Hard-coded family/routing special cases will keep expanding.

Option B: Build a native/general MLX backend and keep MFLUX as the bridge

Pros:

  • Matches the real product goal: MLX as a first-class local engine on Apple Silicon.
  • Decouples AbstractVision’s Apple strategy from one specialized runtime.
  • Allows the catalog to represent “native MLX runnable” separately from “MFLUX runnable”.

Cons:

  • Higher implementation and integration cost.
  • Needs real feasibility proof across at least one non-MFLUX family.
  • Requires a careful transition plan so current MFLUX users do not regress.

Option C: Treat MLX as download-only for now and postpone runtime work

Pros:

  • Avoids more short-term runtime complexity.
  • Keeps backend surface smaller.

Cons:

  • Fails the product goal for Apple-local MLX execution.
  • Leaves users with downloadable artifacts that are not a first-class runtime path.
  • Pushes more unsupported/manual assembly burden onto callers.

Decision

Chosen approach: build a true native/general MLX backend for AbstractVision, while keeping MFLUX as the short-term bridge and compatibility runtime during the transition.

Why:

  • The long-term goal is to handle MLX vision models as a real engine category, not only through MFLUX-specific support tables.
  • MFLUX is good enough as a bridge, but too narrow to be the terminal abstraction for Apple-local model execution.
  • The repo already shows that MLX download variants matter to users; the missing piece is a real engine boundary and runtime support strategy.
  • Truth-alignment work is still required, but only as a prerequisite to shipping a trustworthy MLX engine, not as the end goal of this task.

Dependencies

  • ADRs:
  • docs/adr/0005_curated_capability_registry_and_download_catalog.md
  • docs/adr/0006_operator_control_configuration_precedence_and_explicit_network_use.md
  • Backlog tasks:
  • Planned: docs/backlog/planned/020_adapter_aware_model_graph_and_catalog.md
  • Completed: docs/backlog/completed/007_local_hf_backend_strategy_diffusers.md
  • Completed: docs/backlog/completed/013_stable_diffusion_cpp_gguf_backend.md
  • Completed: docs/backlog/completed/015_vision_install_profiles_and_pending_defaults.md
  • Completed: docs/backlog/completed/019_best_effort_preload_warmup_for_local_backends.md

Implementation plan

  • Define the native MLX backend boundary:
  • provider/backend id;
  • configuration shape;
  • cache discovery expectations;
  • supported local model layout requirements.
  • Keep MFLUX as a separate backend/provider during the transition; do not silently collapse mlx into mflux.
  • Add a first native MLX backend implementation that can load:
  • at least one non-MFLUX family;
  • at least one currently curated Apple-local family from configuration rather than from a narrow hard-coded family switchboard.
  • Tighten current runtime truth while the MLX backend is being built:
  • fail fast on mlx/... everywhere until the native backend exists;
  • stop advertising unsupported MFLUX families as runnable;
  • keep download-only Apple variants visible only with explicit non-runnable state.
  • Add runtime-state metadata that distinguishes:
  • cached/downloadable;
  • runnable through MFLUX;
  • runnable through native MLX;
  • download-only.
  • Update CLI, playground, and AbstractCore routing so mlx becomes a real backend/provider once the native engine exists.
  • Revisit preset and registry generation so Apple-local download truth does not drift across _PRESETS, the registry, and backend family tables.

Success criteria

  • AbstractVision has a true native MLX backend/provider beyond MFLUX.
  • Generic mlx model/provider selection is a real supported path, not just an error or alias.
  • At least one non-MFLUX family is runnable through the native MLX backend.
  • Apple-local catalog surfaces distinguish native-MLX runnable, MFLUX-runnable, and download-only entries clearly.
  • Unsupported MFLUX-only families are not advertised as runnable.
  • Runtime extras remain optional and base abstractvision stays import-light.
  • Docs explain when to choose native MLX, MFLUX, Diffusers, and stable-diffusion.cpp/GGUF.

Test plan

  • PYTHONPATH=src python -m unittest tests.test_model_downloads -q
  • PYTHONPATH=src python -m unittest tests.test_cli_smoke -q
  • PYTHONPATH=src python -m unittest tests.test_playground_server -q
  • PYTHONPATH=src python -m unittest tests.test_abstractcore_plugin -q
  • Add dedicated backend tests for:
  • native MLX backend model loading;
  • unsupported mlx/... routing before engine availability;
  • supported mlx/... routing after engine availability;
  • separation between MFLUX-runnable and native-MLX-runnable catalog entries.
  • Manual Apple Silicon smoke tests for:
  • one MFLUX-backed model;
  • one native-MLX-backed non-MFLUX family;
  • one download-only MLX entry with a clear non-runnable explanation.

Deprecation report

Deprecated: 2026-05-25

This plan is superseded by the completed MLX-Gen runtime migration in ../completed/0024_mlx_gen_runtime_migration.md.

Reason: - The selected Apple-local runtime identity is mlx-gen, not a generic mlx provider. - Generic mlx remains a download/artifact target and is intentionally rejected as a runnable image provider to avoid confusion with AbstractCore's text-generation MLX provider. - The old MFLUX implementation boundary is now a compatibility layer around the mlx-gen package, with legacy mflux aliases preserved for existing users.

Outcome: - Do not implement this task as written. - Any future Apple-local vision work should start from the canonical mlx-gen provider/engine and open a fresh backlog item only if a runtime beyond MLX-Gen becomes real and needs a new package boundary.