Skip to content

011 abstractcore tool integration and artifact refs

Task 011: AbstractCore tool integration (vision tools returning artifact refs)

Date: 2026-01-23
Status: Completed
Priority: P0

Main goals

  • Provide a first-class integration path so abstractcore users can call generative vision as normal tools:
  • T2I: text_to_image
  • I2I: image_to_image
  • T2V: text_to_video
  • I2V: image_to_video
  • (optional) multi-view image
  • Ensure every tool returns a small JSON artifact reference (not raw bytes) so results can flow through:
  • AbstractCore tool calling
  • AbstractRuntime durable runs (vars remain JSON-serializable)
  • AbstractFlow workflows
  • AbstractCode UI / third-party UIs

Secondary goals

  • Keep the integration optional: import abstractvision must not require abstractcore.
  • Provide a minimal “tool factory” that third parties can reuse (not tied to AbstractCode).

Context / problem

Generative media outputs are large. For a good end-user and third-party experience, we want a “golden path” where: - an agent/workflow can request an image/video, - the output is stored durably, - downstream components receive a small ref object they can display/open/forward.

The AbstractFramework already uses artifact references ({"$artifact": "..."}) for large payloads (AbstractRuntime). AbstractVision should integrate with that model directly.

Constraints

  • No hard dependency on abstractcore in the base install.
  • Tool outputs must be JSON-serializable and stable (avoid backend-native objects).
  • No implicit model downloads in tools; any heavy dependencies remain behind explicit extras.

Research, options, and references

  • Option A: Expose only a Python API (VisionManager) and let each app wrap it into tools.
  • Pros: minimal surface.
  • Cons: every integrator reinvents tool schemas and output shapes; inconsistent third-party UX.
  • Option B (chosen): Ship an official abstractvision.integrations.abstractcore module that provides tool wrappers.
  • Pros: one canonical tool schema + one canonical artifact output shape; simplest adoption.

References: - AbstractRuntime artifact refs: abstractruntime/src/abstractruntime/storage/artifacts.py - AbstractRuntime tool execution: abstractruntime/src/abstractruntime/integrations/abstractcore/tool_executor.py

Decision

Chosen approach: - Add abstractvision.integrations.abstractcore that exposes a factory such as: - make_vision_tools(*, vision_manager, store, model_id, registry=None) -> list[callable] - Each tool: - validates the requested task against the capabilities registry (best-effort) - calls the configured backend via VisionManager - stores the output via the store (local store or runtime artifact store adapter) - returns a GeneratedMediaRef containing at least "$artifact" and content_type

Why: - This becomes the clean, intuitive, “one import” integration point for end users. - Tool outputs are portable across apps and UIs because they use the framework artifact ref contract.

Dependencies

  • Backlog tasks:
  • Completed: docs/backlog/completed/005_core_api_tasks_and_abstractions.md
  • Completed: docs/backlog/completed/008_asset_outputs_and_third_party_integration.md
  • Completed: docs/backlog/completed/004_capability_schema_and_validation.md (prefer early validation errors)

Implementation plan

  • Implement the integration module with minimal surface area:
  • tool names, docstrings, and JSON schema designed for third parties
  • tools created as closures capturing (vision_manager, store, model_id, registry)
  • Define a stable return shape (documented in Task 008):
  • always include "$artifact"
  • include content_type, sha256, filename when available
  • Provide example wiring in docs (Task 010):
  • register tools in an AbstractCore agent/session and call them
  • Add unit tests with:
  • a fake backend that returns deterministic bytes
  • a temp local store
  • assertions that tool outputs are small JSON and include $artifact

Success criteria

  • A third party can register the tools and receive artifact refs without writing glue code.
  • Tool outputs are compatible with AbstractRuntime artifact references and safe to persist in run state.

Test plan

  • python -m unittest discover -s tests -p "test_*.py" -q

Report (fill only when completed)

Summary

  • Added an optional AbstractCore integration module:
  • src/abstractvision/integrations/abstractcore.py
  • Implemented make_vision_tools(...) which returns @abstractcore.tool-decorated callables:
  • vision_text_to_image
  • vision_image_to_image
  • vision_multi_view_image
  • vision_text_to_video
  • vision_image_to_video
  • All tools:
  • validate support via VisionModelCapabilitiesRegistry.require_support(model_id, task)
  • require VisionManager.store so outputs are always returned as artifact refs ({"$artifact": ...})
  • accept image inputs either as artifact refs (image_artifact) or base64 (image_b64) for compatibility.

Tool argument conventions (v0): - Image inputs: - preferred: image_artifact={"$artifact": "..."} (no base64 in tool calls) - fallback: image_b64="..." (raw base64 or data URL) - Mask inputs: - mask_artifact / mask_b64 (optional) - Multi-view reference image: - reference_image_artifact / reference_image_b64 (optional)

Unit tests: - tests/test_abstractcore_tool_integration.py

Validation

  • Tests:
  • python -m unittest discover -s tests -p "test_*.py" -q