Skip to Content
New EngineExtensionsOverview

Spiral extensions package domain-specific data access behind SQL table functions, scalar functions, typed values, and materialization paths. They are for datasets where metadata is cheap and values are expensive: images, video, audio, documents, tensors, scientific arrays, feature stores, and custom data systems.

The implemented public pieces today are Blob, JPEG/EXIF, tensor/vector, and temporal primitives. The broader modality pages reserve the intended user-facing shape and are marked Experimental until the runtime packages exist.

Current and planned surfaces

Blobs

External byte ranges used by modality functions.

Images

Implemented JPEG handles, JPEG bytes, EXIF summaries, and image previews.

Video

Experimental: video metadata, clip ranges, sampled frames, and decode policies.

Audio

Experimental: audio clips, windows, transcripts, spectrograms, and embeddings.

Documents

Experimental: PDFs, pages, text chunks, OCR, embeddings, and retrieval artifacts.

Tensors

Experimental: tensor files, embeddings, feature arrays, and model-ready batches.

Custom Extensions

The expected shape for third-party or internal extension packages.

Extension model

Extensions are packages that add table functions, scalar functions, schemas, codecs, materializers, and adapters for a specific modality or external system. An extension should answer three user questions:

  1. What tables or table functions does it expose?
  2. What logical values does it produce?
  3. What materialization path turns those values into bytes, tensors, previews, or artifacts?

Current status

The public CLI can already query registered table and source functions:

SELECT *
FROM spql.meta.table_functions();
 
SELECT *
FROM spql.meta.source_functions();

It can also query registered extension dtypes and their stability contract:

SELECT id, stability, summary
FROM spql.meta.extension_types()
ORDER BY id;

The modality extensions documented here are not public runtime packages yet. Video, audio, document, tensor, and custom-extension pages describe intended contracts: which table functions should exist, what rows they should produce, and how users should think about materialization.

Implemented primitives today include:

  • spql.io.Blob, spql.io.blob, and spql.io.* Blob accessors.
  • spql.image.Jpeg, spql.image.JpegBytes, spql.image.Exif, and spql.image.* JPEG/EXIF scalar functions.
  • vortex.tensor.variable_shape_tensor, vortex.tensor.fixed_shape_tensor, vortex.tensor.vector, and spql.tensor.* vector/FixedTensor math functions.
  • spql.viz.ImageView plus descriptor registrations for planned visualization and modality dtypes.
  • spiral-datetime extension dtype factories date, time, and timestamp.

Common conventions

Extension table functions should:

  • return metadata columns before expensive payload columns
  • use stable ids for files, pages, clips, frames, windows, chunks, and tensors
  • emit deferred references for expensive values
  • document whether paths are local paths, object-store URIs, or catalog ids
  • make ordering and sampling explicit
  • avoid decoding bytes unless the user asks for materialization
Last updated on
SQL SupportBlobs