Tensor Extension - Spiral Docs

Experimental: Contact Us

This surface is directional and may change before public release. Contact Spiral if you want to use it in an early workflow.

The Tensor Extension is intended for feature arrays, embedding stores, tensor files, and model-ready batch output. It should make tensor metadata queryable before large arrays are copied or moved to a device.

Current SPQL surface

The default CLI session registers spiral-tensor as a thin wrapper over upstream Vortex tensor support. The current implementation exposes extension dtype factories, scalar math functions, and Python PyTorch export:

Surface	Purpose
`vector(dim)`	Construct a `vortex.tensor.vector` dtype with `f32` elements.
`vector(element, dim)`	Construct a `vortex.tensor.vector` dtype with explicit float element type.
`tensor(shape)`	Construct a `vortex.tensor.fixed_shape_tensor` dtype with `f32` elements.
`tensor(element, shape[, dim_names[, permutation]])`	Construct a `vortex.tensor.fixed_shape_tensor` dtype with explicit metadata.
`spql.tensor.l2_norm(value)`	Compute row-wise Euclidean norms for vectors or fixed-shape tensors.
`spql.tensor.l2_denorm(normalized, norms)`	Re-apply row-wise L2 norms to normalized tensor values.
`spql.tensor.inner_product(lhs, rhs)`	Compute row-wise inner products.
`spql.tensor.cosine_similarity(lhs, rhs)`	Compute row-wise cosine similarity.
`Relation.to_pytorch(...)`	Stream relation output into PyTorch batch dictionaries from Python.

Python PyTorch export

The Python API exposes a bounded streaming terminal:


stream = relation.to_pytorch(device="cpu", transfer="copy", prefetch=2)
for batch in stream:
    # Numeric and tensor-shaped fields become torch.Tensor values.
    # Non-numeric metadata fields remain Python lists.
    ...

prefetch is the number of relation batches the engine may buffer at the terminal boundary. A slow Python consumer back-pressures query execution. It does not set row batch size; the current stream follows the engine’s relation-array boundaries.

The current implementation requests physical buffer placement for tensor extension fields and then converts through CPU Arrow/Python values. Use transfer="copy"; transfer="zero_copy" is rejected until DLPack export is implemented. device="cpu" uses the host-backed path; PyTorch export accepts CUDA and MPS/Metal device names as accelerator targets. The physical model also represents WebGPU for future non-PyTorch sinks. Accelerator placement returns an explicit unsupported error until native device buffers and copy kernels are implemented.

Planned table functions

Function	Purpose
`tensors.scan(path)`	Read tensor files or embedding stores as rows.
`tensors.schema(path)`	Inspect tensor names, shapes, dtypes, and chunks.
`tensors.slice(tensor_ref, ...)`	Produce deferred tensor slices.
`tensors.batch(tensor_ref, ...)`	Materialize model-ready batches.

These names are preview syntax. The table functions are not registered in the current default CLI session.

Output shape

Tensor scans should produce rows like:

Column	Meaning
`tensor_id`	Stable tensor or embedding id.
`path`	Local path or object URI.
`dtype`	Tensor element type.
`shape`	Logical tensor shape.
`chunk_shape`	Storage or read chunking when available.
`tensor_ref`	Deferred tensor reference.

Example shape


SELECT item_id, embedding
FROM tensors.scan('./embeddings.vortex')
LIMIT 1024;

Join tensor metadata with ordinary relational metadata:


SELECT item_id, tensor_ref
FROM tensors.scan('./embeddings.vortex')
WHERE dtype = 'float32'
LIMIT 1024;

Notes

Device placement, framework adapters, and batching options are explicit. Native GPU buffers and DLPack export remain planned.
Tensor references should remain joinable to ordinary metadata.
Materialization should make shape, dtype, and memory layout visible to users.