Tensor Extension - Spiral Docs

Experimental: Contact Us

This surface is directional and may change before public release. Contact Spiral if you want to use it in an early workflow.

The Tensor Extension is intended for feature arrays, embedding stores, tensor files, and model-ready batch output. It should make tensor metadata queryable before large arrays are copied or moved to a device.

Current SPQL surface

The default CLI session registers spiral-tensor. The current implementation exposes extension dtype factories and scalar math functions:

Surface	Purpose
`vector(dim)`	Construct a `vortex.tensor.vector` dtype with `f32` elements.
`vector(element, dim)`	Construct a `vortex.tensor.vector` dtype with explicit float element type.
`Tensor(ndim)`	Construct a variable-shape `Tensor` dtype, `vortex.tensor.variable_shape_tensor`, with `f32` elements and `ndim` physical dimensions.
`Tensor(element, ndim[, dim_names[, permutation[, uniform_shape]]])`	Construct a variable-shape `Tensor` dtype with explicit metadata.
`FixedTensor(shape)`	Construct a static `FixedTensor` dtype, `vortex.tensor.fixed_shape_tensor`, with `f32` elements.
`FixedTensor(element, shape[, dim_names[, permutation]])`	Construct a static `FixedTensor` dtype with explicit metadata.
`spql.tensor.l2_norm(value)`	Compute row-wise Euclidean norms for vectors or static `FixedTensor` values.
`spql.tensor.l2_denorm(normalized, norms)`	Re-apply row-wise L2 norms to normalized tensor values.
`spql.tensor.inner_product(lhs, rhs)`	Compute row-wise inner products.
`spql.tensor.cosine_similarity(lhs, rhs)`	Compute row-wise cosine similarity.

Tensor is the variable-shape tensor type, with Vortex dtype id vortex.tensor.variable_shape_tensor. Its storage follows the Arrow canonical variable-shape tensor shape: a struct with data as a list of non-null primitive elements and shape as a fixed-size list of non-null int32 dimension sizes. Arrow import and export use arrow.variable_shape_tensor extension metadata for interchange. The uniform_shape argument uses ? or null for dimensions that may vary, for example Tensor(f32, 3, "h:w:c", none, "400x?x3").

The scalar math functions do not accept variable-shape Tensor values yet. They only dispatch to vector and FixedTensor kernels whose per-row element count is uniform.

Planned table functions

Function	Purpose
`tensors.scan(path)`	Read tensor files or embedding stores as rows.
`tensors.schema(path)`	Inspect tensor names, shapes, dtypes, and chunks.
`tensors.slice(tensor_ref, ...)`	Produce deferred tensor slices.
`tensors.batch(tensor_ref, ...)`	Materialize model-ready batches.

These names are preview syntax. The table functions are not registered in the current default CLI session.

Output shape

Tensor scans should produce rows like:

Column	Meaning
`tensor_id`	Stable tensor or embedding id.
`path`	Local path or object URI.
`dtype`	Tensor element type.
`shape`	Logical tensor shape.
`chunk_shape`	Storage or read chunking when available.
`tensor_ref`	Deferred tensor reference.

Example shape


SELECT item_id, embedding
FROM tensors.scan('./embeddings.vortex')
LIMIT 1024;

Join tensor metadata with ordinary relational metadata:


SELECT item_id, tensor_ref
FROM tensors.scan('./embeddings.vortex')
WHERE dtype = 'float32'
LIMIT 1024;

Notes

Device placement, framework adapters, and batching options should be explicit once implemented.
Tensor references should remain joinable to ordinary metadata.
Materialization should make shape, dtype, and memory layout visible to users.