AI/ML datasets usually start as local files, manifests, compressed columns, or small scratch tables used to validate a pipeline. Spiral currently supports local Vortex files, Arrow IPC files, Parquet files, and in-memory tables.
Common paths
File table functions
Query Vortex, Arrow IPC, Parquet, and Hugging Face dataset handles through registered table functions.
Inspect a Vortex file
Open a local Vortex file, check its schema, and query a bounded sample.
Local tables
Create tiny REPL tables and insert literal rows for experiments and smoke tests.
Keys
Understand relation keys, uniqueness, and ordered execution assumptions.
Current file support
| Source | Current access path |
|---|---|
| Vortex file | .open / .load, spql.vortex.read(path) |
| Arrow IPC file | spql.arrow.read_ipc(path) |
| Parquet file | spql.parquet.read(path) |
| Hugging Face dataset handle | hf.dataset(...) metadata handle |
| Small literal data | CREATE TABLE, INSERT, and VALUES |
Planned modality dataset functions live under Extensions and remain marked Experimental until implemented.
Last updated on