Client Configuration

This document describes the configuration options for the Spiral client.

Configuration can be set via:

File: ~/.spiral.toml
Environment variables: Prefixed with SPIRAL__ (double underscore)
Runtime overrides: Override values when initializing the client, using the dot notation

Caching

Spiral supports three independent caches for different file types:

keys_cache: Caches key space files. Enabled memory-only by default. Recommended for workloads with repeated key lookups.
manifests_cache: Caches manifest files. Recommended to enable for better query planning performance.
fragments_cache: Caches fragment (data) files. Not recommended to enable in most cases as fragments are typically very large and benefit less from caching.

Each cache can be independently configured with separate memory limits, disk capacity, and storage paths. Set disk_capacity_bytes to 0 for memory-only caching.

Top level

Main configuration for the Spiral client, that can affect many different parts of the client.

Field	Default	Environment Variable	Description
`dev`	`false`	`SPIRAL__DEV`	Development mode flag
`file_format`	`vortex`	`SPIRAL__FILE_FORMAT`	File format for table storage

Authn

Authentication related settings

Field	Default	Environment Variable	Description
`authn.device_code`	`false`	`SPIRAL__AUTHN__DEVICE_CODE`	Whether to force device code authentication flow. This exist as an override when e.g. ssh-ing into an environment machine.
`authn.token`	`-`	`SPIRAL__AUTHN__TOKEN`	Optional authentication token.

Fragments Cache

Client cache limits and constraints

Field	Default	Environment Variable	Description
`fragments_cache.blob_index_size`	`4 * (1 << 10)`	`SPIRAL__FRAGMENTS_CACHE__BLOB_INDEX_SIZE`	Size of the blob index for each blob on disk. A larger index can hold more entries per blob but increases per-blob I/O.
`fragments_cache.block_size`	`16 * (1 << 20)`	`SPIRAL__FRAGMENTS_CACHE__BLOCK_SIZE`	Block size for the disk cache engine. This is the minimum eviction unit and also limits the maximum cacheable entry size.
`fragments_cache.buffer_pool_size`	`16 * (1 << 20)`	`SPIRAL__FRAGMENTS_CACHE__BUFFER_POOL_SIZE`	Total flush buffer pool size in bytes (RAM). Each flusher gets `buffer_pool_size / flushers` bytes. Entries larger than this are dropped.
`fragments_cache.disk_capacity_bytes`	`20 * (1 << 30)`	`SPIRAL__FRAGMENTS_CACHE__DISK_CAPACITY_BYTES`	Disk cache size in bytes.
`fragments_cache.disk_path`	`spiral`	`SPIRAL__FRAGMENTS_CACHE__DISK_PATH`	Disk cache path override. If not set, defaults to $XDG_CACHE_HOME/spiral or ~/.cache/spiral
`fragments_cache.enabled`	`false`	`SPIRAL__FRAGMENTS_CACHE__ENABLED`	Whether this cache is enabled. Default varies per cache type (see ClientSettings).
`fragments_cache.memory_capacity_bytes`	`2 * (1 << 30)`	`SPIRAL__FRAGMENTS_CACHE__MEMORY_CAPACITY_BYTES`	Memory cache size in bytes.
`fragments_cache.submit_queue_size_threshold`	`16 * (1 << 20)`	`SPIRAL__FRAGMENTS_CACHE__SUBMIT_QUEUE_SIZE_THRESHOLD`	If the total estimated size in the submit queue exceeds this threshold, further entries are silently dropped.
`fragments_cache.write_on_insertion`	`-`	`SPIRAL__FRAGMENTS_CACHE__WRITE_ON_INSERTION`	When true, entries are written to disk immediately on insertion. When false, entries are only written to disk when evicted from memory.

Keys Cache

Client cache limits and constraints

Field	Default	Environment Variable	Description
`keys_cache.blob_index_size`	`4 * (1 << 10)`	`SPIRAL__KEYS_CACHE__BLOB_INDEX_SIZE`	Size of the blob index for each blob on disk. A larger index can hold more entries per blob but increases per-blob I/O.
`keys_cache.block_size`	`16 * (1 << 20)`	`SPIRAL__KEYS_CACHE__BLOCK_SIZE`	Block size for the disk cache engine. This is the minimum eviction unit and also limits the maximum cacheable entry size.
`keys_cache.buffer_pool_size`	`16 * (1 << 20)`	`SPIRAL__KEYS_CACHE__BUFFER_POOL_SIZE`	Total flush buffer pool size in bytes (RAM). Each flusher gets `buffer_pool_size / flushers` bytes. Entries larger than this are dropped.
`keys_cache.disk_capacity_bytes`	`0`	`SPIRAL__KEYS_CACHE__DISK_CAPACITY_BYTES`	Disk cache size in bytes.
`keys_cache.disk_path`	`spiral`	`SPIRAL__KEYS_CACHE__DISK_PATH`	Disk cache path override. If not set, defaults to $XDG_CACHE_HOME/spiral or ~/.cache/spiral
`keys_cache.enabled`	`true`	`SPIRAL__KEYS_CACHE__ENABLED`	Whether this cache is enabled. Default varies per cache type (see ClientSettings).
`keys_cache.memory_capacity_bytes`	`1 << 30`	`SPIRAL__KEYS_CACHE__MEMORY_CAPACITY_BYTES`	Memory cache size in bytes.
`keys_cache.submit_queue_size_threshold`	`16 * (1 << 20)`	`SPIRAL__KEYS_CACHE__SUBMIT_QUEUE_SIZE_THRESHOLD`	If the total estimated size in the submit queue exceeds this threshold, further entries are silently dropped.
`keys_cache.write_on_insertion`	`-`	`SPIRAL__KEYS_CACHE__WRITE_ON_INSERTION`	When true, entries are written to disk immediately on insertion. When false, entries are only written to disk when evicted from memory.

Limits

Client resource limits and constraints

Field	Default	Environment Variable	Description
`limits.batch_readahead`	`NonZeroUsize :: new (min (num_cpus :: get ()`	`SPIRAL__LIMITS__BATCH_READAHEAD`	Maximum number of concurrent shards to evaluate, “read ahead”, when scanning. Defaults to min(number of CPU cores, 32).
`limits.compaction_size_based_threshold_bytes`	`8 * 1024 * 1024`	`SPIRAL__LIMITS__COMPACTION_SIZE_BASED_THRESHOLD_BYTES`	Fragments smaller than this threshold will be considered for compaction when size-based compaction strategy is used.
`limits.compaction_split_compressed_max_bytes`	`1024 * 1024 * 1024`	`SPIRAL__LIMITS__COMPACTION_SPLIT_COMPRESSED_MAX_BYTES`	Maximum total compressed size of fragments accumulated in a single compaction split.
`limits.compaction_task_concurrency`	`1`	`SPIRAL__LIMITS__COMPACTION_TASK_CONCURRENCY`	Maximum number of concurrent tasks in compaction.
`limits.http_max_retries`	`10`	`SPIRAL__LIMITS__HTTP_MAX_RETRIES`	Maximum number of HTTP request retries on transient errors.
`limits.http_max_retry_interval_ms`	`5 * 60 * 1000`	`SPIRAL__LIMITS__HTTP_MAX_RETRY_INTERVAL_MS`	Maximum number of milliseconds to wait between HTTP request retries.
`limits.http_min_retry_interval_ms`	`100`	`SPIRAL__LIMITS__HTTP_MIN_RETRY_INTERVAL_MS`	Minimum number of milliseconds to wait between HTTP request retries.
`limits.io_merge_threshold`	`1024 * 1024`	`SPIRAL__LIMITS__IO_MERGE_THRESHOLD`	Byte range merge distance threshold (in bytes) for IO coalescing. Nearby byte ranges within this distance are merged into single HTTP requests.
`limits.join_set_concurrency`	`num_cpus :: get ()`	`SPIRAL__LIMITS__JOIN_SET_CONCURRENCY`	Maximum number of concurrent tasks in a JoinSet. Defaults to the number of CPU cores.
`limits.key_space_max_rows`	`100_000_000`	`SPIRAL__LIMITS__KEY_SPACE_MAX_ROWS`	Maximum number of rows in a key space. Compaction keeps fragments across column groups aligned to L1 key spaces. When this limit is larger, the L1 key spaces might change more frequently, causing a re-alignment to happen. When this limit is smaller, more fragments may have to be split due to key space boundaries.
`limits.manifest_read_concurrency`	`2 * num_cpus :: get ()`	`SPIRAL__LIMITS__MANIFEST_READ_CONCURRENCY`	Maximum number of concurrent manifest file reads.
`limits.manifest_write_concurrency`	`2 * num_cpus :: get ()`	`SPIRAL__LIMITS__MANIFEST_WRITE_CONCURRENCY`	Maximum number of concurrent manifest file writes.
`limits.object_storage_client_pool_idle_timeout_s`	`None`	`SPIRAL__LIMITS__OBJECT_STORAGE_CLIENT_POOL_IDLE_TIMEOUT_S`	`pool_idle_timeout` for object storage clients.
`limits.object_storage_client_pool_max_idle_per_host`	`None`	`SPIRAL__LIMITS__OBJECT_STORAGE_CLIENT_POOL_MAX_IDLE_PER_HOST`	`pool_max_idle_per_host` for object storage clients.
`limits.prefetch_buffer_size`	`20`	`SPIRAL__LIMITS__PREFETCH_BUFFER_SIZE`	Per-stream channel buffer size (number of KeyTables buffered ahead) Zero disables prefetching entirely.
`limits.read_max_iops_per_file`	`-`	`SPIRAL__LIMITS__READ_MAX_IOPS_PER_FILE`	Maximum number of concurrent HTTP requests when reading a single file. This is an upper bound on the number of concurrent I/O requests to a single VortexReadAt instance. Usually only one of these exists per Vortex file, so this is a rough upper limit on the number of concurrent requsts to each fragment or manifest file. See also: [read_thread_per_core_per_file].
`limits.read_threads_per_core_per_file`	`1`	`SPIRAL__LIMITS__READ_THREADS_PER_CORE_PER_FILE`	The number of read threads per core per file. See also: [read_max_iops_per_file], which is a limit shared across all cores and threads reading from the same file object.
`limits.scan_eval_concurrency`	`16`	`SPIRAL__LIMITS__SCAN_EVAL_CONCURRENCY`	Maximum number of concurrent scan node evaluations (streams preparation) during scans.
`limits.scan_min_rows_per_chunk`	`1000`	`SPIRAL__LIMITS__SCAN_MIN_ROWS_PER_CHUNK`	Minimum number of rows to buffer when executing the scan Small chunks are concatenated together until this threshold is reached.
`limits.spfs_hedge_delay_ms`	`200`	`SPIRAL__LIMITS__SPFS_HEDGE_DELAY_MS`	Delay in milliseconds before issuing a hedged duplicate read request to SPFS to reduce tail latency. When non-zero, a second request is sent after this delay and the first response to arrive is used. Set to `0` to disable hedging entirely.
`limits.spfs_max_retries`	`8`	`SPIRAL__LIMITS__SPFS_MAX_RETRIES`	Maximum number of SPFS operation retries on transient errors. A single SPFS read operation may comprise tens or hundreds of distinct HTTP requests. SPFS write operations may comprise multiple HTTP requests but are typically just one (long) HTTP request.
`limits.transaction_compact_threshold`	`100`	`SPIRAL__LIMITS__TRANSACTION_COMPACT_THRESHOLD`	Automatically compact transaction operations before commit if operation count exceeds this threshold. Set to 0 to disable auto-compaction.
`limits.transaction_manifest_max_rows`	`-`	`SPIRAL__LIMITS__TRANSACTION_MANIFEST_MAX_ROWS`	Maximum number of rows in manifest when compacting transaction.
`limits.transaction_retries`	`3`	`SPIRAL__LIMITS__TRANSACTION_RETRIES`	Maximum number of transaction retries on conflict before giving up.
`limits.write_buffer_max_bytes`	`1024 * 1024 * 1024`	`SPIRAL__LIMITS__WRITE_BUFFER_MAX_BYTES`	Maximum number of bytes to buffer in memory when writing key table batches.
`limits.write_partition_max_bytes`	`128 * 1024 * 1024`	`SPIRAL__LIMITS__WRITE_PARTITION_MAX_BYTES`	Maximum number of bytes in an uncompressed fragment file.

Manifests Cache

Client cache limits and constraints

Field	Default	Environment Variable	Description
`manifests_cache.blob_index_size`	`4 * (1 << 10)`	`SPIRAL__MANIFESTS_CACHE__BLOB_INDEX_SIZE`	Size of the blob index for each blob on disk. A larger index can hold more entries per blob but increases per-blob I/O.
`manifests_cache.block_size`	`16 * (1 << 20)`	`SPIRAL__MANIFESTS_CACHE__BLOCK_SIZE`	Block size for the disk cache engine. This is the minimum eviction unit and also limits the maximum cacheable entry size.
`manifests_cache.buffer_pool_size`	`16 * (1 << 20)`	`SPIRAL__MANIFESTS_CACHE__BUFFER_POOL_SIZE`	Total flush buffer pool size in bytes (RAM). Each flusher gets `buffer_pool_size / flushers` bytes. Entries larger than this are dropped.
`manifests_cache.disk_capacity_bytes`	`20 * (1 << 30)`	`SPIRAL__MANIFESTS_CACHE__DISK_CAPACITY_BYTES`	Disk cache size in bytes.
`manifests_cache.disk_path`	`spiral`	`SPIRAL__MANIFESTS_CACHE__DISK_PATH`	Disk cache path override. If not set, defaults to $XDG_CACHE_HOME/spiral or ~/.cache/spiral
`manifests_cache.enabled`	`false`	`SPIRAL__MANIFESTS_CACHE__ENABLED`	Whether this cache is enabled. Default varies per cache type (see ClientSettings).
`manifests_cache.memory_capacity_bytes`	`2 * (1 << 30)`	`SPIRAL__MANIFESTS_CACHE__MEMORY_CAPACITY_BYTES`	Memory cache size in bytes.
`manifests_cache.submit_queue_size_threshold`	`16 * (1 << 20)`	`SPIRAL__MANIFESTS_CACHE__SUBMIT_QUEUE_SIZE_THRESHOLD`	If the total estimated size in the submit queue exceeds this threshold, further entries are silently dropped.
`manifests_cache.write_on_insertion`	`-`	`SPIRAL__MANIFESTS_CACHE__WRITE_ON_INSERTION`	When true, entries are written to disk immediately on insertion. When false, entries are only written to disk when evicted from memory.

Server

API endpoint configuration for the Spiral control plane

Field	Default	Environment Variable	Description
`server.url`	`https://api.spiraldb.dev`	`SPIRAL__SERVER__URL`	The SpiralDB API endpoint URL

Spfs

SpFS (Spiral File System) configuration

Field	Default	Environment Variable	Description
`spfs.http1_only`	`false`	`SPIRAL__SPFS__HTTP1_ONLY`	Restrict object-storage HTTP clients to HTTP/1.1.
`spfs.http2_initial_connection_window_size`	`4 * 1024 * 1024`	`SPIRAL__SPFS__HTTP2_INITIAL_CONNECTION_WINDOW_SIZE`	HTTP/2 initial connection-level flow-control window size (bytes) for object storage clients. Only applies when `http1_only` is false.
`spfs.http2_initial_stream_window_size`	`1024 * 1024`	`SPIRAL__SPFS__HTTP2_INITIAL_STREAM_WINDOW_SIZE`	HTTP/2 initial stream-level flow-control window size (bytes) for object storage clients. Only applies when `http1_only` is false.
`spfs.url`	`https://spfs.spiraldb.dev`	`SPIRAL__SPFS__URL`	The SpFS (Spiral File System) endpoint URL

Telemetry

Telemetry configuration for the Spiral client

Field	Default	Environment Variable	Description
`telemetry.enabled`	`true`	`SPIRAL__TELEMETRY__ENABLED`	Whether client-side telemetry export is enabled.