Client Configuration

This document describes the configuration options for the Spiral client.

Configuration can be set via:

File: ~/.spiral.toml
Environment variables: Prefixed with SPIRAL__ (double underscore)
Runtime overrides: Override values when initializing the client, using the dot notation

Caching

Spiral supports three independent caches for different file types:

keys_cache: Caches key space files. Enabled memory-only by default. Recommended for workloads with repeated key lookups.
manifests_cache: Caches manifest files. Recommended to enable for better query planning performance.
fragments_cache: Caches fragment (data) files. Not recommended to enable in most cases as fragments are typically very large and benefit less from caching.

Each cache can be independently configured with separate memory limits, disk capacity, and storage paths. Set disk_capacity_bytes to 0 for memory-only caching.

Top level

Main configuration for the Spiral client, that can affect many different parts of the client.

Field	Default	Environment Variable	Description
`dev`	`false`	`SPIRAL__DEV`	Development mode flag
`file_format`	`vortex`	`SPIRAL__FILE_FORMAT`	File format for table storage

Adaptive Io

Adaptive I/O settings. The adaptive I/O system increase or decreases the number of requests to achieve high throughput. It never modifies the maximum number of in-flight bytes. This should be set to the amount of RAM you are willing to devote to I/O.

Field	Default	Environment Variable	Description
`adaptive_io.cooldown_ms_min`	`3_000`	`SPIRAL__ADAPTIVE_IO__COOLDOWN_MS_MIN`	Minimum cooldown duration between multiplicative decreases in milliseconds.
`adaptive_io.enabled`	`false`	`SPIRAL__ADAPTIVE_IO__ENABLED`	Enable adaptive session-wide read I/O admission control.
`adaptive_io.goodput_drop_percentage`	`0.05`	`SPIRAL__ADAPTIVE_IO__GOODPUT_DROP_PERCENTAGE`	If previous goodput is this percent larger than current goodput, we assume our changes are decreasing throughput.
`adaptive_io.inflight_kib_max`	`2 * 1024 * 1024`	`SPIRAL__ADAPTIVE_IO__INFLIGHT_KIB_MAX`	Fixed global in-flight byte budget for adaptive read I/O admission. This is enforced as a hard semaphore cap across all providers.
`adaptive_io.kib_per_provider_max`	`2 * 1024 * 1024`	`SPIRAL__ADAPTIVE_IO__KIB_PER_PROVIDER_MAX`	The maximum per-provider in-flight kilobyte budget for adaptive read I/O. The adaptive I/O system does not increase or decrease this value, it is simply a hard upper limit on the number of in-flight bytes from a particular provider.
`adaptive_io.request_kib_estimate_default`	`256`	`SPIRAL__ADAPTIVE_IO__REQUEST_KIB_ESTIMATE_DEFAULT`	Default estimated request size used when exact range size is unknown.
`adaptive_io.requests_decrease_multiple`	`0.8`	`SPIRAL__ADAPTIVE_IO__REQUESTS_DECREASE_MULTIPLE`	Multiplicative decrease factor applied on bad windows.
`adaptive_io.requests_increase_step`	`4`	`SPIRAL__ADAPTIVE_IO__REQUESTS_INCREASE_STEP`	Additive increase step applied on good windows.
`adaptive_io.requests_per_provider_initial`	`128`	`SPIRAL__ADAPTIVE_IO__REQUESTS_PER_PROVIDER_INITIAL`	Initial per-provider concurrent in-flight read I/O permits.
`adaptive_io.requests_per_provider_max`	`1024`	`SPIRAL__ADAPTIVE_IO__REQUESTS_PER_PROVIDER_MAX`	Maximum per-provider concurrent in-flight read I/O permits. The adaptive I/O system will never allow more than this many concurrent requests.
`adaptive_io.requests_per_provider_min`	`8`	`SPIRAL__ADAPTIVE_IO__REQUESTS_PER_PROVIDER_MIN`	Minimum per-provider concurrent in-flight read I/O permits. The adaptive I/O system will always allow at least this many concurrent requests.
`adaptive_io.signal_window_samples`	`20`	`SPIRAL__ADAPTIVE_IO__SIGNAL_WINDOW_SAMPLES`	Number of most-recent samples considered for control-window decisions. No adaptation occurs until at least this many request complete (successfully or not).
`adaptive_io.tick_interval_ms`	`500`	`SPIRAL__ADAPTIVE_IO__TICK_INTERVAL_MS`	Adaptive controller tick interval in milliseconds. The controller will consider an increase or decreases at most once per tick interval.

Authn

Authentication related settings

Field	Default	Environment Variable	Description
`authn.device_code`	`false`	`SPIRAL__AUTHN__DEVICE_CODE`	Whether to force device code authentication flow. This exist as an override when e.g. ssh-ing into an environment machine.
`authn.token`	`-`	`SPIRAL__AUTHN__TOKEN`	Optional authentication token.

Fragments Cache

Client cache limits and constraints

Field	Default	Environment Variable	Description
`fragments_cache.blob_index_size`	`4 * (1 << 10)`	`SPIRAL__FRAGMENTS_CACHE__BLOB_INDEX_SIZE`	Size of the blob index for each blob on disk. A larger index can hold more entries per blob but increases per-blob I/O.
`fragments_cache.block_size`	`16 * (1 << 20)`	`SPIRAL__FRAGMENTS_CACHE__BLOCK_SIZE`	Block size for the disk cache engine. This is the minimum eviction unit and also limits the maximum cacheable entry size.
`fragments_cache.buffer_pool_size`	`16 * (1 << 20)`	`SPIRAL__FRAGMENTS_CACHE__BUFFER_POOL_SIZE`	Total flush buffer pool size in bytes (RAM). Each flusher gets `buffer_pool_size / flushers` bytes. Entries larger than this are dropped.
`fragments_cache.disk_capacity_bytes`	`20 * (1 << 30)`	`SPIRAL__FRAGMENTS_CACHE__DISK_CAPACITY_BYTES`	Disk cache size in bytes.
`fragments_cache.disk_path`	`spiral`	`SPIRAL__FRAGMENTS_CACHE__DISK_PATH`	Disk cache path override. If not set, defaults to $XDG_CACHE_HOME/spiral or ~/.cache/spiral
`fragments_cache.enabled`	`false`	`SPIRAL__FRAGMENTS_CACHE__ENABLED`	Whether this cache is enabled. Default varies per cache type (see ClientSettings).
`fragments_cache.memory_capacity_bytes`	`2 * (1 << 30)`	`SPIRAL__FRAGMENTS_CACHE__MEMORY_CAPACITY_BYTES`	Memory cache size in bytes.
`fragments_cache.submit_queue_size_threshold`	`16 * (1 << 20)`	`SPIRAL__FRAGMENTS_CACHE__SUBMIT_QUEUE_SIZE_THRESHOLD`	If the total estimated size in the submit queue exceeds this threshold, further entries are silently dropped.
`fragments_cache.write_on_insertion`	`-`	`SPIRAL__FRAGMENTS_CACHE__WRITE_ON_INSERTION`	When true, entries are written to disk immediately on insertion. When false, entries are only written to disk when evicted from memory.

Keys Cache

Client cache limits and constraints

Field	Default	Environment Variable	Description
`keys_cache.blob_index_size`	`4 * (1 << 10)`	`SPIRAL__KEYS_CACHE__BLOB_INDEX_SIZE`	Size of the blob index for each blob on disk. A larger index can hold more entries per blob but increases per-blob I/O.
`keys_cache.block_size`	`16 * (1 << 20)`	`SPIRAL__KEYS_CACHE__BLOCK_SIZE`	Block size for the disk cache engine. This is the minimum eviction unit and also limits the maximum cacheable entry size.
`keys_cache.buffer_pool_size`	`16 * (1 << 20)`	`SPIRAL__KEYS_CACHE__BUFFER_POOL_SIZE`	Total flush buffer pool size in bytes (RAM). Each flusher gets `buffer_pool_size / flushers` bytes. Entries larger than this are dropped.
`keys_cache.disk_capacity_bytes`	`0`	`SPIRAL__KEYS_CACHE__DISK_CAPACITY_BYTES`	Disk cache size in bytes.
`keys_cache.disk_path`	`spiral`	`SPIRAL__KEYS_CACHE__DISK_PATH`	Disk cache path override. If not set, defaults to $XDG_CACHE_HOME/spiral or ~/.cache/spiral
`keys_cache.enabled`	`true`	`SPIRAL__KEYS_CACHE__ENABLED`	Whether this cache is enabled. Default varies per cache type (see ClientSettings).
`keys_cache.memory_capacity_bytes`	`1 << 30`	`SPIRAL__KEYS_CACHE__MEMORY_CAPACITY_BYTES`	Memory cache size in bytes.
`keys_cache.submit_queue_size_threshold`	`16 * (1 << 20)`	`SPIRAL__KEYS_CACHE__SUBMIT_QUEUE_SIZE_THRESHOLD`	If the total estimated size in the submit queue exceeds this threshold, further entries are silently dropped.
`keys_cache.write_on_insertion`	`-`	`SPIRAL__KEYS_CACHE__WRITE_ON_INSERTION`	When true, entries are written to disk immediately on insertion. When false, entries are only written to disk when evicted from memory.

Limits

Client resource limits and constraints

Field	Default	Environment Variable	Description
`limits.batch_readahead`	`NonZeroUsize :: new (min (num_cpus :: get ()`	`SPIRAL__LIMITS__BATCH_READAHEAD`	Maximum number of concurrent shards to evaluate, “read ahead”, when scanning. Defaults to min(number of CPU cores, 32).
`limits.compaction_size_based_threshold_bytes`	`8 * 1024 * 1024`	`SPIRAL__LIMITS__COMPACTION_SIZE_BASED_THRESHOLD_BYTES`	Fragments smaller than this threshold will be considered for compaction when size-based compaction strategy is used.
`limits.compaction_split_compressed_max_bytes`	`1024 * 1024 * 1024`	`SPIRAL__LIMITS__COMPACTION_SPLIT_COMPRESSED_MAX_BYTES`	Maximum total compressed size of fragments accumulated in a single compaction split.
`limits.compaction_task_concurrency`	`1`	`SPIRAL__LIMITS__COMPACTION_TASK_CONCURRENCY`	Maximum number of concurrent tasks in compaction.
`limits.disable_request_merging`	`false`	`SPIRAL__LIMITS__DISABLE_REQUEST_MERGING`	Completely disable merging HTTP requests. Even overlapping or duplicate requests wouldn’t be merged together.
`limits.http_max_retries`	`10`	`SPIRAL__LIMITS__HTTP_MAX_RETRIES`	Maximum number of HTTP request retries on transient errors.
`limits.http_max_retry_interval_ms`	`5 * 60 * 1000`	`SPIRAL__LIMITS__HTTP_MAX_RETRY_INTERVAL_MS`	Maximum number of milliseconds to wait between HTTP request retries.
`limits.http_min_retry_interval_ms`	`100`	`SPIRAL__LIMITS__HTTP_MIN_RETRY_INTERVAL_MS`	Minimum number of milliseconds to wait between HTTP request retries.
`limits.io_merge_threshold`	`1024 * 1024`	`SPIRAL__LIMITS__IO_MERGE_THRESHOLD`	Byte range merge distance threshold (in bytes) for IO coalescing. Nearby byte ranges within this distance are merged into single HTTP requests.
`limits.join_set_concurrency`	`num_cpus :: get ()`	`SPIRAL__LIMITS__JOIN_SET_CONCURRENCY`	Maximum number of concurrent tasks in a JoinSet. Defaults to the number of CPU cores.
`limits.key_space_max_rows`	`100_000_000`	`SPIRAL__LIMITS__KEY_SPACE_MAX_ROWS`	Maximum number of rows in a key space. Compaction keeps fragments across column groups aligned to L1 key spaces. When this limit is larger, the L1 key spaces might change more frequently, causing a re-alignment to happen. When this limit is smaller, more fragments may have to be split due to key space boundaries.
`limits.manifest_read_concurrency`	`min (16 * num_cpus :: get ()`	`SPIRAL__LIMITS__MANIFEST_READ_CONCURRENCY`	Maximum number of concurrent manifest file reads.
`limits.manifest_write_concurrency`	`2 * num_cpus :: get ()`	`SPIRAL__LIMITS__MANIFEST_WRITE_CONCURRENCY`	Maximum number of concurrent manifest file writes.
`limits.object_storage_client_pool_idle_timeout_s`	`None`	`SPIRAL__LIMITS__OBJECT_STORAGE_CLIENT_POOL_IDLE_TIMEOUT_S`	`pool_idle_timeout` for object storage clients.
`limits.object_storage_client_pool_max_idle_per_host`	`None`	`SPIRAL__LIMITS__OBJECT_STORAGE_CLIENT_POOL_MAX_IDLE_PER_HOST`	`pool_max_idle_per_host` for object storage clients.
`limits.prefetch_buffer_size`	`20`	`SPIRAL__LIMITS__PREFETCH_BUFFER_SIZE`	Per-stream channel buffer size (number of KeyTables buffered ahead) Zero disables prefetching entirely.
`limits.read_max_iops_per_file`	`-`	`SPIRAL__LIMITS__READ_MAX_IOPS_PER_FILE`	Maximum number of concurrent HTTP requests when reading a single file. This is an upper bound on the number of concurrent I/O requests to a single VortexReadAt instance. Usually only one of these exists per Vortex file, so this is a rough upper limit on the number of concurrent requsts to each fragment or manifest file. See also: [read_thread_per_core_per_file].
`limits.read_threads_per_core_per_file`	`1`	`SPIRAL__LIMITS__READ_THREADS_PER_CORE_PER_FILE`	The number of read threads per core per file. See also: [read_max_iops_per_file], which is a limit shared across all cores and threads reading from the same file object.
`limits.scan_eval_concurrency`	`16`	`SPIRAL__LIMITS__SCAN_EVAL_CONCURRENCY`	Maximum number of concurrent scan node evaluations (streams preparation) during scans.
`limits.scan_min_rows_per_chunk`	`1000`	`SPIRAL__LIMITS__SCAN_MIN_ROWS_PER_CHUNK`	Minimum number of rows to buffer when executing the scan Small chunks are concatenated together until this threshold is reached.
`limits.spfs_api_connect_timeout_s`	`5`	`SPIRAL__LIMITS__SPFS_API_CONNECT_TIMEOUT_S`	TCP connect timeout (seconds) for SpFS API HTTP clients.
`limits.spfs_api_request_timeout_s`	`30`	`SPIRAL__LIMITS__SPFS_API_REQUEST_TIMEOUT_S`	Total request timeout (seconds) for Spfs
`limits.spfs_hedge_delay_ms`	`200`	`SPIRAL__LIMITS__SPFS_HEDGE_DELAY_MS`	Baseline delay in milliseconds before issuing a hedged duplicate read request to SPFS. When non-zero, a shared latency sketch tracks the 95th percentile for completed HTTP reads and scales the hedge delay by the amount of data being read. Set to `0` to disable hedging entirely.
`limits.spfs_max_retries`	`8`	`SPIRAL__LIMITS__SPFS_MAX_RETRIES`	Maximum number of SPFS operation retries on transient errors. A single SPFS read operation may comprise tens or hundreds of distinct HTTP requests. SPFS write operations may comprise multiple HTTP requests but are typically just one (long) HTTP request.
`limits.spfs_object_storage_connect_timeout_s`	`5`	`SPIRAL__LIMITS__SPFS_OBJECT_STORAGE_CONNECT_TIMEOUT_S`	TCP connect timeout (seconds) for object-storage HTTP clients.
`limits.transaction_compact_threshold`	`100`	`SPIRAL__LIMITS__TRANSACTION_COMPACT_THRESHOLD`	Automatically compact transaction operations before commit if operation count exceeds this threshold. Set to 0 to disable auto-compaction.
`limits.transaction_manifest_max_rows`	`-`	`SPIRAL__LIMITS__TRANSACTION_MANIFEST_MAX_ROWS`	Maximum number of rows in manifest when compacting transaction.
`limits.transaction_retries`	`3`	`SPIRAL__LIMITS__TRANSACTION_RETRIES`	Maximum number of transaction retries on conflict before giving up.
`limits.write_buffer_max_bytes`	`1024 * 1024 * 1024`	`SPIRAL__LIMITS__WRITE_BUFFER_MAX_BYTES`	Maximum number of bytes to buffer in memory when writing key table batches.
`limits.write_partition_max_bytes`	`128 * 1024 * 1024`	`SPIRAL__LIMITS__WRITE_PARTITION_MAX_BYTES`	Maximum number of bytes in an uncompressed fragment file.

Manifests Cache

Client cache limits and constraints

Field	Default	Environment Variable	Description
`manifests_cache.blob_index_size`	`4 * (1 << 10)`	`SPIRAL__MANIFESTS_CACHE__BLOB_INDEX_SIZE`	Size of the blob index for each blob on disk. A larger index can hold more entries per blob but increases per-blob I/O.
`manifests_cache.block_size`	`16 * (1 << 20)`	`SPIRAL__MANIFESTS_CACHE__BLOCK_SIZE`	Block size for the disk cache engine. This is the minimum eviction unit and also limits the maximum cacheable entry size.
`manifests_cache.buffer_pool_size`	`16 * (1 << 20)`	`SPIRAL__MANIFESTS_CACHE__BUFFER_POOL_SIZE`	Total flush buffer pool size in bytes (RAM). Each flusher gets `buffer_pool_size / flushers` bytes. Entries larger than this are dropped.
`manifests_cache.disk_capacity_bytes`	`20 * (1 << 30)`	`SPIRAL__MANIFESTS_CACHE__DISK_CAPACITY_BYTES`	Disk cache size in bytes.
`manifests_cache.disk_path`	`spiral`	`SPIRAL__MANIFESTS_CACHE__DISK_PATH`	Disk cache path override. If not set, defaults to $XDG_CACHE_HOME/spiral or ~/.cache/spiral
`manifests_cache.enabled`	`false`	`SPIRAL__MANIFESTS_CACHE__ENABLED`	Whether this cache is enabled. Default varies per cache type (see ClientSettings).
`manifests_cache.memory_capacity_bytes`	`2 * (1 << 30)`	`SPIRAL__MANIFESTS_CACHE__MEMORY_CAPACITY_BYTES`	Memory cache size in bytes.
`manifests_cache.submit_queue_size_threshold`	`16 * (1 << 20)`	`SPIRAL__MANIFESTS_CACHE__SUBMIT_QUEUE_SIZE_THRESHOLD`	If the total estimated size in the submit queue exceeds this threshold, further entries are silently dropped.
`manifests_cache.write_on_insertion`	`-`	`SPIRAL__MANIFESTS_CACHE__WRITE_ON_INSERTION`	When true, entries are written to disk immediately on insertion. When false, entries are only written to disk when evicted from memory.

Server

API endpoint configuration for the Spiral control plane

Field	Default	Environment Variable	Description
`server.url`	`https://api.spiraldb.dev`	`SPIRAL__SERVER__URL`	The SpiralDB API endpoint URL

Spfs

SpFS (Spiral File System) configuration

Field	Default	Environment Variable	Description
`spfs.http1_only`	`false`	`SPIRAL__SPFS__HTTP1_ONLY`	Restrict object-storage HTTP clients to HTTP/1.1.
`spfs.http2_initial_connection_window_size`	`4 * 1024 * 1024`	`SPIRAL__SPFS__HTTP2_INITIAL_CONNECTION_WINDOW_SIZE`	HTTP/2 initial connection-level flow-control window size (bytes) for object storage clients. Only applies when `http1_only` is false.
`spfs.http2_initial_stream_window_size`	`1024 * 1024`	`SPIRAL__SPFS__HTTP2_INITIAL_STREAM_WINDOW_SIZE`	HTTP/2 initial stream-level flow-control window size (bytes) for object storage clients. Only applies when `http1_only` is false.
`spfs.url`	`https://spfs.spiraldb.dev`	`SPIRAL__SPFS__URL`	The SpFS (Spiral File System) endpoint URL

Telemetry

Telemetry configuration for the Spiral client

Field	Default	Environment Variable	Description
`telemetry.enabled`	`true`	`SPIRAL__TELEMETRY__ENABLED`	Whether client-side telemetry export is enabled.