Client Configuration

This document describes the configuration options for the Spiral client.

Configuration can be set via:

File: ~/.spiral.toml
Environment variables: Prefixed with SPIRAL__ (double underscore)
Runtime overrides: Override values when initializing the client, using the dot notation

Caching

Spiral supports three independent caches for different file types:

keys_cache: Caches key space files. Recommended to enable for workloads with repeated key lookups.
manifests_cache: Caches manifest files. Recommended to enable for better query planning performance.
fragments_cache: Caches fragment (data) files. Not recommended to enable in most cases as fragments are typically very large and benefit less from caching.

Each cache can be independently configured with separate memory limits, disk capacity, and storage paths.

Top level

Main configuration for the Spiral client, that can affect many different parts of the client.

Field	Default	Environment Variable	Description
`file_format`	`vortex`	`SPIRAL__FILE_FORMAT`	File format for table storage
`dev`	`false`	`SPIRAL__DEV`	Development mode flag

Authn

Authentication related settings

Field	Default	Environment Variable	Description
`authn.device_code`	`false`	`SPIRAL__AUTHN__DEVICE_CODE`	Whether to force device code authentication flow. This exist as an override when e.g. ssh-ing into an environment machine.
`authn.token`	`-`	`SPIRAL__AUTHN__TOKEN`	Optional authentication token.

Fragments Cache

Client cache limits and constraints

Field	Default	Environment Variable	Description
`fragments_cache.enabled`	`false`	`SPIRAL__FRAGMENTS_CACHE__ENABLED`	Cache is currently disabled by default.
`fragments_cache.memory_capacity_bytes`	`2 * (1 << 30)`	`SPIRAL__FRAGMENTS_CACHE__MEMORY_CAPACITY_BYTES`	Memory cache size in bytes.
`fragments_cache.disk_capacity_bytes`	`20 * (1 << 30)`	`SPIRAL__FRAGMENTS_CACHE__DISK_CAPACITY_BYTES`	Disk cache size in bytes.
`fragments_cache.disk_path`	`spiral`	`SPIRAL__FRAGMENTS_CACHE__DISK_PATH`	Disk cache path override. If not set, defaults to $XDG_CACHE_HOME/spiral or ~/.cache/spiral
`fragments_cache.block_size`	`16 * (1 << 20)`	`SPIRAL__FRAGMENTS_CACHE__BLOCK_SIZE`	Block size for the disk cache engine. This is the minimum eviction unit and also limits the maximum cacheable entry size.
`fragments_cache.buffer_pool_size`	`16 * (1 << 20)`	`SPIRAL__FRAGMENTS_CACHE__BUFFER_POOL_SIZE`	Total flush buffer pool size in bytes (RAM). Each flusher gets `buffer_pool_size / flushers` bytes. Entries larger than this are dropped.
`fragments_cache.submit_queue_size_threshold`	`16 * (1 << 20)`	`SPIRAL__FRAGMENTS_CACHE__SUBMIT_QUEUE_SIZE_THRESHOLD`	If the total estimated size in the submit queue exceeds this threshold, further entries are silently dropped.
`fragments_cache.blob_index_size`	`4 * (1 << 10)`	`SPIRAL__FRAGMENTS_CACHE__BLOB_INDEX_SIZE`	Size of the blob index for each blob on disk. A larger index can hold more entries per blob but increases per-blob I/O.
`fragments_cache.write_on_insertion`	`-`	`SPIRAL__FRAGMENTS_CACHE__WRITE_ON_INSERTION`	When true, entries are written to disk immediately on insertion. When false, entries are only written to disk when evicted from memory.

Keys Cache

Client cache limits and constraints

Field	Default	Environment Variable	Description
`keys_cache.enabled`	`false`	`SPIRAL__KEYS_CACHE__ENABLED`	Cache is currently disabled by default.
`keys_cache.memory_capacity_bytes`	`2 * (1 << 30)`	`SPIRAL__KEYS_CACHE__MEMORY_CAPACITY_BYTES`	Memory cache size in bytes.
`keys_cache.disk_capacity_bytes`	`20 * (1 << 30)`	`SPIRAL__KEYS_CACHE__DISK_CAPACITY_BYTES`	Disk cache size in bytes.
`keys_cache.disk_path`	`spiral`	`SPIRAL__KEYS_CACHE__DISK_PATH`	Disk cache path override. If not set, defaults to $XDG_CACHE_HOME/spiral or ~/.cache/spiral
`keys_cache.block_size`	`16 * (1 << 20)`	`SPIRAL__KEYS_CACHE__BLOCK_SIZE`	Block size for the disk cache engine. This is the minimum eviction unit and also limits the maximum cacheable entry size.
`keys_cache.buffer_pool_size`	`16 * (1 << 20)`	`SPIRAL__KEYS_CACHE__BUFFER_POOL_SIZE`	Total flush buffer pool size in bytes (RAM). Each flusher gets `buffer_pool_size / flushers` bytes. Entries larger than this are dropped.
`keys_cache.submit_queue_size_threshold`	`16 * (1 << 20)`	`SPIRAL__KEYS_CACHE__SUBMIT_QUEUE_SIZE_THRESHOLD`	If the total estimated size in the submit queue exceeds this threshold, further entries are silently dropped.
`keys_cache.blob_index_size`	`4 * (1 << 10)`	`SPIRAL__KEYS_CACHE__BLOB_INDEX_SIZE`	Size of the blob index for each blob on disk. A larger index can hold more entries per blob but increases per-blob I/O.
`keys_cache.write_on_insertion`	`-`	`SPIRAL__KEYS_CACHE__WRITE_ON_INSERTION`	When true, entries are written to disk immediately on insertion. When false, entries are only written to disk when evicted from memory.

Limits

Client resource limits and constraints

Field	Default	Environment Variable	Description
`limits.batch_readahead`	`NonZeroUsize :: new (min (num_cpus :: get ()`	`SPIRAL__LIMITS__BATCH_READAHEAD`	Maximum number of concurrent shards to evaluate, “read ahead”, when scanning. Defaults to min(number of CPU cores, 32).
`limits.transaction_retries`	`3`	`SPIRAL__LIMITS__TRANSACTION_RETRIES`	Maximum number of transaction retries on conflict before giving up.
`limits.transaction_manifest_max_rows`	`-`	`SPIRAL__LIMITS__TRANSACTION_MANIFEST_MAX_ROWS`	Maximum number of rows in manifest when compacting transaction.
`limits.transaction_compact_threshold`	`100`	`SPIRAL__LIMITS__TRANSACTION_COMPACT_THRESHOLD`	Automatically compact transaction operations before commit if operation count exceeds this threshold. Set to 0 to disable auto-compaction.
`limits.key_space_max_rows`	`100_000_000`	`SPIRAL__LIMITS__KEY_SPACE_MAX_ROWS`	Maximum number of rows in a key space. Compaction keeps fragments across column groups aligned to L1 key spaces. When this limit is larger, the L1 key spaces might change more frequently, causing a re-alignment to happen. When this limit is smaller, more fragments may have to be split due to key space boundaries.
`limits.http_max_retries`	`10`	`SPIRAL__LIMITS__HTTP_MAX_RETRIES`	Maximum number of HTTP request retries on transient errors.
`limits.http_min_retry_interval_ms`	`100`	`SPIRAL__LIMITS__HTTP_MIN_RETRY_INTERVAL_MS`	Minimum number of milliseconds to wait between HTTP request retries.
`limits.http_max_retry_interval_ms`	`5 * 60 * 1000`	`SPIRAL__LIMITS__HTTP_MAX_RETRY_INTERVAL_MS`	Maximum number of milliseconds to wait between HTTP request retries.
`limits.write_buffer_max_bytes`	`1024 * 1024 * 1024`	`SPIRAL__LIMITS__WRITE_BUFFER_MAX_BYTES`	Maximum number of bytes to buffer in memory when writing key table batches.
`limits.write_partition_max_bytes`	`128 * 1024 * 1024`	`SPIRAL__LIMITS__WRITE_PARTITION_MAX_BYTES`	Maximum number of bytes in an uncompressed fragment file.
`limits.scan_eval_concurrency`	`4`	`SPIRAL__LIMITS__SCAN_EVAL_CONCURRENCY`	Maximum number of concurrent scan node evaluations (fragment processing) during scans.
`limits.compaction_task_concurrency`	`1`	`SPIRAL__LIMITS__COMPACTION_TASK_CONCURRENCY`	Maximum number of concurrent tasks in compaction.
`limits.manifest_read_concurrency`	`2 * num_cpus :: get ()`	`SPIRAL__LIMITS__MANIFEST_READ_CONCURRENCY`	Maximum number of concurrent manifest file reads.
`limits.manifest_write_concurrency`	`2 * num_cpus :: get ()`	`SPIRAL__LIMITS__MANIFEST_WRITE_CONCURRENCY`	Maximum number of concurrent manifest file writes.
`limits.fragment_cache_bytes`	`256 * 1024 * 1024`	`SPIRAL__LIMITS__FRAGMENT_CACHE_BYTES`	Maximum in-memory footprint (in bytes) of the fragment cache.
`limits.compaction_split_compressed_max_bytes`	`1024 * 1024 * 1024`	`SPIRAL__LIMITS__COMPACTION_SPLIT_COMPRESSED_MAX_BYTES`	Maximum total compressed size of fragments accumulated in a single compaction split.
`limits.compaction_size_based_threshold_bytes`	`8 * 1024 * 1024`	`SPIRAL__LIMITS__COMPACTION_SIZE_BASED_THRESHOLD_BYTES`	Fragments smaller than this threshold will be considered for compaction when size-based compaction strategy is used.
`limits.join_set_concurrency`	`num_cpus :: get ()`	`SPIRAL__LIMITS__JOIN_SET_CONCURRENCY`	Maximum number of concurrent tasks in a JoinSet. Defaults to the number of CPU cores.
`limits.object_storage_client_pool_max_idle_per_host`	`None`	`SPIRAL__LIMITS__OBJECT_STORAGE_CLIENT_POOL_MAX_IDLE_PER_HOST`	`pool_max_idle_per_host` for object storage clients.
`limits.object_storage_client_pool_idle_timeout_s`	`None`	`SPIRAL__LIMITS__OBJECT_STORAGE_CLIENT_POOL_IDLE_TIMEOUT_S`	`pool_idle_timeout` for object storage clients.
`limits.read_max_iops_per_file`	`-`	`SPIRAL__LIMITS__READ_MAX_IOPS_PER_FILE`	Maximum number of concurrent HTTP requests when reading a single file. This is an upper bound on the number of concurrent I/O requests to a single VortexReadAt instance. Usually only one of these exists per Vortex file, so this is a rough upper limit on the number of concurrent requsts to each fragment or manifest file. See also: [read_thread_per_core_per_file].
`limits.read_threads_per_core_per_file`	`1`	`SPIRAL__LIMITS__READ_THREADS_PER_CORE_PER_FILE`	The number of read threads per core per file. See also: [read_max_iops_per_file], which is a limit shared across all cores and threads reading from the same file object.
`limits.io_merge_threshold`	`1024 * 1024`	`SPIRAL__LIMITS__IO_MERGE_THRESHOLD`	Byte range merge distance threshold (in bytes) for IO coalescing. Nearby byte ranges within this distance are merged into single HTTP requests.
`limits.spfs_max_retries`	`3`	`SPIRAL__LIMITS__SPFS_MAX_RETRIES`	Maximum number of SPFS operation retries on transient errors. A single SPFS read operation may comprise tens or hundreds of distinct HTTP requests. SPFS write operations may comprise multiple HTTP requests but are typically just one (long) HTTP request.

Manifests Cache

Client cache limits and constraints

Field	Default	Environment Variable	Description
`manifests_cache.enabled`	`false`	`SPIRAL__MANIFESTS_CACHE__ENABLED`	Cache is currently disabled by default.
`manifests_cache.memory_capacity_bytes`	`2 * (1 << 30)`	`SPIRAL__MANIFESTS_CACHE__MEMORY_CAPACITY_BYTES`	Memory cache size in bytes.
`manifests_cache.disk_capacity_bytes`	`20 * (1 << 30)`	`SPIRAL__MANIFESTS_CACHE__DISK_CAPACITY_BYTES`	Disk cache size in bytes.
`manifests_cache.disk_path`	`spiral`	`SPIRAL__MANIFESTS_CACHE__DISK_PATH`	Disk cache path override. If not set, defaults to $XDG_CACHE_HOME/spiral or ~/.cache/spiral
`manifests_cache.block_size`	`16 * (1 << 20)`	`SPIRAL__MANIFESTS_CACHE__BLOCK_SIZE`	Block size for the disk cache engine. This is the minimum eviction unit and also limits the maximum cacheable entry size.
`manifests_cache.buffer_pool_size`	`16 * (1 << 20)`	`SPIRAL__MANIFESTS_CACHE__BUFFER_POOL_SIZE`	Total flush buffer pool size in bytes (RAM). Each flusher gets `buffer_pool_size / flushers` bytes. Entries larger than this are dropped.
`manifests_cache.submit_queue_size_threshold`	`16 * (1 << 20)`	`SPIRAL__MANIFESTS_CACHE__SUBMIT_QUEUE_SIZE_THRESHOLD`	If the total estimated size in the submit queue exceeds this threshold, further entries are silently dropped.
`manifests_cache.blob_index_size`	`4 * (1 << 10)`	`SPIRAL__MANIFESTS_CACHE__BLOB_INDEX_SIZE`	Size of the blob index for each blob on disk. A larger index can hold more entries per blob but increases per-blob I/O.
`manifests_cache.write_on_insertion`	`-`	`SPIRAL__MANIFESTS_CACHE__WRITE_ON_INSERTION`	When true, entries are written to disk immediately on insertion. When false, entries are only written to disk when evicted from memory.

Server

API endpoint configuration for the Spiral control plane

Field	Default	Environment Variable	Description
`server.url`	`https://api.spiraldb.dev`	`SPIRAL__SERVER__URL`	The SpiralDB API endpoint URL

Spfs

SpFS (Spiral File System) configuration

Field	Default	Environment Variable	Description
`spfs.url`	`https://spfs.spiraldb.dev`	`SPIRAL__SPFS__URL`	The SpFS (Spiral File System) endpoint URL

Telemetry

Telemetry configuration for the Spiral client

Field	Default	Environment Variable	Description
`telemetry.enabled`	`false`	`SPIRAL__TELEMETRY__ENABLED`	Whether client-side telemetry export is enabled.