Client Configuration

This document describes the configuration options for the Spiral client.

Configuration can be set via:

File: ~/.spiral.toml
Environment variables: Prefixed with SPIRAL__ (double underscore)
Runtime overrides: Override values when initializing the client, using the dot notation

Caching

Spiral supports three independent caches for different file types:

keys_cache: Caches key space files. Recommended to enable for workloads with repeated key lookups.
manifests_cache: Caches manifest files. Recommended to enable for better query planning performance.
fragments_cache: Caches fragment (data) files. Not recommended to enable in most cases as fragments are typically very large and benefit less from caching.

Each cache can be independently configured with separate memory limits, disk capacity, and storage paths.

Top level

Main configuration for the Spiral client, that can affect many different parts of the client.

Field	Default	Environment Variable	Description
`file_format`	`vortex`	`SPIRAL__FILE_FORMAT`	File format for table storage
`dev`	`false`	`SPIRAL__DEV`	Development mode flag

Authn

Authentication related settings

Field	Default	Environment Variable	Description
`authn.device_code`	`false`	`SPIRAL__AUTHN__DEVICE_CODE`	Whether to force device code authentication flow. This exist as an override when e.g. ssh-ing into an environment machine.
`authn.token`	`-`	`SPIRAL__AUTHN__TOKEN`	Optional authentication token

Fragments_cache

Client cache limits and constraints

Field	Default	Environment Variable	Description
`fragments_cache.enabled`	`false`	`SPIRAL__FRAGMENTS_CACHE__ENABLED`	Cache is currently disabled by default.
`fragments_cache.memory_capacity_bytes`	`2 * (1 << 30)`	`SPIRAL__FRAGMENTS_CACHE__MEMORY_CAPACITY_BYTES`	Memory cache size in bytes.
`fragments_cache.disk_capacity_bytes`	`20 * (1 << 30)`	`SPIRAL__FRAGMENTS_CACHE__DISK_CAPACITY_BYTES`	Disk cache size in bytes.
`fragments_cache.disk_path`	`-`	`SPIRAL__FRAGMENTS_CACHE__DISK_PATH`	Disk cache path override. If not set, defaults to $XDG_CACHE_HOME/spiral or ~/.cache/spiral

Keys_cache

Client cache limits and constraints

Field	Default	Environment Variable	Description
`keys_cache.enabled`	`false`	`SPIRAL__KEYS_CACHE__ENABLED`	Cache is currently disabled by default.
`keys_cache.memory_capacity_bytes`	`2 * (1 << 30)`	`SPIRAL__KEYS_CACHE__MEMORY_CAPACITY_BYTES`	Memory cache size in bytes.
`keys_cache.disk_capacity_bytes`	`20 * (1 << 30)`	`SPIRAL__KEYS_CACHE__DISK_CAPACITY_BYTES`	Disk cache size in bytes.
`keys_cache.disk_path`	`-`	`SPIRAL__KEYS_CACHE__DISK_PATH`	Disk cache path override. If not set, defaults to $XDG_CACHE_HOME/spiral or ~/.cache/spiral

Limits

Client resource limits and constraints

Field	Default	Environment Variable	Description
`limits.batch_readahead`	`NonZeroUsize :: new (min (num_cpus :: get ()`	`SPIRAL__LIMITS__BATCH_READAHEAD`	Maximum number of concurrent shards to evaluate, “read ahead”, when scanning. Defaults to min(number of CPU cores, 64).
`limits.transaction_retries`	`3`	`SPIRAL__LIMITS__TRANSACTION_RETRIES`	Maximum number of transaction retries on conflict before giving up.
`limits.transaction_manifest_max_rows`	`-`	`SPIRAL__LIMITS__TRANSACTION_MANIFEST_MAX_ROWS`	Maximum number of rows in manifest when compacting transaction.
`limits.transaction_compact_threshold`	`100`	`SPIRAL__LIMITS__TRANSACTION_COMPACT_THRESHOLD`	Automatically compact transaction operations before commit if operation count exceeds this threshold. Set to 0 to disable auto-compaction.
`limits.key_space_max_rows`	`100_000_000`	`SPIRAL__LIMITS__KEY_SPACE_MAX_ROWS`	Maximum number of rows in a key space. Compaction keeps fragments across column groups aligned to L1 key spaces. When this limit is larger, the L1 key spaces might change more frequently, causing a re-alignment to happen. When this limit is smaller, more fragments may have to be split due to key space boundaries.
`limits.http_max_retries`	`10`	`SPIRAL__LIMITS__HTTP_MAX_RETRIES`	Maximum number of HTTP request retries on transient errors.
`limits.http_min_retry_interval_ms`	`100`	`SPIRAL__LIMITS__HTTP_MIN_RETRY_INTERVAL_MS`	Minimum number of milliseconds to wait between HTTP request retries.
`limits.http_max_retry_interval_ms`	`5 * 60 * 1000`	`SPIRAL__LIMITS__HTTP_MAX_RETRY_INTERVAL_MS`	Maximum number of milliseconds to wait between HTTP request retries.
`limits.write_max_concurrency`	`None`	`SPIRAL__LIMITS__WRITE_MAX_CONCURRENCY`	Maximum number of concurrent write operations when writing table fragments.
`limits.write_max_concurrency_manifest`	`32`	`SPIRAL__LIMITS__WRITE_MAX_CONCURRENCY_MANIFEST`	Maximum number of concurrent manifest writes.
`limits.write_buffer_max_bytes`	`1024 * 1024 * 1024`	`SPIRAL__LIMITS__WRITE_BUFFER_MAX_BYTES`	Maximum number of bytes to buffer in memory when writing key table batches.
`limits.write_partition_max_bytes`	`128 * 1024 * 1024`	`SPIRAL__LIMITS__WRITE_PARTITION_MAX_BYTES`	Maximum number of bytes in an uncompressed fragment file.
`limits.read_max_concurrency`	`16`	`SPIRAL__LIMITS__READ_MAX_CONCURRENCY`	Maximum number of concurrent operations when evaluating multiple nodes during scans.
`limits.fragment_cache_bytes`	`256 * 1024 * 1024`	`SPIRAL__LIMITS__FRAGMENT_CACHE_BYTES`	Maximum in-memory footprint (in bytes) of the fragment cache.
`limits.compaction_task_max_bytes`	`4 * 1024 * 1024 * 1024`	`SPIRAL__LIMITS__COMPACTION_TASK_MAX_BYTES`	Maximum size of fragments accumulated in a single compaction task.
`limits.compaction_fragment_size_bytes_threshold`	`16 * 1024 * 1024`	`SPIRAL__LIMITS__COMPACTION_FRAGMENT_SIZE_BYTES_THRESHOLD`	Fragments smaller than this threshold will be considered for compaction when size-based compaction strategy is used.

Manifests_cache

Client cache limits and constraints

Field	Default	Environment Variable	Description
`manifests_cache.enabled`	`false`	`SPIRAL__MANIFESTS_CACHE__ENABLED`	Cache is currently disabled by default.
`manifests_cache.memory_capacity_bytes`	`2 * (1 << 30)`	`SPIRAL__MANIFESTS_CACHE__MEMORY_CAPACITY_BYTES`	Memory cache size in bytes.
`manifests_cache.disk_capacity_bytes`	`20 * (1 << 30)`	`SPIRAL__MANIFESTS_CACHE__DISK_CAPACITY_BYTES`	Disk cache size in bytes.
`manifests_cache.disk_path`	`-`	`SPIRAL__MANIFESTS_CACHE__DISK_PATH`	Disk cache path override. If not set, defaults to $XDG_CACHE_HOME/spiral or ~/.cache/spiral

Server

API endpoint configuration for the Spiral control plane

Field	Default	Environment Variable	Description
`server.url`	`https://api.spiraldb.dev`	`SPIRAL__SERVER__URL`	The SpiralDB API endpoint URL

Spfs

SpFS (Spiral File System) configuration

Field	Default	Environment Variable	Description
`spfs.url`	`https://spfs.spiraldb.dev`	`SPIRAL__SPFS__URL`	The SpFS (Spiral File System) endpoint URL