Troubleshooting

A guide to troubleshooting common Spiral client-side issues.

Exporting telemetry

Client-side OTEL  telemetry data can be exported to Spiral Cloud, or your own OTEL collector.

To enable exporting to your own collector, set the SPIRAL_OTEL_ENDPOINT environment variable to your OTEL collector endpoint. Standard OTEL environment variables can be used to configure the OTEL exporter, such as OTEL_EXPORTER_OTLP_HEADERS, etc.

To export telemetry data to Spiral Cloud, you must first obtain a telemetry token:

spiral telemetry export

To export telemetry data to Spiral Cloud, you must set the SPIRAL_OTEL_TOKEN environment variable to the obtained token.

Temporary workarounds

Following issues may occur and currently have temporary workarounds. We are actively working on permanent fixes. If the workaround does not work for you, please reach out to us.

• “ArrowInvalid: offset overflow while concatenating arrays” The error occurs on write when working with variable-length data (e.g., bytes, strings, lists) and the total size of the data exceeds the maximum allowed size for a single Arrow array. Spiral currently does not support writing Arrow chunked arrays so all chunks are combined into a single array before writing. To resolve this, we recommend using “large” Arrow types (e.g. pa.large_binary(), pa.large_string(), pa.large_list()) when passing in Arrow arrays. Spiral uses Vortex  as storage and memory format. Unlike Arrow, Vortex has logical types and does not differentiate between “large” types.

• “Too many open files” This error occurs when the operating system limit for open file descriptors is reached. Default limits are usually too low for data-intensive applications. Use ulimit -n to check the current soft limit and ulimit -Hn to check the hard limit. Use ulimit -n <new_limit> to increase the limit for the session or reach out to your system administrator to increase the hard limit. We recommend setting the limit to at least 65536 or higher, but do not exceed the hard limit.