File Systems

Many Spiral resources make use of object storage. Spiral File Systems provide a means to securely federate and accelerate access to underlying object storage.

As with all resources in Spiral, File Systems are scoped to a project. However, unlike many other resource types, each project may only have a single file system. This file system is often referred to as the project’s default file system. By default, any resource that requires a file system will use the project’s default file system, unless otherwise specified.

Configuration

You can see a project’s file system configuration using the Spiral CLI

spiral fs show <project_id>

When resources like Tables are created they configure a prefix in a file system. File systems can only be reconfigured when they have zero used prefixes.

Built-in Providers

Spiral supports several built-in file system providers.

The full set of providers can be listed using the CLI:

spiral fs list-providers

Configure a project’s file system provider:

spiral fs update <project_id> --provider <provider>

If you require a specific provider, please reach out to us, and we should be able to add it for you.

Bring Your Own Bucket

If you have an existing bucket that you would like to use as a Spiral File System, you can update the project’s default file system to use it. See specific provider instructions below for how to configure the bucket and permissions.

Use the CLI to update default file system:

spiral fs update --help

We recommend that you use a dedicated project to register your own bucket as default file system, i.e. avoid creating any resources in this project. This allows you to separate bucket management permissions from resource management permissions. Projects with resources can be configured to use the “bucket” project as their default file system using:

spiral fs update <project_id> --upstream <bucket_project_id>

AWS

To configure an S3 bucket for Spiral, create an IAM policy first.

{
	"Version": "2012-10-17",
	"Statement": [
		{
			"Sid": "AllowListBucketAll",
			"Effect": "Allow",
			"Action": "s3:ListBucket*",
			"Resource": [
				"arn:aws:s3:::{your-bucket-name}"
			]
		},
		{
			"Sid": "AllowObjectSome",
			"Effect": "Allow",
			"Action": [
				"s3:*Object",
				"s3:*ObjectVersion",
				"s3:AbortMultipartUpload",
				"s3:ListMultipartUploadParts"
			],
			"Resource": [
				"arn:aws:s3:::{your-bucket-name}/*"
			]
		}
	]
}

Next, you have to allow Spiral to assume a role with this policy. The easiest and most secure way to do this (avoids any long lived tokens!), is to allow Spiral’s GCP identity to assume the role.

Create an IAM role with a following trust policy and attach the above object storage access policy to it.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "Federated": "accounts.google.com"
            },
            "Action": "sts:AssumeRoleWithWebIdentity",
            "Condition": {
                "StringEquals": {
                    "accounts.google.com:sub": "116500466089430548312",
                    "accounts.google.com:aud": "116500466089430548312",
                    "accounts.google.com:oaud": "https://iss.spiraldb.com"
                }
            }
        }
    ]
}

116500466089430548312 is a unique identifier for Spiral’s service account (spiraldb-filesystems@pyspiral-dev.iam.gserviceaccount.com) in GCP. Spiral exchanges short-lived GCP identity token for AWS temporary credentials. See AWS docs  for more details on this approach.

Use the ARN of the created role to update project’s file system:

spiral fs update --type s3 --bucket <your-bucket-name> --region <your-bucket-region> --role-arn <your-role-arn>

GCP

To configure a GCS bucket for Spiral, grant Storage Object User and Storage Object Viewer roles on the bucket to Spiral service account spiraldb-filesystems@pyspiral-dev.iam.gserviceaccount.com.

Use the CLI to update project’s file system:

spiral fs update --type gcs --bucket <your-bucket-name> --region <your-bucket-region>