Virtual S3 namespaces — design

Oct 12, 2026 · Robert C. Sheets

Summary

A virtual namespace gives a tenant an S3 endpoint of their own, where they can name buckets anything S3 allows: photos, backups, www.example.com, with no <prefix>-- in front. This lets tenants run unconstrained by the naming rules the portal imposes on the shared store, which today force every bucket name to start with the tenant's prefix.

• How: a gateway sits between the client and the store. It picks the namespace from the hostname (<namespace>.s3v.picote.ch), checks the client's signature itself, and maps each bucket name to an opaque stored bucket in the reserved namespace, such as yk3x9--q7m2c8vd4hxt9bnp.
• What it costs: the gateway becomes a second S3 authentication layer. It has to verify every request's signature, sign it again toward the store, and translate bucket names in both directions. That is most of the work and most of the risk.
• Isolation: three independent layers. The hostname picks the namespace; the access key must belong to it; and the store credentials the gateway uses for a tenant reach only that tenant's stored buckets.
• Status: proposed as an experiment, opt-in, path-style only, with a deliberately small set of S3 operations.

Background

The store is one shared S3 namespace, so the portal makes every tenant's bucket <prefix>--<name>. The prefix is 4 to 12 characters from 23456789bcdfghjkmnpqrstvwxz, assigned at random. Every legal bucket name falls in one of three namespaces, decided by the part before its first --:

The rule keeps tenants apart, but tenants pay for it:

• Tools assume fixed bucket names. Backup tools, static site generators and application configs often expect backups or assets, and every one has to be changed to the prefixed name.
• Names are shorter. A 4-character prefix and -- leave 57 of S3's 63 characters.
• Names are global. Two tenants cannot both have photos, and a domain-like name is only ever <prefix>--example.com.
• Moving in is harder. Data from another S3 service arrives under new bucket names, so every reference to it changes too.

The reserved namespace is empty and kept free for a use like this one: no tenant's grant reaches it.

Design overview

A tenant's client talks to its namespace's own endpoint; the gateway behind it is the only new component, and the store sees nothing but ordinary buckets.

A request for /photos at <ns>.s3v.picote.ch passes s3-edge to the gateway, which checks its signature, looks up photos in the registry, and sends it back through s3-edge to the store as y<prefix>--<id>, signed with the tenant's own store credentials. The portal creates namespaces and keys and writes them to the registry; it is not on the request path.

Naming and mapping

A stored bucket's name never depends on the name the tenant chose. Each virtual bucket is stored as y<prefix>--<id>, where <id> is 16 random characters from the prefix alphabet; a mapping records which stored bucket each name points at.

The stored name. The marker y is not in the prefix alphabet, so y<prefix> is never a valid prefix and the name is always in the reserved namespace. y also sorts the virtual buckets after every tenant bucket whose prefix starts with a digit or b to x, so a full listing of the store keeps them out of the way; only prefixes starting with z sort after them. (z itself cannot be the marker: z<prefix> can be a valid prefix.) The longest stored name is 31 characters (1 + 12 + 2 + 16), so the tenant's name can use all 63 that S3 allows. The namespace docs would claim this corner: reserved names whose part before the first -- is y followed by a valid prefix belong to the virtual service.

The namespace ID. The hostname carries a namespace ID, not the prefix: 8 random characters from the same alphabet, kept in the mapping with its tenant. A tenant can then have several namespaces (say production and staging), and replace an endpoint without touching their prefix.

The mapping. In the portal's registry bucket, storage-portal, beside the prefixes:

Creating a bucket, under the namespace's lock:

1. Refuse the name if virtual/buckets/<ns>/<name> exists (S3's BucketAlreadyOwnedByYou).
2. Draw an <id> and create y<prefix>--<id> in the store, versioned like every tenant bucket.
3. Write virtual/buckets/<ns>/<name>. Only now does the bucket exist for the client.

Deleting a bucket runs the other way: delete the stored bucket (the store refuses one that is not empty, and the client gets BucketNotEmpty), then the mapping.

Reconciling. A crash between steps leaves either a stored bucket with no name or a name with no stored bucket. A sweep, like the hourly usage sweep, deletes empty stored buckets in the virtual corner that no mapping names and are over an hour old, and removes mappings whose stored bucket is gone. It never deletes a bucket with objects in it; one of those is logged for an operator.

Uniqueness relies on one gateway replica and an in-memory lock per namespace, as the portal does for prefixes today. More replicas would need a conditional write on the mapping object, which is an open question for SeaweedFS.

Request path

The gateway verifies the client's signature itself and signs the request again toward the store, because the bucket name it rewrites is part of what the client signed. Each request runs these steps in order, and the first that fails ends it:

1. Namespace from the hostname. <ns>.s3v.picote.ch names the namespace. An unknown namespace gets NoSuchBucket, the same answer as an unknown bucket, so probing reveals nothing more.
2. Bucket from the path. Path-style only: /<bucket>/<key>. A request with no bucket is ListBuckets.
3. Credentials. A SigV4 Authorization header or a presigned URL's X-Amz-Credential names a virtual key. The key must exist and belong to this namespace; otherwise InvalidAccessKeyId. SigV2 is refused.
4. Signature. The canonical request is rebuilt exactly as received, with the hostname as the client sent it, and compared in constant time. Clock skew over 15 minutes and expired presigned URLs are refused, as S3 refuses them.
5. Payload. The payload hash is checked against the body as it streams:
   • UNSIGNED-PAYLOAD passes through.
   • A signed SHA-256 is hashed on the way through; on a mismatch the gateway drops the store connection before the body completes, so the store keeps nothing.
   • aws-chunked bodies (STREAMING-AWS4-HMAC-SHA256-PAYLOAD) have every chunk's signature checked and are decoded before forwarding.
   • STREAMING-UNSIGNED-PAYLOAD-TRAILER, which current AWS SDKs send by default with a CRC checksum in a trailer, is decoded and its checksum verified.
6. Authorization. The operation must be on the allowlist (see Rewriting), and an ro key may only read. Anonymous requests are allowed only to GET and HEAD objects in a bucket flagged public-read.
7. Forwarding. The gateway rewrites the bucket to its stored name and signs the request with the tenant's store credentials: an IAM user owner-<prefix>-v, whose only grant is arn:aws:s3:::y<prefix>--*. It goes to s3-edge's in-cluster address, so the store's request log still records it.
8. Response. Bucket names in the response are rewritten back to the tenant's names before it reaches the client.

Signature checking is the riskiest code here. It should be tested against AWS's published SigV4 test suite and against real clients (aws-cli, the Go and Python SDKs, rclone, restic) before any tenant uses it.

Rewriting

The gateway forwards only operations it knows how to translate; anything else gets NotImplemented. A name missed in a response would show the client a stored name, and one missed in a request would reach the wrong bucket or none, so the list is an allowlist, not a denylist.

Not forwarded in the experiment: bucket policies and ACLs (their documents name buckets in ARNs, and the gateway's own read-only flag covers public access), notifications, website hosting, Object Lock, replication, and SSE-C. Each can be added later with its own translation.

Security model

A single bug should not let one tenant reach another's data. Three checks stand between a request and a stored bucket, and each is enforced by a different component:

1. The hostname picks the namespace (gateway). A name is only ever looked up within it.
2. The key must belong to that namespace (gateway). A valid key for namespace A is refused at namespace B's hostname.
3. The store credentials reach one tenant only (the store). The gateway signs each tenant's requests as owner-<prefix>-v, granted only y<prefix>--*. A gateway bug that resolved the wrong tenant's bucket would still be refused by the store.

The gateway holds more power than any single tenant key, but less than the portal: it can create and delete only in the virtual corner, and it has no IAM rights. Its store users are created by the portal, frozen with the same freeze policy as every tenant user, so a store freeze reaches virtual namespaces too.

Operations

The gateway runs as its own service on pico-prd-na-west-2, beside the store, not inside the storage-portal binary: it is on every request's path, while the portal is not, so it deploys, scales and fails on its own. It reuses what the store already has wherever it can.

• Zone. *.s3v.picote.ch, not *.s3.picote.ch: a namespace label there would be indistinguishable from a real bucket addressed hostname-style. One wildcard DNS record points it at the existing s3-public load balancer.
• TLS. One wildcard certificate for *.s3v.picote.ch covers every namespace, because the experiment is path-style only. Hostname-style would need a wildcard per namespace (*.<ns>.s3v.picote.ch), issued by DNS challenge. How s3-edge gets its certificate today decides how this one is issued.
• Routing. s3-edge routes the new zone to the gateway, so its access log, with s3_key_id and the client's address, covers virtual traffic as it covers the rest.
• Logging and usage. The store's audit log names the stored bucket and the user owner-<prefix>-v. The gateway logs each request's namespace, tenant name and stored name, so either can be traced to the other. The portal's key-usage view can read virtual keys from s3-edge's records unchanged.
• The portal. It needs to know the virtual corner: namespaceOf gains a fourth answer, so the admin page lists these buckets by tenant instead of flagging them under "Reserved buckets", and bucket counts and gauges carry the tenant's prefix. Tenants create namespaces and keys, and see their virtual buckets, on their portal page.
• Quotas. Virtual buckets count against the tenant's bucket quota, 25 today. Keys get a quota of their own.
• Freeze. The gateway's store users carry the freeze policy, so a freeze leaves virtual namespaces read-only, like everything else.
• Capacity. Virtual buckets share the store's disks, with the same replication as every bucket; the gateway adds one hop and a re-signature per request.

Scope of the experiment

The experiment proves the model on the operations most tools use, for tenants who opt in, and leaves everything else for later.

What tenants are told: it is experimental. It may change, go down, or be withdrawn, with notice. Their data sits in ordinary buckets in the store, versioned like any other, and can be moved to an ordinary tenant bucket if the experiment ends.

Plan and open questions

The first step is the signature work, because it decides whether the rest is worth building. Each step ends with something testable:

1. Claim the corner. Document y<prefix>--* as the virtual service's part of the reserved namespace, and teach namespaceOf the fourth answer.
2. Signature checking. Verify SigV4 headers, presigned URLs, aws-chunked and unsigned trailers against AWS's test suite. No forwarding yet.
3. Gateway core. Hostname and key resolution, the mapping, bucket create, delete and list, and object get, put, head, delete and list, signed again to the store with the tenant's user.
4. The rest of the allowlist. Multipart, copy, versioning, tagging, CORS, lifecycle, and the rewriting table in full, tested against aws-cli, the Go and Python SDKs, rclone and restic.
5. The portal. Namespace and key creation, the virtual buckets on the tenant's page, the admin page's fourth namespace, and the reconciliation sweep.
6. Invite tenants. A few, with the announcement, before deciding whether to keep it.

Open questions:

• Can SeaweedFS do conditional writes (If-None-Match) on the registry? Without them the mapping stays single-replica.
• Does SeaweedFS support bucket tagging well enough to tag each stored bucket with its tenant name, as a second record beside the mapping?
• How does s3-edge get its certificate today, and can the same mechanism issue *.s3v.picote.ch?
• Should virtual keys be scoped per bucket from the start, like the portal's scoped keys, or begin with rw and ro for the whole namespace?
• Are virtual buckets versioned like tenant buckets, or does the tenant choose, as admins do for system buckets?

Experimental: virtual namespaces · Object storage · picote.ch