Object Storage

How to Prepare named object storage (S3, GCS, and similar) as one of the handful of building blocks worth knowing cold — and File Storage Sync leans on it directly for where actual file content lives. This lesson is what that building block actually does underneath "we'd store it in S3."

Objects, not files

Object storage doesn't have a real directory tree — a "folder" is a convention, not a structural fact. Every object is stored flat, addressed by a key (often something that looks like a path, e.g. users/42/avatar.png), a value (the raw bytes), and metadata (content type, custom tags). There's no partial-object edit: replacing part of an object means uploading a new version of the whole thing, which is exactly why File Storage Sync chunks a file into smaller, independently-addressed objects rather than storing one file as one object — a one-line edit to a large object would otherwise mean re-uploading the entire object.

Durability vs. availability, said precisely

Object storage providers commonly advertise "11 nines" of durability — the odds of losing an object at all, achieved by transparently storing multiple copies across facilities. That's a different number from availability — whether a request for the object succeeds right now — which is typically quoted with far fewer nines (99.9% or so). Conflating the two is a common way this topic loses precision: a design can lose almost no data ever, while still occasionally failing a single read due to a transient issue, and a design should handle the availability case (retry with backoff) even while trusting the durability guarantee.

Presigned URLs: skip the app server for the actual bytes

Routing a large file upload through the application server — client to app server to object storage — means the app server pays the bandwidth and compute cost of every upload, for zero business logic. A presigned URL sidesteps this: the app server generates a short-lived, signed URL authorizing a specific upload (or download) directly against object storage, and the client uploads straight there. The app server's only job is deciding whether to authorize the URL and recording that the object exists — the byte transfer itself never touches it.

Multipart upload

A large object — gigabytes, not kilobytes — uploaded as one HTTP request is fragile: a network blip partway through means restarting from zero. Multipart upload splits the object into independently-uploaded parts, each retryable on its own, then a final call tells object storage to assemble them in order. This is also what makes parallel upload of a single large file possible — several parts in flight to the same object at once — which is part of what makes chunk-level sync designs like File Storage Sync practical at real file sizes.

When object storage is the wrong tool

It has no query language beyond "get this exact key" (or list by prefix) — no filtering by content, no joins, no transactions across objects. Data that's frequently and partially updated, needs to be queried by anything other than its key, or needs transactional guarantees belongs in an actual database; object storage is for large, mostly-immutable blobs where a database would pay real cost to store the bytes without buying any relational benefit from them.