S3 Module

The S3 module connects Sync Engine to Amazon S3 and object-storage services that implement the S3 API. It registers the s3 remote file system and uses the SDK request pipeline for authentication, proxy routing, and other request middleware.

Supported Backends

The module supports:

• Amazon S3
• Cloudflare R2
• Backblaze B2 through its S3-compatible API
• MinIO
• Garage
• Aliyun / Tencent Cloud object storage
• Wasabi
• Ceph Object Gateway, DigitalOcean Spaces, and other S3-compatible services

Compatibility depends on the service implementing the object operations listed in Required Capabilities and Permissions. Some services require path-style URLs instead of virtual-hosted URLs.

Settings and Configuration

Install and enable the S3 module, then select S3 as the storage backend. Configure these module settings:

Setting	Description
Endpoint URL	HTTP(S) endpoint for the S3 service.
Region	Bucket region used by request signing. Defaults to us-east-1.
Access key ID	Access key used for authentication.
Secret access key	Secret paired with the access key ID. Stored in Obsidian's keychain.
Bucket name	Bucket that stores the vault objects.
URL style	Virtual-hosted or path-style object URLs.
Prefix	Key prefix that exposes a subdirectory as the vault root. Defaults to /.
Proxy URL	Optional proxy endpoint for S3 requests. Disabled by default.

Endpoint, region, access key ID, bucket, and prefix values are saved as module settings. Endpoint and proxy values must be valid URLs. The prefix is normalized as a directory key, so vault becomes vault/.

Credentials and Keychain

The access key ID is stored in module settings. The secret access key is entered through Obsidian's secret storage and is not stored as ordinary module settings. The S3 module reads it when resolving credentials for connection checks or sync operations.

URL Styles

Virtual-hosted style puts the bucket in the host:

https://bucket.s3.example.com/path/to/file.md

Path style puts the bucket in the path:

https://s3.example.com/bucket/path/to/file.md

Virtual-hosted style is the default. Use path style when the service does not provide bucket subdomains, or when its DNS and TLS setup requires it.

Prefix

The prefix selects the part of the bucket exposed to Sync Engine. With prefix vault/, a local key such as Notes/today.md is stored as vault/Notes/today.md; the prefix itself is hidden from listings and file stats. / exposes the bucket root.

The prefix is an importable SDK prefixWrapper applied to the S3 file system. Its general wrapper behavior is described in file-system wrappers.

Proxy

Enable Proxy URL when S3 requests must pass through an HTTP(S) proxy or gateway. The proxy replaces the request host while preserving the original path and query. SigV4 signing happens before this rewrite, so the proxy must forward the signed request to the configured S3 endpoint and preserve its signed headers.

Required Capabilities and Permissions

Grant the S3 identity permissions equivalent to these operations for the configured bucket and prefix:

• ListBucket to check the connection and list objects, including paginated listings
• GetObject to download objects, read ranged object data, and retrieve object metadata for file stats
• PutObject to upload objects and folder marker objects
• DeleteObject to delete objects and clean up moved objects
• CopyObject access to copy the source object when a file is moved
• Multipart upload permissions for large streamed writes: initiate, upload parts, complete, and abort multipart uploads
• DeleteObjects for optimized batch deletion

Exact permission names vary between providers. A policy can scope object permissions to the configured prefix, but bucket listing permission must also allow listing that prefix.

Practical Behavior

• S3 has objects, not real directories. mkdir() creates a zero-byte object ending in / as a folder marker.
• A move uses S3 copy followed by delete because S3 has no native rename operation.
• Deleting a missing object is treated as success. Other service errors stop that operation.
• File stats use ETag when available. Without one, the module uses modification time and size as the file UID fallback.
• Asymmetric storage can flatten and anchor remote keys. Use it only when remote files do not need to remain readable in their normal folder structure, and keep the setting consistent across devices.

Implementation

Unified File-System Mapping

S3Fs implements the SDK RootFs contract with unified keys: / for root, folder/file.md for files, and folder/ for folders. Object keys are URL-encoded by path segment. The S3 module applies the shared prefixWrapper around S3Fs when a prefix is configured.

The basic operations map to S3 requests as follows:

File-system operation	S3 operation
read()	GET object
write()	PUT object
stat()	HEAD object
delete()	DELETE object
move()	CopyObject, then DELETE
mkdir()	PUT zero-byte folder marker
list()	ListObjectsV2

SigV4 Middleware

When S3 is the selected backend, registered request middleware wraps every S3 request with AWS Signature Version 4 authentication. It signs the endpoint host, method, canonical path, sorted query, and request headers using the configured access key ID, secret access key, region, and service name s3.

The implementation sends UNSIGNED-PAYLOAD for the payload hash so the request abstraction can pass binary bodies without buffering them for hashing. Custom headers present before signing are included in the signature.

Recursive Listing

list() uses S3 ListObjectsV2 with a prefix rather than making one request per directory. It follows NextContinuationToken until all pages are read, reports each returned key to the unified file-system reporter, and maps keys ending in / to folders. The requested directory entry itself is omitted from its child list.

Range Reads

readStream() works above the request abstraction with ranged GET requests. It requests 2 MiB chunks, keeps at most eight requests in flight, buffers at most eight chunks, and emits chunks in file order even when responses arrive out of order. Empty files return an already-closed stream.

Multipart Uploads

Every writeStream() uses S3 multipart upload. Parts are normally 5 MiB; the final part may be smaller:

1. Initiate a multipart upload.
2. Read 5 MiB parts and upload at most three parts concurrently.
3. Complete the upload in part-number order using each part's ETag.

The final part may be smaller than 5 MiB. On failure, active part uploads finish or settle, then the module attempts to abort the multipart upload.

Batch Deletes

The S3 optimizer groups delete atoms into DeleteObjects requests with at most 1000 keys each. Requests contain XML and a Content-MD5 header. The service result is handled per key: successful keys resolve, while keys reported in an S3 error result reject individually. If the whole batch request fails, every key in that batch fails.