Client-side Encryption v2

The plugin uploads encrypted files to remote, download and decrypt back to local. This document specifies the encryption algorithm implementation in the Encryption module.

Encryption implementation in this module welcomes volunteer auditing.

File Content

File Key

File Name

Master Salt

Root File Key

Name Key

User Password

Master Key

File Key Salt

Chunk Count

File Size

16 Byte Random File Salt

Account Name

Server Endpoint

Terminology

Notations

• IV = Initialization Vector = Nonce
• B = Byte
• b = bit
• 1KiB = 1024B
• || = concatenate

Threat Model & Constraints

The implementation trusts:

• Clients share password outside of the module. And the share channel is out of scope so the module doesn't need to implement its own key exchange.
• The operating system or Obsidian runtime are not compromised.

The implementation does not trust:

• Backend server
• Transmission layer

The implementation protects against:

• Unauthorized access
• Unauthorized modification or file truncation
• Encryption combined with Asymmetric Storage can yield better obfuscation since it flattens hierarchical structure and randomizes file basenames.

The implementation cannot:

• Prevent file rollback to a previous version
• Prevent unintended renaming or movement
• Prevent deletion at server side
• Obfuscate file size or modification time
• Obfuscate total file number
• Obfuscate remote root directory name

Algorithms

• SHA256: generate deterministic and unique salt from user info to obtain master key.
• Argon2id: generate master key from salt and user password, used every user info / password change.
• HKDF-SHA-256: derive root file key and file key from master key, and derive file key from root file key and file salt, used every sync.
• CSPRNG: (crypto.getRandomValues()) generate random file salt, used every file encryption
• AES-GCM-256: main algorithm used to encrypt file content, used every file encryption
• AES-GCM-SIV-256: algorithm used to encrypt file / folder names, used every file encryption
• Base64URL: transform binary data into ASCII for storage / display

Keys & data structure

• user password: the raw string user password
• auth tag: 16B, generated by AES-GCM-256 to verify data integrity
• master salt: 16B, for salting user password to get master key, generated from SHA256 on user info
• master key: 32B, deterministic strong key generated by Argon2id from user password and master salt
• name key: 32B, derived from master key used to encrypt file names
• root file key: 32B, derived from master key used to generate file key for each file
• file key: 32B, derived from root file key and random file salt
• file salt: 16B, pure random, to derive file key each time to encrypt or decrypt a file
• file key salt: 32B, SHA256 of file salt || encrypted file size in 8B
• chunk nonce: 12B, chunk index represented in 12B.
• encrypted file / folder name: Base64URL of AES-GCM-SIV encrypted file name
• encrypted file / folder path: encrypted file / folder name of all ancestor hierarchies joint by /.
• file chunk: 131,088B, 128KiB-chunked and AES-GCM-256 encrypted piece of file (end chunk can be smaller)
• encrypted file content: file salt || all chunks sequentially

Enabling and Disabling

The function is toggled via the Encryption setting. The password is stored in Obsidian's keychain. When a record store exists, changing the toggle opens a migration confirmation, which is similar to the migration of asymmetric storage.

Sync Routine

On each sync, generate a promise to obtain root file key and name key that will be resolved and cached on demand:

• starting check: if found the user password is an empty string or undefined, throw directly
• generate master salt: SHA256 of <server URL>.<account name>.<remote base directory> truncated to 16B.
• generate master key using Argon2id with 32MB memory, 3 iterations, and 1 thread on the user password with master salt
• derive root file key (info: root-file-key-v1, no salt) and name key (info: name-key-v1, no salt) using HKDF-SHA-256
• resolve with root file key and name key

Then come to the traversal and syncing logic:

• encryption should be isolated at the end site, directly in the push task and mkdir task
• decryption should happen immediately when the encrypted file touches local machine, directly during remote traversal, pull task, and getRemoteContent.
• assume all files are encrypted when "Encryption" is enabled, assume all plain when not enabled.

File / Folder Path Encryption

• All paths should be represented in UTF-8 and normalized to Unicode NFC
• Cascade and encrypt the whole path chain, for each file name (should be a standalone name like bar, foo.md):
  • use AES-GCM-SIV-256 to encrypt file name with key: name key, content: file name, nonce: fixed at UTF-8 of file-name-v1 in bytes.
  • obtain the final name using Base64URL on the result yielded above
• For example, when encrypting the full path of foo/bar/a.md, first encrypt foo, then bar folder, and finally a.md.
• Use a global in-memory cache to accelerate file path encryption / decryption for identical paths, limited to 10K entries.
• Do not encrypt user's remote root directory.

File Encryption

• Input raw file and raw file size
• Generate random 16B file salt via CSPRNG
• Calculate encrypted file size as raw size + 16B + ceil(raw size / 128KiB) * 16B
• Calculate file key salt as SHA256 of file salt || encrypted file size in 8B
• Calculate file key using HKDF-SHA-256 of root file key (salt: file key salt, info: file-key-v1).
• Splice the file content into 128KiB chunks, special size for the last chunk, for each chunk:
  • get chunk index starting from 0 represented in 12B as chunk nonce
  • encrypt the 128KiB with file key and chunk nonce using AES-GCM-256
  • each chunk = ciphertext (128KiB = 131,072B) + auth tag (16B) = 131,088B
• Concatenate the file as: file salt || all chunks sequentially

One-pass Decryption

• Input encrypted file and encrypted size in bytes
• Splice the first 16B as file salt
• Calculate file key salt as SHA256 of file salt || encrypted file size in 8B
• Calculate file key using HKDF-SHA-256 of root file key (salt: file key salt, info: file-key-v1).
• Continue to splice into 131,088B pieces (exception for the last chunk), for each piece:
  • count chunk index as chunk nonce
  • decrypt the chunk with chunk nonce + file key with AES-GCM-256, throw data corrupted or wrong password and skip the file if auth tag mismatch
• Concatenate decrypted content

Streamed Decryption

Streamed decryption receives a ReadableStream, generates another ReadableStream, and pipe decrypts it between streams.

• Input the encrypted size in bytes, and accept a sequential binary stream of the file content
• Split and concatenate the chunk internally
• For example, if the first received binary is 2,000,000B in size, the range decrypter:
  • strips first 16B as file salt
  • add up counter for each chunk and strip next 1966380B as 15 completed chunks
  • save the last 33604B as an incomplete chunk and save to a buffer
  • decrypt the 15 completed chunks like one-pass decryption
  • concatenate content, and pipe to new stream.
  • when next binary stream chunk arrives, it concatenates the content in the buffer with the first certain size of bytes in the new binary as the first completed chunk.
  • repeat until the original stream finishes, the class treats the rest content in its buffer as a completed chunk, decrypt directly, pipe to the new stream, then end it.

Streamed Encryption

Streamed encryption follows symmetrical path with streamed decryption.

• Input the unencrypted size in bytes and a ReadableStream of file content.
• Calculate the encrypted size as raw size + 16B + ceil(raw size / 128KiB) * 16B
• Calculate file key salt as SHA256 of file salt || encrypted file size in 8B
• Calculate file key using HKDF-SHA-256 of root file key (salt: file key salt, info: file-key-v1 constant string).
• Creates a TransformStream above the original stream.
• Constantly reads the stream, encrypts and maintains a buffer of raw bytes: if the buffer < 128KiB after reserving a stream chunk, continue reading; if larger than 128KiB, encrypt the max number of 128KiB chunks in the buffer and emit. When stream ends, encrypt and emit the rest in the buffer.
• The first emitted should have file salt prepended.

File / Folder Name Decryption

• Input file name and Base64URL decode to raw bytes
• Decrypt the rest of the file name using AES-GCM-SIV-256 with name key and nonce (use the cache if possible)

Implementation

The implementation should only use:

• Web Crypto
• argon2id export from hash-wasm
• gcmsiv export from @noble/ciphers/aes.js

The encryption function will be cleanly integrated inside the encryption package as a Sync Engine optional module, which ships a RemoteFsWrapper:

Receives the memory database instance and user password in the second argument.

getUid(), checkConnection(), options: keep as-is.

read(): encrypt key before relaying to original, and decrypt when original returns.

readStream(): encrypt key before relaying to original, creates a new transformStream that relays the original stream.

write(): encrypt the key and the content before relying to original.

delete(), mkdir(), stat(): encrypt the key before relying to original.

stat(): encrypt the key before relaying, decrypt the key in Stat when original returns.

list(): encrypt the key before relaying, decrypt the key in Array<Stat> when original returns.

Path segments and derived keys are cached, persistent beyond sync runs. The cache is based on Uni-KV memoryDB obtained from context, similar to Context Wrapper, the cache is reset only when signals changed:

• Store meta: EncryptionDBMeta
• Storage schema: EncryptionDBSchema
• Scope: decryptedToEncrypted and encryptedToDecrypted stores

Behavior:

• memoryDB from context is of different type, cast it to MemoryDatabase<EncryptionDBSchema, EncryptionDBMeta>.
• lastEncryptionUid = RemoteFs.getUid() || ~ || user password
• Cached derived keys are stared in store meta encryptionKeys field
• Only once when the wrapper is activated: check if store meta lastEncryptionUid is aligned with the current one. If not, clear stores and encryptionKeys, and update the meta to the current uid.