Benchmarking

Benchmark Standards

The benchmark aims to obtain reliable and reproducible sync performance and algorithmic correctness statistics between Sync Engine and other syncing plugins. The Obsidian instance must have only the benchmarked plugin installed during a benchmark. And the benchmarked plugin must use the default settings except:

• account configuration
• UI confirmations that prevents a sync from happening non-interactively
• setting that would cause the final result have "errors" per the Correctness Validator (max size limits / don't sync specific files / ...)

Each benchmark contains fixed routines and test cases as specified below, any special deviation from the test routine must be documented clearly.

Benchmark Signature

To ensure reproducibility, each benchmark must contain following metadata:

• Benchmarked plugin
• Obsidian version
• Benchmarked plugin version
• Benchmarking machine operating system (due to benchmarking script assumption, Windows is not supported)
• Benchmarking machine Geekbench 7 CPU single-core score (can search from Geekbench website and estimate average)
• Tested remote backend type and service
• Average ping delay to tested service in milliseconds (use Ping Tester)
• Average upload and download speed of the tested service in MiB/s.

Apparent Statistics

Plugin size: Obtained from the GitHub release page of the corresponding version.

Transparent: Whether the author has made every controllable component transparent.

Startup speed:

On the benchmarking machine, kill Obsidian Electron APP completely and restart for 5 times, each time record down Obsidian settings -> General -> Advanced -> Notify if startup takes longer than expected -> Stopwatch icon -> benchmarked plugin startup time in milliseconds. Then take average over the 5 records.

Sync hidden files: Whether the plugin can (be configured to) sync arbitrary files in folders starting with .. For example, the .obsidian/ directory. (Obsidian vault.* API cannot see those files, only vault.adapter.* can)

Move detection: Whether the plugin can properly identify local and remote file moves instead of making them delete + upload / download.

Chunked transfer: Whether the plugin read / write / download / upload large files in stream or chunks, and does not buffer the whole file in memory. This feature determine whether it can transfer large files on mobile devices without crashing.

Metadata syncing: Whether the plugin can preserve file last modified time / creation time when syncing across devices.

Encryption: Whether the plugin supports client-side encryption.

Test Cases

Upload speed:

Initiate a deterministic test vault using Test Vault Bootstrapper.

Then start a non-interactive sync using the plugin, wait for it to finish, and obtain the plugin's total sync duration.

Download speed:

Reuse the already uploaded test vault above, create a new empty vault and configure the benchmarked plugin. Perform full sync to download those files from remote and record duration.

Daily sync speed

Now we should have two copies of the vault, one (original) is generated from Test Vault Bootstrapper, another one (replica) is newly downloaded from remote.

Perform deterministic sync-on-change using Operation Executor inside the original vault. The scripts contains 10 independent and sequential test cases covering create, modify, delete, and rename (move) operations to simulate small but frequent user edit. Each operation is gated with CLI command to continue. A sync is triggered every case in original vault first, then in replica vault. If the benchmarked plugin has "Sync on change" option, enabled it in original vault, otherwise trigger syncs in original manually.

When all cases finish, calculate the max, min, and median duration of the above 10 syncs.

Sync correctness:

Run Correctness Validator on both original and replica vault, sum up the total count of errors appeared in each vault. This stands for the stability and reliability of the plugin.

Benchmarking Utilities

The benchmarking utilities are located in scripts/benchmark/ in this repo. It is a CLI application majorly interacting with Obsidian CLI with five modules: Test Vault Bootstrapper, Operation Executor, Correctness Validator, and Ping Tester.

Test Vault Bootstrapper

File: scripts/benchmark/test-vault.ts

Initiate a deterministic test vault using scripts/benchmark/test-vault.ts. Clears the vault first, then populates the vault with a large vault simulation with randomly and deeply nested files; consists 1880 small files with sizes ranging from 50 bytes to 50 KiB, 100 medium files with sizes ranging from 500 KiB to 2 MiB, and 20 large files with sizes ranging from 15 MiB to 100 MiB.

Operation Executor

File: scripts/benchmark/daily-test.ts

Operation Executor executes defined command operations in series or in parallel, it follows a declarative syntax to iterate over an array of Commands:

type MaybePromise<T> = Promise<T> | T;
type Command =
	| string
	| { command: string; callback: (result: $.ShellPromise) => MaybePromise<void> };
type Commands = Array<Command | Array<Command>>;

All commands in a sub-array inside the command array are executed concurrently.

The commands are hooked to the running Obsidian instance via executing Obsidian CLI commands:

• obsidian create path=<path> content=<content> overwrite
• obsidian delete path=<path>
• obsidian move path=<path> to=<destination>

The Daily Test routine contains 10 test cases, executed in series to simulate small but frequent user edits. The benchmarked plugin is required to perform a sync for each test case. The duration of each sync is measured for evaluation.

Correctness Validator

File: scripts/benchmark/validator.ts

The validator is used to validate the sync correctness after the set of operations executed by Operation Executor and following syncs. It scans a vault and validates the existence of each file and folder, missing or extra ones all marked as error. For files that are appended in the test cases, it additionally validates the size of the file.

After validation, it outputs the total count of errors, and each error with its reason.

Ping Tester

File: scripts/benchmark/ping-test.ts

The Ping Tester tests average HTTP ping delay when trying to connect to a URL. It launches 20 sequential HTTP HEAD requests toward the target and record the response delay.

After all tests, it sorts the request delays and discard the longest two and shortest two delays. Then calculate the average for the rest 16 samples.