## Sync

Synchronization between devices uses an eventually consistent strategy. Applications can choose how to deal with conflicts, however, due to the nature of the folder event logs and the ability to go back in time a merge on conflict with a last-write wins approach is acceptable.

Synchronization is achieved via an untrusted intermediary server.

## Transport Security

Even though all data is encrypted on the client before being sent over the network, servers **must protect** the data in transit to prevent against MitM attacks that could be used to replay requests and alter the server state of an account.

For development it is convenient to use HTTP rather than configure certificates for TLS however in a production environment servers **must** secure connections with TLS.

## Authentication

Authentication is performed using a `Bearer` token in an `Authorization` HTTP header.

The token takes the format of a base58 encoded signature generated by the accout signing key. If a device signature is required a base58 encoded signature generated by a trusted device signing key is concantenated to the account signature delimited by a period.

The data to be signed for each request is either the request body or if no request body exists (eg: GET/DELETE requests) the path of the URL (without any query parameters) must be signed.

### Examples

Account signature:

```
Bearer 9bxSv8J4V7ebN9cjUiHrgusZS1666iNXkDawQto9Y2X8Rf4tdNGgrgDUwyiG8RZ7P2uRmq2EJzzxDZK7wavywM4s
```

Account and device signature:

```
Bearer 2WE8VSRm5PiVzCNpULMkcRf9UviqRZyVMdC1JuHsCkUWQ9hncjBS7Ym8WejBYWMb3i9ft3Sd4dQQb868gMizEdgUQ.4ZaqcJjzZxmyUSBKFxfvBQpKnJwDxseCcBMKrbuUrE57tNg6CxKzwTdNh5omrxwazEg753gjZf4mceHQLjr73gvt
```

If an implementation wants to add an additional token (eg: JWT, PASETO etc) for permissioned access it **must** be base58 encoded and prepended to the `Bearer` token and delimited from the signature(s) by a colon.

## Reference Implementation

The [sos-net](/workspace/net) crate provides a client and server reference implementation; pre-built binaries can be [downloaded from our website](https://saveoursecrets.com/command-line-tools/).

To start a server first initialize a configuration file:

```
sos-server init config.toml     # create config.toml file
sos-server start config.toml    # start with config.toml file
```

The server API is documented using Open API, to view the endpoints as JSON navigate to `/api/v1/docs/openapi.json` for a web interface navigate to `/api/v1/docs`.

The server is suitable to be hosted on a LAN and is permissionless so should **not be exposed to the internet**, configuring a network for self-hosting is beyond the scope of this document and will vary depending upon the network.

### Configuration

### Storage

Set the storage path to determine the root directory where the server will store data:

```toml
[storage]
path = "sandbox/accounts"
```

Relative paths are resolved from the current working directory when the server is started.

The server will create top-level folders:

```
sandbox/accounts
├── identity
├── logs
└── remote
```

* `identity`: Account identity folders.
* `logs`: Log files.
* `remote`: Account data.

### Access Controls

It is **strongly recommended** to use the `allow` and `deny` access controls to determine which accounts are allowed to store data otherwise your server may be abused to store data on behalf of unknown connections.

When an `allow` configuration is present then only those addresses are allowed to connect, all other addresses will be denied.

```toml
[access]
allow = [
  "0x3ebe1c7c8e56a1e9b813073e30caf1a0cd8e7634"
]
```

If a `deny` configuration is present then any address other than the explicitly denied addresses can connect:

```toml
[access]
deny = [
  "0x7ebe1c7c89e56a1e9b813073e30caf1a0cd8e5541"
]
```

## Device Pairing

Adding a new device to an existing account consists of a device that is already authenticated to the account which we call the **offering device** and another device which is not authenticated called the **accepting device**.

Pairing the devices is performed using an untrusted relay server. Communication between the devices exposes sensitive information (the account signing key) which must not be visible to the server so device pairing uses the [noise protocol](https://noiseprotocol.org/) to ensure the communication between the devices is private. The protocol includes a pre-shared symmetric key in the **pairing URL** so that the server cannot forge client connections.

The **pairing URL** needs to be transferred between the offering device and the accepting device; this can be done by scanning a QR code or typing in the URL. The **pairing URL** uses the `data:` scheme to differentiate from the HTTP/S schemes.

### Pairing Protocol

1) Offering device generates a noise protocol session keypair and encodes the relay server URL, symmetric pre-shared key and noise protocol public key in a **pairing URL** that must be sent to the accepting device.
2) Once the **pairing URL** has been received by the accepting device it begins the noise protocol handshake.
3) After completing the noise protocol handshake the accepting device encrypts and sends a **trusted device** to the offering side. The **trusted device** contains meta data about the device and the public key of the device's signing key.
4) When the offering device receives the **trusted device** it updates the server(s) to trust the new device and sends the encrypted account signing key in reply.
5) The pairing protocol is complete when the accepting device receives the account signing key.

### Device Enrollment

Once the pairing protocol is finished the accepting device can perform device enrollment as it has both the account signing key and a device signing key whose public key has been added as a trusted device to the server(s).

1) Fetch account event logs and write the account to disc.
2) Finish device enrollment by authenticating to the account using the primary password.

## Device Revocation

If a device is lost or stolen the device can be revoked which will remove the device from the list of trusted devices preventing it from syncing with servers.

Whilst server endpoints will return a **forbidden** response for untrusted device signatures; if the lost or stolen device was unlocked and the account was authenticated the owner must consider all of their secrets compromised.

## Event Logs

Several events logs are stored on both the client and server so that complete deterministic, incremental synchronization is possible for an account.

1) Application event log tracks changes to accounts. [^1]
2) Account event log tracks changes to folders.
3) Device event log tracks trusted devices.
4) Folder event log tracks changes to secrets in a folder.
5) File event log tracks changes to external files.

It is important to note that the folder event log stores secret data in the same encrypted format as a vault and is therefore inaccessible to the untrusted server.

The account and file logs **must not be compacted**.

## Divergence 

Event logs can only diverge under two well-defined scenarios that rewrite the event history.

1) Folder event log was compacted.
2) Folder password was changed.

## Conflict

Conflicts can happen when two devices write to the same folder whilst both devices are offline and unable to sync. [^2]

## Identification

An account is identified by the [account address](/doc/overview.md#account-address) derived from the [account signing key](/doc/overview.md#signing-key).

Client requests that access an account **must include a valid signature** and the server **must use the address of the public key from the signature** to identify the account.

Signatures prove account ownership as the account signing key is protected by the [identity vault](/doc/overview.md#identity-vault).

## Devices

Devices are represented by Ed25519 signing keys and handled differently by clients and servers.

Clients store the device signing key in a [device vault](/doc/overview.md#device-vault) and may cache meta data about trusted devices in the vault so that applications can show information about a device (hardware, operating system etc).

The [device vault](/doc/overview.md#device-vault) is **not included in synchronization** and should **never leave the device**.

When a client sends account vaults to create an account on a server it **must include** the public key of the device creating the account and the server **must trust** the device. 

### Trusting Devices

To add trusted devices the account owner can share the account [signing key](/doc/overview.md#signing-key) via a QR code (or hex-encoded string) which will allow the device to communicate with the server and add it's own public key and device meta data as a trusted device. 

Once the server has established that the device is trusted it is able to retrieve the [identity vault](/doc/overview.md#identity-vault) and account folders to perform an initial synchronization. The account owner can then provide the [primary password](/doc/overview.md#primary-password) to sign in to the account on the new device.

The server API endpoint for trusting devices **must only require a signature from the account signing key** for authentication.

### Revoking Devices

If a device has been lost or stolen an account owner can revoke the public key for the device so it is no longer trusted and will not be allowed to communicate with server endpoints that require a signature from a device.

The server API endpoint for revoking devices **must require signatures from both the account signing key and a trusted device** for authentication.

## Protocol

Let's review some terminology first.

* Let **hash** be an SHA2-256 hash
* Let a **commit tree** be a merkle tree that tracks the last leaf hash
* Let **head** be the last leaf hash
* Let **tip** be the merkle proof of **head**
* Let **commit state** be a combined **head** and **tip**
* Let **event log** own a **commit tree**
* Let **patch** be a collections of events
* Let **change set** be a collection of patches used to initialize an account
* Let **diff** contain a **patch** and the **commit state** before and after the patch was applied
* Let **sync status** include the **commit state** of every event log in an account
* Let **sync diff**  include a **diff** of every event log in an account

The terms **local** and **remote** vary by context, for a client **local** is itself and **remote** refers to a server. For a server, **local** is itself and **remote** is the client.

The behavior of clients and servers will differ slightly as they store data differently. For example, servers store vault files as head-only as they only need to prioritize storing the event logs. Whereas clients need full on-disc vaults for quick access to the secret data.

---

Client requests the **sync status** from a remote server as **remote status**.

If a **sync status** cannot be retrieved from the remote server because the account does not exist then the client should send a **change set** to the server to initialize a new account.

When a remote **sync status** is returned the client can proceed to synchronize:

1. Client compares their **local status** to the **remote status** and generates a **sync diff** including all the events that exist on local but not on remote.
2. Client sends it's **sync status** and the **sync diff** to the remote server.
3. Server receives the **sync diff** and merges the changes in each **diff** into the corresponding event logs. Merges **must be checked** such that the patch is only applied if the tip of the event log matches the before proof in the diff. For some types of events the server may need to **replay** the events such that the on disc (and in-memory) representation is correct before replying to the client. In particular, for account level events the server will need to create, update or delete folders. Replaying events **must update** the corresponding event log(s) so that the merkle tree on the server exactly matches the client.
4. Server can now compare it's updated **local status** to the **remote status** (sent by the client earlier) and generate a **sync diff** of events that exist on the server that the client has not yet received. Server replies to the client with it's updated **sync status** and the **sync diff**.
5. Client receives the updated **remote status** and the **sync diff** and merges the diff into it's local storage.

[^1]: The application event log when implemented will allow account deletion to be synchronized with a server.
[^2]: Document the strategy for resolving conflicts.
