
- testdata encryption password: `test`

- <https://www.arqbackup.com/documentation/arq7/English.lproj/dataFormat.html> misses to mention that the derived key needs to be split into an HMAC and Encryption key. <https://www.arqbackup.com/docs/arqcloudbackup/English.lproj/dataFormat.html> describes that and I just gave it a try.

- Encryption is done with AES/CBC and PKCS7 padding
  - 32bit keys
  - encryption key (32bit) and hmac key (32bit) are derived via PBKDF2-SHA256 with 200k rounds from the passphrase
- Compression is done with LZ4 block format, with a 4-byte big-endian length

- "relative paths", e.g. in backup records:
  - are not really relative, as they start with `/`
  - they do include the plan UUID, so it's the full path starting with storage location

```terminal
$ tree testdata/target/D7E09473-74BF-4E6D-86C3-43A033570D80/
testdata/target/D7E09473-74BF-4E6D-86C3-43A033570D80/
├── backupconfig.json
├── backupfolders
│   └── DD994AEF-F9CB-4D39-B12D-1EF5305103C8
│       ├── backupfolder.json
│       └── backuprecords
│           └── 00176
│               └── 3107319.backuprecord
├── backupfolders.json
├── backupplan.json
├── blobpacks
│   └── 1C
│       └── F3DB04-19D7-44EA-8F8A-3C6097E30B22.pack
├── encryptedkeyset.dat
├── keyset_history
│   └── encryptedkeyset_1763107296.dat
├── temp
└── treepacks
    └── 32
        └── BFADB4-9AD4-4991-8AFE-FAFFCCEB11F2.pack
```

- `backupconfig.json` (NOT encrypted, NOT compressed, JSON)
- `backupplan.json` (encrypted, not compressed, JSON)
  - only indirectly documented, via `backupfolders/<UUID>/backuprecords/00161/4294169.backuprecord`
  - for every selected folder in the backup plan, there is a "backup folder" with a UUID under `backupfolders/`
  - `backupFolderPlansByUUID` is an JSON object, where the keys are the UUIDs of the the selected folders.
  - this gives us a list of `backupfolder` UUIDs
- `backupfolders.json` (NOT encrypted, NOT compressed, JSON)
  - ???
- `backupfolders/<UUID>/backupfolder.json`
  - some data about the backup folder. Not sure what this is used for exactly.
- A backup in Arq creates a `backuprecord`, e.g `backuprecords/00176/3107319.backuprecord` (encrypted, compressed, JSON)
  - contains a copy of the backup plan and other meta data
  - `node`, and `node.treeBlobLoc` contains the "entrypoint" into the tree (directories and files)
  - from there we can parse the data and traverse through the tree, parsing `node` objects, loading more data from `treepacks` or `blobpacks`.
  - data for files will be in "blobs", referenced through `BlobLoc` in "blobpacks"


## TIL

- `mac_st` probably stands for macOS `struct stat`
  - https://developer.apple.com/library/archive/documentation/System/Conceptual/ManPages_iPhoneOS/man2/stat.2.html TODO:, find better link
  - https://stackoverflow.com/questions/54263830/what-is-the-meaning-of-each-field-with-the-output-of-stat-on-osx
  - from the docs

    ```plain
    Int32:mac_st_dev: device id
    UInt64:mac_st_ino: inode number
    UInt32:mac_st_mode: file mode
    UInt32:mac_st_nlink: hard-link count
    UInt32:mac_st_uid: owner user id
    UInt32:mac_st_gid: owner group id
    Int32:mac_st_rdev: device type?
    Int32:mac_st_flags: see chflags
    ```



## Concepts

- "storage location": The location for one or more "backup plans"
- "backup plan": Configuration, listed under "back up" in the UI
- "backup folder": >=1 per plan
- "backup record": the "actual" backup, per folder

## CLI

- Point to storage location and provide plan UUID; or point to plan (storage location + UUID)

## Features

### Now

- recommend to use rclone for remote storage locations
  - Hetzner StorageBox allows up to 10 connections (https://docs.hetzner.com/storage/storage-box/access/access-ssh-rsync-borg/#rclone)
    - so you might want to use `--sftp-connections=10` and/or limit `--checkers` and `--transfers`.
    - somehow that's very slow, something is off, retrying or so; `rclone serve nfs :sftp:/home/tisba/arq --sftp-host=u423995.your-storagebox.de --sftp-port=23 --sftp-user=u423995 --read-only --sftp-connections=8 --sftp-disable-hashcheck --sftp-shell-type=none -vv`
      - might be an issue with NFS? `rclone serve s3` seems to be quite okay performance wise
  - `rclone serve nfs :sftp:/home/tisba/arq/06DEFBFC-907F-4918-8F7B-A21D6E56DEF1 --sftp-host=u423995.your-storagebox.de --sftp-port=23 --sftp-user=u423995  --read-only`
  - `mount -t nfs -r -o port=61549,mountport=61549,tcp 127.0.0.1:/ /Users/basti/Documents/src/arq-restore/tmp/remote`
  

### Later

- direct support for other storage providers
  - via opendal
- Add support for exclude patterns (maybe based on https://docs.rs/glob/latest/glob/struct.Pattern.html?)
