# Introduction

Ambient does automated [continuous integration][] for programmers in a safe and secure way.
That means it builds the software and runs its automated tests,
but prevents the software under test from harming the host it runs on.
Ambient achieves this by only running code from the software under test,
or any of its dependencies,
in a virtual machine without network access,
and only the resources configured by the Ambient user.


[continuous integration]: https://en.wikipedia.org/wiki/Continuous_integration


# Installation

See [`INSTALL.md`](https://app.radicle.xyz/nodes/radicle.liw.fi/rad:zwPaQSTBX8hktn22F6tHAZSFH2Fh/tree/INSTALL.md)
in the source tree.


# Getting started

## Built-in help

The `ambient` command has extensive builtin help: give the `-h` or `--help` option to the command,
or any subcommand,
or use `help` subcommand:

~~~
ambient -h
ambient --help
ambient run --help
ambient help
ambient help run
~~~

## Projects file {#projects}

To run CI on a project using Ambient, you need to create a "projects file".

~~~yaml
projects:
  dummy:
    image: /scratch/ambient-images/ambient-boot.qcow2
    source: ~/pers/ambient-ci/ambient-ci
    pre_plan:
      - action: cargo_fetch
    plan:
      - action: cargo_clippy
      - action: cargo_build
      - action: cargo_doc
      - action: cargo_test
~~~

The projects file, in YAML, specifies the projects Ambient should know about.
For each project:

* `image` is the virtual machine image to use
  - a custom image is published at [`files.liw.fi`](https://files.liw.fi/ambient/)
* `source` is the path to the source directory of the project
  - it can be, but does not need to be, a Git checkout
* `pre_plan`, `plan`, and `post_plan` are lists of actions to execute for a CI run
  - `pre_plan` actions are executed before the VM starts
  - `plan` actions are executed inside the running VM
  - `post_plan` actions are executed after the VM finishes

In each kind of plan, only specific actions are allowed.
For `pre_plan`, the following are allowed:

* `cargo_fetch` - download Rust crate dependencies for the software under test
* `http_get` - download specific files from URLs

For `plan`:

* `cargo_fmt`, `cargo_clippy`, `cargo_deny`, `cargo_doc`, `cargo_build`, `cargo_test`
  `cargo_install` - run the corresponding `cargo` command
* `deb` - build a `deb` package
* `custom` - run an executable action from `.ambient` in the root of the source tree

For `post_plan`:

* `dput` - upload built `deb` packages to an APT repository
* `rsync` - upload built artifact files to a server using `rsync`

For post-plan actions, the credentials of the user running `ambient` are used.

See the [actions](#actions) chapter for a complete list.

## Configuration

You should configure Ambient by creating `~/.config/ambient/config.yaml`
(the location obeys the [XDG base directory spec](https://specifications.freedesktop.org/basedir/latest/)):

~~~yaml
tmpdir: /scratch/tmp
projects: ~/liw-dot-files/ambient.yaml
target: "_ewww@webby:/srv/http"
dput_target: apt.liw.fi
executor: /scratch/cargo-cache/x86_64-unknown-linux-musl/debug/ambient-execute-plan
artifacts_max_size: 1G
cache_max_size: 50G
state: /scratch/ambient-state
qemu:
  cpus: 8
  memory: 16G
~~~

The fields are:

* `tmpdir` - location where Ambient temporary files are put
  - these can get quite large, so it can be necessary to override the default `$TMPDIR`
* `projects` - path to the [projects file](#projects)
* `target` - where the `rsync` action uploads files over SSH
* `dput_target` - where `deb` packages are upload using `dput`
* `executor` - the `ambient-execute-plan` program suitable for the VM used
* `artifacts_max_size`, `cache_max_size` - how big the artifacts and cache directories can grow
* `state` - location where Ambient keeps pre-project state
* `qemu.cpus`, `qemu.memory` - number of CPUs and memory to give to the VM

Some of those are optional, as they have defaults.
To see the actual run-time configuration used by `ambient`, run:

~~~
ambient config
~~~

You can specify the configuration file to use with the `--config` option.

## Run Ambient

Given an image and a [projects file](#projects), run Ambient using a command like this:

~~~
ambient run 
~~~

This runs CI for every project in the projects file, if the project source code has changed.
Add the `--force` option to force CI to run.

Any output to the standard output and error from any of the actions is written to the standard output
and gathered in a "run log".
The run log can be viewed later with

~~~
ambient log dummy
~~~

The run log is, for now, quite messy.


# Actions for use in Ambient CI plans {#actions}

An action is an atomic task executed during a CI run. Actions are specified in
the CI plan. The plan is divided into a pre-plan, actual plan, and post-plan.
The pre- and post-plan actions are executed with network access and the actual
plan is executed in an isolated virtual machine with no network access. The
actions allowed in pre- and post-plan are carefully designed and vetted so
that they are safe and secure to run. Most importantly, they do not execute
any code from the software under test.

## The workspace in the virtual machine

Ambient sets up a workspace where CI is run. Until version 0.10, this was
`/workspace`, but after that, it is also known as `/ci`, for brevity. The old
name also works to avoid breaking existing CI plans.

The workspace contains several directories:

* `/ci/src` - the source code; this is the contents of the `source` directory
  in the project specification; changes in the source directory are not
  persisted and vanish when the VM shuts down
* `/ci/dependencies` - dependencies downloaded by pre-plan actions;
  changes in the dependencies directory are not persisted and vanish when the
  VM shuts down
* `/ci/cache` - a cache that persists between CI runs
* `/ci/artifacts` - files built by CI, for publication by post-plan actions

The `cache` and `artifacts` directories have a maximum size. When the CI
run ends, the contents of those directories are written as tar archives
to the corresponding virtual block devices set up by Ambient. This is what
will fail if the directories are too large. See [`artifacts_max_size` and
`cache_max_size`](#configuration) in the configuration file.

You mostly only need to care about the workspace and its layout in the
[`shell` action](#shell). All other actions know what's where, but your
shell script snippets need to use the right locations. Also any [custom
action](#custom) executables need to do that.

Ambient adds actions to unpack the source and other directories at the start
of a CI run, and exporting the cache and artifacts directories at the end. You
don't need to do that manually.


## Pre-plan actions

The following actions are allowed in the pre-plan.

### `dummy`

~~~yaml
- action: dummy
~~~

Do nothing, except write a message to the standard output. This action is
meant for trouble-shooting pre-plans.

### `pwd`

~~~yaml
- action: pwd
~~~

Write out the path to the current working directory. This action is meant
for trouble-shooting.

### `cargo_fetch`

~~~yaml
- action: cargo_fetch
~~~

Download Rust crate dependencies using `cargo fetch`. The downloaded crates
will be available in the VM in a location where `cargo` will find them. The
downloaded crates will be cached so that they do not need to be downloaded
again for the next CI run.

For safety and security, `cargo fetch` is run on a stripped down copy of
the source tree. This prevents the software under test from reconfiguring
`cargo` using files in the source tree. This does not affect what crates are
downloaded.

The crates will be in `/ci/deps` and the `CARGO_HOME` environment variable is
set to that location.

### `http_get`

~~~yaml
- action: http_get
  items:
  - url: https://files.example.com/big.xz
    filename: data.xz
  - url: https://files.example.com/cat.jpg
    filename: cat.jpg
~~~

Download files over HTTP if they are missing locally or have changed on the
server. The downloaded files are available in the dependencies directory in
the VM, `/ci/deps`. If the file exists locally, the download uses the HTTP
header `If-Modified-Since` to avoid downloading it again, unless the file on
the server has a newer modification time.

The downloads are done using HTTP GET. There is no support for authentication.
There is no automatic cleaning of downloaded files.

Both the `url` and `filename` fields are required. The filename may not contain
a directory.

## Actual plan actions

These are also known as "unsafe actions". They're allowed to be dangerous
and to run code from the software under test, because they get executed in
a virtual machine that isolates and constrains the code so it can't harm the
host running Ambient, or any other hosts.

### `mkdir`

~~~yaml
- action: mkdir
  pathname: /ci/src
~~~

Ensure a directory exists. If it already does, do nothing, otherwise create
it. This is meant for Ambient internal use, to create the CI workspace when a
CI run starts. User-provided plans can also use this action.

### `tar_create`

~~~yaml
- action: tar_create
  archive: /ci/artifacts/rsync/source.tar
  directory: /ci/src
~~~

Create a [`tar` archive][] with the contents of a directory. This is meant for
Ambient internal use to export data from the virtual machine via virtual block
devices, i.e., the cache and artifacts directories. User-provided plans can
also use this action.

[`tar` archive]: https://en.wikipedia.org/wiki/Tar_(computing)

### `tar_extract`

~~~yaml
- action: tar_extract
  archive: /dev/vdx
  directory: /ci/yummy
~~~

Extract a [`tar` archive][] to a directory. This is meant for Ambient internal
use to import data into the virtual machine via virtual block devices, e.g.,
the source, dependencies, and cache directories. User-provided plans can also
use this action.

### `shell`

~~~yaml
- action: shell
  shell: |
    echo hello, world
    make
~~~

Execute a shell script snippet using `bash`. The Bash setting `set -xeuo
pipefail` will be in effect. This means that if the snippet uses a variable
that hasn't been set, or a command fails, the action fails.

### `cargo_fmt`

~~~yaml
- action: cargo_fmt
~~~

Check that Rust source code is formatted in the idiomatic style, by running
`cargo fmt --check`.

### `cargo_clippy`

~~~yaml
- action: cargo_clippy
~~~

Check that Rust code is correct and idiomatic by running `cargo clippy`. Any
warnings are treated as errors.

### `cargo_deny`

~~~yaml
- action: cargo_deny
~~~

Check that Rust code only has dependencies, licenses, and known security
problems that are explicitly allowed, by running `cargo deny`.

### `cargo_doc`

~~~yaml
- action: cargo_doc
~~~

Format documentation from Rust code, using `cargo doc`. The formatted
documentation is put in the `cargo` target directory, `CARGO_TARGET`, which
is in the cache directory. It is not automatically put into the artifacts
directory. You have to do that yourself, if you want it.

### `cargo_build`

~~~yaml
- action: cargo_build
~~~

Build a Rust project, including binaries, tests, and examples ("all targets").
This runs `cargo build` with suitable options.

### `cargo_test`

~~~yaml
- action: cargo_test
~~~

Run tests in a Rust project, using `cargo test`.

### `cargo_install`

~~~yaml
- action: cargo_install
~~~

Install a Rust project into the artifacts directory, `/ci/artifacts`.

### `deb`

~~~yaml
- action: deb
~~~

Build a Debian `deb` package. This first creates an "upstream tar" from the
source tree, using `git archive`, and then running `dpkg-buildpackage`, the
standard tool for building Debian packages. The built files, including the
`.changes` file, get put into the artifacts directory, `/ci/artifacts`.

Note that this assumes the source is in Git, that Debian packaging tools are
installed in the VM, and that the `debian` directory includes the necessary
files for Debian packaging.

Also note that this action does not invent a version number for CI builds.

This action is meant to be used together with the [`dput`](#dput) post-plan
action.

This action differs from the [`deb2` action](#deb2) only in the location of
where built packages are put.

### `deb2`

~~~yaml
- action: deb2
~~~

Build a Debian `deb` package. This first creates an "upstream tar" from the
source tree, using `git archive`, and then running `dpkg-buildpackage`, the
standard tool for building Debian packages. The built files, including the
`.changes` file, get put into `debian` directory in the artifacts directory,
`/ci/artifacts/debian`.

Note that this assumes the source is in Git, that Debian packaging tools are
installed in the VM, and that the `debian` directory includes the necessary
files for Debian packaging.

Also note that this action does not invent a version number for CI builds.

This action is meant to be used together with the [`dput2`](#dput2) post-plan
action.

This action differs from the [`deb` action](#deb) only in the location of
where built packages are put.

### `custom`

~~~yaml
- action: custom
  name: dch
  args:
    debemail: liw@liw.fi
    debfullname: "Lars Wirzenius"
~~~

Execute a "custom action". Custom actions are executables (usually shell
scripts, but any executable is OK), located in the `.ambient` directory at the
root of the source tree. The `custom` action runs the executable and passes it
arguments as specified in the `args` field. The whole `args` field is encoded
as JSON and given to the executable via its standard input. In addition, each
argument `name` is separately encoded as JSON and an environment variable
`AMBIENT_CI_name` is set to contain the JSON. The executable can use either,
whichever is more convenient to it.

Note that Ambient does not try to retrieve the custom action executables
from anywhere. The project that uses the `custom` action has to include
the executable in their source tree. This avoids making Ambient do package
management for the executables, but make using them more tedious. The `custom`
action is best considered a compromise while the Ambient project explores
the needs and wants for custom actions. The Ambient project maintains a small
[repository of custom actions][] as part of the compromise.

[repository of custom actions]: https://app.radicle.xyz/nodes/radicle.liw.fi/rad%3Az1LPp3eFHEwG7sZrnc447W1aC5Z

Example custom action script to add a Debian package changelog entry with a
new version suitable for a CI build:

~~~sh
#!/usr/bin/env bash

set -euo pipefail

export DEBEMAIL="$AMBIENT_CI_debemail"
export DEBFULLNAME="$AMBIENT_CI_debfullname"
export CARGO_TARGET_DIR=/workspace/cache
export CARGO_HOME=/workspace/deps
export HOME=/root
export PATH="/root/.cargo/bin:$PATH"

git reset --hard
git clean -fdx

V="$(dpkg-parsechangelog -SVersion | sed 's/-[^-]*$//')"
T="$(date -u "+%Y%m%dT%H%M%S")"
version="$V.ci$T-1"
dch -v "$version" "CI build under Ambient."
dch -r ''
~~~

(The example comes from the repository of custom actions, script `dch`.)

### `setenv`

~~~yaml
- action: setenv
  set:
    foo: bar
~~~

Set environment variables for later actions.

## Post-plan actions

### `dummy`

This is the same action as in the pre-plan.

### `pwd`

This is the same action as in the pre-plan.

### `rsync`

~~~yaml
- action: rsync
~~~

Publish the contents of the entire artifacts directory (`/ci/artifacts` in
the VM) using the `rsync` program. The sync target is specified in Ambient
configuration (see [`rsync_target`](#configuration)); the software under test cannot affect
that. The SSH credentials for the host running Ambient are used by `rsync`.
Any files on the target that are not in the artifacts directory are deleted.

### `rsync2`

~~~yaml
- action: rsync2
~~~

Publish the contents of the `rsync` subdirectory of the artifacts directory
(`/ci/artifacts/rsync` in the VM) using the `rsync` program. The sync target
is specified in Ambient configuration (see [`rsync_target`](#configuration)); the software under
test cannot affect that. The SSH credentials for the host running Ambient
are used by `rsync`. Any files on the target that are not in the artifacts
directory are deleted.

The difference between the `rsync` and `rsync2` actions is the location of
the files to be published. The `rsync` action publishes the entire artifacts
directory, which may also contain packages built by the `deb` action. The
`rsync2` action only publishes the subdirectory, which contains no packages.

### `dput`

~~~yaml
- action: dput
~~~

Upload Debian `deb` packages from the artifacts directory to an APT
repository, using the `dput` program on the host and the SSH credentials of
the host. The packages are found by looking for `.changes` files anywhere inside
`/ci/artifacts`, including subdirectories. The [`deb` action](#deb) puts them
in `/ci/artifacts`.

### `dput2`

~~~yaml
- action: dput2
~~~

Upload Debian `deb` packages from the artifacts directory to an APT
repository, using the `dput` program on the host and the SSH credentials of
the host. The packages are found by looking for `.changes` files anywhere
inside `/ci/artifacts/debian`, including subdirectories. The [`deb2`
action](#deb2) puts them there.

