# procfarm

[![builds.sr.ht status](https://builds.sr.ht/~mplanchard/procfarm/commits/main.svg)](https://builds.sr.ht/~mplanchard/procfarm/commits/main?)
[![procfarm on crates.io](https://img.shields.io/crates/v/procfarm)](https://crates.io/crates/procfarm/)

Farm commands out to a background server for parallel execution.

# Project Status

Unstable, moderately exercised.

- Interface still under active development
- Used in at least one real, complex CI pipeline

# Use Cases

In general, this is meant to be used in situations where you want to
spin off some task(s) to be completed, without directly responding to
their exit status, introspecting their output, etc. It is a
fire-and-forget approach to parallel task running.

Procfarm is a good choice when you want to throw something into the
background and get on with your life.

### Nix Cache Uploads

The main use case this was originally built for was for nix build
uploads using a [post-build hook](https://nix.dev/manual/nix/latest/advanced-topics/post-build-hook.html),
which is to say, a case where you'd like to trigger some command in
the background and let it finish asynchronously, while you get on with
your CI pipeline.

#### GitLab Example

The following is a functional example of using procfarm to do nix cache
uploads in GitLab.

With the following setup, a procfarm instance will run for each job,
and any nix builds will send the appropriate `nix copy` commands for
cache uploads to the servers. The `after_script` ensures that the
background process shuts down, since otherwise GitLab CI will hang
indefinitely while it waits for all spawned processes to terminate.

##### Wrapper Script

A simple wrapper script to handle globbing settings, at say
`/etc/nix/post-build-hook.sh`

```sh
set -eu

set -f
export IFS=' '

# Set this to whatever your cache is, e.g. an s3 binary cache
CACHE=s3://nix-cache?region=us-east-2&compression=zstd&parallel-compression=true

# Set the runtime dir explicitly, to ensure that even if nix is run as
# different user, it will use the same server.
procfarm --runtime-dir=/build/procfarm run -- nix copy --to "$CACHE" $OUT_PATHS
```

Don't forget to mark it exectuable.

##### Update Nix Config

Set your nix conf to call your wrapper script as the post-build-hook:

```conf
# /etc/nix.conf
extra-experimental-features = nix-command flakes
post-build-hook = /etc/nix/post-build-hook.sh
```

##### Update GitLab Config

You don't explicitly need to start the server yourself, but doing so,
while setting the log output to stdout, lets you see the report output
when you stop the server.

```yml
default:
  # Be sure to use the same runtime-dir that you used in the hook script.
  before_script:
    - procfarm start --runtime-dir=/build/procfarm --log=stderr --detach
  after_script:
    - procfarm --runtime-dir=/build/procfarm stop --timeout 300
```


# Goals and Non-Goals

## Goals

### Primary

- Reliability
  - The server should essentially never crash
  - Users should be able to mostly just care about the `run` command
- Performance
  - Should make it easy to run an arbitrary number of processes in
    parallel
  - Procfarm itself should add as close to zero overhead as possible
- Lightweightness
  - Users should not have to worry about procfarm's resource
    utilization, e.g. it should be careful to limit any accumulated
    buffer output to avoid RAM pressure

### Secondary

- Reporting
  - Procfarm should make it easy to monitor server activity
  - When finished, procfarm should be able to emit some useful
    aggregate information about what it did
  - Processes that fail to start or that return error statuses should
    have visibility in the logs

## Non-Goals

- Task-joining
  - Procfarm is explicitly a fire-and-forget operation. It is not GNU
    parallel, it does not care about exit codes or process output,
    except for some amount of error reporting.
- Librarification
  - There is no intention here to provide a library interface

## Windows

This project does not currently support Windows systems. I'm not
against it, but I don't have a Windows box to test anything on. If
you'd like to add Windows support, feel free to do so. I think the
main things that would need adjustment are:

- Writing a new `Transport` layer and using it instead of the default
  `UnixStream`. This is already encapsulated in a trait, so should not
  be too hard.
- It's possible the `syslog` logging variant will not work on Windows.
  If so, it should be replaced with something else on a Windows
  system.
- I don't know how detaching/daemonization works on Windows, so that
  likely will also need some attention.

# Contributing

Community contributions are welcome.

Commits must all include a [signed-off-by](https://git-scm.com/docs/git-commit#Documentation/git-commit.txt---signoff)
trailer indicating that the Contributor has read and agrees to the
terms of the [Developer's Certificate of Origin](./DCO) and the
[Contributor Guidelines](./contributor_guidelines.md).

## Development Environment

If you have nix, it's easy to get set up. You can either run `nix
develop` and get dropped into a shell with all dependencies or, if you
use `direnv`, run `direnv allow`.

## Tests

Run tests with `cargo test`.

# License Statement

Copyright (C) 2026 Matthew Planchard

This program is free software: you can redistribute it and/or modify
it under the terms of the GNU Affero General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU Affero General Public License for more details.

You should have received a copy of the GNU Affero General Public License
along with this program.  If not, see https://www.gnu.org/licenses/.
