RRDNSD

Distributed monitoring for Round Robin DNS load balancing / HA / failover.

badge License REUSE Rust Codeberg

RRDNSD monitors the availability of HTTP[S] services and updates DNS records accordingly. Lightweight and easy to configure. It can run on a small SBC but also scale up to hundreds of services.

For increased reliability it can run on multiple instances (nodes) using a quorum protocol.

Features:

  • ✓ Minimalistic web-based status dashboard

  • ✓ Logging

  • ✓ StatsD metrics

  • ✓ Security sandbox

Description

It is common to configure multiple 'A' or 'AAAA' DNS records for the same FQDN in order to provide simple forms of load balancing and failover. In this way clients can connect to one or another IP address for the same FQDN randomly or in round-robin fashion. If any of the IP address providing the service experience a downtime the DNS record can be removed. Once the DNS update propagates all the way to the clients they will stop using the failing IP address.

This method complements, rather than replacing, the traditional load balancers that sit between clients and servers.

DNS-based failover cannot mitigate a server downtime as quickly as "traditional" TCP/HTTP[S] load balancers due to DNS propagation time.

Yet, it provides resiliency against small and large network failures especially when used for geographically distributed services.

Glossary:

  • Service: a public facing service identified by a unique FQDN in the configuration file

  • Endpoint: an ipaddres providing a service. A service should have many endpoints. An endpoint might provide multiple services.

  • Node: an instance of RRDNSD identified by an unique ipaddr/port pair.

Use case of a simple setup

A client (e.g. a browser) going to access a service running at two IP addresses (endpoints).

RRDNSD monitors the availability of the endpoints. It detects that one endpoint becomes unreachable and updates the DNS resolver in order to delete the related A record. When the downtime ends RRDNSD publishes the A record again.

Use case of redundant nodes

Again the service is available at two endpoints. Three RRDNSD nodes are monitoring the endpoints from different locations on the Internet.

Diagram

Scenario 1

Diagram

The endpoint on the left becomes unreachable from 2 nodes due to large network outage.

Also, the endpoint on the right is unachable from one node due to a localized issue near the rightmost node.

The nodes vote by majority and decide to remove the "A" record from the left endpoint from DNS. They also decide that the endpoind on the right is likely to be in good shape and keep the related A record in DNS.

Scenario 2

A DNS resolver or DNS API receiving updates is shown on the left.

Diagram

The leftmost node is not running (e.g. due to a power failure). The two live nodes are still able to probe the endpoints, communicate to each other and react to changes.

Additionally, the node in the center is unable to reach the resolver/API (e.g. due to network congestion). Yet, RRDNSD is still able to work as the rightmost node can reach the resolver/API.

Deployment

Building natively on Debian Bookworm or Sid

Generates a native .deb package:

sudo apt-get install rust
dpkg-buildpackage -us -uc -b

Building on Debian Bookworm using Podman

This generates a minimalistic container localhost/rrdnsd to run rrdnsd

make podman-build-bookworm

# You can optionally extract the binary:
podman run --rm --entrypoint cat localhost/rrdnsd:bookworm /app/rrdnsd > ./rrdnsd_bookworm

Deployment

Install the Debian package locally, configure /etc/rrdnsd.json, then:

sudo apt install ./rrdnsd.deb
sudo systemctl start rrdnsd.service
sudo journalctl -f

Security:

Do not expose the RRDNSD API port directly on the Internet. Mount it behind a reverse proxy like Nginx and enable TLS.

You can further restrict the API entry points:

  • POST /quorum/v1/* : Required, used only between RRDNSD nodes

  • GET /dash : Web dashboard for administrators, optional

  • GET /health : Basic internal healthcheck, optional

  • POST /api/v1/* : Administration API, optional

Configuration:

A simple setup running on one node:

update_method can be:

  • nsupdate: Use /usr/bin/nsupdate to send an update to one or more DNS resolvers.

  • knsupdate: Similarly to nsupdate, use /usr/bin/knsupdate for Knot DNS

  • dynu: Uses the Dynu API. Configure update_credentials with the Dynu token.

Example configuration file:

WW

puts "Hello, world!"
{
  "_comment": "Keys starting with underscore are ignored.",
  "conf_version": 1,
  "_local_node": "IP address and port of this RRDNSD node.",
  "local_node": "127.0.0.1:3333",
  "_nodes": "IP address and port of this node and its peers, if any.",
  "nodes": [
    "127.0.0.1:3333"
  ],
  "nodes_protocol": "http",
  "services": [
    {
      "fqdn": "rrdnsd.test",
      "_healthcheck": "Protocol, port and HTTP path to test. {} is replaced with the enpoint IP address.",
      "healthcheck": "http://{}:8778/",
      "ipaddrs": [
        "127.0.0.2",
        "127.0.0.3"
      ],
      "ttl": 5,
      "zone": "rrdnsd.test"
    }
  ],
  "probe_interval_ms": 1000,
  "update_method": "nsupdate",
  "_update_resolvers": "List of DNS resolvers and port for nsupdate or knsupdate.",
  "_update_credentials": "Token, password or username:password to authenticate updates",
  "update_credentials": "",
  "update_resolvers": [
    "127.0.0.1:5454"
  ]
}

local_node can be overridden using the environment variable LOCAL_NODE

The default config path is /etc/rrdnsd.json and can be overridden using the environment variable CONF

Development

Roadmap:

  • ✓ Replace unwrap()

  • ❏ Optimize connection reuse and lifetime

  • ❏ Log if a whole service goes down with warn

  • ✓ Add end-to-end (sort of) /health

    • ✓ Fail-open: if more than 50% of endpoints are down keep them in DNS

  • ❏ Support active-standby pattern

  • ❏ Support IPv6 (AAAA records)

  • ❏ Exit gracefully

  • ❏ Support running scripts / hooks

  • ❏ Add an API to:

    • ❏ fetch status

    • ❏ fetch events

    • ❏ add/remove services and endpoints

    • ❏ add/remove services nodes

    • ❏ manual failover

    • ❏ drain endpoints

    • ❏ flag services as under maintenance

    • ❏ "refresh" an FQDN by adding again all live endpoints and then deleting the unreachable ones

  • ❏ Add TCP testing

  • ❏ Service-specific probing interval

  • ❏ Support DNS APIs

    • ✓ Dynu

    • ❏ Digital Ocean

Testing

Run basic unit tests using:

cargo test

Run a test instance:

RUST_BACKTRACE=1 CONF=integ/rrdnsd.json LOCAL_NODE=127.0.0.1:8000 target/debug/rrdnsd

Run full integration tests using the following command. Warning: this starts/stops knotd, runs sudo tc …​

cargo test --test integration_test