<p align="center">
  <img src="./static/nebu_logo2_alpha.png" alt="Nebulous Logo" width="400">
</p>


__A globally distributed container orchestrator__

Think of it as a Kubernetes that can span clouds and regions with a focus on __accelerated compute__.    
Ships as a single binary, performant and lightweight via Rust :crab:    
   
Why not Kubernetes? See [why_not_kube.md](docs/why_not_kube.md)   
   
> [!WARNING]
> Nebulous is in __alpha__, things may break.

## Installation

```sh
curl -fsSL -H "Cache-Control: no-cache" https://raw.githubusercontent.com/agentsea/nebulous/main/remote_install.sh | bash
```
> [!NOTE]
> Only MacOS and Linux arm64/amd64 are supported at this time.

## Usage   

Export the keys of your cloud providers.
```sh
export RUNPOD_API_KEY=...
export AWS_ACCESS_KEY_ID=...
export AWS_SECRET_ACCESS_KEY=...
```

Run a local API server on docker
```sh
nebu serve --docker
```

Or optionally run on Kubernetes with our [helm chart](./deploy/charts/nebulous/)   
   
Connect to the tailnet
```sh
nebu connect
```
    
See what cloud platforms are currently supported.
```sh
nebu get platforms
```

> [!TIP]
> Prefer a pythonic interface? Try [nebulous-py](https://github.com/agentsea/nebulous-py)   
>     
> Prefer a higher level LLM interface? Try [orign](https://github.com/agentsea/orign)

### Containers

Let's run our first container. We'll create a container on runpod with 2 A100 GPUs which trains a model using TRL.   
   
First, let's find what accelerators are available.
```sh
nebu get accelerators
```

Now lets create a container.
```yaml
kind: Container
metadata:
  name: trl-job
  namespace: training
image: "huggingface/trl-latest-gpu:latest"
platform: runpod
command: |
  source activate trl && trl sft --model_name_or_path $MODEL \
      --dataset_name $DATASET \
      --output_dir /output \
      --torch_dtype bfloat16 \
      --use_peft true
env:
  - key: MODEL
    value: Qwen/Qwen2.5-7B 
  - key: DATASET
    value: trl-lib/Capybara 
volumes:  
  - source: /output
    dest: nebu://training-output
    driver: RCLONE_COPY
    continuous: true
accelerators:
  - "2:A100_SXM"
restart: Never
```
Now to create the container

```sh
nebu create container -f mycontainer.yaml
```
> [!TIP]
> See our [container examples](examples/containers) for more.

List all containers
```sh
nebu get containers
```

Get the container we just created.
```sh
nebu get containers trl-job -n training
```

Exec a command in a container
```text
nebu exec trl-job -n training -c "echo hello"
```

Get logs from a container
```sh
nebu logs trl-job -n training
```

Send an http request to a container
```sh
curl http://container-{id}:8000
```

#### Queues

Containers can be assigned to a FIFO queue, which will block them from starting until the queue is free.

```yaml
kind: Container
image: pytorch/pytorch:latest
queue: actor-critic-training
```

#### Volumes

Volumes provide a means to persist and sync data accross clouds. Nebulous uses [rclone](https://rclone.org/) to sync data between clouds backed by an object storage provider.   
   
```yaml
kind: Volume
metadata:
  name: shared-datasets
  namespace: my-app
source: s3://foo/bar
```

This volume could then be used in a container like
```yaml
kind: Container
volumes:
  - source: /output
    dest: nebu://shared-datasets/my-dataset
    driver: RCLONE_SYNC
    continuous: true
```

The dest path `nebu://shared-datasets/my-dataset` would translate to `s3://foo/bar/my-dataset`. Nebulous will take care of configuring the container with the needed credentials to access shared namespace volumes.
> [!TIP]
> By default, Nebulous provisions every namespace with a volume with the same name as the namespace
   
Optionally users can define volumes directly in the container

```yaml
kind: Container
volumes:
  - source: s3://my-other-bucket/foo/bar
    dest: /datasets
    driver: RCLONE_SYNC
env:
  - key: AWS_ACCESS_KEY_ID
    secret_name: my-aws-access-key-secret
  - key: AWS_SECRET_ACCESS_KEY
    secret_name: my-aws-secret-access-key
```
> [!NOTE]
> Note that you will need to provide the necessary credentials for the container to access that s3 path.

Supported drivers are:
- `RCLONE_SYNC`
- `RCLONE_COPY`
- `RCLONE_BISYNC`

#### Meters

Metered billing is supported through [OpenMeter](https://openmeter.cloud/) using the `meters` field.

```yaml
meters:
  - cost: 0.1
    unit: second
    currency: USD
    metric: runtime 
```

Cost plus is supported through the `costp` field.

```yaml
meters:
  - costp: 10
    unit: second
    currency: USD
    metric: runtime 
```
This configuration will add 10% to the cost of the container.

#### Authz

Authz is supported through the container proxy. 

To enable the proxy for a container, set the `proxy_port` field to the container port you want to proxy.
```yaml
proxy_port: 8080
```

Then your service can be accesssed at `http://proxy.<nebu-host>` with the header `x-resource: <name>.<namespace>.<kind>`.   

With the proxy enabled, you can also configure authz rules.

```yaml
authz:
  rules:
    # Match on email
    - name: email-match
      field_match:
        - field: "owner"
          pattern: "${email}"
      allow: true
    
    # Path-based matching for organization resources
    - name: org-path-access
      path_match:
        - pattern: "/api/v1/orgs/${org_id}/**"
        - pattern: "/api/v1/organizations/${org_id}/**"
        - pattern: "/api/v1/models/${org_id}/**"
      allow: true
```

Variables are interpolated from the users auth profile.

> [!TIP]
> See [container examples](examples/containers) for more.

### Secrets

Secrets are used to store sensitive information such as API keys and credentials. Secrets are `AES-256` encrypted and stored in the database.

Create a secret
```sh
nebu create secret my-secret --value $MY_SECRET_VALUE -n my-app
```

Get all secrets
```sh
nebu get secrets -n my-app
```

Get a secret
```sh
nebu get secrets my-secret -n my-app
```

Delete a secret
```sh
nebu delete secrets my-secret -n my-app
```

Secrets can be used in container environment variables.

```yaml
kind: Container
metadata:
  name: my-container
  namespace: my-app
env:
  - key: MY_SECRET
    secret_name: my-secret
```

### Namespaces

Namespaces provide a means to segment groups of resources across clouds.  

```yaml
kind: Container
metadata:
  name: llama-factory-server
  namespace: my-app
```
   
Resources within a given namespace are network isolated using [Tailnet](https://tailscale.com/kb/1136/tailnet), and can be accessed by simply using `http://{kind}-{id}` e.g. `http://container-12345:8000`.
    
Nebulous cloud provides a free hosted [HeadScale](https://github.com/juanfont/headscale) instance to connect your resources, or you can bring your own by simply setting the `TAILSCALE_URL` environment variable.   

#### Tenancy

Nebulous is multi-tenant from the ground up. Tenancy happens at the namespace level, when creating a namespace a user can set the owner to their user or an organization they are a member of.

```sh
kind: Namespace
metadata:
  namespace: my-app
  owner: acme
```

Now all resources created in that namespace will be owned by `acme`.   
   
The authorization heirarchy is
```
owners -> namespaces -> resources
```

### Services [in progress]

Services provide a means to expose containers on a stable IP address, and to balance traffic across multiple containers. Services auto-scale up and down as needed.

```yaml
kind: Service
metadata:
  name: vllm-qwen
  namespace: inference
container:
  image: vllm/vllm-openai:latest
  command: |
    python -m vllm.entrypoints.api_server \
      --model Qwen/Qwen2-7B-Instruct \
      --tensor-parallel-size 1 \
      --port 8000
  accelerators:
    - "1:A100"
platform: gce
min_containers: 1
max_containers: 5
scale:
  up:
    above_latency: 100ms
    duration: 10s
  down:
    below_latency: 10ms
    duration: 5m
  zero:
    below_latency: 10ms
    duration: 10m
```

```sh
nebu create service -f examples/service/vllm-qwen.yaml
```

The IP will be returned in the `status` field.

```sh
nebu get services vllm-qwen -n inference
```

Service can be buffered, which will queue requests until a container is available.

```yaml
buffered: true
```

Services can also scale to zero.

```yaml
min_containers: 0
```

Services can also enforce schemas.

```yaml
schema:
  - name: prompt
    type: string
    required: true
```

Or use a common schema.

```yaml
common_schema: OPENAI_CHAT
```

Services can record all requests and responses.

```yaml
record: true
```

Services can perform metered billing, such as counting the number of tokens in the response.

```yaml
meters:
  - cost: 0.001
    unit: token
    currency: USD
    response_json_value: "$.usage.prompt_tokens"
```

Services also work with clusters.

```yaml
kind: Service
metadata:
  name: vllm-qwen
  namespace: inference
cluster:
  container:
    image: vllm/vllm-openai:latest
    command: |
      python -m vllm.entrypoints.api_server \
        --model Qwen/Qwen2-72B-Instruct \
        --tensor-parallel-size 1 \
        --port 8000
    accelerators:
      - "8:A100"
  num_nodes: 2
```

> [!TIP]
> See [service examples](examples/services) for more.
   
### Clusters [in progress]

Clusters provide a means of multi-node training and inference.

```yaml
kind: Cluster
metadata:
  name: pytorch-test
  namespace: foo
container:
  image: pytorch/pytorch:latest
  command: "echo $NODES && torchrun ..."
  platform: ec2
  env:
    - key: HELLO
      value: world
  volumes:
    - source: s3://nebulous-rs/test
      dest: /test
      driver: RCLONE_SYNC
      continuous: true
  accelerators:
    - "8:B200"
num_nodes: 4
```
```sh
nebu create cluster -f examples/cluster.yaml
```

Each container will get a `$NODES` env var which contains the IP addresses of the nodes in the cluster.   
   
Clusters always aim to schedule nodes as close to each other as possible, with as fast of networking as available.   

> [!TIP]
> See [cluster examples](examples/clusters) for more.

### Processors [in progress]

Processors are containers that work off real-time data streams and are autoscaled based on back-pressure. Streams are provided by [Redis Streams](https://redis.io/docs/latest/develop/data-types/streams/).

Processors are best used for bursty async jobs, or low latency stream processing.

```yaml
kind: Processor
metadata:
  name: translator
  namespace: my-app
stream: my-app:workers:translator
container:
  image: corge/translator:latest
  command: "redis-cli XREAD COUNT 10 STREAMS my-app:workers:translator"
  platform: gce
  accelerators:
    - "1:A40"
min_workers: 1
max_workers: 10
scale:
  up:
    above_pressure: 100
    duration: 10s
  down:
    below_pressure: 10
    duration: 5m
  zero:
    duration: 10m
```
```sh
nebu create processor -f examples/processors/translator.yaml
```

Processors can also scale to zero.

```yaml
min_workers: 0
```

Processors can enforce schemas.

```yaml
schema:
  - name: text_to_translate
    type: string
    required: true
```

Send data to a processor stream

```sh
nebu send processor translator --data '{"text_to_translate": "Dlrow Olleh"} -n my-app'
```

Read data from a processor stream

```text
nebu read processor translator --num 10
```

List all processors

```sh
nebu get processors
```

Processors can use containers across different platforms. [in progress]

```yaml
container:
  image: corge/translator:latest
  command: "redis-cli XREAD COUNT 10 STREAMS my-app:workers:translator"
  platforms:
    - gce
    - runpod
  accelerators:
    - "1:A40"
```

> [!TIP]
> See [processor examples](examples/processors) for more.

## SDK

:snake: Python https://github.com/agentsea/nebulous-py    
   
:crab: Rust https://crates.io/crates/nebulous/versions

## Roadmap

- [x] Support non-gpu containers
- [ ] Services
- [ ] Clusters
- [ ] Processors
- [ ] Support for AWS EC2
- [ ] Support for GCE
- [ ] Support for Azure
- [ ] Support for Kubernetes

## Contributing

Please open an issue or submit a PR.

## Developing

Add all the environment variables shown in the [.env_](.env_) file to your environment.

Run a postgres and redis instance locally. This can be done easily with docker.

```sh
docker run -d --name redis -p 6379:6379 redis:latest
docker run -d --name postgres -p 5432:5432 postgres:latest
```

To configure the secrets store you will need an encryption key. This can be generated with the following command.
```sh
openssl rand -base64 32 | tr -dc '[:alnum:]' | head -c 32
```
Then set this to the `NEBU_ENCRYPTION_KEY` environment variable.   
     
To optionally use OpenMeter for metered billing, you will need to open an account with either [their cloud](https://openmeter.cloud/) or run their [open source](https://github.com/openmeterio/openmeter) and set the `OPENMETER_API_KEY` and `OPENMETER_URL` environment variables.   
     
To optionally use Tailnet, you will need to open an account with [Tailscale](https://tailscale.com/) or run your own [HeadScale](https://github.com/juanfont/headscale) instance and set the `TAILSCALE_API_KEY` and `TAILSCALE_TAILNET` environment variables.
   
Install locally
```
make install
```
   
Run the server
```
nebu serve
```
   
Login to the auth server. When you do, set the server to `http://localhost:3000`.
```
nebu login
```
   
Now you can create resources

```sh
nebu create container -f examples/containers/trl_small.yaml
```
   
When you make changes, simply run `make install` and `nebu serve` again.

## Inspiration

- [Kubernetes](https://kubernetes.io/)
- [Aurea](https://github.com/aurae-runtime/aurae)
- [RunPod](https://runpod.io/)
- [Prime Intellect](https://primeintellect.com/)
