# dtox - DTO Generator

This tool automates the generation of source code and other artifacts from a set of Data Transfer Object (DTO) definition files. It uses a template-based approach, allowing for flexible code generation for any language or framework.

The generator discovers DTO files, processes templates by replacing placeholders with DTO-specific information, and handles both per-DTO file generation and aggregation of content into single files.

## Installation

`dtox` is now distributed as a regular Rust CLI in addition to the Docker image. If you already have Rust installed, you can install the latest tagged release on macOS, Windows, or Linux with a single command:

```sh
cargo install --locked --git https://gitlab.com/hti-oss/rxdatasets/dtox.git --tag 0.7.0
```

Drop the `--tag` flag if you prefer tracking `master`, or add `--force` to upgrade an existing installation. Helper scripts live in [`scripts/install.sh`](scripts/install.sh) (Bash) and [`scripts/install.ps1`](scripts/install.ps1) (PowerShell) if you would rather not memorize the `cargo install` flags.

See [INSTALL.md](INSTALL.md) for platform-specific walkthroughs (Rust/Rustup prerequisites, PATH updates, and verification steps). Once installed, run `dtox --version` to verify the binary is on your `PATH`.

## Usage

Here is a summary of the command-line options, which you can also view by running `dtox --help`.

| Option                  | Description                                                                                              | Default |
| ----------------------- | -------------------------------------------------------------------------------------------------------- | ------- |
| `-i`, `--input-dir <path>`    | Path to the directory containing DTO definition files and configuration CSVs.                              |         |
| `-o`, `--output-dir <path>`   | Path to the directory where generated files will be stored.                                              |         |
| `-t`, `--template-dir <path>` | Path to the directory containing templates. If not specified, the input directory is used.                 |         |
| `-l`, `--list-file <path>`    | Optional path to a file with a newline-separated list of DTO filenames to process.                         |         |
| `-v`, `--verbose`       | Enable verbose logging.                                                                                  | `false` |
| `--keep`                | Do not clean the output directory before generation.                                                     | `false` |
| `--no-gitignore`        | Do not respect .gitignore files when processing templates (includes all files).                         | `false` |
| `--generate-completion` | Generate a shell completion script for `bash`, `powershell`, etc. and print it to stdout.                |         |

## How It Works

The generator operates by discovering DTOs, loading configuration, and processing templates.

### DTO Discovery

The tool finds DTO definition files in the specified `--input-dir`.

*   **File Naming:** DTO files must follow the pattern `[name].v[version].[extension]`. For example, `product.v2.dto` or `order.v1.proto`.
*   **File Content:** Each DTO file must contain a line specifying its minor version, like `minor version 3`. Suggestion is to use a format-specific comment, such as `// minor version 3` for protobuf
*   **Selective Processing:** You can optionally provide a file via `--list-file` (e.g., `dtos.lst`) containing a newline-separated list of DTO filenames to process. If this is used, only the files listed will be processed - and in that order.

### Configuration Files

You can provide DTO-specific values for placeholders using special CSV files in the input directory.

*   **File Naming:** Configuration files must be named `__[placeholder_name]__.csv`. For example, a file named `__csharp_namespace__.csv` will provide values for the `__csharp_namespace__` placeholder.
*   **File Format:** The CSV file should contain two columns: the DTO identifier and the value. The DTO identifier is `[name].v[version]`. Comment lines or lines that are not valid CSV are ignored.

    ```csv
    # C# Namespaces for DTOs
    product.v2,MyCompany.Products.V2
    order.v1,MyCompany.Billing.V1
    ```

### Template Processing

The tool processes templates found in the `--template-dir` (which defaults to `--input-dir`). There are two main processing modes.

#### 1. Per-DTO Generation

For each discovered DTO, the tool copies all subdirectories from the template directory into a temporary location. It then performs placeholder replacements on all filenames and file contents within that temporary directory before merging the result into the output directory.

#### 2. Foreach-DTO Generation

For creating aggregate files (e.g., a registration file or an index), you can use special markers in any template file. These are lines containing the following strings:

*   `FOREACH-DTO-START`
*   `FOREACH-DTO-END`

The block of text between these two markers will be duplicated for every DTO, with placeholders replaced accordingly. The marker **lines** themselves are removed - but it's suggested to use use format-specific comments such as eg. `<!-- FOREACH-DTO-START -->` in XML.

Example:
```csharp
// FOREACH-DTO-START
builder.Services.AddDtoServer<__DtoName____DtoVersion__>();
// FOREACH-DTO-END
```

#### Available Placeholders

| Placeholder           | Description                                                              | Example (from `product.v2.dto`) |
| --------------------- | ------------------------------------------------------------------------ | ------------------------------- |
| `__dtoname__`         | The lowercase name of the DTO.                                           | `product`                       |
| `__DtoName__`         | The PascalCase name of the DTO.                                          | `Product`                       |
| `__dtoversion__`      | The version of the DTO.                                                  | `v2`                            |
| `__DtoVersion__`      | The PascalCase version of the DTO.                                       | `V2`                            |
| `__dtominorversion__` | The minor version from inside the DTO file.                              | `3`                             |
| `__dtofilepath__`     | The full path to the source DTO definition file.                         | `/path/to/product.v2.dto`       |
| `__dtofilecontents__` | Injects the entire content of the source DTO file.                       | `#minor version 3\n...`          |
| `__custom__`          | Any custom placeholder defined in a `__custom__.csv` configuration file. | `MyCompany.Products.V2`         |

You can also define custom placeholders directly in your DTO files using the pattern `// __placeholder_name__ value`. See the **Multi-Value Placeholders** section below for details.

### Multi-Value Placeholders with FOREACH-DTO-VALUE

You can extract multiple values for the same placeholder from a DTO file and iterate over them in templates using `FOREACH-DTO-VALUE` blocks.

#### Defining Multi-Value Placeholders

In your DTO file, specify the same placeholder multiple times on different lines:

```protobuf
// minor version 1
// __indexname__ bysku
// __indexname__ byname  
// __indexname__ bycategory

message Product {
    string sku = 1;
    string name = 2;
}
```

#### Using FOREACH-DTO-VALUE in Templates

Create a template with `FOREACH-DTO-VALUE-START` and `FOREACH-DTO-VALUE-END` markers:

```csharp
public class ProductIndexes {
    public void RegisterIndexes() {
        // FOREACH-DTO-VALUE-START __indexname__
        RegisterIndex("__indexname__");
        // FOREACH-DTO-VALUE-END
    }
}
```

**Generated Output:**

```csharp
public class ProductIndexes {
    public void RegisterIndexes() {
        RegisterIndex("bysku");
        RegisterIndex("byname");
        RegisterIndex("bycategory");
    }
}
```

#### Paired Placeholders (Format B)

You can define multiple placeholders together on the same line, and they will stay paired during iteration:

**DTO File:**

```protobuf
// __aliasname__ __aliaspattern__ byitem item_sku
// __aliasname__ __aliaspattern__ bycustomer customer_id
```

**Template:**

```csharp
// FOREACH-DTO-VALUE-START __aliasname__
AddAlias("__aliasname__", "__aliaspattern__");
// FOREACH-DTO-VALUE-END
```

**Generated Output:**

```csharp
AddAlias("byitem", "item_sku");
AddAlias("bycustomer", "customer_id");
```

**Key Features:**

* Placeholders appearing on the same line are automatically paired
* Both placeholders in a pair must be used consistently together
* Order matters: `__name__ __pattern__` is different from `__pattern__ __name__`
* Mix single-value placeholders (like `__dtoname__`) with multi-value placeholders in FOREACH-DTO-VALUE blocks

#### Nesting FOREACH Blocks

You can nest `FOREACH-DTO-VALUE` inside `FOREACH-DTO` for per-DTO multi-value generation:

```csharp
// FOREACH-DTO-START
public class __DtoName__Indexes {
    // FOREACH-DTO-VALUE-START __indexname__
    RegisterIndex("__dtoname__", "__indexname__");
    // FOREACH-DTO-VALUE-END
}
// FOREACH-DTO-END
```

**Important:** You cannot nest `FOREACH-DTO` inside `FOREACH-DTO-VALUE`.

### Unique Value Iteration with FOREACH-UNIQUE-DTO-VALUE

When your DTO files contain duplicate values for multi-value placeholders, you can use `FOREACH-UNIQUE-DTO-VALUE` to iterate only over unique values, automatically filtering duplicates while preserving first-occurrence order.

#### Use Case: Enum Generation

A common scenario is generating enums from DTO metadata that may contain duplicates:

**DTO File (order.v1.proto):**

```protobuf
// minor version 1
// __statusvalue__ Pending
// __statusvalue__ Processing
// __statusvalue__ Pending     // duplicate
// __statusvalue__ Completed
// __statusvalue__ Processing  // duplicate
// __statusvalue__ Cancelled
```

**Template:**

```csharp
public enum OrderStatus {
    // FOREACH-UNIQUE-DTO-VALUE-START __statusvalue__
    __statusvalue__,
    // FOREACH-UNIQUE-DTO-VALUE-END __statusvalue__
}
```

**Generated Output:**

```csharp
public enum OrderStatus {
    Pending,
    Processing,
    Completed,
    Cancelled,
}
```

**Key Features:**

* **Automatic deduplication:** Duplicates are filtered automatically based on first occurrence
* **Order preservation:** Values appear in the order they first occur in the DTO
* **Works with paired placeholders:** Uniqueness is determined by the primary placeholder
* **Integrates with FOREACH-DTO:** Can be nested inside `FOREACH-DTO` blocks for per-DTO unique iteration
* **Coexists with FOREACH-DTO-VALUE:** Both can be used in the same template for different purposes

#### Deduplication with Paired Placeholders

When using paired placeholders (Format B), uniqueness is determined by the primary (first) placeholder:

**DTO File:**

```protobuf
// __aliasname__ __aliaspath__ bysku product/sku
// __aliasname__ __aliaspath__ bysku product/id    // duplicate primary
// __aliasname__ __aliaspath__ byname product/name
```

**Template:**

```csharp
// FOREACH-UNIQUE-DTO-VALUE-START __aliasname__
AddUniqueAlias("__aliasname__", "__aliaspath__");
// FOREACH-UNIQUE-DTO-VALUE-END __aliasname__
```

**Generated Output:**

```csharp
AddUniqueAlias("bysku", "product/sku");    // First occurrence kept
AddUniqueAlias("byname", "product/name");
```

The second `bysku` entry is filtered out because the primary placeholder (`__aliasname__`) is a duplicate, even though the secondary placeholder (`__aliaspath__`) has a different value.

#### Combining FOREACH-DTO-VALUE and FOREACH-UNIQUE-DTO-VALUE

You can use both in the same template when you need all values in one place and unique values in another:

```csharp
public class ProductIndexes {
    // Register all indexes (including duplicates for tracking)
    public void RegisterAllIndexes() {
        // FOREACH-DTO-VALUE-START __indexname__
        RegisterIndex("__indexname__");
        // FOREACH-DTO-VALUE-END __indexname__
    }
    
    // Register only unique indexes (for schema creation)
    public void CreateUniqueIndexes() {
        // FOREACH-UNIQUE-DTO-VALUE-START __indexname__
        CreateIndex("__indexname__");
        // FOREACH-UNIQUE-DTO-VALUE-END __indexname__
    }
}
```

#### When to Use FOREACH-UNIQUE-DTO-VALUE

Use `FOREACH-UNIQUE-DTO-VALUE` when:
* Generating enums from potentially duplicate data
* Creating schema definitions (tables, indexes, etc.)
* Building configuration that requires unique keys
* Deduplicating import statements or dependencies

Use regular `FOREACH-DTO-VALUE` when:
* You need to preserve all occurrences (including duplicates)
* Counting or tracking frequency of values
* Maintaining audit trails or logs

### Conditional Inclusion with IFDEF-DTO-VALUE / IFANY-DTO-VALUE

The `IFDEF-DTO-VALUE` and `IFANY-DTO-VALUE` markers allow you to conditionally include blocks of content based on whether a placeholder is defined in the DTO file. Both marker names work identically and are completely interchangeable.

#### Syntax

```
// IFDEF-DTO-VALUE-START __placeholder__
... content to include if __placeholder__ exists ...
// IFDEF-DTO-VALUE-END
```

Or equivalently:

```
// IFANY-DTO-VALUE-START __placeholder__
... content to include if __placeholder__ exists ...
// IFANY-DTO-VALUE-END
```

**Important Notes:**
* If `__placeholder__` exists in the DTO (has at least one value), the content between the markers is included
* If `__placeholder__` doesn't exist, the entire block (including the marker lines) is removed
* Unlike `FOREACH-DTO-VALUE`, this does NOT iterate - content is included at most once
* The markers themselves are removed from the output
* Placeholder substitution happens AFTER the inclusion decision
* `IFDEF` and `IFANY` are completely interchangeable - use whichever reads better in your context

#### Example: Optional Features

**DTO with optional metadata (product.v1.proto):**

```protobuf
// minor version 1
// __indexname__ bysku
// __cacheable__ true

message Product {
    string sku = 1;
    string name = 2;
}
```

**DTO without optional metadata (order.v1.proto):**

```protobuf
// minor version 1
// __indexname__ byid

message Order {
    string id = 1;
}
```

**Template with conditional block:**

```csharp
public class __DtoName____DtoVersion__ {
    public string Name = "__dtoname__";
    
    // IFDEF-DTO-VALUE-START __cacheable__
    public bool IsCacheable = true;
    public string CacheKey = "__DtoName___cache";
    // IFDEF-DTO-VALUE-END
    
    public List<string> Indexes = new List<string>();
}
```

**Generated output for Product (has __cacheable__):**

```csharp
public class ProductV1 {
    public string Name = "product";
    
    public bool IsCacheable = true;
    public string CacheKey = "Product_cache";
    
    public List<string> Indexes = new List<string>();
}
```

**Generated output for Order (no __cacheable__):**

```csharp
public class OrderV1 {
    public string Name = "order";
    
    public List<string> Indexes = new List<string>();
}
```

The caching-related fields only appear in the Product class because `__cacheable__` is defined in its DTO file.

#### Integration with FOREACH-DTO

`IFDEF-DTO-VALUE` works seamlessly inside `FOREACH-DTO` blocks, allowing each DTO to have different optional content:

```csharp
// FOREACH-DTO-START
public class __DtoName____DtoVersion__Config {
    // IFDEF-DTO-VALUE-START __cacheable__
    public TimeSpan CacheDuration = TimeSpan.FromMinutes(5);
    // IFDEF-DTO-VALUE-END
    
    // IFDEF-DTO-VALUE-START __audit__
    public bool EnableAuditLog = true;
    // IFDEF-DTO-VALUE-END
}
// FOREACH-DTO-END
```

Each DTO will generate its own config class with only the features it declares.

#### When to Use IFDEF-DTO-VALUE

Use `IFDEF-DTO-VALUE` when:
* Different DTOs have different optional features or capabilities
* You want to keep templates DRY while supporting optional functionality
* You need conditional compilation patterns (e.g., feature flags)
* Some DTOs require extra configuration or initialization

This is particularly useful for:
* Optional caching strategies
* Conditional validation rules
* Feature-specific code generation
* DTO-specific optimizations

### Conditional Inclusion with IFNDEF-DTO-VALUE / IFNO-DTO-VALUE

The `IFNDEF-DTO-VALUE` and `IFNO-DTO-VALUE` markers provide the opposite behavior of `IFDEF`/`IFANY` - they include content only when a placeholder is **NOT** defined in the DTO file. Both marker names work identically and are completely interchangeable.

#### Syntax

```
// IFNDEF-DTO-VALUE-START __placeholder__
... content to include if __placeholder__ does NOT exist ...
// IFNDEF-DTO-VALUE-END
```

Or equivalently:

```
// IFNO-DTO-VALUE-START __placeholder__
... content to include if __placeholder__ does NOT exist ...
// IFNO-DTO-VALUE-END
```

**Important Notes:**
* If `__placeholder__` does NOT exist in the DTO, the content between the markers is included
* If `__placeholder__` exists (has at least one value), the entire block is removed
* This is the logical opposite of `IFDEF`/`IFANY`
* The markers themselves are removed from the output
* `IFNDEF` and `IFNO` are completely interchangeable

#### Example: Default Values

**DTO with custom cache (product.v1.proto):**

```protobuf
// minor version 1
// __custom_cache__ redis
// __custom_timeout__ 3600
```

**DTO without custom cache (order.v1.proto):**

```protobuf
// minor version 1
```

**Template:**

```csharp
// FOREACH-DTO-START
public class __DtoName____DtoVersion__Config {
    public string Name = "__dtoname__";
    
    // IFNDEF-DTO-VALUE-START __custom_cache__
    public string CacheStrategy = "default";
    // IFNDEF-DTO-VALUE-END
    
    // IFNO-DTO-VALUE-START __custom_timeout__
    public int TimeoutSeconds = 30;
    // IFNO-DTO-VALUE-END
}
// FOREACH-DTO-END
```

**Generated output for Product (has custom values):**

```csharp
public class ProductV1Config {
    public string Name = "product";
}
```

**Generated output for Order (no custom values):**

```csharp
public class OrderV1Config {
    public string Name = "order";
    
    public string CacheStrategy = "default";
    
    public int TimeoutSeconds = 30;
}
```

The default values only appear when the custom placeholders are not defined.

#### Combining IFDEF and IFNDEF

You can mix both types of conditional markers in the same template:

```csharp
// FOREACH-DTO-START
public class __DtoName____DtoVersion__ {
    // IFDEF-DTO-VALUE-START __custom_validation__
    public IValidator Validator = new CustomValidator();
    // IFDEF-DTO-VALUE-END
    
    // IFNDEF-DTO-VALUE-START __custom_validation__
    public IValidator Validator = new DefaultValidator();
    // IFNDEF-DTO-VALUE-END
}
// FOREACH-DTO-END
```

#### When to Use IFNDEF-DTO-VALUE

Use `IFNDEF-DTO-VALUE` when:
* You want to provide default values for DTOs that don't customize them
* Creating fallback behavior when optional features aren't specified
* Generating boilerplate only for DTOs without custom implementations
* Implementing "opt-out" patterns (default is on, custom disables it)

This is particularly useful for:
* Default configuration values
* Standard initialization code
* Fallback implementations
* Base-case code generation

### Gitignore Support

By default, `dtox` respects `.gitignore` files when processing templates. This helps prevent binary files, build artifacts, and other unwanted files from being included in the generated output.

**Default Behavior (Gitignore Enabled):**
- Automatically skips files and directories specified in `.gitignore`
- Prevents binary files from causing UTF-8 encoding errors
- Follows standard git ignore rules (`.gitignore`, `.git/info/exclude`, global gitignore)
- Logs when gitignore rules are being applied

**Disabling Gitignore:**
Use the `--no-gitignore` flag to include all files, even those normally ignored:

```bash
dtox --input-dir ./templates --output-dir ./output --no-gitignore
```

**Common Use Cases:**
- **Enabled (default):** Normal template processing where you want to exclude build artifacts, IDE files, etc.
- **Disabled (`--no-gitignore`):** When you need to include specific files that are gitignored, or when working with repositories that don't use git

## Using `just` (Task Runner)

This project includes a `justfile` to simplify common development tasks. `just` is a command runner similar to `make` or `invoke`.

First, [install `just`](https://github.com/casey/just#installation).

Then you can use the following commands from the project directory:

*   `just build`: Compiles the project.
*   `just run -- --input-dir <path>`: Runs the generator. Pass arguments after `--`.
*   `just completion-bash`: Generates the Bash completion script (`dtox-completion.bash`).
*   `just completion-pwsh`: Generates the PowerShell completion script (`_dtox.ps1`).

Running `just` with no arguments will list all available commands.

## Shell Completion

This tool can generate shell completion scripts for various shells like Bash and PowerShell. You can generate them manually or use the `just` recipes above.

### Bash

1.  **Generate the completion script:**
    Run your compiled application with the `--generate-completion bash` flag and redirect the output to a file.

    ```sh
    # Assuming your executable is in the default debug location
    ./target/debug/dtox --generate-completion bash > dtox-completion.bash
    ```

2.  **Load the script for the current session:**
    Source the generated file to enable completion in your current terminal session.

    ```sh
    source dtox-completion.bash
    ```

3.  **Load the script permanently:**
    To make completion available in all new shell sessions, you can either copy the script to a standard completion directory or source it from your shell's profile file (`~/.bashrc` or `~/.bash_profile`).

    **Option A: Copy to completion directory (recommended on Linux):**
    ```sh
    # The exact path may vary based on your distribution
    sudo cp dtox-completion.bash /etc/bash_completion.d/dtox
    ```

    **Option B: Source from `.bashrc`:**
    Add the following line to your `~/.bashrc` file, replacing `/path/to/` with the actual path to the script.
    ```sh
    echo 'source /path/to/dtox-completion.bash' >> ~/.bashrc
    ```

    You will need to restart your shell for the changes to take effect.

### PowerShell (pwsh)

1.  **Check your execution policy:**
    PowerShell requires scripts to be signed or the execution policy to be changed. You can check your current policy with `Get-ExecutionPolicy`. If it's `Restricted`, you may need to change it.

    ```powershell
    # Run this command in a PowerShell terminal with administrator privileges
    Set-ExecutionPolicy RemoteSigned -Scope CurrentUser
    ```

2.  **Generate the completion script:**
    ```powershell
    # Assuming your executable is in the default debug location
    .\target\debug\dtox.exe --generate-completion powershell > _dtox.ps1
    ```

3.  **Load the script permanently:**
    To load the script every time you open PowerShell, add it to your profile. You can find your profile path by checking the `$PROFILE` variable.

    ```powershell
    # Open your profile in Notepad (it will be created if it doesn't exist)
    notepad $PROFILE
    ```

    Add the following line to the profile file, replacing `C:\path\to\` with the actual path to the script.
    ```powershell
    . C:\path\to\_dtox.ps1
    ```

    Save the file and restart your PowerShell session. You can now use tab-completion for the `dtox` command and its arguments.
