![markdown-ai-banner-primary](https://github.com/user-attachments/assets/7974c422-2ad1-442b-8c08-c82026100341)

<br />

# Rust Crate: Markdown AI Citation Removal

[![Crates.io](https://img.shields.io/crates/v/markdown-ai-cite-remove.svg)](https://crates.io/crates/markdown-ai-cite-remove)
[![Documentation](https://docs.rs/markdown-ai-cite-remove/badge.svg)](https://docs.rs/markdown-ai-cite-remove)
[![License](https://img.shields.io/crates/l/markdown-ai-cite-remove.svg)](LICENSE)
[![Tests](https://img.shields.io/badge/tests-58%20passing-brightgreen)](tests/)

<br />

> *"Five years of AI evolution and flirting with AGI. Zero libraries to remove `[1][2][3]` from markdown 🙄. Remember me well when they take over."*

<br />

**Remove AI-generated citations and annotations from Markdown text at the speed of Rust**

High-performance Rust library for cleaning ChatGPT, Claude, Perplexity, and other AI markdown responses. Removes inline citations `[1][2]`, reference links `[1]: https://...`, and bibliography sections with 100% accuracy.

<br />

> 📚 [CLI Guide](docs/guides/CLI_GUIDE.md) • [Benchmarking Guide](docs/performance/BENCHMARKING.md) • [FAQ](docs/guides/FAQ.md) • [Documentation Index](docs/DOCUMENTATION_INDEX.md)

<br />
<br />

---

## ⚡ Performance-First

- **100+ MB/s** throughput on typical documents
- **Zero-copy** processing where possible
- **Regex-optimized** with lazy compilation
- **Thread-safe** stateless design

<br />

---

## 🎯 Use Cases

<br />

### Real-World Applications

<br />

#### **1. Blog Publishing Pipeline**

```rust
// Clean AI-generated blog posts before publishing
use markdown_ai_cite_remove::clean;

let ai_draft = fetch_from_chatgpt();
let clean_post = clean(&ai_draft);
publish_to_cms(clean_post);
```

<br />

#### **2. Documentation Generation**

```rust
// Remove citations from AI-generated documentation
let docs = generate_docs_with_ai();
let clean_docs = clean(&docs);
write_to_file("README.md", clean_docs);
```

<br />

#### **3. Content Aggregation**

```rust
// Clean multiple AI responses
use markdown_ai_cite_remove::CitationCleaner;

let cleaner = CitationCleaner::new();
let responses = vec![chatgpt_response, claude_response, perplexity_response];

let cleaned: Vec<String> = responses
    .iter()
    .map(|r| cleaner.clean(r))
    .collect();
```

<br />

#### **4. Streaming API Processing**

```rust
// Clean AI responses in real-time
async fn process_stream(stream: impl Stream<Item = String>) {
    let cleaner = CitationCleaner::new();

    stream
        .map(|chunk| cleaner.clean(&chunk))
        .for_each(|cleaned| {
            send_to_client(cleaned).await;
        })
        .await;
}
```

<br />

#### **5. Batch File Processing (CLI)**

```bash
# Clean all markdown files in a directory
find ./ai_output -name "*.md" -exec md-cite-clean {} -o ./cleaned/{} \;
```

<br />
<br />

### Common Scenarios

- ✅ Clean AI chatbot responses (ChatGPT, Claude, Perplexity, Gemini)
- ✅ Prepare markdown for blog posts and articles
- ✅ Remove citations before website publishing
- ✅ Process streaming API responses in real-time
- ✅ Batch document cleaning for content pipelines
- ✅ Clean documentation generated by AI tools
- ✅ Prepare content for CMS ingestion
- ✅ Remove annotations from research summaries

<br />

---

## 📦 Installation

<br />

### Prerequisites

**Minimum Requirements:**
- Rust 1.70 or later
- Cargo (comes with Rust)

**Optional (for enhanced benchmarking):**
- Gnuplot (for benchmark visualization)
  - macOS: `brew install gnuplot`
  - Ubuntu/Debian: `sudo apt-get install gnuplot`
  - Windows: Download from http://www.gnuplot.info/

<br />

### Library Installation

Add to your `Cargo.toml`:

```toml
[dependencies]
markdown-ai-cite-remove = "0.1"
```

<br />

### CLI Installation

Install the command-line tool globally:

```bash
# Install from crates.io (when published)
cargo install markdown-ai-cite-remove --features cli

# Or install from local source
cargo install --path . --features cli
```

Verify installation:

```bash
md-cite-clean --version
```

<br />

---

## 🚀 Quick Start

<br />

### Quick Reference

| Task | Command |
|------|---------|
| Clean from stdin | `echo "Text[1]" \| md-cite-clean` |
| Clean a file | `md-cite-clean input.md` |
| Save to file | `md-cite-clean input.md -o output.md` |
| Verbose output | `md-cite-clean input.md -o output.md --verbose` |
| Run tests | `cargo test` |
| Run benchmarks | `cargo bench` |
| View docs | `cargo doc --open` |

<br />

### Library Usage

```rust
use markdown_ai_cite_remove::clean;

let markdown = "AI research shows promise[1][2].\n\n[1]: https://example.com\n[2]: https://test.com";
let cleaned = clean(markdown);
assert_eq!(cleaned.trim(), "AI research shows promise.");
```

<br />

### CLI Usage

<br />

#### Basic Examples

**1. Process from stdin to stdout (pipe mode):**

```bash
echo "Text[1] here." | md-cite-clean
# Output: Text here.
```

<br />

**2. Clean a file and print to stdout:**

```bash
md-cite-clean document.md
```

<br />

**3. Clean a file and save to another file:**

```bash
md-cite-clean input.md -o output.md
```

<br />

**4. Clean with verbose output (shows processing details):**

```bash
md-cite-clean input.md -o output.md --verbose
# Output:
# Reading from file: input.md
# Cleaning markdown (input size: 1234 bytes)...
# Cleaned markdown (output size: 1100 bytes)
# Writing to file: output.md
# Done!
```

<br />
<br />

#### Advanced CLI Usage

**Process multiple files:**

```bash
# Using a loop
for file in *.md; do
  md-cite-clean "$file" -o "cleaned_${file}"
done
```

<br />

**Integration with other tools:**

```bash
# Clean AI output from curl
curl https://api.example.com/ai-response | md-cite-clean

# Clean and preview
md-cite-clean document.md | less

# Clean and count words
md-cite-clean document.md | wc -w

# Chain with other markdown processors
md-cite-clean input.md | pandoc -f markdown -t html -o output.html
```

<br />

**Using in scripts:**

```bash
#!/bin/bash
# clean_ai_docs.sh - Clean all AI-generated markdown files

INPUT_DIR="./ai_output"
OUTPUT_DIR="./cleaned"

mkdir -p "$OUTPUT_DIR"

for file in "$INPUT_DIR"/*.md; do
  filename=$(basename "$file")
  md-cite-clean "$file" -o "$OUTPUT_DIR/$filename" --verbose
done

echo "Cleaned $(ls -1 "$INPUT_DIR"/*.md | wc -l) files"
```

<br />

---

## 🔧 Features

- ✅ Remove inline numeric citations `[1][2][3]`
- ✅ Remove named citations `[source:1][ref:2][cite:3][note:4]`
- ✅ Remove reference link lists `[1]: https://...`
- ✅ Remove reference section headers `## References`, `# Citations`, `### Sources`
- ✅ Remove bibliographic entries `[1] Author (2024). Title...`
- ✅ Preserve markdown formatting (bold, italic, links, lists, etc.)
- ✅ Whitespace normalization
- ✅ Configurable cleaning options

<br />

---

## 📖 Documentation

- **[FAQ](docs/guides/FAQ.md)** - Frequently asked questions and troubleshooting
- **[CLI Usage Guide](docs/guides/CLI_GUIDE.md)** - Complete command-line tool documentation
- **[Benchmarking Guide](docs/performance/BENCHMARKING.md)** - Understanding performance metrics
- **[Documentation Index](docs/DOCUMENTATION_INDEX.md)** - Complete guide to all documentation
- **[API Documentation](https://docs.rs/markdown-ai-cite-remove)** - Full API reference
- **[Examples](examples/)** - Working code examples

<br />

---

## 📚 Advanced Usage

<br />

### Custom Configuration

```rust
use markdown_ai_cite_remove::{CitationCleaner, CleanerConfig};

// Remove only inline citations, keep reference sections
let config = CleanerConfig::inline_only();
let cleaner = CitationCleaner::with_config(config);
let cleaned = cleaner.clean("Text[1] here.\n\n[1]: https://example.com");

// Remove only reference sections, keep inline citations
let config = CleanerConfig::references_only();
let cleaned = cleaner.clean("Text[1] here.\n\n[1]: https://example.com");

// Full custom configuration
let config = CleanerConfig {
    remove_inline_citations: true,
    remove_reference_links: true,
    remove_reference_headers: true,
    remove_reference_entries: true,
    normalize_whitespace: true,
    remove_blank_lines: true,
    trim_lines: true,
};
```

<br />

### Reusable Cleaner Instance

```rust
use markdown_ai_cite_remove::CitationCleaner;

let cleaner = CitationCleaner::new();

// Reuse for multiple documents
let doc1 = cleaner.clean("First document[1].");
let doc2 = cleaner.clean("Second document[2][3].");
let doc3 = cleaner.clean("Third document[source:1].");
```

<br />

---

## 🧪 Examples

See the [`examples/`](examples/) directory for more:

- [`basic_usage.rs`](examples/basic_usage.rs) - Simple examples
- [`custom_config.rs`](examples/custom_config.rs) - Configuration options

Run examples:

```bash
cargo run --example basic_usage
cargo run --example custom_config
```

<br />

---

## 🏎️ Performance

<br />

### Running Benchmarks

```bash
# Run all benchmarks
cargo bench

# Run specific benchmark
cargo bench chatgpt_format

# Save baseline for comparison
cargo bench -- --save-baseline main

# Compare against baseline
cargo bench -- --baseline main

# View results (after running benchmarks)
open target/criterion/report/index.html  # macOS
xdg-open target/criterion/report/index.html  # Linux
start target/criterion/report/index.html  # Windows
```

<br />

**Note about benchmark output:**
- Tests shown as "ignored" during `cargo bench` is **normal behavior** - regular tests are skipped during benchmarking to avoid interference
- Outliers (3-13% of measurements) are normal due to OS scheduling and CPU frequency scaling
- "Gnuplot not found" warning is harmless - Criterion uses an alternative plotting backend
- **With Gnuplot installed**: Interactive HTML reports with charts are generated in `target/criterion/report/`

<br />

### Performance Characteristics

Typical performance on modern hardware (Apple Silicon M-series):

| Benchmark | Time | Throughput | Notes |
|-----------|------|------------|-------|
| Simple inline citations | ~580 ns | 91 MiB/s | Single sentence |
| Complex document | ~2.5 μs | 287 MiB/s | Multiple sections |
| Real ChatGPT output | ~18 μs | 645 MiB/s | 11.8 KB document |
| Real Perplexity output | ~245 μs | 224 MiB/s | 54.9 KB document |
| Batch (5 documents) | ~2.2 μs | 43 MiB/s | Total for all 5 |
| No citations (passthrough) | ~320 ns | 393 MiB/s | Fastest path |

<br />

**Key Insights:**
- **Throughput**: 100-650 MB/s depending on document complexity
- **Latency**: Sub-microsecond to ~250 μs for large documents
- **Scalability**: Linear with document size
- **Memory**: ~200-300 bytes per operation

<br />

---

## 🧪 Testing

This library has **100%+ test coverage** with comprehensive edge case testing.

<br />

### Running Tests

```bash
# Run all tests (unit + integration + doc tests)
cargo test

# Run with output visible
cargo test -- --nocapture

# Run specific test
cargo test test_real_world_chatgpt

# Run only unit tests
cargo test --lib

# Run only integration tests
cargo test --test integration_tests

# Run tests with all features enabled
cargo test --all-features
```

<br />

### Test Coverage

- **58 total tests** covering all functionality
- **18 unit tests** - Core logic, patterns, configuration
- **36 integration tests** - Real-world scenarios, edge cases
- **4 doc tests** - Documentation examples

<br />

**What's tested:**
- ✅ All citation formats (numeric, named, reference links)
- ✅ Real AI outputs (ChatGPT, Perplexity)
- ✅ Edge cases (empty strings, no citations, only citations)
- ✅ Unicode and emoji support
- ✅ Large documents (1000+ citations)
- ✅ Configuration variations
- ✅ Markdown preservation (formatting, links, images)

<br />

### Understanding Test Output

When running `cargo bench`, you'll see tests marked as "ignored" - **this is normal**. Rust automatically skips regular tests during benchmarking to avoid timing interference. All tests pass when running `cargo test`.

<br />

---

## 🔧 Troubleshooting

<br />

### Common Issues

<br />

**Q: Why do tests show as "ignored" when running `cargo bench`?**

A: This is normal Rust behavior. When running benchmarks, regular tests are automatically skipped to avoid interfering with timing measurements. All tests pass when you run `cargo test`. See [BENCHMARKING.md](docs/performance/BENCHMARKING.md) for details.

<br />

**Q: What does "Gnuplot not found, using plotters backend" mean?**

A: This is just an informational message. Criterion (the benchmarking library) can use Gnuplot for visualization, but falls back to an alternative plotting backend if it's not installed. Benchmarks still run correctly. To install Gnuplot:
- macOS: `brew install gnuplot`
- Ubuntu/Debian: `sudo apt-get install gnuplot`
- Windows: Download from http://www.gnuplot.info/

<br />

**Q: Why are there performance outliers in benchmarks?**

A: Outliers (typically 3-13% of measurements) are normal due to:
- Operating system scheduling
- CPU frequency scaling
- Background processes
- Cache effects

This is expected and doesn't indicate a problem. Criterion automatically detects and reports outliers.

<br />

**Q: The CLI tool isn't found after installation**

A: Make sure Cargo's bin directory is in your PATH:

```bash
# Add to ~/.bashrc, ~/.zshrc, or equivalent
export PATH="$HOME/.cargo/bin:$PATH"

# Then reload your shell
source ~/.bashrc  # or source ~/.zshrc
```

<br />

**Q: How do I know if citations were actually removed?**

A: Use the `--verbose` flag to see before/after sizes:

```bash
md-cite-clean input.md -o output.md --verbose
```

<br />

### Getting Help

- **Issues**: Report bugs or request features on [GitHub](https://github.com/opensite-ai/markdown-ai-cite-remove/issues)
- **Documentation**: Run `cargo doc --open` for full API docs
- **Examples**: Check the `examples/` directory for working code

<br />

---

## 🤝 Contributing

Built by [OpenSite AI](https://opensite.ai) for the developer community.

Contributions welcome! Please feel free to submit a Pull Request.

<br />

### Development Setup

```bash
# Clone the repository
git clone https://github.com/opensite-ai/markdown-ai-cite-remove.git
cd markdown-ai-cite-remove

# Run tests
cargo test

# Run benchmarks
cargo bench

# Build documentation
cargo doc --open

# Format code
cargo fmt

# Run linter
cargo clippy
```

<br />

---

## 📄 License

Licensed under either of:

- Apache License, Version 2.0 ([LICENSE-APACHE](LICENSE-APACHE) or http://www.apache.org/licenses/LICENSE-2.0)
- MIT license ([LICENSE-MIT](LICENSE-MIT) or http://opensource.org/licenses/MIT)

at your option.
