# ethl

**WIP**

`ethl` is a powerful Ethereum log event ETL (Extract, Transform, Load) tool designed for efficient and reliable log processing. It provides:

- RPC utilities for streaming logs with support for multiple providers, fallback mechanisms, and retries.
- Tools to archive events into an opinionated Arrow + Parquet format for optimized storage and querying.
- Features enabling fast replay of specific events and continuous indexing for real-time use cases.

This tool was built to address the need for high-performance event processing and storage in Ethereum-based applications.

### Getting Started

To get started with `ethl`, add it to your `Cargo.toml` via `cargo install ethl`:

```toml
[dependencies]
ethl = "0.1"
```

Then, include it in your project:

```rust
use ethl;
```

For detailed examples and usage, check the [documentation](https://docs.rs/ethl).

### Development nodes (anvil, hardhat, ganache)

`ethl` treats RPC provider responses as authoritative. Local development nodes
expose two failure modes that can silently corrupt an event cache:

1. **Silent empty responses past tip.** Some nodes return `{"result": []}` for
   `eth_getLogs` on block ranges past their actual tip rather than erroring.
   `ethl` guards against this by calling `eth_blockNumber` whenever a batch
   returns no logs and refusing to advance the cursor when `to_block` exceeds
   the reported tip (yields `RpcError::CursorPastTip`). The overhead is one
   extra RPC per empty batch; batches that return logs are unaffected.

2. **Synthetic blocks from local mining.** Dev nodes mine new blocks in
   response to transactions you send, including simulations. Those blocks
   share a number with real chain blocks but contain only your local
   transactions — they are well-formed blocks as far as the API is concerned,
   so `ethl` cannot distinguish them from fork-proxied real blocks. If you
   index against a node that has been used for writes, **treat the cache as
   poisoned**: roll `block_height` back to a value from before the node was
   first used and rebuild from a real RPC.
