# xi-unicode

This crate contains unicode utilites adapted for working with non-contiguous
bytes (such as a rope.)

Much of the contents of this repo are generated automatically by scripts from
Unicode data files.

This *current* file is the result of some archaeology; documentation on how to
rebuild the various files was missing, and I am attempting to reconstruct it.

## data

Constructing the various tables require the various data files. These are
available through the [components of the Unicode standard][components]
directory, for a given unicode version. In particular, we require
`LineBreak.txt`.

This file should be placed in a directory: I use `data`.

- `src/tables.rs` is generated with the script located at `tools/mk_tables.py`,
  and can be built with,

  ```sh
  $ python3 tools/mk_tables.py data > src/tables.rs
  ```
  where `data` is the path to the created data directory.

- the unit tests in `src/lib.rs` are also generated by this script, by passing
  the `--tests` and `--tests-str` flags (separately, to separate invocations) of
  the script, and then copying the output over into the body of these tests.


[components]: https://www.unicode.org/versions/components-14.0.0.html
