Skip to content

Commit

Permalink
y
Browse files Browse the repository at this point in the history
  • Loading branch information
gbenson committed Jun 1, 2024
1 parent ab0004c commit cd00fd9
Showing 1 changed file with 31 additions and 16 deletions.
47 changes: 31 additions & 16 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -32,24 +32,39 @@ to care about it.

this shiz aims to do the same but for HTML, such that:

> <code>
X
> <code> X
becomes:

> <code style="background-color: #ccbfee;">&lt;<code><code style="background-color: #beedc6;">html<code><code style="background-color: #f6d9ab;">&gt;<code><code style="background-color: #f4aeb1;">&lt;<code><code style="background-color: #a4dcf3;">head<code><code style="background-color: #ccbfee;">&gt;<code><code style="background-color: #beedc6;">&lt;<code><code style="background-color: #f6d9ab;">meta<code><code style="background-color: #f4aeb1;">_<code><code style="background-color: #a4dcf3;">http<code><code style="background-color: #ccbfee;">equiv<code><code style="background-color: #beedc6;">=<code><code style="background-color: #f6d9ab;">utf<code><code style="background-color: #f4aeb1;">8<code><code style="background-color: #a4dcf3;">&gt;<code><code style="background-color: #ccbfee;">&lt;/<code><code style="background-color: #beedc6;">meta<code><code style="background-color: #f6d9ab;">&gt;<code>
- <code style="background-color: #f4aeb1;">a<code>
- <code style="background-color: #a4dcf3;">b<code>
- <code style="background-color: #ccbfee;">c<code>
- <code style="background-color: #beedc6;">d<code>
- <code style="background-color: #f6d9ab;">e<code>
- <code style="background-color: #f4aeb1;">f<code>
- <code style="background-color: #a4dcf3;">g<code>
- <code style="background-color: #ccbfee;">h<code>
- <code style="background-color: #beedc6;">i<code>
- <code style="background-color: #f6d9ab;">j<code>
- <code style="background-color: #f4aeb1;">k<code>
- <code style="background-color: #a4dcf3;">l<code>
> <code style="background-color: #ccbfee;">&lt;</code>
> <code style="background-color: #beedc6;">html</code>
> <code style="background-color: #f6d9ab;">&gt;</code>
> <code style="background-color: #f4aeb1;">&lt;</code>
> <code style="background-color: #a4dcf3;">head</code>
> <code style="background-color: #ccbfee;">&gt;</code>
> <code style="background-color: #beedc6;">&lt;</code>
> <code style="background-color: #f6d9ab;">meta</code>
> <code style="background-color: #f4aeb1;">_</code>
> <code style="background-color: #a4dcf3;">http</code>
> <code style="background-color: #ccbfee;">equiv</code>
> <code style="background-color: #beedc6;">=</code>
> <code style="background-color: #f6d9ab;">utf</code>
> <code style="background-color: #f4aeb1;">8</code>
> <code style="background-color: #a4dcf3;">&gt;</code>
> <code style="background-color: #ccbfee;">&lt;/</code>
> <code style="background-color: #beedc6;">meta</code>
> <code style="background-color: #f6d9ab;">&gt;</code>
> <code style="background-color: #f4aeb1;">a</code>
> <code style="background-color: #a4dcf3;">b</code>
> <code style="background-color: #ccbfee;">c</code>
> <code style="background-color: #beedc6;">d</code>
> <code style="background-color: #f6d9ab;">e</code>
> <code style="background-color: #f4aeb1;">f</code>
> <code style="background-color: #a4dcf3;">g</code>
> <code style="background-color: #ccbfee;">h</code>
> <code style="background-color: #beedc6;">i</code>
> <code style="background-color: #f6d9ab;">j</code>
> <code style="background-color: #f4aeb1;">k</code>
> <code style="background-color: #a4dcf3;">l</code>

tokenizers for generation need to be able to decode reversibly,
Expand Down

0 comments on commit cd00fd9

Please sign in to comment.