Code

Contextual tokenizer#1415

Code

Open

Jemoka wants to merge 20 commits intodevstanfordnlp/stanza:devfrom contextual_tokenizerstanfordnlp/stanza:contextual_tokenizerCopy head branch name to clipboard

+200-21

Commits

Commits on Sep 2, 2024

• initial work on contextual-aware tokenizer

  Jemoka

  committed
  271dcd9

  Copy full SHA for 271dcd9

• last mile tokenizer changes for running (still slow)

  Jemoka

  committed
  7e9388d

  Copy full SHA for 7e9388d

• small optmizitain changes

  Jemoka

  committed
  58e90b8

  Copy full SHA for 58e90b8

• remove extra spaces

  Jemoka

  committed
  1b9a9ce

  Copy full SHA for 1b9a9ce

• Merge branch 'contextual_tokenizer' of github.com:stanfordnlp/stanza into contextual_tokenizer

  Jemoka

  committed
  eef83c3

  Copy full SHA for eef83c3

• use mwt info

  Jemoka

  committed
  23d4ab5

  Copy full SHA for 23d4ab5

Commits on Sep 4, 2024

• make the tokenizer a smidge more efficinet

  Jemoka

  committed
  38a3398

  Copy full SHA for 38a3398

Commits on Sep 7, 2024

• now use an LSTM

  Jemoka

  committed
  e4b691a

  Copy full SHA for e4b691a

• Merge branch 'contextual_tokenizer' of github.com:stanfordnlp/stanza into contextual_tokenizer

  Jemoka

  committed
  393e655

  Copy full SHA for 393e655

Commits on Sep 9, 2024

• various tokenizer changes for a smaller model

  Jemoka

  committed
  ca1423b

  Copy full SHA for ca1423b

• some edits to second pass classifier

  Jemoka

  committed
  a8934d5

  Copy full SHA for a8934d5

• bump second pass start steps

  Jemoka

  committed
  b687c6f

  Copy full SHA for b687c6f

• fixing some ordering problems

  Jemoka

  committed
  49ef2e6

  Copy full SHA for 49ef2e6

• split on even using draft positions

  Jemoka

  committed
  58345ba

  Copy full SHA for 58345ba

• initially, an identity transform

  Jemoka

  committed
  337405c

  Copy full SHA for 337405c

• Merge branch 'contextual_tokenizer' of github.com:stanfordnlp/stanza into contextual_tokenizer

  Jemoka

  committed
  e46c639

  Copy full SHA for e46c639

• whopps, it was applied in the othre direction

  Jemoka

  committed
  e7ae33a

  Copy full SHA for e7ae33a

Commits on Sep 10, 2024

• add some dropout

  Jemoka

  committed
  d6e4276

  Copy full SHA for d6e4276

• include character information in the tokenizer model

  Jemoka

  committed
  7545f39

  Copy full SHA for 7545f39

Commits on Sep 12, 2024

• add a tiny bit of dropout

  Jemoka

  committed
  6fe35de

  Copy full SHA for 6fe35de