mongolian-text-classification

Mongolian cyrillic text classification with modern tensorflow and some fine tuning on TugsTugi's BERT model.

Load Mongolian BERT in Tensorflow 2

Generate text using Mongolian BERT

Visualize Mongolian BERT using bertviz and pytorch model

Fine tuning TugsTugi's Mongolian BERT model

On TPU mode, loading checkpoints from the file system doesn't supported by the bert and bucket should be used.

Fine tuning mongolian BERT on TPU, You need own bucket in order to finetune on TPU

Fine tune a mongolian BERT on GPU, a lot of computation needed, a low batch size matters due to memory limit

Classifiers using simple neural networks

No 01, Simplest classifier

No 02, Pretrained Word2Vec initialization from Facebook's fasttext, kind of transfer learningish. Embedding layer is not trainable in this case and with trainable embedding layer

No 03, 1D Convolution and multiple 1D convnets

No 04, LSTM

Visualize RNN neuron firing in text generation

No 05, LSTM with Attention, visualization of attention scores in text classification

No 06, Classification with Mongolian BERT and Tensorflow 2.0, with frozen bert layers

No 07, Classification with Mongolian BERT large and HuggingFace and Tensorflow 2

Mongolian sentence interpolation experiments

Sequence loss in keras and tf2

Variational Auto Encoder for Mongolian text

Other experiments

Predict next word, greedy text generation

Series included(or will) followings

word2vec initialization, 1D Convolution, RNN variants, Attention, Some weights visualization for reasoning, Transformer, Techniques to handle longer texts and so on...

useful references and resources

• Mongolian BERT models https://github.com/tugstugi/mongolian-bert
• Mongolian NLP https://github.com/tugstugi/mongolian-nlp
• Eduge classification baseline using SVM https://colab.research.google.com/github/tugstugi/mongolian-nlp/blob/master/misc/Eduge_SVM.ipynb
• News crawler https://github.com/codelucas/newspaper

Images and screenshots