Has the work of adding BERT done?

[SparkJiao]

Hi,
do you have finished the word of adding bert? Could you please share the results?
Thank you very much!

[matthew-z]

I have added BERT, but it is very different the original one:

1. I only used the BERT embedding of the last word piece of each word as I have not converted the labels into word-piece based.

2. Question and context are not encoded together by BERT in my implementation (the BERT paper concat them as input). Currently, they are encoded separately.

As the result, this BERT version only achieved about 78 F1. Due to the hardware limitation, I only trained it with Mixed Precision (I am not sure if it is also a reason)

[SparkJiao]

Sorry to say, I have got a similar worse performance on biadf++ using allennlp.
I think there are two problems here.
The first is that we need to calculate an average hidden state of the word pieces for one word instead of the last piece. This method comes from here: https://arxiv.org/abs/1812.03593. This may be easy to modified but recently I don't have much time.
The second thing is that because of limitation of max sequence length, a sentence may be split as many segments, and for details: google-research/bert#66 (comment). But I think this may be difficult to implement using Allennlp.
And I have also used bert embedding like https://github.com/huggingface/pytorch-pretrained-BERT/blob/master/examples/extract_features.py, and outperformed than elmo in another reading comprehension model. But due to some difficulty, I didn't used slide window like mentioned in the google-research.
Hope these will help you!
If I have completed my model, I will come back to do something ~

[matthew-z]

Thank you for sharing your experience! It is very helpful.

I will also try to improve it and will let you know when I have any good news.

[jind11]

Any updates from you on the BERT related experiments? Thank you for sharing your experience!