Have anyone reproduced the result?

[qiaopt]

I have tried to reproduce the result, by got QWK much less than that in the paper. Here is my log for prompt 1, fold_0:

log.txt

Did i do something wrong?

[DamonCC]

I also got a similar result, with Kappa scores ranging from 0.5 to 0.6. fold == 0, prompt == 1, all other parameters are default values.

How did you get the final result? As I saw in the source code, the author run 50 epochs on one fold and get the final dev score and test score. Should I run 50 epochs individually in each fold, and average their test results as the final experimental result?

[jkdufair]

I am also trying to replicate the results and getting similar outcomes. I've also tried varying the seed as described in the paper, to no avail.

[jkdufair]

I was able to replicate the results with QWKs in the .8 range. You'll need to utilize the embeddings file, as described here. Additionally, when you download the file from the link in the FAQ, the embeddings values are separated by commas but this repo expects them to be separated by spaces. I was able to accomplish this with

sed -ri ':a;s/(\ [^,]*),/\1 /;ta' embeddings.w2v.txt

@kavehtp Perhaps you want to update the FAQ to reflect this?

Thanks for making this repo available!

[jkdufair]

Also, my understanding from the paper is that the best results used a combination of CNN & RNN (LSTM). When I was able to replicate, I passed --cnndim 50 as well. I do not believe CNN is defaulted in parameters.

[nahos]

What versions of python,theano,keras and tensorflow did you use? I am facing issues with tensorflow.

[NNNNNaaaaaa]

Also, my understanding from the paper is that the best results used a combination of CNN & RNN (LSTM). When I was able to replicate, I passed --cnndim 50 as well. I do not believe CNN is defaulted in parameters.

Even with --cnndim 50 and the embeddings file, I still get the highest QWK with 0.556 for prompt 1, fold_0. Did you use any parameters? And how did you deal with words tagged "<unk> <num> <pad>"? Thank you so much!

[philiphaddad97]

Did anyone get the same result as mentioned in the paper?