Accuracy 0 with latest code

[ashish1405]

Hello ilovin,

I'm trying to run your exact code(latest, standard) to see If I can perform trainning. But even after 300 epoch the accuracy is 0. Epoch 354/10000, accuracy = 0.000,avg_train_cost = 22.038, lastbatch_err = 0.980, time = 328.230,lr=0.00000000

I've seen issue #1 and #2, and checked that your latest code contains all the changes still the accuracy is 0.
python 2.7 and 3.5.2
tensorflow 1.2.1 without WrapCTC
train data set size 128000

Can you please help. Thanks

[ilovin]

could you upload the decoded results?
In general, if you cannot see significant change(of the decoded result) in 15 epochs, you may need to
adjust the lr etc.

[ashish1405]

Hello Scott,

Thanks, I did a fresh checkout and ran 15 epocs, current lr is 1e-3. Below is my output
https://gist.github.com/ashish1405/350a27ab7fda2f92fcb3d07f34cc6f9f

Can you please suggest, what to do next?

[ilovin]

maybe the lr decay too fast, you may fix the lr(set decay rate to 1.0) to 1e-3, 5e-4 or 1e-4 and run for another 15 epochs

[ashish1405]

Thanks Scott, I tried both lr=5e-4 and 1e-4 (with decay rate 1.0) still it is not able to converge still after 20 epocs

[QaisarRajput]

Facing the same issue. accuracy does not increase.

[ilovin]

You may have a try the dev version, it adds the CNN layer to extract features, and it is much easy to converge.

[ashish1405]

But Scott the dev version uses WrapCTC. will it work with tensorflow 1.2?

[ilovin]

---update---
some people say although the warpCTC test cannot pass, it still works. have not testified it.

it may not.
Take look at #3

[ashish1405]

Ok thanks, will try to run on dev and TF 1.0.1

[QaisarRajput]

@ashish1405 if it works please let me know. i am getting a hard time installing warp-ctc with tensorflow.

[QaisarRajput]

It worked for me. i had to build and install warpCTC with tensorflow version 1.0.1.

[ashish1405]

wow that is great. Will try with that. Any issues you faced in installing wrapCTC?
BTW @ilovin do you have any idea, will this work with dual line text?

[ashish1405]

Yes indeed it works with wrapCTC. Had some minor issues in building wrapCTC. We built wrapCTC on ubuntu. Initially we came across an issue undefined symbol: _ZN10tensorflow7strings6StrCatB5cxx11ERKNS0_8AlphaNumE. So to over come that we need to build Tensorflow with below options

build -c opt --copt=-mavx --copt=-mavx2 --copt=-mfma --copt=-mfpmath=both --copt=-msse4.2 --cxxopt="-D_GLIBCXX_USE_CXX11_ABI=0" -k //tensorflow/tools/pip_package:build_pip_package

the D_GLIBCXX_USE_CXX11_ABI is important. Also added the same flag to wrapCTC compilation.

BTW tried with the dev branch, still need to check with the master branch of wrapCTC.

[AllenDuane]

Facing the same issue. accuracy does not increase at all.
@ashish1405 Have you solved this problem?

[ashish1405]

Simon please use dev branch and compile wrapCTC with tensorflow 1.0