Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

when i train the model,it raise an error,please help me! #3

Open
lzx168 opened this issue Dec 18, 2017 · 2 comments
Open

when i train the model,it raise an error,please help me! #3

lzx168 opened this issue Dec 18, 2017 · 2 comments

Comments

@lzx168
Copy link

lzx168 commented Dec 18, 2017

20/10000 [..............................] - ETA: 1:00:01 - loss: 2282.6917
40/10000 [..............................] - ETA: 31:47 - loss: 1886.1443
60/10000 [..............................] - ETA: 22:25 - loss: 1565.0140
80/10000 [..............................] - ETA: 17:43 - loss: 1411.5662
100/10000 [..............................] - ETA: 14:52 - loss: 1321.03732017-12-18 09:00:46.923191: W tensorflow/core/framework/op_kernel.cc:1152] Invalid argument: Saw a non-null label (index >= num_classes - 1) following a null label, batch: 10 num_classes: 73 labels: 1,43,27,1,6,22,1,27,21,5,54,41,8,28,37,4,32,38,1,13,34,2,41,28,1,15,19,1,46,13,2,24,6,1,13,65,4,19,47,1,51,10,1,31,28,8,63,39,4,37,21,4,42,12,3,17,38,5,57,68,1,13,39,1,13,68,1,13,30,1,30,16,71
Traceback (most recent call last):
File "/home/dell/sourcecode/aa/speech_to_text_train.py", line 120, in
batch_size=20,epochs=1000);
File "/usr/local/lib/python2.7/dist-packages/keras/engine/training.py", line 1631, in fit
validation_steps=validation_steps)
File "/usr/local/lib/python2.7/dist-packages/keras/engine/training.py", line 1213, in _fit_loop
outs = f(ins_batch)
File "/usr/local/lib/python2.7/dist-packages/keras/backend/tensorflow_backend.py", line 2332, in call
**self.session_kwargs)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 778, in run
run_metadata_ptr)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 982, in _run
feed_dict_string, options, run_metadata)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 1032, in _do_run
target_list, options, run_metadata)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 1052, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Saw a non-null label (index >= num_classes - 1) following a null label, batch: 10 num_classes: 73 labels: 1,43,27,1,6,22,1,27,21,5,54,41,8,28,37,4,32,38,1,13,34,2,41,28,1,15,19,1,46,13,2,24,6,1,13,65,4,19,47,1,51,10,1,31,28,8,63,39,4,37,21,4,42,12,3,17,38,5,57,68,1,13,39,1,13,68,1,13,30,1,30,16,71
[[Node: ctc/CTCLoss = CTCLoss[ctc_merge_repeated=true, preprocess_collapse_repeated=false, _device="/job:localhost/replica:0/task:0/cpu:0"](ctc/Log/_1659, ctc/ToInt64/_1661, ctc/ToInt32_2/_1663, ctc/ToInt32_1/_1665)]]
[[Node: training/Adam/gradients/ctc/CTCLoss_grad/mul/_1693 = _Recvclient_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:0", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_20095_training/Adam/gradients/ctc/CTCLoss_grad/mul", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/gpu:0"]]

Caused by op u'ctc/CTCLoss', defined at:
File "/home/dell/sourcecode/aa/speech_to_text_train.py", line 110, in
loss_out=Lambda(ctc_lambda_function,output_shape=(1,),name="ctc")([y_true_input,logit,logit_length_input,y_true_length_input])
File "/usr/local/lib/python2.7/dist-packages/keras/engine/topology.py", line 603, in call
output = self.call(inputs, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/keras/layers/core.py", line 651, in call
return self.function(inputs, **arguments)
File "/home/dell/sourcecode/aa/speech_to_text_train.py", line 99, in ctc_lambda_function
return K.ctc_batch_cost(y_true_input,logit,logit_length_input,y_true_length_input);
File "/usr/local/lib/python2.7/dist-packages/keras/backend/tensorflow_backend.py", line 3745, in ctc_batch_cost
sequence_length=input_length), 1)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/ctc_ops.py", line 145, in ctc_loss
ctc_merge_repeated=ctc_merge_repeated)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/gen_ctc_ops.py", line 164, in _ctc_loss
name=name)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/op_def_library.py", line 768, in apply_op
op_def=op_def)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 2336, in create_op
original_op=self._default_original_op, op_def=op_def)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 1228, in init
self._traceback = _extract_stack()

InvalidArgumentError (see above for traceback): Saw a non-null label (index >= num_classes - 1) following a null label, batch: 10 num_classes: 73 labels: 1,43,27,1,6,22,1,27,21,5,54,41,8,28,37,4,32,38,1,13,34,2,41,28,1,15,19,1,46,13,2,24,6,1,13,65,4,19,47,1,51,10,1,31,28,8,63,39,4,37,21,4,42,12,3,17,38,5,57,68,1,13,39,1,13,68,1,13,30,1,30,16,71
[[Node: ctc/CTCLoss = CTCLoss[ctc_merge_repeated=true, preprocess_collapse_repeated=false, _device="/job:localhost/replica:0/task:0/cpu:0"](ctc/Log/_1659, ctc/ToInt64/_1661, ctc/ToInt32_2/_1663, ctc/ToInt32_1/_1665)]]
[[Node: training/Adam/gradients/ctc/CTCLoss_grad/mul/_1693 = _Recvclient_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:0", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_20095_training/Adam/gradients/ctc/CTCLoss_grad/mul", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/gpu:0"]]

Process finished with exit code 1

@moneypi
Copy link

moneypi commented Jan 20, 2018

same problem,anyone help?

@zhaowz
Copy link

zhaowz commented Mar 2, 2018

the author use python3, so char_index will have 2665 character.
in python 2.x,you need decode word:
all_words=[]
for i in np.arange(0,len(texts)):
for j in np.arange(1,len(texts[i])):
all_words.append(texts[i][j])
all_words=[i.decode() for i in all_words]

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants