something wrong

[haha010508]

saving model to ./chkpt/model_epoch70.pt
do evaluation ......

do scoring ......

epoch = 70 cosine eer% = 99.95%

Drawing tsne of latent space ......
select speakers with utts large than 250
n_samples=6403, n_dim=(512,), n_labels=22
Computing t-SNE embedding epoch
Traceback (most recent call last):
File "train.py", line 294, in
tsne.main(path0,epoch)
File "/MG-master/tsne.py", line 70, in main
result = tsne.fit_transform(data)
File "/pytorch/lib64/python3.6/site-packages/sklearn/manifold/_t_sne.py", line 932, in fit_transform
embedding = self._fit(X)
File "/pytorch/lib64/python3.6/site-packages/sklearn/manifold/_t_sne.py", line 704, in _fit
dtype=[np.float32, np.float64])
File "/pytorch/lib64/python3.6/site-packages/sklearn/base.py", line 421, in _validate_data
X = check_array(X, **check_params)
File "/pytorch/lib64/python3.6/site-packages/sklearn/utils/validation.py", line 63, in inner_f
return f(*args, **kwargs)
File "/pytorch/lib64/python3.6/site-packages/sklearn/utils/validation.py", line 664, in check_array
allow_nan=force_all_finite == 'allow-nan')
File "/pytorch/lib64/python3.6/site-packages/sklearn/utils/validation.py", line 106, in _assert_all_finite
msg_dtype if msg_dtype is not None else X.dtype)
ValueError: Input contains NaN, infinity or a value too large for dtype('float32').

input data have NaN? why? can you help to fix it?

[Caiyq2019]

Whether it works well in ealier epoch?

[haha010508]

Whether it works well in ealier epoch?

yes, the result eer is below:epoch = 0 cosine-eer% = 10.06
epoch = 0 cosine-eer% = 10.28
epoch = 0 cosine-eer% = 10.20
epoch = 5 cosine-eer% = 8.15
epoch = 10 cosine-eer% = 7.52
epoch = 15 cosine-eer% = 7.03
epoch = 20 cosine-eer% = 6.89
epoch = 25 cosine-eer% = 6.97
epoch = 30 cosine-eer% = 6.86
epoch = 35 cosine-eer% = 6.67
epoch = 40 cosine-eer% = 6.59
epoch = 45 cosine-eer% = 6.72
epoch = 50 cosine-eer% = 6.78
epoch = 55 cosine-eer% = 6.70
epoch = 60 cosine-eer% = 6.64
epoch = 65 cosine-eer% = 6.64
epoch = 70 cosine-eer% = 99.95

[Caiyq2019]

The results look not bad.
This problem is complicated, and it is a training problem. When the input data distribution is complex, or the target distribution is complex, this flow-based network architecture occasionally has singular values. Lowering the learning rate will help. And we are considering replacing a more stable network structure. We will update the code if it is done.