Colored Master Model

[TryAnotherName]

Hello. I need a model to define only one character in an image. The character dictionary consists of numbers only.
Also I would like to be able to choose a color or grayscale option for using the image. I was only able to set up CRNN-TPS to work with these configurations. I tried to set up Master or SATRN models, but during training in the logs, loss = nan.
2022-11-07 19:39:56,883 - mmocr - INFO - Epoch [5][238/275] lr: 1.000e-04, eta: 4:18:18, time: 1.043, data_time: 0.028, memory: 10195, loss_ce: nan, loss: nan
Here is an example image from my dataset:

Here is my config file for SATRN model:
log_config = dict(interval=5, hooks=[dict(type='TextLoggerHook')]) dist_params = dict(backend='nccl') log_level = 'INFO' load_from = None resume_from = None workflow = [('train', 1)] opencv_num_threads = 0 mp_start_method = 'fork' img_norm_cfg = dict(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225]) train_pipeline = [ dict(type='LoadImageFromFile'), dict( type='ResizeOCR', height=32, min_width=100, max_width=100, keep_aspect_ratio=False, width_downsample_ratio=0.25), dict(type='ToTensorOCR'), dict( type='NormalizeOCR', mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225]), dict( type='Collect', keys=['img'], meta_keys=[ 'filename', 'ori_shape', 'img_shape', 'text', 'valid_ratio', 'resize_shape' ]) ] test_pipeline = [ dict(type='LoadImageFromFile'), dict( type='MultiRotateAugOCR', rotate_degrees=[0, 90, 270], transforms=[ dict( type='ResizeOCR', height=32, min_width=100, max_width=100, keep_aspect_ratio=False, width_downsample_ratio=0.25), dict(type='ToTensorOCR'), dict( type='NormalizeOCR', mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225]), dict( type='Collect', keys=['img'], meta_keys=[ 'filename', 'ori_shape', 'img_shape', 'valid_ratio', 'resize_shape', 'img_norm_cfg', 'ori_filename' ]) ]) ] dataset_type = 'OCRDataset' root = 'Datasets/Single_digit_recognition/' img_prefix = 'Datasets/Single_digit_recognition/train/images/' train1 = dict( type='OCRDataset', img_prefix='Datasets/Single_digit_recognition/train/images/', ann_file='Datasets/Single_digit_recognition/train/labels.txt', loader=dict( type='AnnFileLoader', repeat=1, file_format='txt', file_storage_backend='disk', parser=dict( type='LineStrParser', keys=['filename', 'text'], keys_idx=[0, 1], separator=';')), pipeline=None, test_mode=False) train2 = dict( type='OCRDataset', img_prefix='Datasets/Single_digit_recognition/train2/images/', ann_file='Datasets/Single_digit_recognition/train2/labels.txt', loader=dict( type='AnnFileLoader', repeat=1, file_format='txt', file_storage_backend='disk', parser=dict( type='LineStrParser', keys=['filename', 'text'], keys_idx=[0, 1], separator=';')), pipeline=None, test_mode=False) train3 = dict( type='OCRDataset', img_prefix='New_Datasets/Digit_recognition/train/images/', ann_file='New_Datasets/Digit_recognition/train/labels.txt', loader=dict( type='AnnFileLoader', repeat=1, file_format='txt', file_storage_backend='disk', parser=dict( type='LineStrParser', keys=['filename', 'text'], keys_idx=[0, 1], separator=';')), pipeline=None, test_mode=False) test = dict( type='OCRDataset', img_prefix='New_Datasets/Digit_recognition/test/images/', ann_file='New_Datasets/Digit_recognition/test/labels.txt', loader=dict( type='AnnFileLoader', repeat=1, file_format='txt', file_storage_backend='disk', parser=dict( type='LineStrParser', keys=['filename', 'text'], keys_idx=[0, 1], separator=';')), pipeline=None, test_mode=True) val1 = dict( type='OCRDataset', img_prefix='Datasets/Single_digit_recognition/val/images/', ann_file='Datasets/Single_digit_recognition/val/labels.txt', loader=dict( type='AnnFileLoader', repeat=1, file_format='txt', file_storage_backend='disk', parser=dict( type='LineStrParser', keys=['filename', 'text'], keys_idx=[0, 1], separator=';')), pipeline=None, test_mode=True) val2 = dict( type='OCRDataset', img_prefix='Datasets/Single_digit_recognition/val2/images/', ann_file='Datasets/Single_digit_recognition/val2/labels.txt', loader=dict( type='AnnFileLoader', repeat=1, file_format='txt', file_storage_backend='disk', parser=dict( type='LineStrParser', keys=['filename', 'text'], keys_idx=[0, 1], separator=';')), pipeline=None, test_mode=True) val3 = dict( type='OCRDataset', img_prefix='New_Datasets/Digit_recognition/val/images/', ann_file='New_Datasets/Digit_recognition/val/labels.txt', loader=dict( type='AnnFileLoader', repeat=1, file_format='txt', file_storage_backend='disk', parser=dict( type='LineStrParser', keys=['filename', 'text'], keys_idx=[0, 1], separator=';')), pipeline=None, test_mode=True) train_list = [train1, train3] test_list = [test] val_list = [val3] optimizer = dict(type='Adam', lr=0.0003) optimizer_config = dict(grad_clip=None) lr_config = dict(policy='step', step=[3, 4]) runner = dict(type='EpochBasedRunner', max_epochs=6) checkpoint_config = dict(interval=1) max_seq_len = 1 label_convertor = dict( type='AttnConvertor', dict_list=['0','1','2','3','4','5','6','7','8','9'], with_unknown=True, max_seq_len=1) model = dict( type='SATRN', backbone=dict(type='ShallowCNN', input_channels=3, hidden_dim=512), encoder=dict( type='SatrnEncoder', n_layers=12, n_head=8, d_k=64, d_v=64, d_model=512, n_position=100, d_inner=2048, dropout=0.1), decoder=dict( type='NRTRDecoder', n_layers=6, d_embedding=512, n_head=8, d_model=512, d_inner=2048, d_k=64, d_v=64), loss=dict(type='TFLoss'), label_convertor=dict( type='AttnConvertor', dict_list=['0','1','2','3','4','5','6','7','8','9'], with_unknown=True, max_seq_len=1), max_seq_len=1) data = dict( samples_per_gpu=64, workers_per_gpu=0, val_dataloader=dict(samples_per_gpu=64), test_dataloader=dict(samples_per_gpu=64), train=dict( type='UniformConcatDataset', datasets=train_list, pipeline=[ dict(type='LoadImageFromFile'), dict( type='ResizeOCR', height=32, min_width=100, max_width=100, keep_aspect_ratio=False, width_downsample_ratio=0.25), dict(type='ToTensorOCR'), dict( type='NormalizeOCR', mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225]), dict( type='Collect', keys=['img'], meta_keys=[ 'filename', 'ori_shape', 'img_shape', 'text', 'valid_ratio', 'resize_shape' ]) ]), val=dict( type='UniformConcatDataset', datasets=val_list, pipeline=[ dict(type='LoadImageFromFile'), dict( type='MultiRotateAugOCR', rotate_degrees=[0, 90, 270], transforms=[ dict( type='ResizeOCR', height=32, min_width=100, max_width=100, keep_aspect_ratio=False, width_downsample_ratio=0.25), dict(type='ToTensorOCR'), dict( type='NormalizeOCR', mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225]), dict( type='Collect', keys=['img'], meta_keys=[ 'filename', 'ori_shape', 'img_shape', 'valid_ratio', 'resize_shape', 'img_norm_cfg', 'ori_filename' ]) ]) ]), test=dict( type='UniformConcatDataset', datasets=test_list, pipeline=[ dict(type='LoadImageFromFile'), dict( type='MultiRotateAugOCR', rotate_degrees=[0, 90, 270], transforms=[ dict( type='ResizeOCR', height=32, min_width=100, max_width=100, keep_aspect_ratio=False, width_downsample_ratio=0.25), dict(type='ToTensorOCR'), dict( type='NormalizeOCR', mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225]), dict( type='Collect', keys=['img'], meta_keys=[ 'filename', 'ori_shape', 'img_shape', 'valid_ratio', 'resize_shape', 'img_norm_cfg', 'ori_filename' ]) ]) ])) evaluation = dict(interval=1, metric='acc')

I would be grateful if you tell me what I need to specify in the config files.

[gaotongxiao]

I didn't quite get your question. Did you want to train Master/SATRN on your dataset in grayscale mode?

[gaotongxiao]

BTW it'd be better to reformat your config for better readability

[TryAnotherName]

I didn't quite get your question. Did you want to train Master/SATRN on your dataset in grayscale mode?

Not exactly. From what I understand, the default config uses dict(type='LoadImageFromFile') and that means loading images in grayscale. I would like to be able to use images in color format, but that's not that important. My main problem is that if you specify max_seq_len=1 in the MASTER / SATRN config of the models, when training, you get loss = nan. And I only need to recognize 1 character, since the localization task has already been completed in advance.

[TryAnotherName]

BTW it'd be better to reformat your config for better readability

Sorry for that. I didn’t find a way to conveniently copy the code here directly into the comment.
Here is my config file for SATRN model in .txt format.
recog_satrn_config.txt

[gaotongxiao]

From what I understand, the default config uses dict(type='LoadImageFromFile') and that means loading images in grayscale.

No, it's not. LoadImageFromFile loads images in colors by default. CRNN only accepts grayscale images and its pipeline specifies color_type as grayscale.

mmocr/configs/_base_/recog_pipelines/crnn_pipeline.py

Line 4 in a36de7b

dict(type='LoadImageFromFile', color_type='grayscale'),

I would like to be able to use images in color format, but that's not that important. My main problem is that if you specify max_seq_len=1 in the MASTER / SATRN config of the models, when training, you get loss = nan. And I only need to recognize 1 character, since the localization task has already been completed in advance.

max_seq_len=1 is not a good choice MMOCR 0.x. During training, when converting the target string into an index tensor, the first character will always be <BOS>.

mmocr/mmocr/models/textrecog/convertors/attn.py

Line 96 in 0421659

src_target[0] = self.start_idx

That is, the actual target string for the model will always be a meaningless <BOS>. Worse still, TFLoss is also built on such an assumption, resulting in a deliberate truncation of the first target character. To retain the length consistency between predictions and targets, TFLoss also discards the last character in the model's prediction (because the model won't output <SOS> at the beginning).

mmocr/mmocr/models/textrecog/losses/ce_loss.py

Lines 123 to 126 in 6512f0a

	def format(self, outputs, targets_dict):
	outputs = outputs[:, :-1, :].contiguous()
	targets = targets_dict['padded_targets']
	targets = targets[:, 1:].contiguous()

Discarding one character each results in two invalid tensors and consequently your 'nan' loss. As a solution, you should set your max_seq_len to 3 to reserve the room for <BOS> and <EOS> tokens.

All these implicit assumptions were deemed as a design flaw in MMOCR 0.x, which has already been addressed in MMOCR 1.0. Now you are able to turn off these <BOS> and <EOS> tokens in Dictionary and max_seq_len=1 should work as expected. You are welcome to upgrade to this better and overall more consistent version, if time permits.

[TryAnotherName]

From what I understand, the default config uses dict(type='LoadImageFromFile') and that means loading images in grayscale.

No, it's not. LoadImageFromFile loads images in colors by default. CRNN only accepts grayscale images and its pipeline specifies color_type as grayscale.

mmocr/configs/_base_/recog_pipelines/crnn_pipeline.py

Line 4 in a36de7b

dict(type='LoadImageFromFile', color_type='grayscale'),

I would like to be able to use images in color format, but that's not that important. My main problem is that if you specify max_seq_len=1 in the MASTER / SATRN config of the models, when training, you get loss = nan. And I only need to recognize 1 character, since the localization task has already been completed in advance.

max_seq_len=1 is not a good choice MMOCR 0.x. During training, when converting the target string into an index tensor, the first character will always be <BOS>.

mmocr/mmocr/models/textrecog/convertors/attn.py

Line 96 in 0421659

src_target[0] = self.start_idx

That is, the actual target string for the model will always be a meaningless <BOS>. Worse still, TFLoss is also built on such an assumption, resulting in a deliberate truncation of the first target character. To retain the length consistency between predictions and targets, TFLoss also discards the last character in the model's prediction (because the model won't output <SOS> at the beginning).

mmocr/mmocr/models/textrecog/losses/ce_loss.py

Lines 123 to 126 in 6512f0a

	def format(self, outputs, targets_dict):
	outputs = outputs[:, :-1, :].contiguous()
	targets = targets_dict['padded_targets']
	targets = targets[:, 1:].contiguous()

Discarding one character each results in two invalid tensors and consequently your 'nan' loss. As a solution, you should set your max_seq_len to 3 to reserve the room for <BOS> and <EOS> tokens.

All these implicit assumptions were deemed as a design flaw in MMOCR 0.x, which has already been addressed in MMOCR 1.0. Now you are able to turn off these <BOS> and <EOS> tokens in Dictionary and max_seq_len=1 should work as expected. You are welcome to upgrade to this better and overall more consistent version, if time permits.

WOW. I did not expect such a detailed and exhaustive answer. Thank you very much! I will try)