New configs for model testing

[airogachev]

It is possible to find configs for some datasets and models in the repo. However it's not clear how should a new config be properly added.
E.g. I tried to add a CTW1500 cofig for testing DBNet and it looks as follows:

_base_ = [
    '_base_dbnet_resnet50-dcnv2_fpnc.py',
    '../_base_/datasets/ctw1500.py',
    '../_base_/default_runtime.py',
    '../_base_/schedules/schedule_sgd_1200e.py',
]

# dataset settings
ctw_det_train = _base_.ctw_det_train
ctw_det_train.pipeline = _base_.train_pipeline
ctw_det_test = _base_.ctw_det_test
ctw_det_test.pipeline = _base_.test_pipeline

train_dataloader = dict(
    batch_size=4,
    num_workers=4,
    persistent_workers=True,
    sampler=dict(type='DefaultSampler', shuffle=True),
    dataset=ctw_det_train)

val_dataloader = dict(
    batch_size=1,
    num_workers=1,
    persistent_workers=True,
    sampler=dict(type='DefaultSampler', shuffle=False),
    dataset=ctw_det_test)

test_dataloader = val_dataloader

auto_scale_lr = dict(base_batch_size=4)

Then I used it for testing:

CUDA_VISIBLE_DEVICES=4 python tools/test.py configs/textdet/dbnet/dbnet_resnet50-dcnv2_fpnc_1200e_ctw1500.py weights/dbnet_resnet50-dcnv2_fpnc_1200e_icdar2015_20220828_124917-452c443c.pth --save-preds

And got icdar/precision: 0.2755 icdar/recall: 0.4091 icdar/hmean: 0.3293

Thus it's not clear whether the config is properly added and model just doesn't perform well at this case or whether I passed some parameters in a wrong way.

So, to sum up, it is enough to take existing config of some particular model and just change the way/name of the dataset or should I change some other parameters as well?

[gaotongxiao]

Your config looks alright, but the hyperparameters that work on icdar2015 may not work well on other datasets such as ctw1500. Usually, you will have to change the image size in the pipeline, and sometimes you may even have to change the learning schedule.

For some examples, you can examine the diff of PSE_CTW vs PSE_IC15, and PAN_CTW vs PAN_IC15 to get a sense of it

[airogachev]

TBH, I don't see the difference for test part, it looks like we have to use the same image size as we do not refit the model

[gaotongxiao]

Their test-time scales are different.

Take PSENet as an example:

mmocr/configs/textdet/psenet/psenet_resnet50_fpnf_600e_ctw1500.py

Line 24 in f30c16c

dict(type='Resize', scale=(1280, 1280), keep_ratio=True),

vs

mmocr/configs/textdet/psenet/_base_psenet_resnet50_fpnf.py

Line 65 in f30c16c

dict(type='Resize', scale=(2240, 2240), keep_ratio=True),