Improve documentation concerning the new config files #3225

orbiskcw · 2021-07-06T21:42:12Z

📚 Documentation Improvements

In short

Concerning: https://detectron2.readthedocs.io/en/latest/tutorials/configs.html
Problem: Documentation does not seem to have been updated to reflect the new config files (.py rather than .yaml)
Solution: Update the documentation

Problem description

FAIR recently published new Mask R-CNN baselines and this was my first introduction to the new config file that no longer relies on YAML files but on 'raw' .py files. I am trying to load the new baselines using the config files mentioned in the MODEL_ZOO (see this table). For example:

from detectron2 import model_zoo
model = model_zoo.get("new_baselines/mask_rcnn_regnetx_4gf_dds_FPN_400ep_LSJ.py", trained=True)

This gives

RuntimeError: new_baselines/mask_rcnn_regnetx_4gf_dds_FPN_400ep_LSJ not available in Model Zoo!

I have installed Detectron2 using the installation instructions. When looking up the documentation on configs, it seems that this has not been updated to reflect the new configs and still solely mentions YAML files.

Proposed solution

It could be that the CONFIG_PATH_TO_URL_SUFFIX dictionary in _ModelZooUrls class still has to be updated and that this is actually a bug (see here), but I find it hard to estimate wheter this is meant behavior (i.e. the new config file should be loaded differently) or a bug due to my limited understanding of the new config files. Either way, I therefore feel like the documentation on readthedocs should be updated to reflect the change from .yaml to .py.

The text was updated successfully, but these errors were encountered:

cwognum · 2021-07-06T22:12:53Z

I think this is actually a bug 🐛

@ppwwyyxx Could you confirm whether this is a bug or expected behavior? I noticed you authored the last commit in the model_zoo.py file and made the changes there for the new .py config files.

VishalBalaji321 · 2021-07-07T07:49:58Z

This is actually a bug. I had similar issues but when I added the required 'new-baseline' config manually into model_zoo.py file, this was solved.

cwognum · 2021-07-07T12:30:18Z

I created a pull request to add the mapping from config to model checkpoint for the new baselines. However, I do not think that solves this particular issue. Even though the transition from YAML to PY configs is easy, the documentation could (and should) still be updated to reflect these changes.

Summary: Related to #3225, but does **not** solve it. # Overview of changes FAIR recently published new [Mask R-CNN baselines](https://ai.facebook.com/blog/advancing-computer-vision-research-with-new-detectron2-mask-r-cnn-baselines/). The new config files have not yet been mapped to the corresponding model checkpoint files in the `model_zoo._ModelZooUrls` class. This was a relatively easy fix that simply follows the table given in the [MODEL_ZOO table](https://github.com/facebookresearch/detectron2/blob/master/MODEL_ZOO.md#new-baselines-using-large-scale-jitter-and-longer-training-schedule). If we run the following code snippet below: ```python from detectron2 import model_zoo model = model_zoo.get("new_baselines/mask_rcnn_regnetx_4gf_dds_FPN_400ep_LSJ.py", trained=True) ``` This used to give: ``` RuntimeError: new_baselines/mask_rcnn_regnetx_4gf_dds_FPN_400ep_LSJ not available in Model Zoo! ``` With these new changes, we get what we would expect and actually download and load the model: ``` model_final_f1362d.pkl: 173MB [01:13, 2.37MB/s] The checkpoint state_dict contains keys that are not used by the model: proposal_generator.anchor_generator.cell_anchors.{0, 1, 2, 3, 4} ``` I am not entirely sure whether the warning is expected or something that requires fixing. # Additional proposal I personally am not a big fan of the naming. "new_baslines" seems too general to me and deviates from the already established naming conventions. This makes it less intuitive and not very robust to future updates. I would suggest moving the configs to `configs/COCO-InstanceSegmentation`. Similarly, I would then also move the checkpoint files accordingly to `https://dl.fbaipublicfiles.com/detectron2/COCO-InstanceSegmentation/`. I did not implement these changes yet because I cannot change where the model files are stored and because I can imagine there to be an internal motivation not to change the naming anymore. Pull Request resolved: #3228 Reviewed By: vaibhava0 Differential Revision: D29590936 Pulled By: ppwwyyxx fbshipit-source-id: ffebeff845acfa2d9beb6d0e5472ff8670fbb457

orbiskcw · 2021-07-08T09:21:56Z

Thanks for the help so far!

I indeed do think the documentation should still be updated. I find it especially confusing that the two config types seem to use different keys. For example cfg.MODEL.WEIGHTS vs. cfg.train.init_checkpoint (see model_zoo.py). Could there perhaps be a method to convert between the two formats?

orbiskcw · 2021-07-09T09:56:11Z

@ppwwyyxx I have been trying to compose a minimal script to understand the usage of the new .py config files in the Detectron2 framework. I see the importance of instantiate(). I expected something like this to work:

cfg = get_config('new_baselines/mask_rcnn_regnety_4gf_dds_FPN_400ep_LSJ.py', True)

model = instantiate(cfg.model)
dataset = instantiate(cfg.dataloader.train)
optimizer = instantiate(cfg.optimizer)

trainer = SimpleTrainer(model, dataset, optimizer)
trainer.train(0, 700000)

I think this is the absolute bare minimum. It does not do any evaluation and does not even save the model. However, this already causes issues that make me suspect that the new config files are not fully supported yet. For example; instantiate(cfg.optimizer) gives TypeError: get_default_optimizer_params() missing 1 required positional argument: 'model'.

There also does not yet seems to be a BaseTrainer sub class that uses the new config format, which generally makes up a quite essential component to a Detectron2 training loop. Is it a correct assumption that the new config files are thus not fully supported yet?

cwognum · 2021-07-09T09:59:31Z

I share @orbiskcw his intuition. I would be willing to work on this and to create a pull request. Perhaps we can create a new LazyDefaultTrainer class? But I do think we need some pointers and guidance from FAIR on this as it involves some possibly influential changes. @ppwwyyxx Thoughts?

ppwwyyxx · 2021-07-09T10:24:46Z

https://github.com/facebookresearch/detectron2/blob/master/tools/lazyconfig_train_net.py is the training script to use these configs. We'll add documentation soon.

orbiskcw · 2021-07-09T10:35:39Z

Awesome! Thank you for the quick response. I completely overlooked that file. Sorry for the inconvenience.

mohsendaoud1 · 2021-07-09T14:16:21Z

@orbiskcw Can you share your training code please

orbiskcw · 2021-07-09T14:27:17Z

It's not worth sharing. The provided training script that was just mentioned contains all the information you need. I just updated some configurations so that we can train it on our own data. I added code for the custom dataset and used args.opts for setting some additional parameters. Most importantly:

Set a cfg.train.init_checkpoint (previously cfg.MODEL.WEIGHTS)
Changed SyncBN to just BN because we train with a single GPU
Set cfg.dataloader.train.dataset.names (previously cfg.DATASETS.TRAIN) to the name of our own dataset as registered in the DatasetCatalog. Similarly for cfg.dataloader.test.dataset.names (previously cfg.DATASETS.TEST).

sharing my script will not help you as I barely changed anything and what I did change is application specific

mohsendaoud1 · 2021-07-09T16:32:50Z

i am new with code thats why i need it

orbiskcw · 2021-07-14T13:18:52Z

@ppwwyyxx Saving and loading a config file that had a lambda does not seem to work yet. For example:

cfg = get_config('new_baselines/mask_rcnn_regnety_4gf_dds_FPN_400ep_LSJ.py')
LazyConfig.save(cfg, "tmp.yaml")
cfg = LazyConfig.load("tmp.yaml")
model = instantiate(cfg.model)
print(model)

Gives:

yaml.constructor.ConstructorError: while constructing a Python object
cannot find module 'detectron2._cfg_loader8ac1.mask_rcnn_R_50_FPN_100ep_LSJ.py' (No module named 'detectron2._cfg_loader8ac1')
  in "tmp.yaml", line 100, column 18

Manually removing the lambdas from the generated .yaml file does allow it to be loaded and prints the model architecture as expected. However, the model now has no batch normalization. In this case it has to do with this line of code. Although I am not a 100% sure what the best way is to do this, I believe this particular case could be fixed by some changes in get_norm() or FastRCNNConvFCHead.

Generally, however, I am not sure whether and how lambdas could (and should) be supported with the lazy config format. What are your thoughts on this?

ppwwyyxx · 2021-07-14T22:24:27Z

Lambda should not be saved in a yaml - I don't think there is a reasonable way to do that.

Summary: fix #3225 Pull Request resolved: fairinternal/detectron2#553 Reviewed By: wat3rBro Differential Revision: D29731838 Pulled By: ppwwyyxx fbshipit-source-id: 69ed27681747cf00a2fa682c245369c3d1ee8945

ppwwyyxx · 2021-07-21T01:31:03Z

We added documentation in https://detectron2.readthedocs.io/en/latest/tutorials/lazyconfigs.html. Feel free to suggest improvements or missing information.

orbiskcw · 2021-07-21T08:18:07Z

Thank you for this! It reads quite nicely. There is two points I would like to add to the documentation:

What are the best practices for saving and loading config files locally? Take for example the use case in which I want to continue training a provided baseline but want to experiment with some of the configurations and save that updated config. I have seen a total of three options and was wondering how they compare to one another:
- First of all, we can use LazyConfig.load() and LazyConfig.save() to save the 'code' config to a .yaml file.
- Secondly, we could reproduce (part of) the config structure in Python and have our configs stored as code
- Finally, we could use the provided methods in the OmegaConf library (see here)
A little section on how the new config compares to the old config would also be nice. I understand that this is more challenging than I make it sound because there is no 1-to-1 mapping providing an overview like the one used for the yacs config is impossible with this format. However, perhaps some common use cases could still be illustrated to make the transition more smooth. Some examples I had trouble figuring out when I started using the new configs:
- Where to update the batch size?
- Where to update the custom dataset?
- How to change the augmentations?

cwognum · 2021-07-21T08:21:18Z

I agree with the comments above. An additional question would be whether there are any concrete plans to further expand the support for these new config files throughout the repository? For example, by adding a new DefaultTrainer-like class for lazy configs. If there were to be some sort of overview or roadmap of changes planned / needed, I would happily pick up some of those tasks!

ppwwyyxx · 2021-07-23T10:00:01Z

What are the best practices for saving and loading config files locally

I agree we should explain more on this topic. Among your solutions: (1) and (3) are actually almost the same. They share the same limitation that the config object can be made unserializable if users choose to use complex objects. I'm not so sure on how (2) can be done.

A little section on how the new config compares to

As you see there isn't a simple 1-1 mapping. Because config follows the code directly, the new config is most suitable for users with enough expertise so that they can directly work with arguments in the code. But still, I think we can list a few examples on the commonly used options.

concrete plans to further expand the support

New features will likely not have support for old config
We'd like to get help on translating more existing yaml's to python, especially those that are sufficiently different from what we have already translated. Translation might uncover new issues.
Some code in lazyconfig_train_net.py can very well be put into some "trainer". So far there is no plan. It is more of a preference but now we feel it's simple enough so that working on this script directly is easier, especially given that the new config intended more for power users.

VishalBalaji321 · 2021-07-29T17:42:16Z

Hey guys, I am training a simple mask R CNN model for instance segmentation. I found that the new-baselines pre-trained weights offer significantly better accuracy but I am not able to use it as such. I went through the documentation of @ppwwyyxx and lazyconfig_train_net.py but as a beginner, I couldn't understand much. It would be really helpful if you guys could provide me with a simple script how to initiate the entire config file including parameters max_iters, steps and gamma from scratch and how to start training the model.

Cheers,
Vishal Balaji

orbiskcw · 2021-07-29T21:00:33Z

Hey @VishalBalaji321, it seems like the lazyconfig_train_net.py CLI can do exactly what you are looking for. I am not sure what information you are missing. Please note that using the default argument parser used in the lazyconfig_train_net.py script gives you the opts parameter which is then used to specify the overrides. Meaning, that you can simply override the config using the path.key=value syntax. For example:

# Overrides the maximum number of iterations by 100 and the learning rate by 0.1
python lazyconfig_train_net.py --config-file ... train.max_iter=100 optimizer.lr=0.1

Of course, you coud also do this in code. Similarly to what is done here.

# Overrides the maximum number of iterations by 100 and the learning rate by 0.1
cfg.train.max_iter = 100
cfg.optimizer.lr = 0.1

Without any concrete questions, I can't really help you more than that. I'm afraid that, like @ppwwyyxx said as well, this new config requires you to understand the code a bit better than before. But I have been working with it for a little while now and I actually have been enjoying it more and more. It's quite powerful!

VishalBalaji321 · 2021-07-30T09:18:54Z

It's not worth sharing. The provided training script that was just mentioned contains all the information you need. I just updated some configurations so that we can train it on our own data. I added code for the custom dataset and used args.opts for setting some additional parameters. Most importantly:

Set a cfg.train.init_checkpoint (previously cfg.MODEL.WEIGHTS)

Changed SyncBN to just BN because we train with a single GPU

Set cfg.dataloader.train.dataset.names (previously cfg.DATASETS.TRAIN) to the name of our own dataset as registered in the DatasetCatalog. Similarly for cfg.dataloader.test.dataset.names (previously cfg.DATASETS.TEST).

sharing my script will not help you as I barely changed anything and what I did change is application specific

Thanks a lot for the previous reply @orbiskcw , it did help me a lot. Now I am facing issues with 'SyncBN' normalization. Could you pls explain how to change it to just 'BN' for single GPU? (For Info: I am planning on using newbaselines/R101_FPN_400ep model

orbiskcw · 2021-07-30T09:44:23Z

Happy to have helped. I am pretty sure that the change depends on the model you are using. For me, it was

# Using the python syntax here, but as I have shown earlier 
# it looks similar if you provide these overrides using the CLI
cfg.model.backbone.bottom_up.norm = "BN" 
cfg.model.backbone.norm = "BN"

To help you find the right keys, I would recommend doing one of two things:

Either clone the Detectron2 repository and using your favorite IDE go through the hierarchical definition of the .py configs manually to search for the configuration you are interested in.
Or load the config using LazyConfig.load(args.config_file) and immediately save it to a .yaml file with LazyConfig.save('/local/path/to/config.yaml). This makes it a bit easier because everything is one place.

Note that because the lazy configs closely follow the code, not all configurations you are looking for will be present in the configs or easy to find with either of these two options. I'm afraid there is no other solution than to just bite the bullet and dig in the Detectron2 code! I think you will find it is nicely structured and quite easy to comprehend once you get going! 😉

orbiskcw · 2021-07-30T19:25:39Z

@VishalBalaji321 In case you have not seen this yet, the documentation was updated yesterday: https://detectron2.readthedocs.io/en/latest/tutorials/lazyconfigs.html It explains a lot of the things I tried to explain to you as well, but better! 😉

@ppwwyyxx Great work by you and your colleagues. It feels a lot more complete now. Nothing comes to mind anymore that could be added.

orbiskcw · 2021-08-12T11:44:23Z

@ppwwyyxx One more question that recently came up while working with the detectron2 repo: What is the best way to extend a config from the model zoo? I expected something like this to work:

# In /path/to/custom_config.py

from detectron2.model_zoo.configs.new_baselines.mask_rcnn_regnetx_4gf_dds_FPN_400ep_LSJ import  (
    dataloader,
    lr_multiplier,
    model,
    optimizer,
    train,
)

train.max_iter = 10

But if I then try to load the new config with:

LazyConfig.load('/path/to/custom_config.py')

I get:

  File "/home/cas/Documents/repositories/detectron2-model-training/detectron2_model_training/config.py", line 98, in load
    return LazyConfig.load(config_path)
  File "/home/cas/Documents/repositories/detectron2-model-training/.venv/lib/python3.8/site-packages/detectron2/config/lazy.py", line 202, in load
    exec(compile(content, filename, "exec"), module_namespace)
  File "configs/default.py", line 1, in <module>
    from detectron2.model_zoo.configs.new_baselines.mask_rcnn_regnetx_4gf_dds_FPN_100ep_LSJ import (  # noqa
  File "/home/cas/Documents/repositories/detectron2-model-training/.venv/lib/python3.8/site-packages/detectron2/config/lazy.py", line 145, in new_import
    return old_import(name, globals, locals, fromlist=fromlist, level=level)
  File "/home/cas/Documents/repositories/detectron2-model-training/.venv/lib/python3.8/site-packages/detectron2/model_zoo/configs/new_baselines/mask_rcnn_regnetx_4gf_dds_FPN_100ep_LSJ.py", line 1, in <module>
    from .mask_rcnn_R_50_FPN_100ep_LSJ import (
  File "/home/cas/Documents/repositories/detectron2-model-training/.venv/lib/python3.8/site-packages/detectron2/config/lazy.py", line 145, in new_import
    return old_import(name, globals, locals, fromlist=fromlist, level=level)
  File "/home/cas/Documents/repositories/detectron2-model-training/.venv/lib/python3.8/site-packages/detectron2/model_zoo/configs/new_baselines/mask_rcnn_R_50_FPN_100ep_LSJ.py", line 13, in <module>
    train.init_checkpoint = ""
AttributeError: 'dict' object has no attribute 'init_checkpoint'

Could you confirm whether this is expected behavior or a bug? Could you clarify if it is possible to do this and if so, what the conditions / requirements are?

orbiskcw · 2021-08-12T12:01:44Z

Something like this does work, but feels very cumbersome:

config = get_config('new_baselines/mask_rcnn_regnetx_4gf_dds_FPN_400ep_LSJ.py')

train = config.train
optimizer = config.optimizer
model = config.model
dataloader = config.dataloader
lr_multiplier = config.lr_multiplier
del config

train.max_iter = 10

orbiskcw · 2021-08-24T13:25:01Z

@ppwwyyxx Do you see the chance to still respond to the above question?

ppwwyyxx · 2021-08-24T18:06:16Z

The failure is currently expected because when doing LazyConfig.load we only turn relative imports into config loading.

However it would be nice to enable from detectron2.model_zoo.configs.xxx import .... This should be possible by modifying our custom import function at

detectron2/detectron2/config/lazy.py

Lines 124 to 130 in fe4d35b

    
           def new_import(name, globals=None, locals=None, fromlist=(), level=0): 
        
               if ( 
        
                   # Only deal with relative imports inside config files 
        
                   level != 0 
        
                   and globals is not None 
        
                   and (globals.get("__package__", "") or "").startswith(_CFG_PACKAGE_NAME) 
        
               ):

which currently only handles relative import.

orbiskcw · 2021-08-25T17:31:10Z

Clear! I have taken a look at the method you referenced, but this is some Python wizardry I am not really familiar with and I wouldn't know where to start to support absolute imports. I might take a closer look at it once I have a bit more time, but for now I will stick to the workaround. Thank you for your reply! 👍

FeriBolour · 2022-03-31T22:27:53Z

Hello Everyone,

Thank you for all the info mentioned in this issue. It really helped me in understanding how to use LazyConfig to use the architectures in the new_baselines.

However I am still having an issue which I don't know if it is due to my lack of understanding of the new configs.

I am using the lazyconfig_train_net.py for my own custom coco format dataset. I've already used this dataset with the DefaultTrainer class and it works perfectly. However I am having an issue with this dataset when it comes to this script.

For now, I haven't changed anything in lazyconfig_train_net.py except the main function. This is how the main function looks like now:

def main(args):
    register_coco_instances("CottonImaging_train", {}, '/home/avl/Projects/Cotton Imaging Project/Data/Datasets02272022/7030_images/TrainingSet2_imbalanced_70.json', "/home/avl/Projects/Cotton Imaging Project/Data/Datasets02272022/7030_images/training_images")
    
    cfg = get_config("new_baselines/mask_rcnn_R_101_FPN_400ep_LSJ.py")
    cfg.model.backbone.bottom_up.stages = detectron2.modeling.ResNet.make_default_stages(depth=101, norm='BN', stride_in_1x1=True)
    cfg.model.backbone.norm = "BN"
    cfg.dataloader.train.dataset= detectron2.data.get_detection_dataset_dicts(names='CottonImaging_train')
    cfg.train.max_iter=10000
    cfg.train.output_dir='./MaskRCNN ResNet101 400ep LSJ'
    cfg.model.roi_heads.num_classes = 4
    cfg.optimizer.lr=0.001
    cfg.dataloader.train.total_batch_size = 12
    default_setup(args, cfg)    
    do_train(cfg)

When I run this script, this is the error I get:

[03/31 17:04:17 d2.data.dataset_mapper]: [DatasetMapper] Augmentations used in training: [ResizeScale(min_scale=0.1, max_scale=2.0, target_height=1024, target_width=1024), FixedSizeCrop(crop_size=[1024, 1024]), RandomFlip()]
[03/31 17:04:18 fvcore.common.checkpoint]: No checkpoint found. Initializing model from scratch
[03/31 17:04:18 d2.engine.train_loop]: Starting training from iteration 0
ERROR [03/31 17:04:19 d2.engine.train_loop]: Exception during training:
Traceback (most recent call last):
  File "/home/avl/.local/lib/python3.8/site-packages/detectron2/engine/train_loop.py", line 149, in train
    self.run_step()
  File "/home/avl/.local/lib/python3.8/site-packages/detectron2/engine/train_loop.py", line 391, in run_step
    data = next(self._data_loader_iter)
  File "/home/avl/.local/lib/python3.8/site-packages/detectron2/data/common.py", line 179, in __iter__
    for d in self.dataset:
  File "/home/avl/.local/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 521, in __next__
    data = self._next_data()
  File "/home/avl/.local/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 1203, in _next_data
    return self._process_data(data)
  File "/home/avl/.local/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 1229, in _process_data
    data.reraise()
  File "/home/avl/.local/lib/python3.8/site-packages/torch/_utils.py", line 425, in reraise
    raise self.exc_type(msg)
omegaconf.errors.ConfigAttributeError: Caught ConfigAttributeError in DataLoader worker process 0.
Original Traceback (most recent call last):
  File "/home/avl/.local/lib/python3.8/site-packages/torch/utils/data/_utils/worker.py", line 287, in _worker_loop
    data = fetcher.fetch(index)
  File "/home/avl/.local/lib/python3.8/site-packages/torch/utils/data/_utils/fetch.py", line 44, in fetch
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/home/avl/.local/lib/python3.8/site-packages/torch/utils/data/_utils/fetch.py", line 44, in <listcomp>
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/home/avl/.local/lib/python3.8/site-packages/detectron2/data/common.py", line 43, in __getitem__
    data = self._map_func(self._dataset[cur_idx])
  File "/home/avl/.local/lib/python3.8/site-packages/detectron2/utils/serialize.py", line 23, in __call__
    return self._obj(*args, **kwargs)
  File "/home/avl/.local/lib/python3.8/site-packages/detectron2/data/dataset_mapper.py", line 168, in __call__
    annos = [
  File "/home/avl/.local/lib/python3.8/site-packages/detectron2/data/dataset_mapper.py", line 169, in <listcomp>
    utils.transform_instance_annotations(
  File "/home/avl/.local/lib/python3.8/site-packages/detectron2/data/detection_utils.py", line 284, in transform_instance_annotations
    bbox = BoxMode.convert(annotation["bbox"], annotation["bbox_mode"], BoxMode.XYXY_ABS)
  File "/home/avl/.local/lib/python3.8/site-packages/detectron2/structures/boxes.py", line 80, in convert
    arr = box.clone()
  File "/home/avl/.local/lib/python3.8/site-packages/omegaconf/listconfig.py", line 176, in __getattr__
    self._format_and_raise(
  File "/home/avl/.local/lib/python3.8/site-packages/omegaconf/base.py", line 190, in _format_and_raise
    format_and_raise(
  File "/home/avl/.local/lib/python3.8/site-packages/omegaconf/_utils.py", line 821, in format_and_raise
    _raise(ex, cause)
  File "/home/avl/.local/lib/python3.8/site-packages/omegaconf/_utils.py", line 719, in _raise
    raise ex.with_traceback(sys.exc_info()[2])  # set end OC_CAUSE=1 for full backtrace
omegaconf.errors.ConfigAttributeError: ListConfig does not support attribute access
    full_key: [130].annotations[0].bbox[clone]
    object_type=list

I register my dataset the same way and use it for DeafultTrainer and I don't get any errors.

Here's how my annotations look like for reference if needed:

I'd appreciate any kind of help with this.
Please let me know if I need to provide more info.

ppwwyyxx · 2022-03-31T22:34:54Z

cfg.dataloader.train.dataset= detectron2.data.get_detection_dataset_dicts(names='CottonImaging_train')

The value of this config should be a config object, but not the actual data itself. So you'd want

    from detectron2.config import LazyConfig as L
    cfg.dataloader.train.dataset = L(detectron2.data.get_detection_dataset_dicts)(names='CottonImaging_train')

See examples in

detectron2/configs/common/data/coco.py

Lines 15 to 16 in 6886f85

    
           dataloader.train = L(build_detection_train_loader)( 
        
               dataset=L(get_detection_dataset_dicts)(names="coco_2017_train"),

FeriBolour · 2022-04-03T20:43:19Z

cfg.dataloader.train.dataset= detectron2.data.get_detection_dataset_dicts(names='CottonImaging_train')
The value of this config should be a config object, but not the actual data itself. So you'd want
    from detectron2.config import LazyConfig as L
    cfg.dataloader.train.dataset = L(detectron2.data.get_detection_dataset_dicts)(names='CottonImaging_train')
See examples in

detectron2/configs/common/data/coco.py

Lines 15 to 16 in 6886f85

dataloader.train = L(build_detection_train_loader)(

dataset=L(get_detection_dataset_dicts)(names="coco_2017_train"),

Thank You for the very quick reply.
It solved my problem and I was able to successfully train my model.

Now a quick question that might be easy. I just want to know if there are any specific functions that I can use.

How can I predict on single images with my saved model?
Almost anywhere I look, they use the DefaultPredictor class which I don't think we can use with these config files. (If we can, I'd appreciate if you tell me how).
Are there any other classes like the DefaultPredictor that I could use for this purpose. If not, how would you suggest that I do the predictions.

Thank You

EDIT:
I tried to solve this issue by using the following script:

cfg = get_config("new_baselines/mask_rcnn_R_50_FPN_400ep_LSJ.py")
cfg.model.backbone.bottom_up.stages = detectron2.modeling.ResNet.make_default_stages(depth=50, norm='BN', stride_in_1x1=True)
cfg.model.backbone.norm = "BN"
cfg.model.backbone.bottom_up.stem = detectron2.modeling.backbone.BasicStem(in_channels=3, norm='BN', out_channels=64)
cfg.model.roi_heads.num_classes = 4
cfg.train.init_checkpoint = './model_final.pth'
model = instantiate(cfg.model)
model.to(cfg.train.device)
model = create_ddp_model(model)
DetectionCheckpointer(model).load(cfg.train.init_checkpoint)

model.eval()

def mapper(dataset_dict):
    dataset_dict = copy.deepcopy(dataset_dict)
    dicts = []
    for i in range(len(dataset_dict)):
        image = utils.read_image(dataset_dict[i]["file_name"], format="BGR")
        temp = image.copy()
        utils.check_image_size(dataset_dict[i], temp)
        image = torch.from_numpy(temp)
        dicts.append({
           "image": image,
           'height': 720,
           'width': 1280,
        })
    return dicts

dataset_dict = detectron2.data.get_detection_dataset_dicts('CottonImaging_test')
data= mapper(dataset_dict)

with torch.no_grad():
  outputs = model(data)

But I get the following error:

---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
/tmp/ipykernel_3575/3931202212.py in <module>
      3 
      4 with torch.no_grad():
----> 5   outputs = model(data)

~/.local/lib/python3.8/site-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
   1049         if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks
   1050                 or _global_forward_hooks or _global_forward_pre_hooks):
-> 1051             return forward_call(*input, **kwargs)
   1052         # Do not call functions when jit is used
   1053         full_backward_hooks, non_full_backward_hooks = [], []

~/.local/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py in forward(self, batched_inputs)
    144         """
    145         if not self.training:
--> 146             return self.inference(batched_inputs)
    147 
    148         images = self.preprocess_image(batched_inputs)

~/.local/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py in inference(self, batched_inputs, detected_instances, do_postprocess)
    197         assert not self.training
    198 
--> 199         images = self.preprocess_image(batched_inputs)
    200         features = self.backbone(images.tensor)
    201 

~/.local/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py in preprocess_image(self, batched_inputs)
    223         """
    224         images = [x["image"].to(self.device) for x in batched_inputs]
--> 225         images = [(x - self.pixel_mean) / self.pixel_std for x in images]
    226         images = ImageList.from_tensors(images, self.backbone.size_divisibility)
    227         return images

~/.local/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py in <listcomp>(.0)
    223         """
    224         images = [x["image"].to(self.device) for x in batched_inputs]
--> 225         images = [(x - self.pixel_mean) / self.pixel_std for x in images]
    226         images = ImageList.from_tensors(images, self.backbone.size_divisibility)
    227         return images

RuntimeError: The size of tensor a (720) must match the size of tensor b (3) at non-singleton dimension 0

I don't know if this is the right way to do the predictions. But if it is, I don't exactly understand how to resolve this error.

ppwwyyxx · 2022-04-05T08:40:37Z

As documented in https://detectron2.readthedocs.io/en/latest/tutorials/models.html#model-input-format the image of model's input has to be in (C, H, W) format while in your code it's (H, W, C).

DefaultPredictor would need a patch like #3755 to work with new configs.

FrancescoFarinola · 2022-04-13T10:17:45Z

If anyone is looking for a solution for doing inference on single images remember to:

Convert the image to your model.input_format (usually BGR since openCV converts images as default in BGR)
im = cv2.imread("path/to/image.jpg")
Transform the BGR image into a tensor with image = torch.from_numpy(im)
Permute the tensor as the model input format is (C, H, W) so: image_tensor = image.permute(2,0,1)
Do inference - remember that when doing inference on custom models you need to pass a list of dicts, so if you are doing inference on a single image:

model.eval()
with torch.no_grad():
      outputs = model(d)

where d is d = [{'image' : image_tensor}]

Bill4869 · 2022-06-16T03:42:33Z

Hello.
Sorry for bothering. I have looked at the documentation but I still could not test pretrained new baselines model on one image.
I was wondering if anyone could tell me how to do it?

Thank you in advances.

bouachalazhar · 2023-05-09T09:21:13Z

Hello. Sorry for bothering. I have looked at the documentation but I still could not test pretrained new baselines model on one image. I was wondering if anyone could tell me how to do it?

Thank you in advances.

Same problem

rezat96 · 2023-07-22T21:52:20Z

Hi, I have carefully read the docs and the tutorial, but is there an equivalent of cfg.INPUT.MASK_FORMAT = 'bitmask' for the new baselines? I could use it with previous pipelines but with the new one I cant train on my custom dataset @ppwwyyxx

yanghoonkim · 2023-10-04T11:05:20Z

@rezat96 cfg.dataloader.train.mapper['instance_mask_format'] = 'bitmask'

CA4GitHub · 2023-10-23T17:36:42Z

As documented in https://detectron2.readthedocs.io/en/latest/tutorials/models.html#model-input-format the image of model's input has to be in (C, H, W) format while in your code it's (H, W, C).

DefaultPredictor would need a patch like #3755 to work with new configs.

Is there a way to convert new configs to old configs that will work with DefaultPredictor?

orbiskcw added the documentation Problems about existing documentation or comments label Jul 6, 2021

cwognum mentioned this issue Jul 7, 2021

Added the mapping from config to model checkpoint for the new baselines #3228

Closed

ppwwyyxx mentioned this issue Jul 16, 2021

Loading configurations for new baselines fails #3254

Closed

facebook-github-bot closed this as completed Jul 21, 2021

AKMourato mentioned this issue Sep 1, 2021

Compatibility of demo code with the new baselines for Instance Segmentation with Mask R-CNN #3416

Open

This was referenced Oct 24, 2022

Custom data set training using pre-trained weights IDEA-Research/detrex#78

Closed

Inference doc for new baselines #4597

Open

Testing new baselines model #4337

Closed

buckeye17 mentioned this issue Jan 4, 2023

Loading pre-trained model configuration from Python file #4737

Closed

Improve documentation concerning the new config files #3225

Improve documentation concerning the new config files #3225

Comments

orbiskcw commented Jul 6, 2021 • edited Loading

📚 Documentation Improvements

In short

Problem description

Proposed solution

cwognum commented Jul 6, 2021

VishalBalaji321 commented Jul 7, 2021

cwognum commented Jul 7, 2021

orbiskcw commented Jul 8, 2021 • edited Loading

orbiskcw commented Jul 9, 2021

cwognum commented Jul 9, 2021

ppwwyyxx commented Jul 9, 2021

orbiskcw commented Jul 9, 2021

mohsendaoud1 commented Jul 9, 2021

orbiskcw commented Jul 9, 2021 • edited Loading

mohsendaoud1 commented Jul 9, 2021

orbiskcw commented Jul 14, 2021 • edited Loading

ppwwyyxx commented Jul 14, 2021

ppwwyyxx commented Jul 21, 2021

orbiskcw commented Jul 21, 2021

cwognum commented Jul 21, 2021

ppwwyyxx commented Jul 23, 2021 • edited Loading

VishalBalaji321 commented Jul 29, 2021

orbiskcw commented Jul 29, 2021 • edited Loading

VishalBalaji321 commented Jul 30, 2021

orbiskcw commented Jul 30, 2021 • edited Loading

orbiskcw commented Jul 30, 2021

orbiskcw commented Aug 12, 2021 • edited Loading

orbiskcw commented Aug 12, 2021

orbiskcw commented Aug 24, 2021

ppwwyyxx commented Aug 24, 2021

orbiskcw commented Aug 25, 2021

FeriBolour commented Mar 31, 2022

ppwwyyxx commented Mar 31, 2022

FeriBolour commented Apr 3, 2022 • edited Loading

ppwwyyxx commented Apr 5, 2022

FrancescoFarinola commented Apr 13, 2022

Bill4869 commented Jun 16, 2022

bouachalazhar commented May 9, 2023

rezat96 commented Jul 22, 2023

yanghoonkim commented Oct 4, 2023

CA4GitHub commented Oct 23, 2023

orbiskcw commented Jul 6, 2021 •

edited

Loading

orbiskcw commented Jul 8, 2021 •

edited

Loading

orbiskcw commented Jul 9, 2021 •

edited

Loading

orbiskcw commented Jul 14, 2021 •

edited

Loading

ppwwyyxx commented Jul 23, 2021 •

edited

Loading

orbiskcw commented Jul 29, 2021 •

edited

Loading

orbiskcw commented Jul 30, 2021 •

edited

Loading

orbiskcw commented Aug 12, 2021 •

edited

Loading

FeriBolour commented Apr 3, 2022 •

edited

Loading