📝 A List of Welcoming Features

[kennymckormick]

Call for Contributors: https://openmmlab.medium.com/be-an-openmmlab-contributor-c9087428062a

Here we maintain a list of welcoming features proposed by the community. Our developer may consider integrating them into MMAction2. Besides, we highly encourage users to implement them and raise a PR.

1. A Custom Training Pipeline for Spatio-Temporal Action Detection. How can I train a spatio-temporal action detection network only for action recognition? #1443 , how to make my custom ava dataset  #1318 , about AVA #1292 , Spatio-temporal region proposals #604
2. Support AVA-Kinetics. How to train or Fine-tuning a specific class on the AVA-Kinetics #853
3. Support HACS. support hacs dataset? #1331
4. Support Charades. (The feature is partially implemented in [Feature] Support Charades Dataset #788 , thanks @congee524 . Anyone who would like to work on this feature can refer to the PR [Feature] Support Charades Dataset #788. )
5. Lack of SOTA Action Detection model. Lack of SOTA temporal action detection models #1124
6. TensorRT Support. export #1424
7. Support SGN. Implementation of Semantics Guided Network(SGN) #1463 (It is easy to implement, I guess it may take at most 1 or 2 days)

[kennymckormick]

@gengenkai , you can check if there are some features you are interested in.

[hukkai]

I did run action detection models on AVA-Kinetics. The changes on codes are not very complicated, but the data processing is rather painful. Many videos in the Kinetics 700 dataset are no longer available. Many videos can be corrupt during download, i.e., shorter than 10 seconds. This is not a serious problem for classification tasks but for detection, the missing part can have annotations...

The CVDF repo (https://github.com/cvdfoundation/kinetics-dataset) provides Kinetics dataset (400, 600, 700-2020) but not Kinetics700. Also there are many videos in the repo which is corrupted.

If there could be a good solution to prepare the dataset, that is super helpful and I can submit a PR for AVA-kinetics training.

[kennymckormick]

Hi, @hukkai , if you are eager to implement this feature in MMAction2, you can provide me a list of missing videos to be. I will see if these videos exist in my database and provide them to you (after ECCV ddl).

[hukkai]

Hi @kennymckormick I can implement the AVA-Kinetics dataset and training. Training on AVA-Kinetics indeed helps the performance on AVA a lot.

For the AVA-Kinetics, I plan to use YouTube-dl to download from YouTube. Some videos are missing and I can provide the list. Then how do you want to combine these missing videos with these which are still on YouTube? Will mmaction2 provide a link to download them?

[kennymckormick]

Hi @kennymckormick I can implement the AVA-Kinetics dataset and training. Training on AVA-Kinetics indeed helps the performance on AVA a lot.

For the AVA-Kinetics, I plan to use YouTube-dl to download from YouTube. Some videos are missing and I can provide the list. Then how do you want to combine these missing videos with these which are still on YouTube? Will mmaction2 provide a link to download them?

I guess we can not directly share the download link, host a dataset of dozens of GB is expensive. A practical approach would be: users who require the missing data send me the missing list, and I will share the missing videos to them if I have.

[irvingzhang0512]

@hukkai You can take a look at kinetics-dataset, maybe CVDF has already provide a link to download ava-kinetics videos and annotations

[hukkai]

@irvingzhang0512 My first attempt is the CVDF repo. One issue is that they have K400, K600 and K700-2020 but not K700. Some videos are different for K700-2020 and K700 which is required by AVA-Kinetics. This is not a big issue. The problem is that many videos in that repo are corrupted. I also put an issue (cvdfoundation/kinetics-dataset#4 (comment)) but it seems that they only handle the val/test set.

[kennymckormick]

Hi @kennymckormick I can implement the AVA-Kinetics dataset and training. Training on AVA-Kinetics indeed helps the performance on AVA a lot.

For the AVA-Kinetics, I plan to use YouTube-dl to download from YouTube. Some videos are missing and I can provide the list. Then how do you want to combine these missing videos with these which are still on YouTube? Will mmaction2 provide a link to download them?

Do you still need any help in finding missing videos for implementing AVA-Kinetics in MMAction2 ?

[hukkai]

Hi @kennymckormick I can implement the AVA-Kinetics dataset and training. Training on AVA-Kinetics indeed helps the performance on AVA a lot.
For the AVA-Kinetics, I plan to use YouTube-dl to download from YouTube. Some videos are missing and I can provide the list. Then how do you want to combine these missing videos with these which are still on YouTube? Will mmaction2 provide a link to download them?

Do you still need any help in finding missing videos for implementing AVA-Kinetics in MMAction2 ?

No, I will implement in a way that remove missing or incomplete videos from the training list. I will submit a pull request recently.

[rlleshi]

I had written a multimodal_demo script some time ago that late fuses the scores of rgb-, skeleton- & audio-based models.
Does it make sense to make a PR out of it?

[borijang]

Is development of SOTA temporal action detection models (such as ASM-Loc) still an active task?

[coldmanck]

Hi @hukkai @kennymckormick , is there any update on supporting AVA-Kinetics? Thanks!