Skip to content

Commit

Permalink
add UniBEV to benchmark
Browse files Browse the repository at this point in the history
  • Loading branch information
@TillBeemelmanns
TillBeemelmanns committed Oct 12, 2024
1 parent b1719e8 commit 98904d8
Show file tree
Hide file tree
Showing 17 changed files with 720 additions and 217 deletions.
3 changes: 2 additions & 1 deletion .gitignore
View file
Original file line number Diff line number Diff line change
Expand Up @@ -2,4 +2,5 @@ __pycache__/
npy/
nuscenes/
multicorrupt/
tmp
tmp
.venv/
28 changes: 17 additions & 11 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -126,26 +126,32 @@
- [01.02.2024] **v0.0.1** Initial Release with **10 corruption types** and **5 evaluated models**

## Benchmark Results
### Resistance Ability (RA) computed with NDS metric
### 📊 Relative Resistance Ability (RRA) computed with NDS metric and baseline BEVfusion
| Model | Clean | Beams Red. | Brightness | Darkness | Fog | Missing Cam. | Motion Blur | Points Red. | Snow | Spatial Mis. | Temporal Mis. | mRRA |
|:----------------------------------------------------------------|:----------|:-------------|:-------------|:-----------|:----------|:---------------|:--------------|:--------------|:----------|:---------------|:----------------|:----------|
| ⭐ [CMT](https://github.com/junjie18/CMT) | 0.729 | **18.642** | -1.138 | -0.096 | **9.398** | **2.041** | -0.841 | **8.213** | **9.887** | **17.053** | **8.448** | **7.161** |
| [Sparsefusion](https://github.com/yichen928/SparseFusion) | 0.732 | 4.264 | **3.179** | **1.821** | 4.429 | 0.297 | 0.280 | 3.242 | 1.887 | 3.699 | 7.228 | 3.033 |
| [IS-Fusion](https://github.com/yinjunbo/IS-Fusion) | **0.737** | 3.684 | 2.291 | 1.267 | 3.890 | 0.920 | **3.994** | 1.691 | -2.351 | 4.513 | 7.177 | 2.708 |
| ⚓︎ [BEVfusion](https://github.com/mit-han-lab/bevfusion) | 0.714 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 |
| [TransFusion](https://github.com/XuyangBai/TransFusion) | 0.708 | -7.210 | 1.799 | 1.146 | -0.552 | 0.340 | -5.412 | -3.296 | -4.220 | -3.626 | 3.850 | -1.718 |
| [UniBEV_avg](https://github.com/tudelft-iv/UniBEV) | 0.684 | 7.617 | -3.758 | -4.595 | -1.228 | -5.170 | -11.777 | -0.144 | -6.909 | 2.812 | 1.617 | -2.154 |
| [UniBEV_cat](https://github.com/tudelft-iv/UniBEV) | 0.678 | 6.534 | -4.303 | -5.279 | -0.199 | -5.438 | -12.505 | -0.979 | -6.596 | 1.436 | 0.799 | -2.653 |
| [UniBEV_cnw](https://github.com/tudelft-iv/UniBEV) | 0.685 | 5.030 | -3.729 | -4.383 | -1.104 | -5.749 | -13.119 | -0.428 | -8.055 | 1.582 | 1.025 | -2.893 |
| [DeepInteraction](https://github.com/fudan-zvg/DeepInteraction) | 0.691 | -6.361 | -3.150 | -7.215 | -25.037 | -16.386 | -7.077 | -2.188 | -5.149 | 0.212 | 0.145 | -7.221 |

### 📊 Resistance Ability (RA) computed with NDS metric
| Model | Clean | Beams Red. | Brightness | Darkness | Fog | Missing Cam. | Motion Blur | Points Red. | Snow | Spatial Mis. | Temporal Mis. | mRA |
|:----------------------------------------------------------------|:----------|:-------------|:-------------|:-----------|:----------|:---------------|:--------------|:--------------|:----------|:---------------|:----------------|:----------|
| ⭐ [CMT](https://github.com/junjie18/CMT) | 0.729 | **0.786** | 0.937 | 0.948 | **0.806** | 0.974 | 0.841 | **0.925** | **0.833** | **0.809** | **0.788** | **0.865** |
| [UniBEV_cat](https://github.com/tudelft-iv/UniBEV) | 0.678 | 0.759 | 0.975 | 0.967 | 0.791 | 0.970 | 0.798 | 0.910 | 0.761 | 0.753 | 0.788 | 0.847 |
| [UniBEV_avg](https://github.com/tudelft-iv/UniBEV) | 0.684 | 0.760 | 0.972 | 0.965 | 0.776 | 0.964 | 0.797 | 0.909 | 0.752 | 0.757 | 0.787 | 0.844 |
| [UniBEV_cnw](https://github.com/tudelft-iv/UniBEV) | 0.685 | 0.740 | 0.971 | 0.966 | 0.775 | 0.957 | 0.784 | 0.905 | 0.742 | 0.747 | 0.781 | 0.837 |
| [Sparsefusion](https://github.com/yichen928/SparseFusion) | 0.732 | 0.689 | 0.975 | 0.963 | 0.767 | 0.954 | 0.848 | 0.879 | 0.770 | 0.714 | 0.777 | 0.834 |
| [BEVfusion](https://github.com/mit-han-lab/bevfusion) | 0.714 | 0.676 | 0.967 | 0.969 | 0.752 | 0.974 | 0.866 | 0.872 | 0.774 | 0.705 | 0.742 | 0.830 |
| [IS-Fusion](https://github.com/yinjunbo/IS-Fusion) | **0.737** | 0.680 | 0.960 | 0.952 | 0.758 | 0.953 | **0.873** | 0.860 | 0.733 | 0.715 | 0.771 | 0.826 |
| [TransFusion](https://github.com/XuyangBai/TransFusion) | 0.708 | 0.633 | **0.993** | **0.988** | 0.754 | **0.985** | 0.826 | 0.851 | 0.748 | 0.685 | 0.777 | 0.824 |
| [DeepInteraction](https://github.com/fudan-zvg/DeepInteraction) | 0.691 | 0.655 | 0.969 | 0.929 | 0.583 | 0.842 | 0.832 | 0.882 | 0.759 | 0.731 | 0.768 | 0.795 |

### Relative Resistance Ability (RRA) computed with NDS metric and baseline BEVfusion
| Model | Clean | Beams Red. | Brightness | Darkness | Fog | Missing Cam. | Motion Blur | Points Red. | Snow | Spatial Mis. | Temporal Mis. | mRRA |
|:----------------------------------------------------------------|:----------|:-------------|:-------------|:-----------|:----------|:---------------|:--------------|:--------------|:----------|:---------------|:----------------|:----------|
| ⭐ [CMT](https://github.com/junjie18/CMT) | 0.729 | **18.642** | -1.138 | -0.096 | **9.398** | **2.041** | -0.841 | **8.213** | **9.887** | **17.053** | **8.448** | **7.161** |
| [Sparsefusion](https://github.com/yichen928/SparseFusion) | 0.732 | 4.264 | **3.179** | **1.821** | 4.429 | 0.297 | 0.280 | 3.242 | 1.887 | 3.699 | 7.228 | 3.033 |
| [IS-Fusion](https://github.com/yinjunbo/IS-Fusion) | **0.737** | 3.684 | 2.291 | 1.267 | 3.890 | 0.920 | **3.994** | 1.691 | -2.351 | 4.513 | 7.177 | 2.708 |
| ⚓ [BEVfusion](https://github.com/mit-han-lab/bevfusion) | 0.714 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 |
| [TransFusion](https://github.com/XuyangBai/TransFusion) | 0.708 | -7.210 | 1.799 | 1.146 | -0.552 | 0.340 | -5.412 | -3.296 | -4.220 | -3.626 | 3.850 | -1.718 |
| [DeepInteraction](https://github.com/fudan-zvg/DeepInteraction) | 0.691 | -6.361 | -3.150 | -7.215 | -25.037 | -16.386 | -7.077 | -2.188 | -5.149 | 0.212 | 0.145 | -7.221 |

## Metrics
We adhere to the official nuScenes metric definition for computing the NDS and mAP metrics on the MultiCorrupt dataset. To quantitatively compare the performance between the corrupted dataset and the clean nuScenes datasets, we use a metric called the *Resistance Ability* (RA). This metric is calculated across the different severity levels with
Expand Down Expand Up @@ -349,7 +355,7 @@ logfile="/workspace/evaluation_log.txt"


## TODOs
- [ ] Add visualization for results
- [ ] 📝 Add visualization for results


## Contribution
Expand Down
5 changes: 4 additions & 1 deletion evaluation/evaluations_nds.csv
View file
Original file line number Diff line number Diff line change
Expand Up @@ -4,4 +4,7 @@ DeepInteraction,0.6909,0.6139,0.5237,0.4544,0.5863,0.5014,0.4267,0.6522,0.6242,0
TransFusion,0.7084,0.6306,0.5437,0.4766,0.5708,0.4761,0.4095,0.6554,0.619,0.5331,0.6727,0.5878,0.4944,0.7017,0.698,0.6946,0.6506,0.5996,0.3529,0.7025,0.6996,0.6978,0.7056,0.7041,0.7007,0.6222,0.4591,0.2636,0.5447,0.5353,0.5087
Sparsefusion,0.7315,0.6523,0.5622,0.4901,0.6061,0.5149,0.4461,0.6881,0.6578,0.5838,0.6997,0.6234,0.5374,0.7117,0.6981,0.6836,0.6762,0.6261,0.3811,0.7122,0.7039,0.6978,0.7205,0.7146,0.7039,0.6593,0.5139,0.338,0.5859,0.5704,0.5337
BEVfusion,0.7144,0.6341,0.5471,0.4085,0.5903,0.4943,0.4266,0.6684,0.6379,0.5628,0.6861,0.6217,0.5475,0.7038,0.6961,0.6873,0.6555,0.5981,0.3584,0.699,0.691,0.6861,0.6967,0.6913,0.6851,0.6396,0.4899,0.3199,0.5782,0.5572,0.5233
IS-Fusion,0.7365,0.6537,0.5609,0.4892,0.6221,0.5214,0.4359,0.6827,0.6489,0.5691,0.7046,0.6445,0.5803,0.7164,0.7023,0.6877,0.6697,0.6107,0.3943,0.7101,0.6998,0.6925,0.7198,0.7091,0.6917,0.6694,0.5068,0.3266,0.5619,0.5463,0.5115
IS-Fusion,0.7365,0.6537,0.5609,0.4892,0.6221,0.5214,0.4359,0.6827,0.6489,0.5691,0.7046,0.6445,0.5803,0.7164,0.7023,0.6877,0.6697,0.6107,0.3943,0.7101,0.6998,0.6925,0.7198,0.7091,0.6917,0.6694,0.5068,0.3266,0.5619,0.5463,0.5115
UniBEV_cat,0.6782,0.6097,0.5283,0.4644,0.5813,0.5126,0.4390,0.6470,0.6263,0.5775,0.6341,0.5371,0.4521,0.6666,0.6584,0.6487,0.6346,0.5910,0.3832,0.6645,0.6559,0.6461,0.6717,0.6629,0.6493,0.6141,0.5101,0.4199,0.5811,0.5292,0.4390
UniBEV_avg,0.6842,0.6147,0.5330,0.4677,0.5801,0.5117,0.4619,0.6530,0.6313,0.5821,0.6367,0.5432,0.4569,0.6691,0.6597,0.6505,0.6346,0.5902,0.3674,0.6706,0.6600,0.6501,0.6757,0.6658,0.6537,0.6183,0.5151,0.4264,0.5849,0.5269,0.4323
UniBEV_cnw,0.6853,0.6122,0.5292,0.4646,0.5794,0.5065,0.4492,0.6514,0.6295,0.5802,0.6348,0.5344,0.4427,0.6673,0.6549,0.6450,0.6363,0.5931,0.3648," 0.6716",0.6617,0.6518,0.6764,0.6670,0.6524,0.6181,0.5061,0.3981,0.5798,0.5229,0.4224
3 changes: 3 additions & 0 deletions evaluation/ra_results.csv
View file
Original file line number Diff line number Diff line change
Expand Up @@ -5,3 +5,6 @@ TransFusion,0.8783173348390739,0.9960474308300394,0.9916713721061547,0.918407679
Sparsefusion,0.9012987012987013,0.9849624060150376,0.9736158578263842,0.9244019138755981,0.9729323308270676,0.9565276828434722,0.9406698564593301,0.800956937799043,0.8285714285714285,0.8917293233082706,0.7025290498974709,0.9768967874231032,0.9622693096377306,0.8559125085440874,0.9543403964456596,0.8522214627477784,0.8992481203007519,0.7797676008202323,0.7038961038961039,0.7685577580314422,0.46206425153793573,0.9622693096377306,0.9539302802460696,0.5209842788790157,0.9345181134654819,0.7346548188653451,0.7980861244019138,0.7295967190704031,0.6098427887901572,0.6699931647300068,0.8335748462064251
BEVfusion,0.8952967525195967,0.9752239641657334,0.9784434490481522,0.9175531914893617,0.9851623740201567,0.9603863381858903,0.9356103023516237,0.8093505039193729,0.8262877939529676,0.8875979843225084,0.6857502799552071,0.9676651735722285,0.967245240761478,0.8372060470324747,0.9743840985442329,0.8702407614781635,0.8929171332586786,0.7799552071668533,0.691909294512878,0.7658174692049272,0.44778835386338184,0.958986562150056,0.9603863381858903,0.5016797312430011,0.9620660694288914,0.7663773796192609,0.7877939529675251,0.7325027995520716,0.597144456886898,0.5718085106382979,0.8296845837999254
IS-Fusion,0.9088934147997284,0.9773251866938221,0.9641547861507127,0.9093007467752884,0.9727087576374746,0.9566870332654446,0.9269517990495586,0.7629327902240325,0.8446707399864222,0.8875763747454174,0.6881194840461643,0.9627970128988458,0.9501697216564833,0.8291921249151392,0.9535641547861508,0.8750848608282416,0.8810590631364562,0.7417515274949084,0.7079429735234215,0.7615750169721656,0.443448744059742,0.9391717583163611,0.9402579769178546,0.5353699932111337,0.9337406653088933,0.7879158180583843,0.7727087576374746,0.6945010183299388,0.5918533604887983,0.6642226748133062,0.8255216112242589
UniBEV_cat,0.9054851076378648,0.9904158065467413,0.9797994691831318,0.9357121792981422,0.9828959009141846,0.9349749336478914,0.9539958714243586,0.856826894721321,0.8571217929814214,0.898997345915659,0.7521380123857269,0.9774402831023298,0.9671188439988204,0.8714243585962842,0.9708050722500736,0.7919492774992628,0.923473901503981,0.7803007962253022,0.7558242406369802,0.778973754054851,0.6191388970805072,0.9573872014155116,0.9526688292539074,0.5650250663521085,0.9565025066352109,0.66661751695665,0.8515187260395164,0.6473016809200826,0.6473016809200826,0.6847537599528162,0.8471296569350241
UniBEV_avg,0.9036831335866705,0.9875767319497222,0.9801227711195556,0.927506577024262,0.9779304296989184,0.9305758550131541,0.9543992984507453,0.8548669979538146,0.8478515054077754,0.8984215141771412,0.7528500438468284,0.9731072785735164,0.9646302250803859,0.8626132709733995,0.9641917567962583,0.7939199064600994,0.9226834258988599,0.770096463022508,0.7478807366267174,0.7790119847997662,0.6232095878398128,0.9554223911137093,0.95016077170418,0.5369774919614148,0.9507453960830166,0.6677871967261034,0.8507746273019584,0.6318327974276527,0.6750950014615609,0.6835720549546915,0.8439832407678066
UniBEV_cnw,0.9019407558733401,0.987012987012987,0.9800087552896541,0.9284984678243104,0.9737341310375018,0.926309645410769,0.9505326134539617,0.8460528235809134,0.845469137603969,0.8933313877134101,0.738508682328907,0.9732963665547936,0.96556252736028,0.8654603823143148,0.9556398657522254,0.7798044651977236,0.9185758062162556,0.7630234933605721,0.7390923683058513,0.7722165474974464,0.580913468553918,0.9519918283963227,0.9511162994309063,0.5323216109732963,0.9411936378228513,0.645994454983219,0.846636509557858,0.6163723916532905,0.6554793521085656,0.6779512622209252,0.8368014008463446
8 changes: 4 additions & 4 deletions evaluation/ra_results_beamsreducing.csv
View file
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
CMT,DeepInteraction,TransFusion,Sparsefusion,BEVfusion,IS-Fusion
0.9363511659807956,0.8918801563178463,0.8783173348390739,0.9012987012987013,0.8952967525195967,0.9088934147997284
0.8056241426611798,0.6616008105369808,0.6480801806888763,0.7025290498974709,0.6857502799552071,0.6881194840461643
0.6168724279835391,0.41091330149080907,0.37210615471485037,0.46206425153793573,0.44778835386338184,0.443448744059742
CMT,DeepInteraction,TransFusion,Sparsefusion,BEVfusion,IS-Fusion,UniBEV_cat,UniBEV_avg,UniBEV_cnw
0.9363511659807956,0.8918801563178463,0.8783173348390739,0.9012987012987013,0.8952967525195967,0.9088934147997284,0.9054851076378648,0.9036831335866705,0.9019407558733401
0.8056241426611798,0.6616008105369808,0.6480801806888763,0.7025290498974709,0.6857502799552071,0.6881194840461643,0.7521380123857269,0.7528500438468284,0.738508682328907
0.6168724279835391,0.41091330149080907,0.37210615471485037,0.46206425153793573,0.44778835386338184,0.443448744059742,0.6191388970805072,0.6232095878398128,0.580913468553918
8 changes: 4 additions & 4 deletions evaluation/ra_results_brightness.csv
View file
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
CMT,DeepInteraction,TransFusion,Sparsefusion,BEVfusion,IS-Fusion
0.9654320987654321,0.9820523954262557,0.9960474308300394,0.9849624060150376,0.9752239641657334,0.9773251866938221
0.9399176954732511,0.9724996381531338,0.9939299830604177,0.9768967874231032,0.9676651735722285,0.9627970128988458
0.9060356652949245,0.9515125199015777,0.9891304347826086,0.9622693096377306,0.958986562150056,0.9391717583163611
CMT,DeepInteraction,TransFusion,Sparsefusion,BEVfusion,IS-Fusion,UniBEV_cat,UniBEV_avg,UniBEV_cnw
0.9654320987654321,0.9820523954262557,0.9960474308300394,0.9849624060150376,0.9752239641657334,0.9773251866938221,0.9904158065467413,0.9875767319497222,0.987012987012987
0.9399176954732511,0.9724996381531338,0.9939299830604177,0.9768967874231032,0.9676651735722285,0.9627970128988458,0.9774402831023298,0.9731072785735164,0.9732963665547936
0.9060356652949245,0.9515125199015777,0.9891304347826086,0.9622693096377306,0.958986562150056,0.9391717583163611,0.9573872014155116,0.9554223911137093,0.9519918283963227
8 changes: 4 additions & 4 deletions evaluation/ra_results_dark.csv
View file
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
CMT,DeepInteraction,TransFusion,Sparsefusion,BEVfusion,IS-Fusion
0.9672153635116597,0.9544072948328268,0.9916713721061547,0.9736158578263842,0.9784434490481522,0.9641547861507127
0.9470507544581619,0.9284990591981473,0.9875776397515528,0.9622693096377306,0.967245240761478,0.9501697216564833
0.9308641975308642,0.9051961210015921,0.9850367024280067,0.9539302802460696,0.9603863381858903,0.9402579769178546
CMT,DeepInteraction,TransFusion,Sparsefusion,BEVfusion,IS-Fusion,UniBEV_cat,UniBEV_avg,UniBEV_cnw
0.9672153635116597,0.9544072948328268,0.9916713721061547,0.9736158578263842,0.9784434490481522,0.9641547861507127,0.9797994691831318,0.9801227711195556,0.9800087552896541
0.9470507544581619,0.9284990591981473,0.9875776397515528,0.9622693096377306,0.967245240761478,0.9501697216564833,0.9671188439988204,0.9646302250803859,0.96556252736028
0.9308641975308642,0.9051961210015921,0.9850367024280067,0.9539302802460696,0.9603863381858903,0.9402579769178546,0.9526688292539074,0.95016077170418,0.9511162994309063
8 changes: 4 additions & 4 deletions evaluation/ra_results_fog.csv
View file
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
CMT,DeepInteraction,TransFusion,Sparsefusion,BEVfusion,IS-Fusion
0.9566529492455419,0.8377478651034882,0.9184076792772444,0.9244019138755981,0.9175531914893617,0.9093007467752884
0.8953360768175582,0.701114488348531,0.8464144551101073,0.8559125085440874,0.8372060470324747,0.8291921249151392
0.5670781893004115,0.21016066000868433,0.4981648785996612,0.5209842788790157,0.5016797312430011,0.5353699932111337
CMT,DeepInteraction,TransFusion,Sparsefusion,BEVfusion,IS-Fusion,UniBEV_cat,UniBEV_avg,UniBEV_cnw
0.9566529492455419,0.8377478651034882,0.9184076792772444,0.9244019138755981,0.9175531914893617,0.9093007467752884,0.9357121792981422,0.927506577024262,0.9284984678243104
0.8953360768175582,0.701114488348531,0.8464144551101073,0.8559125085440874,0.8372060470324747,0.8291921249151392,0.8714243585962842,0.8626132709733995,0.8654603823143148
0.5670781893004115,0.21016066000868433,0.4981648785996612,0.5209842788790157,0.5016797312430011,0.5353699932111337,0.5650250663521085,0.5369774919614148,0.5323216109732963
8 changes: 4 additions & 4 deletions evaluation/ra_results_missingcamera.csv
View file
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
CMT,DeepInteraction,TransFusion,Sparsefusion,BEVfusion,IS-Fusion
0.9849108367626886,0.9050513822550297,0.9905420666290231,0.9729323308270676,0.9851623740201567,0.9727087576374746
0.9751714677640604,0.841800550007237,0.9853190287972896,0.9543403964456596,0.9743840985442329,0.9535641547861508
0.9614540466392318,0.7791286727456941,0.9805194805194805,0.9345181134654819,0.9620660694288914,0.9337406653088933
CMT,DeepInteraction,TransFusion,Sparsefusion,BEVfusion,IS-Fusion,UniBEV_cat,UniBEV_avg,UniBEV_cnw
0.9849108367626886,0.9050513822550297,0.9905420666290231,0.9729323308270676,0.9851623740201567,0.9727087576374746,0.9828959009141846,0.9779304296989184,0.9737341310375018
0.9751714677640604,0.841800550007237,0.9853190287972896,0.9543403964456596,0.9743840985442329,0.9535641547861508,0.9708050722500736,0.9641917567962583,0.9556398657522254
0.9614540466392318,0.7791286727456941,0.9805194805194805,0.9345181134654819,0.9620660694288914,0.9337406653088933,0.9565025066352109,0.9507453960830166,0.9411936378228513
8 changes: 4 additions & 4 deletions evaluation/ra_results_motionblur.csv
View file
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
CMT,DeepInteraction,TransFusion,Sparsefusion,BEVfusion,IS-Fusion
0.9445816186556928,0.9480387899840789,0.9496047430830039,0.9565276828434722,0.9603863381858903,0.9566870332654446
0.8441700960219478,0.8323925314806774,0.8297571993224166,0.8522214627477784,0.8702407614781635,0.8750848608282416
0.7348422496570645,0.7148646692719641,0.6979107848673066,0.7346548188653451,0.7663773796192609,0.7879158180583843
CMT,DeepInteraction,TransFusion,Sparsefusion,BEVfusion,IS-Fusion,UniBEV_cat,UniBEV_avg,UniBEV_cnw
0.9445816186556928,0.9480387899840789,0.9496047430830039,0.9565276828434722,0.9603863381858903,0.9566870332654446,0.9349749336478914,0.9305758550131541,0.926309645410769
0.8441700960219478,0.8323925314806774,0.8297571993224166,0.8522214627477784,0.8702407614781635,0.8750848608282416,0.7919492774992628,0.7939199064600994,0.7798044651977236
0.7348422496570645,0.7148646692719641,0.6979107848673066,0.7346548188653451,0.7663773796192609,0.7879158180583843,0.66661751695665,0.6677871967261034,0.645994454983219
Loading

0 comments on commit 98904d8

Please sign in to comment.