First of all thank you authors for this great work.
When I followed the readme guide to reproduce the paper, I wanted to Evaluate the trained detectors in the Transfer Learning task, and I executed the provided sample script code as follows.
python3 ./tools/train_net.py \ --eval-only \ --num-gpus 1 \ --config-file ./configs/COCO-InstanceSegmentation/CLIP_fast_rcnn_R_50_C4_ovd.yaml \ MODEL.WEIGHTS ./pretrained_ckpt/regionclip/regionclip_finetuned-coco_rn50.pth \ MODEL.CLIP.OFFLINE_RPN_CONFIG ./configs/COCO-InstanceSegmentation/mask_rcnn_R_50_C4_1x_ovd_FSD.yaml \ MODEL.CLIP.BB_RPN_WEIGHTS ./pretrained_ckpt/rpn/rpn_coco_48.pth \ MODEL.CLIP.TEXT_EMB_PATH ./pretrained_ckpt/concept_emb/coco_48_base_cls_emb.pth \ MODEL.CLIP.OPENSET_TEST_TEXT_EMB_PATH ./pretrained_ckpt/concept_emb/coco_65_cls_emb.pth \ MODEL.ROI_HEADS.SOFT_NMS_ENABLED True \
However, the Average Precision of RN50 and COCO (Generalized: Novel + Base) is very low, and the test result is basically 0, as follows.
[05/07 22:13:34 d2.evaluation.evaluator]: Inference done 4832/4836. Dataloading: 0.0011 s / iter. Inference: 4.8330 s / iter. Eval: 0.0004 s / iter. Total: 4.8347 s / iter. ETA=0:00:19
[05/07 22:13:44 d2.evaluation.evaluator]: Inference done 4833/4836. Dataloading: 0.0011 s / iter. Inference: 4.8341 s / iter. Eval: 0.0004 s / iter. Total: 4.8358 s / iter. ETA=0:00:14
[05/07 22:13:50 d2.evaluation.evaluator]: Inference done 4834/4836. Dataloading: 0.0011 s / iter. Inference: 4.8344 s / iter. Eval: 0.0004 s / iter. Total: 4.8361 s / iter. ETA=0:00:09
[05/07 22:13:55 d2.evaluation.evaluator]: Total inference time: 6:29:18.203654 (4.835066 s / iter per device, on 1 devices)
[05/07 22:13:55 d2.evaluation.evaluator]: Total inference pure compute time: 6:29:10 (4.833392 s / iter per device, on 1 devices)
[05/07 22:13:58 d2.evaluation.coco_evaluation]: Preparing results for COCO format ...
[05/07 22:13:58 d2.evaluation.coco_evaluation]: Saving results to ./output/inference/coco_instances_results.json
[05/07 22:14:00 d2.evaluation.coco_evaluation]: Evaluating predictions with unofficial COCO API...
Loading and preparing results...
DONE (t=1.64s)
creating index...
index created!
[05/07 22:14:02 d2.evaluation.fast_eval_api]: Evaluate annotation type bbox
[05/07 22:14:08 d2.evaluation.fast_eval_api]: COCOeval_opt.evaluate() finished in 5.88 seconds.
[05/07 22:14:08 d2.evaluation.fast_eval_api]: Accumulating evaluation results...
[05/07 22:14:10 d2.evaluation.fast_eval_api]: COCOeval_opt.accumulate() finished in 1.59 seconds.
Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.000
Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.000
Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.000
Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.000
Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.000
Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.000
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.000
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.000
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.001
Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.001
Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.001
Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.002
[05/07 22:14:10 d2.evaluation.coco_evaluation]: Evaluation results for bbox:
| AP | AP50 | AP75 | APs | APm | APl |
|:-----:|:------:|:------:|:-----:|:-----:|:-----:|
| 0.001 | 0.005 | 0.001 | 0.001 | 0.002 | 0.002 |
[05/07 22:14:10 d2.evaluation.coco_evaluation]: AP50_split_target AP: 0.0
[05/07 22:14:10 d2.evaluation.coco_evaluation]: AP50_split_base AP: 6.675366123124714e-05
[05/07 22:14:10 d2.evaluation.coco_evaluation]: AP50_split_all AP: 4.9295011370767116e-05
[05/07 22:14:10 d2.evaluation.coco_evaluation]: Per-category bbox AP:
| category |
AP |
category |
AP |
category |
AP |
| person |
0.074 |
bicycle |
0.000 |
car |
0.000 |
| motorcycle |
0.000 |
airplane |
0.000 |
bus |
0.000 |
| train |
0.000 |
truck |
0.000 |
boat |
0.000 |
| bench |
0.000 |
bird |
0.000 |
cat |
0.000 |
| dog |
0.000 |
horse |
0.000 |
sheep |
0.000 |
| cow |
0.000 |
elephant |
0.000 |
bear |
0.000 |
| zebra |
0.000 |
giraffe |
0.000 |
backpack |
0.000 |
| umbrella |
0.000 |
handbag |
0.000 |
tie |
0.000 |
| suitcase |
0.000 |
frisbee |
0.000 |
skis |
0.000 |
| snowboard |
0.000 |
kite |
0.000 |
skateboard |
0.000 |
| surfboard |
0.000 |
bottle |
0.000 |
cup |
0.000 |
| fork |
0.000 |
knife |
0.000 |
spoon |
0.000 |
| bowl |
0.000 |
banana |
0.000 |
apple |
0.000 |
| sandwich |
0.000 |
orange |
0.000 |
broccoli |
0.000 |
| carrot |
0.000 |
pizza |
0.000 |
donut |
0.000 |
| cake |
0.000 |
chair |
0.000 |
couch |
0.000 |
| bed |
0.000 |
toilet |
0.000 |
tv |
0.000 |
| laptop |
0.000 |
mouse |
0.000 |
remote |
0.000 |
| keyboard |
0.000 |
microwave |
0.000 |
oven |
0.000 |
| toaster |
0.000 |
sink |
0.000 |
refrigerator |
0.000 |
| book |
0.000 |
clock |
0.000 |
vase |
0.000 |
| scissors |
0.000 |
toothbrush |
0.000 |
|
|
| [05/07 22:14:10 d2.engine.defaults]: Evaluation results for coco_2017_ovd_all_test in csv format: |
|
|
|
|
|
| [05/07 22:14:10 d2.evaluation.testing]: copypaste: Task: bbox |
|
|
|
|
|
| [05/07 22:14:10 d2.evaluation.testing]: copypaste: AP,AP50,AP75,APs,APm,APl |
|
|
|
|
|
| [05/07 22:14:10 d2.evaluation.testing]: copypaste: 0.0011,0.0049,0.0006,0.0006,0.0017,0.0019 |
|
|
|
|
|
What should I do, or does the author have any good advice? I didn't modify any parameters, just follow the steps.
Look forward to receiving your reply, thanks again.
First of all thank you authors for this great work.
When I followed the readme guide to reproduce the paper, I wanted to Evaluate the trained detectors in the Transfer Learning task, and I executed the provided sample script code as follows.
python3 ./tools/train_net.py \ --eval-only \ --num-gpus 1 \ --config-file ./configs/COCO-InstanceSegmentation/CLIP_fast_rcnn_R_50_C4_ovd.yaml \ MODEL.WEIGHTS ./pretrained_ckpt/regionclip/regionclip_finetuned-coco_rn50.pth \ MODEL.CLIP.OFFLINE_RPN_CONFIG ./configs/COCO-InstanceSegmentation/mask_rcnn_R_50_C4_1x_ovd_FSD.yaml \ MODEL.CLIP.BB_RPN_WEIGHTS ./pretrained_ckpt/rpn/rpn_coco_48.pth \ MODEL.CLIP.TEXT_EMB_PATH ./pretrained_ckpt/concept_emb/coco_48_base_cls_emb.pth \ MODEL.CLIP.OPENSET_TEST_TEXT_EMB_PATH ./pretrained_ckpt/concept_emb/coco_65_cls_emb.pth \ MODEL.ROI_HEADS.SOFT_NMS_ENABLED True \However, the Average Precision of RN50 and COCO (Generalized: Novel + Base) is very low, and the test result is basically 0, as follows.
[05/07 22:14:10 d2.evaluation.coco_evaluation]: AP50_split_base AP: 6.675366123124714e-05
[05/07 22:14:10 d2.evaluation.coco_evaluation]: AP50_split_all AP: 4.9295011370767116e-05
[05/07 22:14:10 d2.evaluation.coco_evaluation]: Per-category bbox AP:
What should I do, or does the author have any good advice? I didn't modify any parameters, just follow the steps.
Look forward to receiving your reply, thanks again.