Evaluation always produces mAP of 0.0 when using backbones other than Resnet50

[jpxrc]

First and foremost, thank you for the awesome package! The dataset I am using is of satellite images consisting of 29 different classes. I have been able to train and evaluate a retinanet model on this dataset using the default 'densenet50' backbone on a subset of the 29 classes.

However, when I switch over to training and evaluating a model with a different backbone network such as 'densenet121', all of the mAP scores for each class is zero. I'm not receiving any issues when training (I am also using the random-transform flag for each epoch), or when converting the model (I also supply the --backbone='densenet121' flag) and it converts successfully. I can also see that losses being optimized during training so it's definitely detecting and classifying the objects in the images.

I even tried using the original resnet50 model trained on a subset of classes to see if it would pick up those classes on the full dataset with 29 classes and it still produces an output of zero. I looked at the validation_annotations.csv file for both cases and the formatting is identical so I don't think it has to do with the annotation files.

I have attached the validation_annotations.csv file, the classes.csv file (converted to .txt files in order to attach them here)
common_classes.txt
common_validation_annotations.txt

Any ideas what could be going on?

EDIT: I just did a comparison of a Resnet50 model and Densenet121 model both trained on the same dataset that I know for sure works and the problem is definetely with the densenet121 implementation because the Resnet50 model is producing output during evaluation.

[hgaiser]

Densenet is a community contribution and I have never really used it. If you find out what the problem is then a PR will be welcome :)

[birham-red-bd]

I have the same problem when i changed the backbone to the shuffleNet. The loss is decreasing, but the MAP is always zero. However when i trained model with resnet50 backbone, everything is okay. I still did not find the problem. Can anyone give me some advice?

[hgaiser]

Do you evaluate during training, or after training using the evaluate tool? For densenet and mobilenet I noticed there is a bug when preprocessing the images in the evaluate tool. Those backbones require a mode='tf' in the preprocess_image call which isn't there currently.

[birham-red-bd]

yeah, I evaluate the map during training. The dataset i used is pascal voc2007. And I did not use pretrained model, because i did not find the pretrained weights of shufflenet-V2 on imagenet.

[hgaiser]

@songhuizhong I'm not sure what you mean with shufflenet-V2, we don't have that backbone in our repository.

[birham-red-bd]

It is the model I built by myself. here is the article introducing the shufflenet-v2 :https://arxiv.org/abs/1807.11164

[hgaiser]

In that case I can't help. I only have experience with ResNet backbones. If you find out the solution to this then a PR is welcome.

[birham-red-bd]

well, i find the problem, it is my fault, the reason why map is 0 is because the feature map input into feature pyramid network is in wrong order.
This is the wrong one.
layer_names = ['1x1conv5_out','stage4/block1/relu_1x1conv_1','stage3/block1/relu_1x1conv_1']
and this is the correct one.
layer_names = ['stage3/block1/relu_1x1conv_1','stage4/block1/relu_1x1conv_1','1x1conv5_out']
But I can only trained this model with the highest mAP of 44.59% on dataset of VOC2007 test. The training dataset is voc2012 trainval. The input size is changed to 512*512, because i want to use larger batch size of 28. And i trained this model without using pretrained weights on imageNet.

[hgaiser]

Alright, I'll assume this issue resolved then.

[hgaiser]

Actually, the original issue was something else entirely, DensNet, so I'm reopening..