This repo is the implementation of Training-free Boost for Open-Vocabulary Object Detection with Confidence Aggregation

Open-vocabulary object detection (OVOD) aims at localizing and recognizing visual objects from novel classes unseen at the training time. Whereas, empirical studies reveal that advanced detectors generally assign lower scores to those novel instances, which are inadvertently suppressed during inference by commonly adopted greedy strategies like Non-Maximum Suppression (NMS), leading to sub-optimal detection performance for novel classes. This paper systematically investigates this problem with the commonly-adopted two-stage OVOD paradigm. Specifically, in the region-proposal stage, proposals that contain novel instances showcase lower objectness scores, since they are treated as background proposals during the training phase. Meanwhile, in the object-classification stage, novel objects share lower region-text similarities (i.e., classification scores) due to the biased visual-language alignment by seen training samples. To alleviate this problem, this paper introduces two advanced measures to adjust confidence scores and conserve erroneously dismissed objects: (1) a class-agnostic localization quality estimate via overlap degree of region/object proposals, and (2) a text-guided visual similarity estimate with proxy prototypes for novel classes. Integrated with adjusting techniques specifically designed for the region-proposal and object-classification stages, this paper derives the aggregated confidence estimate for the open-vocabulary object detection paradigm AggDet.

• Installation

  Following the Installation instructions of CoDet to setup environment.

  Setup environment

  conda create --name aggdet python=3.8 -y && conda activate aggdet
  pip install torch==1.12.1+cu116 torchvision==0.13.1+cu116 --extra-index-url https://download.pytorch.org/whl/cu116
  git clone https://github.com/WarlockWendell/AggDet.git

  Install Apex and xFormer (You can skip this part if you do not use EVA-02 backbone)

  pip install ninja
  pip install -v -U git+https://github.com/facebookresearch/xformers.git@7e05e2caaaf8060c1c6baadc2b04db02d5458a94
  git clone https://github.com/NVIDIA/apex && cd apex
  pip install packaging
  pip install -v --disable-pip-version-check --no-cache-dir --no-build-isolation --global-option="--cpp_ext" --global-option="--cuda_ext" ./ && cd ..

  Install detectron2 and other dependencies

  cd AggDet/third_party/detectron2
  pip install -e .
  cd ../..
  pip install -r requirements.txt

• Datasets

  Please refer to DATA.md for more details.

• Pretrained weights

  You can download the pre-trained weights from the official GitHub repos of Detic and CoDet, and put them under the <AGGDET_ROOT>/ckpt/models directory.

  model	dataset	download
  Detic_RN50	COCO	model
  CoDet_RN50	COCO	model
  Detic_SwinB	LVIS	model
  CoDet_RN50	LVIS	model
  CoDet_SwinB	LVIS	model
  CoDet_EVA02	LVIS	model

Take Detic with a ResNet50 backbone on the OV-COCO dataset as an example.

python train_net.py --eval-only --config-file configs/Detic_RN50_COCO.yaml

You can modify the following parameters in the yaml file to adjust the parameters described in the paper.

OVERLAP_TOPK: 3
ALPHA: 0.05
BETA: 0.75

For example, use the following command to test the baseline model:

python train_net.py --eval-only --config-file configs/Detic_RN50_COCO.yaml  \
MODEL.OVERLAP_TOPK=0 MODEL.ALPHA 0.0 MODEL.BETA 0.0

You can change the config-file to change the model and dataset. Refer to REPRODUCE.md for more details.

@article{
  title={Training-free Boost for Open-Vocabulary Object Detection with Confidence Aggregation},
  author={Yanhao Zheng, Kai Liu},
  journal={arXiv preprint arXiv:2404.08603},
  year={2024}
}

AggDet is built upon the awesome works Codet, EVA and Detic. Many thanks for their wonderful work.

This project is licensed under the Apache License 2.0 - see the LICENSE file for details.