Official pytorch implementation of the paper: "HOLMES: HOLonym-MEronym based Semantic inspection for Convolutional Image Classifiers"
Paper accepted at xAI 2023
This repository contains the code for running the experiments of HOLMES: HOLonym-MEronym based Semantic inspection.
This code includes:
-
Code to run HOLMES on VGG16. The script
run_pipeline.py
runs all the HOLMES pipeline steps. -
Code to run only one of the HOLMES pipeline steps at a time. The scripts
run_scraping_only.py
,run_training_only.py
andrun_explanations_only.py
respectively run the HOLMES scraping, training and explanations step. -
Code to evaluate the HOLMES explanations. The script
run_evaluation.py
runs the evaluation of the part-based explanations and collects the related measurements. -
Code to extract and organize the ImageNet validation samples into subfolders named to be compatible with the HOLMES experiments. The script
extract_and_rename.py
extracts the ImageNet validation samples and organizes them into subfolders named with the associated ImageNet class (i.e., Arabian camel, airliner, etc.).
- This code was tested and run in the Python3 (v3.8.2) environment. The required libraries and dependecies were installed with the pip package manager, version 21.0.1.
- Prior to installation, make sure to have a version equal or greater than 21.0.1, by executing the command:
python -m pip -V
- In case your pip version is lesser than 21.0.1, upgrade it by executing the command:
python -m pip install --upgrade pip
- Execute the setup script, which will install all the required libraries and dependecies for running the HOLMES experiment.
setup.sh
- Then, if your Operating System is Unix based, execute the following commands (skip if on Windows OS):
sudo apt-get update
sudo apt-get install chromium-chromedriver
sudo python unix_additional_setup.py install
- Download the ILSVRC 2012 ImageNet validation set (50,000 validation images) from here and put the downloaded archive (ILSVRC2012_img_val.tar) into the
Holonyms
folder. - Execute the script
extract_and_rename.py
from theHolonyms
folder.
python extract_and_rename.py
- Run the HOLMES algorithmic pipeline. You can run
run_pipeline.py
to execute the complete HOLMES pipeline.
python run_pipeline.py
- Alternatively, you can run
run_scraping_only.py
,run_training_only.py
andrun_explanations_only.py
(in this order) to execute each step separately.
python run_scraping_only.py
python run_training_only.py
python run_explanations_only.py
- IMPORTANT: in case you want to skip the first two steps and execute either the explanations step or the evaluation, follow the instruction at the end of the Report Section and then execute the
download.py
script with the following arguments:
python download.py https://drive.google.com/drive/folders/1UkJEN8tcIf9kODo5j-jFx8tdiU2URlxa?usp=sharing Checkpoints
python download.py https://drive.google.com/drive/folders/14w6ohQYxBg7_9FL4E7a2iOYfeYHgGP1M?usp=sharing Checkpoints
python download.py https://drive.google.com/drive/folders/1ETY0igoirK9wWMgpCP9gsXexyn21ZYNP?usp=sharing Checkpoints
In this way you will obtain the same trained models weights obtained during the HOLMES experiment, and they will be located in the Checkpoints
folder.
- After the execution of the whole pipeline, run
run_evaluation.py
to collect the insertion/deletion curves measurements for a quantitative evaluation of the per-part HOLMES generated explanations.
python run_evaluation.py
- At the end of the execution of each step, a json file will be generated in the main directory, containing the measurements which are needed for starting the next steps. For example, after having executed the scraping, the training and the explanations steps, you will have:
scrape.json
train.json
exp.json
- Similarly, after having executed the evaluation script, you will have:
curves.json
- IMPORTANT: In case you want to skip one of the steps and execute one of next ones, copy into the main directory the json files associated to the previous steps from the
Results
folder, which already contains the final measurements recorded during the HOLMES experiment at the end of each step.
The Notebooks folder includes:
Meronyms_Extraction.ipynb
displays the Meronyms Extraction procedure and how the Holonyms and the Meronyms for the HOLMES experiment where selected.Results_analysis.ipynb
displays the results obtained during the HOLMES whole experiment.Explanations_examples(Animals).ipynb
shows a subset of the explanations generated for the Animals classes.Explanations_examples(Man-made).ipynb
shows a subset of the explanations generated for the Man-made object classes.