fairness-benchmark-public

This repository contains the source code that accompanies the FHIBE (Fair Human-centric Image Benchmark) paper, linked here.

Dataset and pre-computed results

Use of this repository requires a copy of the FHIBE dataset, along with some pre-computed results as provided by our FHIBE Website https://fairnessbenchmark.ai.sony/.

You'll need to first create an account, then download the dataset with name:

fhibe.{version}_downsampled_public.tar.gz

This version of the dataset includes the needed metadata, unlike the fhibe.{version}_fullres, which includes the full resolution dataset only.

Please extract this compressed archive via the following command:

tar -xzf fhibe.{version}_downsampled_public.tar.gz

The unarchived dataset and metadata structure:

fhibe.{version_name}_downsampled_public
├── data
│   ├── aggregated_results
│   │   ├── aggregated_scores
│   │   │   ├── fhibe_face_scores.csv
│   │   │   └── fhibe_scores.csv
│   │   └── predictions
│   │       └── face_verification
│   │           ├── arcface
│   │           │   └── model_outputs.json
│   │           ├── curricular_face
│   │           │   └── model_outputs.json
│   │           └── facenet
│   │               └── model_outputs.json
│   ├── annotator_metadata
│   │   ├── QAannotator_demographics.csv
│   │   └── annotator_demographics.csv
│   ├── processed
│   │   ├── fhibe_downsampled
│   │   │   └── fhibe_downsampled.csv
│   │   └── fhibe_face_crop_align
│   │       ├── fhibe_face_crop_align.csv
│   │       └── masks
│   ├── protocol
│   │   └── allfhibe_allgenimposter.txt
│   └── raw
│       └── fhibe_downsampled # FHIBE Dataset
└── results
    ├── face_parsing
    │   └── fhibe_face_crop_align
    │       └── dml_csr
    │           ├── model_outputs.json
    │           └── results_f1.json
    └── face_verification
        └── fhibe_face_crop_align
            ├── arcface
            │   └── model_outputs.json
            ├── curricular_face
            │   └── model_outputs.json
            └── facenet
                └── model_outputs.json

Note that the FHIBE Downsampled dataset is in fhibe.{version_name}_downsampled_public/data/raw/fhibe_downsampled.

Throughout the codebase, references will be made to some of the metadata json and csvs above. Make sure to update these filepath references to point to the relative location of the dataset and metadata in comparison to this codebase.

Utility evaluation

This portion of the codebase is for executing fairness benchmarks on the FHIBE dataset across a variety of tasks.

Refer to readme in fairness-benchmark-public/utility_evaluation/README.md for further details.

Bias Diagnosis Foundation Models

The code included in this section has a series of config yaml files; you'll need to update the directories to point to the dataset+metadata archive you downloaded earlier.

Refer to readme in fairness-benchmark-public/BiasDiagnosis_foundation_models/README.md

Bias Diagnosis Narrow Models

Refer to readme in fairness-benchmark-public/BiasDiagnosis_narrow_models/README.md