BackdoorBench is a comprehensive benchmark for backdoor learning in deep neural networks (DNNs). As DNNs are increasingly applied in critical scenarios, the security issues associated with them have gained greater attention. Backdoor learning is an emerging field focused on studying the vulnerabilities of DNNs. Despite the continuous development of new attack and defense methods, the evaluations of these methods often lack thoroughness in verifying their claims and actual performance.

To address this issue, we have developed BackdoorBench, which includes an extensible modular codebase currently comprising implementations of 20 state-of-the-art attack algorithms and 32 state-of-the-art defense algorithms. The benchmark also establishes a standardized protocol for complete backdoor learning.

BackdoorBench conducts comprehensive evaluations of 11,492 pairs of attacks against defenses, using five poisoning ratios, across five models and four datasets. The evaluations are further analyzed from different perspectives to study the effects of attack and defense algorithms, poisoning ratio, model, and dataset in backdoor learning.

All the codes and evaluations of BackdoorBench are publicly available. The whole framework is shown as follows.