If You Uploaded Osme Models Please Delete Some Models

As requested past IBM, this repository is moved to https://github.com/IBM/CLEVER-Robustness-Score, but we aim to go on both repositories synced upward. The code is released nether Apache License v2.

CLEVER: A Robustness Metric For Deep Neural Networks

CLEVER (Cross-Lipschitz Eastxtreme Value for nEasttwork Robustness) is a metric for measuring the robustness of deep neural networks. It estimates the robustness lower bound by sampling the norm of gradients and fitting a limit distribution using extreme value theory. CLEVER score is attack-agnostic; a higher score number indicates that the network is likely to be less venerable to adversarial examples. CLEVER can be efficiently computed even for large state-of-the-fine art ImageNet models like ResNet-fifty and Inception-v3.

For more details, please come across our paper:

1. Tsui-Wei Weng*, Huan Zhang*, Pin-Yu Chen, Jinfeng Yi, Dong Su, Yupeng Gao, Cho-Jui Hsieh and Luca Daniel,"Evaluating the Robustness of Neural Networks: An Extreme Value Theory Approach", ICLR 2018

2. Tsui-Wei Weng*, Huan Zhang*, Pivot-Yu Chen, Aurelie Lozano, Cho-Jui Hsieh and Luca Daniel, "On Extensions of CLEVER: A Neural Network Robustness Evaluation Algorithm", IEEE GlobalSIP 2018

* Equal contribution

Cite our works

              @inproceedings{weng2018clever,   author = "Tsui-Wei Weng AND Huan Zhang AND Pin-Yu Chen AND Jinfeng Yi AND Dong Su AND Yupeng Gao AND Cho-Jui Hsieh AND Luca Daniel",   title = "Evaluating the Robustness of Neural Networks: An Extreme Value Theory Approach",   booktitle = "International Conference on Learning Representations (ICLR)",   year = "2018",   month = "may" }                          

              @inproceedings{weng2018cleverExtension,   author = "Tsui-Wei Weng AND Huan Zhang AND Pin-Yu Chen AND Aurelie Lozano AND Cho-Jui Hsieh AND Luca Daniel",   title = "On Extensions of CLEVER: A Neural Network Robustness Evaluation Algorithm",   booktitle = "IEEE Global Conference on Point and Information Processing (GlobalSIP)",   year = "2018",   month = "november" }                          

News

• Aug half dozen, 2018: CLEVER evaluation with input transformations (e.g., staircase function or JPEG pinch) is implemented via BPDA (Astern Laissez passer Differentiable Approximation)
• Aug 16, 2018: added 2nd order CLEVER evaluation implementation, which tin can be used to evaluate robustness on classifiers that are twice-differentiable.
• Oct 23, 2018: Our work on extension of CLEVER is accepted by GlobalSIP 2018: On Extensions of CLEVER: A Neural Network Robustness Evaluation Algorithm

Discussion with Ian Goodfellow and Our Clarifications

Nosotros received some inquires on Ian Goodfellow's comment "Gradient Masking Causes CLEVER to Overestimate Adversarial Perturbation Size" on our paper. Nosotros thank Ian for the word but the comments are inappropriate and non applicable to our newspaper. CLEVER is intended to be a tool for network designer and to evaluate network robustness in the "white-box" setting. Especially, the argument that on digital computers all functions are not Lipschitz continuous and carry like a staircase function (where the gradient is zero nigh everywhere) is wrong. Under the white-box setting, gradients can be computed via automatic differentiation, which is well supported by mature packages like TensorFlow. See our answer and discussions with Ian Goodfellow on gradient masking and implmentation on digital computers.

Setup and train models

The code is tested with python3 and TensorFlow v1.iii, v1.4 and v1.5. The following packages are required:

              sudo apt-get install python3-pip python3-dev sudo pip3 install --upgrade pip sudo pip3 install vi pillow scipy numpy pandas matplotlib h5py posix_ipc tensorflow-gpu                          

Then clone this repository:

              git clone https://github.com/huanzhang12/CLEVER.git cd CLEVER                          

Gear up the MNIST and CIFAR-x data and models with different activation functions:

              python3 train_models.py python3 train_2layer.py python3 train_nlayer.py --model mnist --modeltype cnn --activation tanh 32 32 64 64 200 200  python3 train_nlayer.py --model cifar --modeltype cnn --activation tanh 64 64 128 128 256 256                          

To download the ImageNet models:

              python3 setup_imagenet.py                          

To prepare the ImageNet dataset, download and unzip the following archive:

http://download.huan-zhang.com/datasets/adv/img.tar.gz

and put the imgs folder in ../imagenetdata, relative to the CLEVER repository. This path tin be changed in setup_imagenet.py.

              cd .. mkdir imagenetdata && cd imagenetdata wget http://download.huan-zhang.com/datasets/adv/img.tar.gz tar zxf img.tar.gz cd ../CLEVER                          

How to run

Step ane: Collect gradients

The start pace for computing CLEVER score is to collect slope samples. The following control collects gradient samples for ten images in MNIST dataset; for each epitome, 3 target attack classes are chosen (random, top-ii and to the lowest degree likely). Images that are classified incorrectly will exist skipped, and so you might become less than x images. The default network used has a seven-layer AlexNet-like CNN structure.

              python3 collect_gradients.py --dataset mnist --numimg 10                          

Results will be saved into folder lipschitz_mat/mnist_normal by default (which tin can exist changed by specifying the --saved <binder proper noun> parameter), as a few .mat files.

Run python3 collect_gradients.py -h for boosted help information.

Updated: For model with input transformation, use an additional parameter --transform. Currently three input transformations are supported (bit-depth reduction, JPEG pinch and PNG pinch, corresponding to defend_reduce, defend_jpeg, defend_png options). For instance:

              python3 collect_gradients.py --dataset cifar --numimg 10 --transform defend_jpeg                          

Y'all should wait roughly the same CLEVER score with input transformations, as input transformations do not increase model'due south intrinsic robustness and can be broken by BPDA. Encounter defence force.py for the implementations of input transformations.

Updated: To run 2nd society clever score, run.sh can be used and prepare order = two:

              ./run.sh model modeltype nsamp niters activation order target gpuNum                          

For example, to become thousand samples of 2nd social club clever with 100 iterations on a mnist 7-layer cnn model with tanh activation and random target:

              ./run.sh mnist normal grand 100 tanh 2 rand                          

To get samples for the original clever score (1st order approximation), set order = 1.

Stride two: Compute the CLEVER score

To compute CLEVER score using the collected gradients, run clever.py with data saving folder as a parameter:

              python3 clever.py lipschitz_mat/mnist_normal                          

Run python3 clever.py -h for additional assistance data.

Step iii: How to interpret the score?

At the finish of the output of clever.py, you volition see three [STATS][L0] lines similar to the post-obit:

              [STATS][L0] info = least, least_clever_L1 = 2.7518, least_clever_L2 = i.1374, least_clever_Li = 0.080179 [STATS][L0] info = random, random_clever_L1 = two.9561, random_clever_L2 = one.1213, random_clever_Li = 0.075569 [STATS][L0] info = top2, top2_clever_L1 = i.6683, top2_clever_L2 = 0.70122, top2_clever_Li = 0.050181                          

The scores shown are the average scores for all (in the case above, 10) images, with three unlike target attack classes: least probable, random and peak-two (the class with second largest probability). Three scores are provided: CLEVER_L2, CLEVER_Linf and CLEVER_L1, representing the robustness for L2, L_infinity and L1 perturbations. CLEVER score for Lp norm roughly reflects the minimum Lp norm of adversarial perturbations. A higher CLEVER score indicates amend network robustness, as the minimum adversarial perturbation is probable to have a larger Lp norm. As CLEVER uses a sampling based method, the scores may vary slightly for unlike runs.

More Examples

For example, the post-obit command will evaluate the CLEVER scores on one ImageNet image, for a fifty-layer ResNet model. We set the number of gradient samples per iterations to 512, and run 100 iterations:

              python3 collect_gradients.py --dataset imagenet --model_name resnet_v2_50 -N 512 -i 100 python3 clever.py lipschitz_mat/imagenet_resnet_v2_50/                          

For this image (139.00029510.jpg, which is the first prototype given the default random seed) in dataset, the original class is 139 (bustard), least likely class is 20 (chickadee), acme-2 class is 82 (ptarmigan), random course target is 708 (pay-telephone). (These can be observed in [DATAGEN][L1] lines of the output of collect_gradients.py). We become the following CLEVER scores:

              [STATS][L0] info = least, least_clever_L1 = eight.1393, least_clever_L2 = 0.64424, least_clever_Li = 0.0029474  [STATS][L0] info = random, random_clever_L1 = 4.6543, random_clever_L2 = 0.61181, random_clever_Li = 0.0023765  [STATS][L0] info = top2, top2_clever_L1 = 0.99283, top2_clever_L2 = 0.13185, top2_clever_Li = 0.00062238                          

The L2 CLEVER score for the pinnacle-2, random and to the lowest degree-probable classes are 0.13185, 0.61181 and 0.64424, respectively. Information technology indicates that information technology is very easy to attack this image from class 139 to 82. Nosotros then run the CW attack, which is the strongest L2 set on to date, on this image with the same iii target classes. The distortion of adversarial images are 0.1598, 0.82025, 0.85298 for the three targets. Indeed, to misclassify the epitome to class 82, merely a very small distortion (0.1598) is needed. Also, the CLEVER scores are (usually) less than the L2 distortions observed on adversarial examples, merely are not too small to be useless, reflecting the nature that CLEVER is an estimated robustness lower bound.

CLEVER also has an untargeted version, which is essentially the smallest CLEVER score over all possible target classes. The post-obit examples shows how to compute untargeted CLEVER score for x images from MNIST dataset, on the 2-layer MLP model:

              python3 collect_gradients.py --data mnist --model_name ii-layer --target_type xvi --numimg 10 python3 clever.py --untargeted ./lipschitz_mat/mnist_2-layer/                          

Target type 16 (bit 4 ready to ane) indicates that we are collecting gradients for untargeted CLEVER score (encounter python3 collect_gradients.py -h for more details). The results volition look like the following:

              [STATS][L0] info = untargeted, untargeted_clever_L1 = 3.4482, untargeted_clever_L2 = 0.69393, untargeted_clever_Li = 0.035387                          

For datasets which have many classes, it is very expensive to evaluate the untargeted CLEVER scores. Nonetheless, unremarkably the robustness of the top-2 targeted class tin can roughly reflect the untargeted robustness, equally it is usually i of the easiest classes to change to.

Built-in Models

In the examples shown higher up we have used several different models. The code on this repository has a large number of born models for robustness evaluation. Model tin be selected by changing the --model_name parameter to collect_gradiets.py. For MNIST and CIFAR dataset, the post-obit models are available: "two-layer" (MLP), "normal" (7-layer CNN), "distilled" (vii-layer CNN with defensive distillation), "brelu" (7-layer CNN with Bounded ReLU). For ImageNet, available options are: "resnet_v2_50", "resnet_v2_101", "resnet_v2_152", "inception_v1", "inception_v2", "inception_v3", "inception_v4", "inception_resnet_v2", "vgg_16", "vgg_19", "mobilenet_v1_025", "mobilenet_v1_050", "mobilenet_v1_100", "alexnet", "densenet121_k32", "densenet169_k32", "densenet161_k48" and "nasnet_larget". A total of 18 ImageNet models have been built in so far.

How to evaluate my own model?

Models for MNIST, CIFAR and ImageNet datasets are defined in setup_mnist.py, setup_cifar.py and setup_imagenet.py. For MNIST and CIFAR, you tin modify the model definition in setup_mnist.py and setup_cifar.py directly. For ImageNet, a protobuf (.pb) model with frozen network parameters is expected, and new ImageNet models can exist added into setup_imagenet.py past calculation a new AddModel() entry, similar to other ImageNet models. Please read the comments on AddModel() in setup_imagenet.py for more details.

The following two links provide examples on how to prepare a frozen protobuf for ImageNet models:

Fix DenseNet models

Prepare AlexNet model

Known Problems

If you lot run into the following error:

              posix_ipc.ExistentialError: Shared memory with the specified name already exists                          

Please delete those residual files in /dev/shm

              rm -f /dev/shm/*all_inputs rm -f /dev/shm/*input_example rm -f /dev/shm/*randsphere rm -f /dev/shm/*calibration                          

For systemd based Linux distributions (for case, Ubuntu 16.04+), it is necessary to ready RemoveIPC=no in /etc/systemd/logind.conf and restart systemd-logind (sudo systemctl restart systemd-logind.service) to avoid systemd from removing shared memory objects later user logout (which prevents CLEVER running in background).

tayloryety1980.blogspot.com

Source: https://github.com/huanzhang12/CLEVER

Share this post