An interesting demonstration of the potential offered by adversarial attacks conducted in black-box mode is the one described in the paper Practical Black-Box Attacks against Machine Learning (arXiv: 1602.02697v4), in which the possibility of carrying out an attack against remotely hosted DNNs is demonstrated, without the attacker being aware of the configuration characteristics of the target NN.

In these cases, the only information available to the attacker is that of the output returned by the neural network based on the type of input provided by the attacker. In practice, the attacker observes the classification labels returned by the DNN in relation to the attacking inputs. And it is here that an attack strategy becomes interesting. A local substitute model is, in fact, trained in place of the remotely hosted NN, using inputs synthetically...