The Digital Red Queen project used AI to evolve competing programs in the Core War game via continuous self-play. This created increasingly resilient code & offers a sandbox to study adversarial AI.

Any online statistical model AI can be beaten by making the same request repeatedly until it just gives you whatever you want, it's absolutely trash for that purpose.

Using a machine learning algorithm to find methods to attack in closed environment might yield some benefits, though.