While Artificial Intelligence (AI) has tremendous potential as a defense against real-world cybersecurity threats, understanding the capabilities and robustness of AI remains a fundamental challenge, especially in adversarial environments. In this talk, I address two interrelated problems that are essential to the successful deployment of AI in security settings. (1) Discovering real-world vulnerabilities of deep neural networks and countermeasures to mitigate threats. I will present ShapeShifter, the first targeted physical adversarial attack that fools state-of-the-art object detectors, and SHIELD, a real-time defense that removes adversarial noise by stochastic data compression. (2) Developing theoretically-principled methods for choosing machine models to defend against unknown future attacks. I will introduce a novel game theory concept called “diversified strategy” to help make the optimal decision with limited risk. Finally, I will share my vision on making AI more robust under different threat models, and research directions on deploying AI in security-critical and high-stakes problems.