Reinforcement Learning Is All a Game for Kaiqing Zhang

Competing to win—whether it’s a video or board game—requires the player to be strategic and see the bigger picture. They need to understand the other players’ positions and strengths in order to be successful.

This is how Kaiqing Zhang—a self-described games enthusiast—approaches his research, which lies at the intersection of machine learning, reinforcement learning, game theory, and control theory.

“Specifically, I am interested in problems in machine learning when there are multiple learners interacting with each other, and in building theoretical foundations for it,” says Zhang, an assistant professor of electrical and computer engineering.

Zhang offers up self-driving cars as an example. In this kind of emerging technology, there are multiple components—known as agents—making decisions simultaneously, and each agent has its own objective. These goals include arriving at the destination as soon as possible, while also attempting to avoid congestion and collision.

Zhang says this is a typical scenario in the area of game theory, and his focus is on the “learning” part of it—when these agents are machine learners, and what should they do.

One of his current projects is considering multi-agent dynamic decision-making scenarios where each agent may make myopic decisions, and then looking at their collective behavior when they are all put together.

His work has applications in intelligent and distributed cyber-physical systems, including robotics, smart grid and transportation systems.

As a new affiliate faculty member of the University of Maryland Institute for Advanced Computer Studies (UMIACS), Zhang says he is looking forward to taking advantage of UMIACS’ “impressive resources,” especially because reinforcement learning (RL) requires heavy computation.

“UMIACS attracted me with its unique interdisciplinary environment and rich computational resources, which are both important for collaboration and for conducting top-notch machine learning research,” says Zhang, who also holds an appointment in the Institute for Systems Research and is a member of the Maryland Robotics Center.

Additionally, Zhang recently joined the University of Maryland Center for Machine Learning, where he hopes to work with other “brilliant researchers” to advance the foundations of RL, robotics and machine learning in general.

Zhang first became interested in RL after hearing about AlphaGo, a computer program that plays against humans in the classic boardgame Go, which he played as a child. Go originated in China more than 4,000 years ago and is thought to be the oldest board game still played to this day. Developed by the London-based DeepMind Technologies, the program AlphaGo beat human competitors at the Go Championships in 2015 and 2017.

“The key algorithm behind AlphaGo’s success was reinforcement learning,” says Zhang, who was inspired to pursue his dissertation in multi-agent and reinforcement learning after AlphaGo’s success.

Zhang’s thesis, “Reinforcement Learning for Multi-Agent and Robust Control Systems: Towards Large-scale and Reliable Autonomy,” would go on to receive a thesis award from the Coordinated Science Lab at the University of Illinois at Urbana-Champaign (UIUC) in 2022. He earned his doctorate in electrical and computer engineering from UIUC in 2021.

Before joining UMD, Zhang was a postdoctoral scholar at the Massachusetts Institute of Technology in the Laboratory for Information and Decision Systems and the Computer Science & Artificial Intelligence Laboratory.

He was also a research fellow at the Simons Institute at University of California, Berkeley. Zhang’s postdoc work on differentiable simulators—as his first attempt toward research on robotics, an important application domain of reinforcement learning and control—won the Outstanding Paper Award at the International Conference on Machine Learning (ICML) in 2022, one of the two most impactful and leading machine learning conferences.

He is the recipient of several awards and fellowships, including the Hong, McCully, and Allen Fellowship, the Simons-Berkeley Research Fellowship, and the YEE Fellowship Award.

—Story by Melissa Brachfeld, UMIACS communications group