In 2016, engineers at OpenAI spent months teaching artificial intelligence systems to play video games. Or, to be more precise, they spent months watching their AI agents learn to play video games. This was back in the days before artificial intelligence was a subject of nonstop hype and anxiety. OpenAI had been founded by Elon Musk, Sam Altman, and other tech savants just a year before and still operated more like a think tank than like the tech colossus it was to become.
The researchers were training their system on a video game called CoastRunners, in which a player controls a motorboat that races other boats around a track and picks up extra points as it hits targets along the route. The OpenAI team was using an approach called reinforcement learning, or RL. Instead of providing the agent with a full set of instructions, as one would in a traditional computer program, the researchers allowed it to figure out the game through trial and error. The RL agent was given a single overarching incentive, or a “reward function” in AI parlance: to rack up as many points as possible. So any time it stumbled on moves that generated points, it would then strive to replicate those winning moves. The researchers assumed that, as the agent bumbled around the track, it would begin learning strategies that would ultimately help it zoom expertly to the finish line.
That’s not what happened. Instead, as the RL agent steered its boat chaotically around the track, it eventually found a sheltered lagoon containing three targets. Soon the agent began piloting the boat in an endless loop around the lagoon, bouncing off bulkheads and other vessels and smashing the targets again and again, generating points galore. It turns out the CoastRunners game doesn’t require the player to cross the finish line to win, so the RL agent didn’t bother with that nicety. In a report titled “Faulty Reward Functions in the Wild,” the researchers wrote, “Despite repeatedly catching on fire, crashing into other boats, and going the wrong way on the track, our agent manages to achieve a higher score using this strategy than is possible by completing the course in the normal way.” In fact, through its out-of-the-box strategy of not trying to win the race, the AI system outscored human players by 20 percent.
To read more, click here.