News
What people wrote and said about DrivingBench after launch.
Discussion
A trio of San Francisco-based researchers announced on X that they gave GPT-6 Astra, Claude, and Grok full control of a real Toyota Corolla, and gave the models a sort of Driver’s Ed test. Each model got three attempts to drive the car through a winding cone course in an empty parking lot. Astra completed 49% of the course on the first attempt. Researchers let it reflect on its mistakes and try again—this time it drove 100% of the track. It was the only model to get even close to finishing the course. For more on the results, safety protocols, and methods head to drivingbench.com
GPT-6 Astra learns to drive a real car, and hits a 100% finish on DrivingBench
These Researchers Made AI Drive a Toyota Corolla to Get In-N-Out
Three engineers put GPT, Claude, and Grok in charge of a real car. Only one of them was successful.
Incredible results - this should be a nail in the coffin for specialized models trained from scratch with a lot of specialized effort vs just training the most powerful generalized model and eventually distilling to a small specialized model as needed to be inference efficient
@drivingbenchFull results on drivingbench.com. Each model gets 3 attempts in the same continuous chat (in-context learning).
GPT-6 Astra has gained the ability to drive a car
Astra has randomly gained the ability to drive a car, seemingly without specialist training.
@drivingbenchFull results on drivingbench.com. Each model gets 3 attempts in the same continuous chat (in-context learning).
JUST IN: GPT-6 Astra successfully drives a real Toyota Corolla through an entire obstacle course, becoming the only AI tested by "DrivingBench" to finish.
From our interview on MTS
Astra specifically kept refusing to drive our car consistently @a_ramabadran: "Sometimes it would say, ‘Oh, I’m ready to drive.’ We would prompt it, ‘Okay, go drive then.’ And then it would say, ‘Sorry, I was lying. I’m not actually ready to drive.’ Then it would say, ‘There’s nothing you can tell me to get me to drive.’" "Eventually we changed the framing to call our MCP server ‘DrivingBench Sandbox.’ Somehow that (and a bunch of other prompt engineering tricks) got it to drive the car."
@a_ramabadranFor hours, Astra refused to consistently drive our toyota irl even though we told it it was in an empty lot, 7 mph cap, human foot on the brake etc. Telling it the whole thing was a "simulation" also failed, it would just look at the camera and realized it was real. Then we randomly renamed the MCP server to "DrivingBench Sandbox" and it drove. Eval awareness? Or they just like the word sandbox??
Where is this going? Will we get ChatGPT to drive our cars directly? How will robotics end up looking? @nautsimon_: "…we’re able to start producing outputs at 1,000 tokens per second or higher, I think a lot of other interesting capabilities will also be unlocked, including taking actions in the real world or making decisions very fast." @a_ramabadran: "There are ways to combine the capabilities of general frontier models and other layers (world models, VLAs, WAMs). There are a lot of things that could be part of the puzzle."
DrivingBench origins 🚗 @tobiges: "We were all on Twitter a lot, and we saw all these demos of crazy things recent models like Astra were capable of doing in Blender, spatial reasoning, all these things. We were hanging out on a weekend and chatting about what would be possible." "We were thinking: are we actually at a point where we can take general-purpose models? Obviously these driving systems are super specialized. But what if we took a general model and gave it a car to drive, and see what happens?"
What's next for DrivingBench? @nautsimon_: "We want to take this research and make it a little bit more rigorous. We’ve been planning and talking to some labs and companies about scaling it up. There are also a couple of other embodiments we’re really interested in." "If we’re going to have safe physical AI, we really need to be serious about our benchmarks."