It’s in the adorably anthropomorphic Google car, the one conspicuously maneuvering itself around Mountain View, CA. It’s in global auto supplier Delphi Automotive’s attention-grabbing Audi SUV, the car that recently drove itself cross-county. It was at the game-changing 2007 Urban Challenge, in the winning Tartan Racing car that successfully navigated itself through 55 miles of tough city conditions. “It” is Carnegie Mellon talent and technology, long making widespread strides in bringing autonomous driving to reality.
Most recently, Delphi acquired Ottomatika, the CMU spin-off that supplied the brains for its roaming Audi. Founded in 2013 to commercialize pioneering autonomous vehicle (AV) technology developed at the university, Ottomatika offers software and systems development for self-driving vehicles of all kinds. The company was founded by Raj Rajkumar, CMU’s George Westinghouse Professor of Electrical and Computer Engineering. Rajkumar is a Tartan Racing veteran as well as co-director of the CMU/General Motors team responsible for CMU’s own remarkable self-driving Cadillac SRX, as are a number of Ottomatika employees.
Delphi already provides automakers with components currently used in assistive safety technologies like collision mitigation and has since the late 1990’s, when it introduced radar systems for its adaptive cruise control. As its customers grow increasingly hungry for competitive technologies to stay ahead in the AV race, however, the giant supplier (sales last year topped $17 billion) turned to CMU expertise.
“The founding and purchase of Ottomatika validates Carnegie Mellon’s pioneering strengths in automation, robotics and software engineering,” says Rajkumar, describing his company as leveraging CMU’s ground-breaking expertise and long-term experience in vehicle automation. Delphi apparently agrees.
No doubt. CMU talent has been at the forefront of autonomous vehicle technology for more than 30 years.
“This is our future. … I’m so impressed to see what Carnegie Mellon has done in moving this forward.”
Driverless
A gallon of gas costs 89 cents, and The Oprah Winfrey Show is launching nationwide. Nintendo video games have just hit the U.S. market, and incredible things are quietly happening at Carnegie Mellon University. It’s a sunny day in 1986, and the lawn of Flagstaff Hill, a popular park near campus, is littered with students reading, chatting, and sunbathing. A ponderous, dark blue Chevy van rolls slowly along the narrow path twisting up the hill. The students barely glance at the massive vehicle lumbering noisily along beside them. After all, this is CMU. They’re used to seeing it here, with its strange sensors, oversized back, and pack of grad students crammed inside …. and they’re used to the amazing fact that it actually has no driver.
Though it moves slowly and stops hard, the van gets around. It also tools around in the city’s north suburbs, mingling with the neighbors. Within a few years, it’s running errands, like carting cold soda. One day, a police officer arrives to question the students and their advisor, who lives in the area. The young professor graciously offers to give the man a ride. The dubious officer declines but, before he leaves shaking his head, glances inside.
No wonder. Those inside aren’t so comfortable. Computers and monitors clutter one side of the hot, cramped vehicle, and switching gear and an extra generator line the other. Four passengers are working, perched on little stools in the aisle. “Painful,” quips one of them, postdoc Martial Hebert. “It’s surprising how much the quality of the software improves when the graduate students know they’ll be riding along,” their advisor, Charles Thorpe (CS’85), is known to say with a smile.
The lack of luxury can be forgiven. A driverless car like this is unprecedented. It’s the brainchild of this group from CMU’s fledgling Robotics Institute (RI), founded less than a decade before in 1979 and still small enough to hold its seminar in a conference room.
These scientists are working under a U.S. Department of Defense project on strategic computing, using “computers of the future” to solve real world problems. In this case, robot vehicles for the Army. At this point, very few researchers are even considering such an unimaginable task. “It was a really intense period,” Thorpe recalls today. “We felt like we were inventing the future.”
They were. Three decades later, the RI is the largest academic center of robotics in the world, described in terms that include “world-renowned,” “pacesetter,” and “the world’s best robotics research facility.” Society is teetering at the edge of a far-reaching transportation revolution.
Daily headlines trumpet the latest achievements in autonomous vehicle technology, eerily reminiscent of advances pioneered at CMU so long before. Automakers race to equip their cars with ever more intelligent and autonomous systems, including adaptive cruise control, parking assist, collision avoidance, lane maintenance, and more—things the RI team was “playing with 25 years ago.” Many have working autonomous vehicle (AV) prototypes.
The latest in CMU’s decades-long line of AV creations is a Cadillac SRX, developed by Ottomatika founder Rajkumar’s team in collaboration with General Motors. In 2013, the car very publicly ferried House Transportation and Infrastructure Committee Chairman Bill Shuster 33 miles from the Pittsburgh suburbs to the airport, with no human intervention. (When it’s operating autonomously, a human sits in the driver’s seat as a safety precaution—an easy job because the human rarely has to take over and the vehicle has never had a collision.)
A year later, Shuster invited congressional members to experience the marvel for themselves, as CMU demonstrated the car in Washington, D.C.
“Incredible, really amazing to see this technology,” said Shuster. “This is our future. … I’m so impressed to see what Carnegie Mellon has done in moving this forward.”
Other companies have thrown their hats in the AV ring. Tech giant Google, for example, has set loose a number of its weeble-like “pods” around Mountain View and declares it will deliver a fully autonomous car by 2020. Uber recently opened a research facility in Pittsburgh. It’s all the tip of the iceberg.
Road Trip
The social and economic impact of this transformation could be shocking. First and foremost, AV technology could save lives. It’s estimated that 1.2 million people worldwide die in automobile accidents each year, 33,000 in the United States alone. “Superhuman” AV technology would eliminate the driver errors—including distracted and drunk driving—that cause more than 90% of these accidents, as estimated by McKinsey & Company. Self-driving cars would spell freedom for those unable to drive, including the handicapped, elderly, and children. AVs could arrive when summoned and park themselves in centralized garages, reducing car ownership, traffic congestion, and emissions, while adding precious free time to the lives of commuters everywhere.
But robot car utopia may not be free or painless. Employment patterns could radically shift if professional drivers, auto manufacturing employees, and others like them displaced. Government and insurance companies face a tangled regulatory and liability web, not to mention the potential problems of privacy and security. Where and when should AVs drive with humans? Who pays when one strikes another? Who is at fault if a pedestrian is hit—the driver, automaker, or technology supplier?
Nobody knows—yet things are moving along at breakneck speed. Auto supplier Delphi recently trumpeted that its Audi, run by Ottomatika software, had driven itself cross-country on the longest and its “first coast-to-coast automated drive.” But two of the young Carnegie Mellon men who’d spent hours perched inside that Chevy van in the ’80s and early ’90s couldn’t help but laugh. They remembered a very different “first.”
This time it’s 1995. CMU research scientist Dean Pomerleau (CS’92) and his graduate student Todd Jochem (CS’96) are rolling down a Kansas highway. They’re halfway through a 2,849-mile cross-country trip dubbed “No Hands Across America,” the fruition of more than a decade of CMU research.
Jochem, taking his turn in the driver’s seat, checks his screen, and stretches. He and Pomerleau wave with both hands as a reporter pulls alongside to snap a picture of their self-driving wonder car. The reporter then speeds off to position himself ahead for another photo. As he fades out of sight, Pomerleau and Jochem pass a state trooper and watch with dismay as he pulls out behind them. Jochem waits nervously as he has the car change lanes. Thankfully, the trooper passes. Within minutes, they see him again—pulling over to ticket the reporter, who is already standing out on the shoulder to get his picture. Spotting the driverless car, the reporter heedlessly dashes past the cop, who is apparently obstructing the shot. The pair laugh and wave again as they head off to Jay Leno and Los Angeles.
Pomerleau and Jochem have outfitted a Pontiac Trans Sport minivan with a portable computer, video camera, and GPS receiver used to determine speed. With little funding for such a pie-in-the-sky trip, the vehicle and equipment are donated, and they’re literally selling T-shirts along the way. It’s the first long-distance autonomous trip undertaken. The system steers itself 98.7% of the time.
Within two years of this trip, the team had added autonomous braking and throttle and, unlike other AV projects of the day, the vehicle required no roadway modifications.
“That time period—’94–’97—was like an inflection point of the technology,” observes Jochem. “We went from driving 20 mph on Flagstaff Hill to driving 100 mph without a break on interstate highways.”
At the big 1997 government demonstration in California, CMU brought five AVs—two cars, two buses, and a minivan. “We’d go down this eight-mile stretch of highway,” Jochem recounts. “They would pass each other, communicate back and forth to avoid obstacles, change lanes, do all this stuff.”
“It was a really intense period, We felt like we were inventing the future.”
One of the buses carried the U.S. Secretary of Transportation. “The ride went perfectly,” recalls Thorpe. “At the end, the TV cameras were rolling and the Secretary gave a speech of congratulations.” And announced the end of the program due to budget cuts.
Law Abiding
Two decades later, autonomous vehicle activity has exploded. Why the wait? Why now? One big push came from the government, as it visibly renewed its interest with the 2002 announcement of the DARPA challenges. These events invited teams to compete in long-distance AV races. CMU’s Red Team made it the farthest in the first desert Grand Challenge. And its Tartan Racing team then won the 2007 Urban Challenge, driven on a complex, city course against teams that included Stanford, Virginia Tech, and MIT. It was a contest that according to the Boston Globe, “became a Woodstock for roboticists and other engineers.”
At the same time, technology advanced and drove down costs, making sensors more affordable and computing more powerful. Google began its own widely hyped initiative in 2008. The competition was on.
John Dolan (E’91), RI principal systems scientist, was a key player on that winning Tartan Racing team and the group currently responsible for the autonomous Cadillac. “We’re looking at a broader range of things now,” he points out. “Being able to make tactical decisions about what other traffic is doing and respond appropriately at a higher level of awareness.” So just how do robot vehicles manage that?
They use a combination of “superhuman” sensors, including cameras, radar, and lasers, along with mapping and powerful software. Every research group uses its own mix and match, debating over the appropriate forms of sensing, with advantages and disadvantages to each. Cameras (vision) can detect detail but have high computation needs. Radar can deal with rain and snow but can’t discern shape. Lidar (laser) can identify shape but gets confused with dust and rain and is still quite expensive.
The CMU car utilizes embedded automotive-grade sensors and hidden computers to control steering, speed, and braking, as well avoid obstacles like cones and pedestrians.
“Our goal is to create a fully autonomous vehicle in as many contexts as possible,” Dolan explains. “And unlike the Google car, with its expensive laser sensor on top, ours looks like an actual vehicle.”
The team has the car driving in the suburbs, where it communicates with retrofitted traffic lights. To test its systems in the intricate urban environment, the team navigates a loop near the Pittsburgh campus, chock full of traffic and “unpredictable pedestrians,” as Dolan says. “We’re trying to make the vehicle production-relevant.”
In shades of the past, the car had a recent brush with the law. As it made a routine turn, a policewoman pulled over the safety “driver.” As he sat anxiously, she approached. Noticing nothing out of the ordinary, she merely informed him that he couldn’t turn from the middle lane. But the car was apparently right, according a road sign, which he pointed out. With the misunderstanding cleared up, he was on his way.
Traffic Conditions
As the technology rapidly improves, no one really knows how things will progress. Although Google boldly promises a finished product in 2020, the automakers are largely taking the opposite approach. “The automakers are very cautious on one hand, while Google, fairly blue-sky about it, just doesn’t have that whole culture of reliability testing,” says Dolan. “They’re going to have to balance out.”
And even with Google pushing the envelope, automakers will only produce what the public is ready to adopt. At this point, only four states and Washington, D.C., have passed clear legislation allowing automated vehicles. Whether the public has realized it, however, incremental acceptance is already in full swing. Many, including Rajkumar, Hebert, Thorpe, Jochem, and Dolan, agree that autonomous highway driving could well be a reality in five years.
“The technology is already out there on the cars,” says Jochem, now an entrepreneur and consultant. “There are pieces of self-driving steering. There’s definitely throttle and brake. All that needs to happen is to put those together and turn ’em on all at the same time. Call it autonomous cruise.”
Are we ready?
“This technology and the industry have moved much faster than the regulatory standpoint,” warns Hebert, who has moved from the back of that blue Chevy van to RI director. “This is a really big deal, actually. This is a case, like social media, where technology is plowing on and the regulatory, legal framework is not in place at all.”
And there are other complex issues that will likely be fully explored only when the transition is well under way, like the safe handoff from human to machine and back again. Astro Teller, a 1998 CMU alumnus and head of Google X, commented that Google initially removed its steering wheel because humans aren’t a “reliable backup.” California law said no way. But as automakers advance incrementally, will their cars be fully ready as humans stop paying attention? Says Thorpe, “Some say we’re gonna sneak up on this. First they’ll automate the braking, then the steering. But as soon as you’ve done that, people will fall asleep or be facing backwards.”
Then there’s the unavoidable “twilight zone,” as Dolan terms it, while humans and automated vehicles learn to co-exist. Experts envision a day when all vehicles will communicate with one another, when, for example, piling up behind red lights will be long forgotten as cars efficiently take their turns. With more than 250 million cars on American roads, however, it could take decades before they’re all turned over. In the meantime, Dolan’s team is studying socially acceptable autonomous driving. How, for instance, can an AV recognize a driver waving it onward or slowing down as a cue to enter traffic?
The complex and unpredictable urban environment makes the crystal ball even foggier, and many see a much longer timeline for AV urban driving, despite Google’s proclamations.
“My gut says 15 to 20 years out because it’s so complicated,” says Jochem. “It’s not that you don’t know where the intersections are or if the light is green. You don’t know who’s going to pop out from behind a car or if that guy is suddenly going to stop because a bicycle cuts him off.”
“Until we’ve actually categorized those problems and dealt with them one by one, it’s hard to know,” agrees Dolan. “It’s a big problem that really has not been solved.”
Perhaps. Though with Carnegie Mellon ingenuity fueling innovation in so many places—from incredible university research to the work of talented alumni to spin-offs like Ottomatika—it’s better to say, “not yet.”
