Amazon's cashierless store: applying artificial intelligence to everyday life

To continue developing the technology, Kumar’s engineers set up a top-secret lab store, called Otter, on the ground floor of the team’s new building on the corner of Fifth Avenue and Bell Street in downtown Seattle. The Otter lab is accessible only through a pair of locked doors. At first, the shelves were stocked with fake food made of clay and Styrofoam; shredded green construction paper stood in for lettuce. Employees were frequently asked to visit and try to trick the technology. They wore heavy coats, walked with canes, or pushed wheelchairs. They put items back in the wrong places, generating an automatic “messy item” alert that directed clerks to restock the items on the correct shelf. On one day, everyone was asked to bring umbrellas to see if they would block the camera’s view; in another, employees all wore Seattle Seahawks jerseys to confuse the algorithm that distinguishes shoppers based on the color of their clothing.

When the fake food was eventually replaced with the real thing, employees were asked to do the shopping, but in specific situations: For example, Puerini recalls, “You’re at a meeting and need to buy a salad and a drink for lunch,” or “You’re rushing to pick up your kids from daycare and need to quickly buy milk, strawberries, and cereal for tomorrow’s breakfast.” Another day, parents were asked to bring their young children, who tend to be fidgety and will run around and grab things, further stress-testing the system. To augment these real-life experiments, the company also developed digital simulations of stores and populated them with computer-generated shoppers.

The complexity and cost of causing employees to question

Kumar's engineers were trying to solve one of the thorniest problems in retail history: how to figure out what people bought, item by item, at checkout. After years of trying, the team concluded that visual identification of products with overhead cameras alone was impossible. Variations in lighting conditions throughout the day, the depth at which products were placed on shelves, hands and bodies obscuring custom product stickers, or an out-of-control toddler could easily confuse the system.

Eventually they decided to add scales and put more cameras inside the racks. (“The weight provided an additional signal we could use, but most of the heavy lifting was done by the cameras and vision algorithms,” Kumar says.) Amazon then combined the data to figure out who bought what.

But humans still need to oversee those judgments. When the system is unsure about a purchase, an independent team steps in to review it, a so-called low-confidence event. The creation of these groups led some employees to question the entire effort. It became “very tricky,” said one former participant. “If we had a whole group of people watching the monitoring, would scaling be possible?” (Amazon says human intervention is rare.)

People also had another role to play: They had to develop meal kit recipes and prepare daily lunches (lamb sandwiches, chicken sausages and salads, etc.). To prepare for the opening of a scaled-down prototype store at Amazon’s new downtown Seattle campus in late 2016, the company hired chefs and staff from restaurant chains. It opened commercial-grade test kitchens both at the prototype store and near an old warehouse in South Seattle. Uncharacteristically, Amazon went on a spending spree. It bought German commercial ovens, which cost tens of thousands of dollars each. When it smelled something funny in the pilot kitchen, Amazon even splashed out on a professional sniffer to solve the mystery. (The culprit was pickled daikon.)

Those kitchens, and Amazon’s penchant for rigorous and sometimes inhumane efficiency in its operations, presented another set of unexpected challenges. Because food safety was a top priority, it was so cold in the commercial kitchens that Amazon initially resisted workers’ requests to stand on mats on the facility’s frozen concrete floors during their shifts, one employee recalled. After a senior manager at headquarters spent a day observing the kitchens’ operations, the company issued hoodies and other cold-weather gear to kitchen staff. It turns out that the people involved in the service industry proved as difficult to manage as Kumar’s algorithms.

The original Go store opened to Amazon employees in December 2016, but the public opening, scheduled for early 2017, was delayed another 12 months. When 20 or more shoppers were in the store at once, the system was forced to freeze. It lost track of products as shoppers picked them up and placed them on different shelves. Shoppers themselves got confused, too. “We noticed many customers hesitated at the exit, asking employees if they could really leave,” Puerini says. “During testing, we put up a large poster that said ‘Yes, really, you can walk out!’ That poster remains in place.”

Amazon also tinkered with food prep. It began relying less on its own kitchens and buying more food from outside suppliers, including Taylor Farms, which makes salads, sandwiches, and more for Starbucks and 7-Eleven. Those pricey German ovens apparently still sit unused in the prototype stores.

Early on, the Go team envisioned opening thousands of stores in every major urban area. “We wanted to be able to put stores on every corner,” says one former executive. “We wanted to be as ubiquitous as Starbucks.” But now, seven years into the project, Amazon has just opened its 14th store, at the Embarcadero Center in downtown San Francisco. The company has also dramatically slowed down openings of Amazon Books and launched Amazon 4-Star Stores, another new format that includes a curated selection of merchandise and Amazon gadgets. These physical retail experiments have barely impacted the company’s financial results, exactly as Bezos envisioned them in 2012. It’s easy to imagine the CEO constantly cutting bait: He’s talked in the past about trying out concepts seven years ago before they paid off financially.

Most expensive R&D project

At their current pace, Go stores will far exceed that before they recoup their investment. People familiar with the project estimate that Amazon has spent hundreds of millions of dollars, including $2 million to $3 million on pilot stores. One former employee claimed it was one of the most expensive R&D projects in the company’s history, though Kumar disputed that, saying the stores use off-the-shelf hardware and Amazon’s existing cloud computing infrastructure. Still, it’s more expensive to operate than a 7-Eleven, given the dense layout of cameras and sensors and the technical support staff on call at all hours of the week, and, beyond the Slurpee machines, there’s little custom technology.

In traditional Amazon fashion, Kumar claimed that it’s “still early” for the Go project, noting that “customers like the experience of not having to stop to pay.” Analysts largely agreed, comparing the experience to going through TSA Precheck at an airport: Once you get used to it, you don’t want to go back. This gives the project “a lot of freedom to try other types of things,” Kumar said.

It’s those other things that deceive investors and company watchers. Take Amazon’s history of failures, like its early auction business, which led to the successful introduction of third-party sellers, and the Fire Phone, many of whose failures Amazon engineers later applied to Alexa. “Like so many things Amazon does, I’m sure it doesn’t see Go as a convenience store, it doesn’t see Go as a bookstore, it sees it as a data experiment,” said Neil Stern, a senior partner at retail consultant McMillanDoolittle. “The store itself is not their goal.”

Kumar himself was cautious about future plans, but he noted that the Go technology could be adapted beyond convenience stores. “If it makes sense for something else, then we’ll do it,” he said.

Meanwhile, Amazon’s commitment to physical retail appears to be expanding. For the past few years, Kessel has overseen Prime Now and AmazonFresh, the company’s fast-delivery and fresh-food businesses. When Bezos acquired the Whole Foods Market franchise in the summer of 2017, Kessel also took charge of about 500 Whole Foods stores and thousands of traditional checkout lanes that require the old-fashioned act of waiting to pay.

Then there was the mid-sized grocery store that Bezos unveiled on Capitol Hill in the fall of 2015. Earlier this year, Amazon quietly presented new plans to the city of Seattle, reviving the idle store in an effort to allay community concerns. Plans for an on-site kitchen were dropped, and "light-speed checkout lanes" were added to the blueprints. While the stores, which span more than 10,000 square feet, are much larger than the traditional Go format, the new concept is still tightly packed.

Still, if you stand on the sidewalk and squint through a gap in the frosted glass, you can see shelves similar to those at an Amazon Go store.