Amazon's cashierless store: applying artificial intelligence to everyday life

In the fall of 2015, an Amazon executive invited Jeff Bezos to assess their work—a top-secret project to overhaul grocery stores. They rented a warehouse south of Seattle and converted part of the ground floor into a 15,000-square-foot mock supermarket, complete with plywood walls, shelves and turnstiles designed to mimic the technology that could scan shoppers’ smartphones as they walked by.

The Amazon CEO and several assistants pretended to shop, pushing grocery carts down aisles filled with canned goods and plastic fruits and vegetables. Inside, there were also dedicated counters where Amazon employees posing as baristas, butchers and cheesemongers took orders and added items to Bezos' imaginary bill.

Then, according to a person who was there, Bezos gathered the project leads and told them that, while they were all doing great work, the experience was disjointed. It was understandable that customers would have to wait for meat, seafood, and fruit to be weighed and added to their bills, but the store’s main selling point was the time-saving checkout process. Bezos asked the team to focus on getting rid of the checkout process and cash registers. “It was an Amazon thing,” another employee recalled ruefully. “We love it, let’s change everything!”

Almost four years later, 14 Amazon Go stores have opened in Chicago, New York, San Francisco, and Seattle. They’re about a quarter the size of the original model and are located in downtown office districts, offering a small selection of sandwiches, meal kits, and convenience store items like soda, jam, and chips. As Bezos had hoped, there are no cash registers. Customers open a special app and scan a collection screen at the entrance, they walk out the door with their items, and Amazon magically charges their credit cards. By all accounts, the company intends to open more of these stores in the coming months and years.

A technological miracle

From a technological perspective, the Go stores are a marvel, succinct demonstrations of Amazon’s ability to devote vast resources to applying the latest advances in artificial intelligence to everyday problems. They also illustrate the company’s propensity for pursuing technology for technology’s sake (see the Fire Phone). Because these stores offer all the options of a 7-Eleven, but with far more complexity and cost. Cameras suspended from the ceiling at various angles track shoppers as they wander the aisles, while scales embedded in the shelves count products down to the gram to determine which items a shopper has picked up. Behind the scenes, that determination is made by sophisticated image-recognition algorithms, while Amazon workers can review surveillance in their offices to ensure shoppers are being charged accurately. Each store also has a local worker on hand to help people download the Go app, restock shelves, and check IDs where alcohol is sold.

Was it all worth it? Aside from lunchtime, it seems like some Go stores have been all but forgotten. Employees familiar with Amazon’s internal projections said the company had to resort to raffles, giving away tote bags and other branded merchandise, because the Chicago location fell short of expectations. Yet as the project’s turbulent history suggests, the Go stores are less a sudden impulse at the company and more of an ongoing experiment. Of course, the potential prizes are huge, too, given that the grocery industry is worth $12 trillion. Amazon, with its unlimited resources and appetite for risk, may be in the best position to do so.

Analysts and investors have asked Bezos for years whether Amazon would open stores. His response is usually, “We would, but only if we had a really differentiated idea.” As he told an interviewer in 2012, “One thing we don’t do well at Amazon is offer products ourselves.”

That summer, Bezos began to think seriously about the opportunities offered by physical retail, which accounts for 90% of all retail sales in the United States, according to the Census Bureau. He could see that for Amazon to grow in size, it was imperative that it enter new industries. (The development of the Alexa voice assistant and the creation of the Amazon Studios division took place around the same time.) To lead the effort, Bezos hired senior vice president Steve Kessel, who had overseen the development of Amazon’s Kindle and dragged the publishing industry into the digital book age.

Kessel enlisted the help of Gianna Puerini, who had overseen Amazon’s homepage and product recommendations division, to lead the product’s development. Puerini, who had previously overseen Amazon’s homepage and product recommendations division, retired and was flipping houses in the Seattle area. So Puerini (who retired again earlier this year) set up the development division in a nondescript six-story building in the Southlake neighborhood, a few blocks from Amazon’s headquarters. Because the project was secret even from other Amazon employees, one of her first tasks was to choose a code name so boring that no one would pay attention to it, a former colleague said. For the next few years, the team would use the name IHM, for “Inventory Health Management.”

To oversee engineering, Kessel recruited Dilip Kumar, Bezos's shadow or technical advisor. Kumar occasionally performed stand-up comedy at local open-mic nights, but colleagues at work described him as extremely passionate and combative.

Annoying checkout process

IHM employees say the first few months were filled with open-ended brainstorming and debate. They considered whether they should go with a Macy's-style department store, a Walmart-style supercenter or even an electronics store. One scrapped idea involved a two-story store, with Amazon's disc-shaped warehouse robots assembling orders on the top floor, and then conveyor belts and robots delivering them to customers' waiting vehicles below.

A few months later, Kumar, Puerini, and their colleagues acknowledged that most stores in the real world already worked pretty well, with one notable exception: supermarkets, which have a tedious checkout process. The average American buys groceries twice a week, and the experience of waiting at the checkout line creates the most inefficiency in offline shopping. "We realized that there were a lot of benefits to shopping in a physical store, but waiting in line was not one of them," Kumar said.

Many companies have tried to solve this hassle. Apple stores have dedicated employees holding tablets that can read credit cards, while China's Bingo Box uses RFID chips attached to product packaging for self-checkout. The IHM team wanted to eliminate the bottleneck entirely, and Amazon's tradition meant making sure the team worked backwards from customer needs, so they began issuing press releases or "PR releases" at Amazon announcing the opening of stores without checkout processes. Then they started working on the actual technology to make the store a reality.

Kumar recruited computer vision and machine learning scientists from across Amazon without telling them what they were working on. He set deadline after deadline, using upcoming presentations to report to Bezos or Kessel. Engineers worked 70 to 80 hours a week, constantly answering emails, and even in their free time on nights and weekends, they needed to write Amazon's classic six-page documents, which outline proposals in narrative memos. "We were like living in a cave," said one employee.

Initially, the IHM team envisioned large stores of about 30,000 square feet, roughly the size of a suburban supermarket. But after a few months, the team decided that such a large market was too ambitious and cut the size of the proposed store in half.

Puerini's team created the first store mockups using children's building blocks, bookshelves, and other items found around the office. As the project neared its desired launch in mid-2015, the company remodeled a warehouse south of Seattle so it could show the mockups to Bezos. For its first brick-and-mortar store, it also anonymously leased the ground floor of a new luxury apartment building in Seattle's Capitol Hill neighborhood. Permits filed with the city included plans for large produce and dairy coolers, as well as an on-site kitchen for preparing fresh food.

But after Bezos’ visit, the project stopped moving in that direction and began focusing on a more streamlined experience, one that resembled Amazon’s one-click ordering, even if the stores lacked a farmers market or boutique butcher.

After Bezos’s demo, Kessel called a team meeting and announced their pivot to convenience stores. Some engineers breathed a sigh of relief that they could reduce complexity by eliminating items of varying weights, like produce and meat. Others left the project crestfallen, exhausted from the nonstop pace of work or frustrated by the narrowing vision. Over the next three years, the Capitol Hill storefront sat abandoned, the windows of the store in one of Seattle’s hottest neighborhoods mysteriously covered with brown wrapping paper.

Amazon Books goes online

Bezos and Kessel were growing impatient. So in March 2015, as Puerini and Kumar redesigned their concept, they formed a separate group under Kessel’s leadership to open cashierless bookstores. Books are a commodity concept, as opposed to food. They are consistently priced, easy to stock, and Amazon’s oldest product category. And because people tend to browse bookstores at a more leisurely pace, there’s no point in trying to replace cashiers with technology.

That fall, as the company prepared its first Amazon Books in an upscale Seattle mall, speculation about how the company would move into physical retail grew so feverish that a reporter used a pole with a camera to peek inside. Around the same time, Bezos snuck in the back door, got his first glimpse inside, and expressed delight. He said he felt as if Amazon’s business was in full swing.

For the longtime members of the IHM project, watching Amazon Books take off in a matter of months was dizzying. They had been working on it for three years, but their project didn’t even have an official name. In early 2016, Puerini’s team came up with the Go brand to convey the sense of speed that the cashierless stores offered. “Even though the word itself is only two words,” she said. “It means you can just pick up the item.”