Advertisement

Inside Amazon's surveillance-powered no-checkout convenience store

Inside Amazon's surveillance-powered no-checkout convenience store
From TechCrunch - January 21, 2018

By now many have heard of Amazons most audacious attempt to shake up the retail world, the cashless, cashierless Go store. Walk in, grab what you want, and walk out. I got a chance to do just that recently, as well as pick the brain of one of its chief architects.

My intention going in was to try to shoplift something and catch these complacent Amazon types napping. But it became clear when I went in that this wasnt going to be an option. I was never more than a foot or two from an Amazon PR rep, and as Dilip Kumar, the projects VP of Technology, convinced me, theyd already provided against such crude attacks on their system.

As you might have seen in the promo video, you enter the store (heretofore accessible to Amazon employees only) through a gate that opens when you scan a QR code generated by the Amazon Go app on your phone. At this moment (well, actually the moment you entered or perhaps even before) your account is associated with your physical presence and cameras begin tracking your every move.

The many, many cameras.

I wondered when the idea of Amazons cashierless store was first proposed how it would be accomplished. Cameras on the ceiling, behind the display cases, on pedestals? What kind? Proximity and weight sensors, face recognition? Where would this all be collated and processed?

Amazons approach wasnt as complex as I expected, or rather not in the way I expected. Mainly the system is made up of dozens and dozens of camera units mounted to the ceiling, covering and recovering every square inch of the store from multiple angles. Id guess there are maybe a hundred or so in the store I visited, which was about the size of an ordinary bodega or gas station mart.

These are ordinary RGB cameras, custom made with boards in the enclosure to do some basic grunt computer vision work, presumably things like motion detection, basic object identification, and so on.

Theyre augmented by separate depth-sensing cameras (using a time-of-flight technique, or so I understood from Kumar) that blend into the background like the rest, all matte black.

The images captured from these cameras are sent to a central processing unit (for lack of a better term, not knowing exactly what it is), which does the real work of quickly and accurately identifying different people in the store and objects being picked up or held. Picking something up adds it to your virtual shopping cart, and you can pop it in a tote or shopping bag as fast as you like. No need to hold it up for the system to see.

This is where the secret sauce is, Kumar told me, and I believe him. As banal a problem as it may seem to determine which similarly dressed person picked up which nearly identical yogurt cup, its very difficult to get right at the speed and accuracy level needed in order to base an entire business on it.

A student, after all, with the resources available these days, could probably design a version of this store in a few weeks that would work 80 percent of the time. But to get it right 99.9 percent of the time, frictionlessly and instantly, is a challenge that requires a great deal of work.

Notably, there is no facial recognition used (I asked). Amazon perhaps sensed early on that this would earn them rebuke from privacy-conscious shoppers, though the idea of those people coming to this store strikes me as unlikely. Instead, the system uses other visual cues and watches for continuity between camerasyoure never not in sight of a lens, so its easy for the system to see a shopper move from one camera to another and make the connection.

Should there be a technical problem with a camera or it gets sauce on its lens somehow, the system doesnt break down entirely. Its been tested with cameras missing, though naturally it wouldnt be long before a replacement is put in place and the system re-re-calibrates.

Advertisement

Continue reading at TechCrunch »