AI, and owls, will be watching on us as we shop

Roberto Saracco November 21, 2017 Blog 386 Views

A shopper in the Poly demo room picks up a can from the shelf and his movements are tracked by owls cameras and understood by AI. Credit: Poly

Amazon Go, the automated shop that started operation in Seattle in 2017, is suffering from several glitches that have put a stop to the plan of extending it to several location. The goal of killing the queues at the check out point by having the shop recognising what you have taken is quite elusive and pretty difficult to reach with a 100% reliability. The problem is that in this area you either achieve a 100% reliability or you cannot run it. 99% is simply not good enough.

Merchants like BingoBox, in China, have taken the approach of having the customer scanning the bar code attached to the product as she places it into the cart. That works, of course, but it does not provide a seamless experience as the one pursued by Amazon Go, even though it is a tad better than scanning the product bar code at a self checking stand on the way out.

A US start up -Poly-, cofounded by an Italian -Alberto Rizzoli -, is using advanced technology trying to come up with a 100% reliable solution. They are using Artificial Intelligence, AI, to analyse the images captured by video cameras all around a store. The AI software is able to distinguish each customer, detects when a customer picks up a product and places it in the cart (or in his pocket…) and also if he places it back on the shelves (even if it is placed on a wrong one). The software creates a sort of “transient” digital identity of the customer to follow her as she wanders in the shop. This transient digital identity can be associated to a credit card that the customer shows at the entrance of the shop so that she can just walk out of the store as she is done. No check out required. She will receive an itemised bill on the phone associated to her credit card.

The AI system is obviously able to identify the product chosen, although this is way more difficult than one can imagine. It takes hundreds, sometimes thousands of photos of the product in different light conditions, and from different perspectives, to have a certain recognition. The process is so complex and time consuming that Poly is working on using a robot to handle the product and take the required sequence of photos.

An “owl” developed by Poly to pick up images directly from the shelf

There is also the issue of “seeing” what the customer is picking up. Clearly if she places herself between the object and the camera there is no way to identify the object. This is were “owls” come in. They are special cameras placed directly on the shelf that can identify the hand picking up the object.

One may wonder if it wouldn’t be easier to tag each product with an RFID and just check it as it is placed in the cart (and may be once more on the way out). RFID tags are really cheap but photos are cheaper. The problem with RFID tags is that you have to place them manually on each product and this is labour intensive. Image recognition would be cheaper, eventually.

The challenges remain, because of that 100% reliability constraint, but AI is clearly promising as a solution.

However, that solution may not be acceptable by a few of us. When we are in a store we are monitored by the occasional clerk and at the check out point, no doubt about that. Yet, having cameras that are following us step by step, looking at our indecision about a product, seeing how we discuss the choice with our friend may be a bit too much.

Poly makes it a point to tell customers that all images are finalised to automating the check out and are discarded as we step out of the shop. Yet, uneasiness remains.

A second point is about potential job loss. An automated shop, in principle, may operate with no human staff at all (or very very limited one). Robots may take care of refilling the shelves as they run low on products (something the image recognition applications will be prompt in detecting) and no-one is needed at the check out.

Poly likes to tell a different story. Rather than using personnel to man the check out points use that same personnel to provide information to customers as the shop. I can say that would be a good idea, since I have had to search for some assistance and having a hard time finding one in many large store. At the same time I understand that the store keepers have made a conscious decision to decrease the clerks in the store to cut cost and I wouldn’t be surprised if they would keep decreasing personnel as more advanced technology will make it possible.