Robotic Order Preparation

Source code:

Use Case

A lot of processes are happening in an online grocery store. Some of them could be partly or even fully automatized. The focus of our OMiRob Case is on the automatization of the order preparation processes in an online grocery store’s warehouse using robotic workers. Therefore, it comprises parts of the ordering process and the order preparation process of an online grocery store.

We defined the following 3 correlated Use Cases:

Use Case 1:

First, a modelling method needs to be designed and developed. With that modelling method it should be possible to model the construction of a warehouse, i.e. where are the shelves, where are the ways between them and which products are stored in which shelf. Also, it should be possible to model doors, like the door between a freezing room and a “normal” storage room.

In order to make the warehouse plan machine processable, it needs to be exported into a XML-File.This XML Export will then be imported into our system and transformed into the system intern graph-like structure.

The dijkstra Algorithm is used to calculate the shortest Paths between each Orderitems.

Use Case 2:

As the next step, the robot needs to be steered; i.e. to let him walk (right, left, forward, backward) or turn around.

When these foundations are working fine, Use Case 1 and that part that was just described previously can be combined. Therefore, the route needs to be transformed into instructions for the robot. In other words, the robot should be able to walk from A to B by using the XML file from the pre-modeled plan of the warehouse and the route that was calculated by using all that data. So, the robot should walk from the starting point to a pre-defined goal by walking along the shortest calculated path to that destination.

Once that is working as expected, receiving the order as an input is the next open issue. The order will be described as a json and sent via REST request.


As we are talking about groceries, the chronology of taking the products may also be an important issue. So, all in all the order of the products should firstly be by the kind of the products – like if they need to be frozen, refrigerated or none of them. Logically, the products which neither need to be cooled nor frozen, should be the first category the robot is taking. After that category, the products which need to be cooled should be taken and after that products, all products which need to be frozen should be collected by the robotic worker. In each of these categories there needs to be further sorting. If some additional parameters are denoted, these need to be considered first. If not, the sorting will be simply done by weight. An ontology that saves all that information related to each product of the inventory will be created.

Use Case 3:

As third step, the robot needs to be able to recognize the products of the warehouse inventory in order to identify products from the shopping list he processes. The products are abstracted with QR-Codes.

Experiment

Modelling Method:

The foundation of the movement is a model of the warehouse, where the robot will operate. To fulfill this connection we decided to develop a modeling method in order to create models of the warehouses.

The modeling method was created and developed with ADOxx Development Toolkit. The method is based on an empty sample library, which was provided by the University of Vienna at the course Metamodelling. As the first and main step we brainstormed about the necessary classes and relations between. Afterwards all the classes were added and we additionally defined some needed attributes.


The classes are:

  • PackStation: The PackStation is the start- and endpoint of the robot. Therefore only one PackStation at time can be used in a model.
  • Shelf: Inside each Shelf is a product stored. The product as well as the corresponding productid needs to be defined inside the details of the Shelf. Every Shelf is connected to at least one Hub.
  • Hub: The Hubs are the connections between the other classes. They implies a change in direction, a door or a connected Shelf.
  • Door: The Doors are a kind of obstacle. A door can occur at a cooled area inside the warehouse or the freezer. We assumed that these doors are opening automatically if the robot is coming closer. Therefore the robot stops for a second to wait till the door is open.

The relation between the classes is Path. The path has an attribute called Length where the distance needs to be entered. The length is measured in meter and allows decimals.
To ensure the correctness of the model we also implemented some AdoScripts. After an insertion of an instance of PackStation, the script checks whether there is already one included. If this is the case, the new Instance is deleted and the User is informed about the correct usage. The second check is, whether the used product-ID is already in use as that IDs must be unique. In case of an already existing one, it is overwritten with 0 and the user needs to change the ID to a unique one.

Using this modelling method in the ADOxx Modelling Toolkit, a model of type WarehouseArchitecture has to be created in order to model the warehouse.

There are some special points which should be taken into account during creation:

  • There can only be one PackStation

  • The alignment of the Shelves and Doors need to be correct to navigate properly
  • The length of the paths need to be corresponding to the real warehouse

The finished model should then be exported as XML. Below, an example model can be seen:

Product Ontology

An ontology that saves all necessary information related to each product of the inventory of the online grocery store was created with Protege.

Three entities (Product, ProductGroup and CoolingStatus) were defined within the ontology. The main entity is Product. Each product belongs to a ProductGroup and has a CoolingStatus, which are both references to another entity. Additional data that is stored for each Product is: the unique id, the price, the weight, a description and how it is sold (i.e. per piece, per 500 grams, etc.). An example of an instance of class Product with its assertions can be seen below:


These additional informations are mainly used in two different parts of the project:

  1. To display the products appropriately within the webshop GUI.
  2. To sort the products according to their cooling status and their weight.


In order to make the ontology machine processable, it was exported as RDF/XML file. The RDF/XML file was imported and transformed into the internal data structure. This was done by querying the ontology using the Apache JENA library.

Order Simulation

In order to simulate the online grocery store and a real order, a simple REST server was written in Java. The REST server has two operations implemented:

  1. A GET on /products which returns all products that are stored in the ontology including the details for each of them.
  2. A POST to /products/order that is used to simulate a real order. One can post an order which will be further processed by the robotic worker.


Also, a Swagger documentation was created to make it easier to get the products and their details from the ontology and to post a new order.

For the order simulation part, also a minimalistic online grocery store was implemented in TypeScript using Angular 5 and nodejs. Each product of the ontology including its details and an image is shown there. Also, each product has an input field to type in how many of these products are wanted. At the end of the page there is a “purchase” button. When clicking that button all products that have an amount greater than zero are sent as an order. The input will also be checked, so when there are wrong inputs the GUI will show an appropriate error message. When everything worked fine, the order will be stored in a queue and processed by the robot as soon as possible. Also, an appropriate message including the overall price and the id of the order will be visible in the GUI.  

Product Sorting

After receiving the order per REST, the items need to be sorted. We defined several criteria how the items are sorted. To get the necessary corresponding information from the items, the ontology is used. In first place the items are sorted by their cooling status. This cooling status can take the values none, refrigerated or frozen. Frozen items will be picked up as last ones to ensure that they stay as frozen as possible. Within the cooling status, the items are sorted by their weight. The sorting algorithm defines the order of the items with descending weight.
When the sorting process is done, the sorted order is forwarded to the function, which calculates the path between the items.

Finding the Path

In order to find the shortest Path between each item in the sorted order, we used a graph and the dijkstra algorithm.
The first step to achieve the calculation is to import the XML-Export from the prior defined model. The system iterates through the XML nodes and transforms it into our internal graph-like structure. We have two classes – Graph and GraphNode – where the corresponding informations, like product ID or connections between the nodes, are stored. With this Graph, the shortest path from point A to B will be calculated. We implemented a recursive function which implements the logic of the dijkstra algorithm to fulfill that task and to get the overall path for the robot.
If the calculation is finished, the overall path, which consists of some smaller paths, will be returned.

Navigation

In order to enable the NAO to navigate through the warehouse it is firstly necessary to have an ADOxx model exported to XML. As already described, the XML file is imported and transformed to the internal data structure.
So, what’s especially necessary for that part is to get the coordinates out of the XML model and to save the x and y value for each graph node. Also, the robot needs to know the direction he is looking into. Therefore, the directions North, East, South and West are used. The starting direction of the robot is East.
There are two methods to navigate the robot from node to node.

  1. The current location of the robot is compared to the next location the robot needs to go to. In detail, it is checked whether there is a change in the x-axis or in the y-axis. It is not designated to have a change on both axis at the same time as this would mean that the robot needs to walk diagonally. That’s something that needs to be considered when modelling the warehouse. So, this method will return the direction the robot needs to go to.
  2. When having the current and the next direction, it needs to be considered whether the robot need a turn, in which direction and of how many degrees.

Simulation of Orderitems

Instead of real shelves storing real products, QR Codes were used to abstract the product collecting part. The robot stops in front of the “shelf” (= barcode) and scans the barcode instead of taking an item out of the shelf.

DevOps

The modelling method, some example models, the ontology, the used QR codes and the source code of the server as well as of the webshop GUI can be found here: https://gitlab.dke.univie.ac.at/edu-semtech/nao-warehouse

The main project, the server, is a Maven-based Java project. Therefore, it is not bound to a specific IDE.

In order to start the development with the NAO, an appropriate SDK needs to be installed. For this project, the Java NAOqi SDK was used, but there are also SDKs for Python and C++. Important to know is, that the Java NAOqi SDK requires a 32bit Java version as 64bit versions are not supported. Therefore, we used the 32bit Java version 1.8. Details about the Java NAOqi SDK and the required first steps can be found here. Also, the necessary steps to connect to the robot and to call a remote service can be found there. The NAOqi API Documentation can be found here.

In this following image you can see which remote services were used within this project and how to connect to the robot using Java:



The ROBOT_URL is basically the IP address of the robot. Pressing the chest button of the robot will make the robot tell you his IP address. In this project, the default ROBOT_URL is “tcp://192.168.75.41:9559”, the address of the red NAO from the OmiLab. In order to change the ROBOT_URL, directly changing it in the source code of the server (file OrderPreparation.java) is necessary.

As already mentioned, the ontology was created with Protege and the modelling method was developed with ADOxx. When making changes on the ontology, especially on the productids, it is important to keep the ADOxx models in sync with the ontology, same the other way around. The path to the exported ontology (in RDF/XML) as well as the path to the exported ADOxx model (in XML) need to be adapted accordingly (in file OrderPreparation.java).

After all these configurations were done, building the maven project is necessary and will produce a jar file. Running java -jar JARFILE is necessary in order to start the program.

When starting the program, the REST server will be started and the connection to the robot will be established. The links to the Swagger documentation as well as the GUI can be found on http://localhost:8087/static. The REST_URI is defined in the file Server.properties and currently defined as http://localhost:8087/rest. So when change on the url are wanted, the file Server.properties needs to be adapted appropriately. Of course, before building and running the program.

The Webshop GUI was implemented in TypeScript with Angular 5 and nodejs. When changes on that part are wanted, angular-cli version 1.6.6, nodejs and npm are needed to be installed. Using npm install will install the needed node modules. ng serve will provide a development server. The website will be reachable at http://localhost:4200/. The results of code changes are immediately visible in the browser. When all changes are done, using ng build is necessary. This will generate a dist directory which needs to be copied to the Java servers subdirectory static. This is necessary to enable the server to deliver the GUI so no additional web server is needed.

Results

The video below shows the results of the project. In purpose of demonstration we simulated a warehouse with walls. We modelled this environment with our modeling method as shown at the beginning. The system needs the XML-Export of the Model and the Ontology as input. Through the User-Interface we ordered Cherries and Cheese. After sending the Order the products will be sorted appropriately, the shortest Path will be calculated and the NAO starts to collect one item after another. After collecting the last item he returns to the PackStation.

Lessons Learned

While testing the motion range of the NAO we recognized, that the NAO doesn’t walk straight on long distances, instead it uses to walk a bit right or left. The drift depends on the Robot as well as on how warm the engines are. Also the underground seems to be from importance. We also noticed a big difference between the red and the blue NAO of the OMiLab. The blue robot has a very strong tendency to his left side while walking. The red one has not had such a strong drift.
In addition, the predefined function to turn the NAO around (with a desired angle) is not very accurate. The difference is also varying between the robots and other factors. It was to be expected, that the robot will not be able to move extremely accurate, but we didn’t expect that it will be such different from how it should be. The slightly difference in every movement increased the overall variance a lot.


Future Work

We dimensioned the Use Cases that they are realizable within the course. Additionally, there are some possibilities of how the project can be enhanced.
The first idea is to connect the ontology and the ADOxx Model. The used product IDs in the ontology and the model need to match. Therefore, this connection would be a useful improvement.
The next step would be to include image recognition to identify the items, which should be picked up. We abstracted this with QR-Codes. After the individual items are identified, the NAO should reach towards the items and grab one. The big challenge is that the items can have very different textures and depending on that, the NAO needs to hold the items tighter or very careful to not damage it. An example would be an egg which is very fragile.
Another part would be, that the robot should carry the items in some kind and bring them all together to the PackStation. What should be considered at that point is, that grabbing an item would change the center of gravity and the robot could maybe fall or could unable to move, because NAO tries to compensate the weight. This depends on the weight of the items, but it is a possibility.