esieabot-ai

../_images/esieabot-ai.jpg

esieabot-ai is an optional library that makes it easy to use image recognition functions on your esieabot. Currently, it can detect and locate ArUco markers. It is still under development. The calibration is not perfect yet.

ArUco marker

../_images/ArUco.png

ArUco markers are simplistic markers used in augmented reality applications. Each marker represents a number, in this case esieabot-ai recognizes 4 by 4 ArUco markers, covering numbers from 0 to 250. Thanks to image processing, the position and orientation angle of the markers is recognized. This makes it possible to geolocate the esieabot in space using only its camera. You can generate markers on this site: https://chev.me/arucogen/.

Installation

First, you need to connect the camera to your Raspberry Pi. To do this, follow these instructions:

Warning

You must turn off your Raspberry Pi before connecting the camera. If you do not, you will damage the camera.

../_images/step-8a.png ../_images/step-8b.png ../_images/step-8c.png ../_images/step-8d.png

Connect the camera by gently pulling out the black part of the connector, insert the cable as shown, and put the black part back in place. Do the same on the camera side if it is not already done. The black side of the cable should face the camera sensor.

To install esieabot-ai, run the following command: sudo apt install esieabot-ai-api.

Note

esieabot-ai-api cannot run in parallel with another application using the camera, such as esieabot-webcamera which will be uninstalled automatically.

Test

You can test image recognition by accessing the following address from your web browser: http://<address_of_your_esieabot>/ai/get_image

Note

Your esieabot’s address is its IP address, which you can obtain with the command ip a. If you’re connected to its hotspot, the address is always 10.42.0.1. Otherwise, you’ll need to look up the ip address of your robot on your network. If have have an esieabot with a screen, you can also find the address on the screen.

To find out which ArUco markers are detected, open the following page: http://<your_robot_address>/ai/get_markers. By default, you will get information for markers with a side length of 51mm. If you want to specify a different size, you must add “?marker_size_mm=” followed by the marker size in millimeters at the end of the URL.

Warning

It is not possible to use the get_image function and the get_markers function at the same time. So you can’t display what the camera sees and at the same time request the list of markers.

Note

It takes about 150ms (on Raspberry Pi Zero 2W) or 300ms (on Raspberry Pi Zero W) to process a full image. You should take into account in your programs that processing is not instantaneous.

Debugging

If the tests don’t work, you can go to the /esieabot/logs folder to find a esieabot-ai-api.py.log file. This file contains error messages which may be of interest to you. If it doesn’t exist, esieabot-ai hasn’t been installed correctly. In any case, if the application doesn’t work, it’s probably because you have another application monopolizing the camera.

If image recognition doesn’t work, you can run the command libcamera-still --list-cameras to see if the camera is recognized by the system.

Use in C

You can use the esieabot-ai library directly in a C program like this :

#include "esieabot-ai.h"
#include <stdio.h>
#include <stdlib.h>

int main() {
    while (1) { // Infinite loop of the program
        struct marker *markers = get_markers(51); // Retrieve markers with 51mm length (change according to your markers!)
        struct marker *m;
        for (m = markers; m->id != -1; m++) { // For all found markers...
            printf("Marker found: number %d at %dcm distance. x=%d y=%d\n", m->id, m->z, m->x, m->y);
        }
    }
    return 0;
}

This simple program loops esieabot-ai to retrieve the list of markers as an array and iterate through it. To compile it, add -lesieabot-ai -lcurl to the compile command.

Use in Python

You can also use the esieabot-ai library in python. However, you will have to make a request yourself. To do this, you must first import the requests library.

import requests
import math

# 51mm marker size. You must change this value to the size of your markers.
response = requests.get('http://127.0.0.1:5001/get_markers?marker_size_mm=51').json()

for marker in response["markers"]:
    distance = math.sqrt(marker["x"] ** 2 + marker["y"] ** 2 + marker["z"] ** 2)
    print("Marker found: number {} at {}cm distance. x={} y={} z={}".format(marker["id"], distance, marker["x"], marker["y"], marker["z"]))

This program will, as in the C example, retrieve the list of markers, but in the form of a dictionary. The program will thus iterate in it for each element.

Structures

  • marker is a structure containing the following fields:

    • int id is the marker id

    • int x is its position on the X axis in cm

    • int y is its position on the Y axis in cm

    • int z is its position on the Z axis in cm

    • int pitch is its rotation (pitch axis) in degrees

    • int roll is its rotation (roll axis) in degrees

    • int yaw is its rotation (yaw axis) in degrees

Angles are returned in degrees, between -180 and 180.

Note

A marker facing up and aligned perpendicular to the camera will have a roll of 180° but a pitch and yaw of 0°

Orientation

../_images/yaw-pitch-roll.png