Hello everyone,
I am working with a project that demands a companion computer processing images captured by a camera and send a flag (or string) with the information needed from the image and send the interest image itself. I’m trying to use the mavlink protocol to send this through a wifi connection between the computer and a ground station. I managed to get mavlink telemetry this way, but now I want to include some messages triggered by the image processed in the same mavlink “package” to read from the ground station. Someone know if that is possible and which is the better way to do it?
At this time I have a raspberry pi connected by serial ports to the flight controller and by wifi to a computer in ground station.
An example of mission I need to do is: a drone is flying above a street and I want to know how many cars of each color is passing there. The camera mounted on the drone keeps a video capture and everytime it sees a car the companion computer process the image and tell which color is that car. This information may be sent to the ground station with the flag or string meaning the color and the image used to determine the information.
I tryed to fill this topic with a lot of information to make clear what I want to do. If there is any missing information please let me know.
Yer, that’s a good question, and I’m not 100% sure how to answer it. You could create your own MAVLink dialect, and add messages that you require, also see Defining XML Enums/Messages · MAVLink Developer Guide. I think that would work to transmit the meta information such as detections but not the actual image/video stream.
If you need/want something more flexible, you could also consider writing a server in any other “technology”, gRPC, REST, ROS 2, mqtt, choose any that you like or are familiar with.
Hi Julian.
Sorry for the late answer, I was working on other stuff and this one went stand-by.
Using a dialect I could change the message to be sent in flight? Example: the camera sees a red car, so the message will be “red”. After that it sees a blue car, so the message is “blue”. And then there is no car in image, so the message will be “none”. I can program the message to change based on the information that companion computer send to pixhawk?
I’m not sure I understand what you mean with that. If you use MAVLink message definitions (even your own dialect) then the types/structure of that message has to be defined beforehand at build time.
So I can’t use the companion to choose which message I want to send?
The problem I would like to solve is how to use mavlink communication to send a message that could be changed in flight, based on a image processing algorith that is running inside a companion computer.
So if the companion computer process a photo and say that a blue car is on the image I would like to include a message in mavlink package that goes to ground station with telemetry data saying that a blue car is there. When I have no cars in image I can send a message saying that nothing important is on the image.
I used pymavlink to receive mavlink telemetry from the UAV in my computer. There were data like “roll angle = 10º”; “pitch rate = 4º/s”. I would like to include like “car on image = blue” or “car in image = none”.
Right, so the way to do that is usually to define an enum. E.g.:
enum Color {
red = 0;
green = 1;
blue = 2;
};
And then the type would be something like uint8_t in the MAVLink message.
Great! But this raises other doubts. First one is where I define this? A new dialect?
Then, it will be changeable in flight? And the last, how I can make the companion changes this message according to what it’s processing?
I apologize for the excess of questions, but I am not a dev (a begginer, actually) and until this problem with the message I was using px4 firmware only to configure some basic things on a frame.
So I will have something like this?
<enum name="CAR_COLOR">
<description>Car color detected</description>
<entry value="0" name="CAR_COLOR_NONE">
<description>No cars in image</description>
</entry>
<entry value="1" name="CAR_COLOR_RED">
<description>Red car in image</description>
</entry>
<entry value="2" name="CAR_COLOR_GREEN">
<description>Green car in image</description>
</entry>
<entry value="3" name="CAR_COLOR_BLUE">
<description>Blue car in image</description>
</entry>
This entry values could be modified by a input from companion computer through a serial port connected to pixhawk?
I don’t know if this helps, but the hardware I have now is a Pixhawk 2.4.8 connect by serial port TELEM2 to a Raspberry PI 4 that manages a camera and process the photos. I need to change the mavlink message based on what this process program is telling (like the car color).
Yes, and then you need to actually use that enum in a message.
Ok, but I didn’t understand if I can change this enum value using the Raspberry Pi connected in the TELEM2 port.
Yes, you can set the enum value on the RPi side.
