NOTE This article is Machine-Translated. If you have any questions about this article, please send an E-mail to DJI, we will correct it in time. DJI appreciates your support and attention.
To help developers use automatic flight control in application scenarios without GPS or RTK satellites, use Precise Hovering and Obstacle Avoidance, DJI has opened the drone's visual perception system. Developers use API in OSDK can obtain the camera’s stream of the DJI drone's visual perception system, generate time-diff-maps and cloud-maps, combined with image recognition algorithms, developer could develop the applications that meet the specific needs.
- Applications developed on Linux and ROS systems allow developers use DJI's advanced perception features.
- When using the advanced perception function of DJI OSDK, developer need to use a USB cable to receive the raw image data from the drone's visual perception system.
In order to satisfy the function of developers using OSDK to obtain the camera’s stream, OSDK provides the function to obtaining the camera’s stream, supports obtaining the FPV camera and main camera's H.264 tream and RGB images.
- M210 series support H.264 bitstream or RGB images of FPV cameras and main cameras; M300 RTK support H.264 bitstreams of FPV cameras and others cameras.
- The callback functions for obtaining the FPV camera's stream and obtaining the main camera's stream run in separate threads, OpenCV's module imshow only supports running in one thread, so developers are allowed to obtain FPV cameras or main camera's H.264 stream and RGB images.
- After obtaining the camera’s stream, please install a decoder such as FFmpeg for decoding.
- The details of the H.264 criterion please refer to H.264 Criterion.
OSDK supports developers to obtain the main camera’s stream which on the M210 series and M300 RTK. Developers or users can mount different models of cameras and specified the frame rate, obtain the required code stream according the mode and module of the camera.
NOTE The frame rate of both FPV camera and main camera’s stream is 30 FPS.
The process of obtaining the H.264 stream of the camera on the M210 series and M300 RTK is as follows:
startH264Stream ()to specify the camera that the stream, the callback function, the H.264 stream and user's information needs to receive which from the camera;
liveViewSampleCbwhich designed by developer will perform corresponding operations such as storage, decoding and forwarding works.
The visual perception system of the DJI's drone is composed of the visual perception sensor and visual perception algorithm. During the flight of the drone, the perception sensor can obtain the state of the surrounding environment, assist drone to brake, avoid obstacle and accuracy hover.
- If the drone is disturbed by external interference during hovering, the drone will return to the original hovering point.
- When using the stereo perception function, if the drone is disconnected from the drone remote control, the drone will hover.
The image types that the stereoscopic perception function supports is as follows:
The image generated by the drone's visual sensor contains two metadata: frame index and time stamp.
When using Advanced Sensing functions, developers need to call the specified API to subscribe (or unsubscribe) the image of the camera through the USB during the application execution, and use the callback function in the dedicated reading thread to get subscribed images.
- It is recommended to install OpenCV to display the image when acquiring the image of drone's vision sensor.
- In order to blocking the main thread, please create a separate thread to process the image of the visual sensor of the drone. When using Advanced Sensing functions on the ROS system, please using
image_viewto subscribe and display the image.
The following steps to use the KCFcpp library to achieve the Target Tracking.
- When using the KCFcpp library to compile sample code for target tracking, please use the command
cmake -DADVANCED_SENSING=ON -DTARGET_TRACKING_SAMPLE=ON maketo compile the program.
- After entering g from the keyboard, developer can create a KCF tracker; enter s choose the object to track, ESC for exit.
In order to specify the target easily, OSDK provides the
TrackingUtility class, which is used to obtain the target object and tracking status.
tracker = new KCFTracker (true, true, false, false);
The target tracking algorithm calculates the distance based on the position of the image (unit: 1 px).
/* Get the object */
According to the moving state of the target object, calculate the yaw angle and pitch angle that the gimbal needs to change, and send the angle to
DJI :: OSDK :: Gimbal :: SpeedData gimbalSpeed;
Obtain RGB images by polling in the main thread.
vehicle-> advancedSensing-> startFPVCameraStream ()interface to create a thread for reading the original code stream of the camera and decoding it into an image.
vehicle-> advancedSensing-> newFPVCameraImageIsReady ()interface to check the status of the camera’s stream. If there is a stream available in the camera, then call
vehicle-> advancedSensing-> getFPVCameraImage (fpvImage )Get the image.
NOTE If the developer installs the OpenCV library, the
cv::imshow ()interface can be called through the
show_rgbfunction to display the RGB Image which decoded.
vehicle-> advancedSensing-> stopFPVCameraStream ()interface to disconnect the camera and destroy the thread that reads the camera code stream.
Get the RGB image through the callback function.
vehicle-> advancedSensing-> startFPVCameraStream (& show_rgb, NULL)interface to create a thread to get the camera code stream, and register the callback function
show_rgbin this interface to process the received code stream.
vehicle-> advancedSensing-> stopFPVCameraStream ()interface to disconnect from the camera and destroy the thread that reads the camera code stream.
Control the application to receive the H.264 stream of the specified camera.
LiveView::LiveViewErrCode startH264Stream(LiveView::LiveViewCameraPosition pos, H264Callback cb, void *userData);
Control the application stop getting the H.264 stream of the specified camera.
LiveView::LiveViewErrCode stopH264Stream(LiveView::LiveViewCameraPosition pos);
After obtaining the H.264 stream, the application developed based on OSDK would option the H.264 stream for developer.
typedef void (*H264Callback)(uint8_t* buf, int bufLen, void* userData);
- After obtains the H.264 stream, developer could use
ffplay FPV.h264to play the H.264 file which obtained.
- Using the sample (djiosdk-liveview-sample) offered by the OSDK, developer could obtain the h.264 stream and recorded as the H.264 file, which named 'userData'.
- Using Sample camera-stream-callback-sample and camera-stream-poll-sample. Developer could decode the H.264 stream and implement the required functionality.
The developer can use Elecard StreamEye Tools, H264Visa and other H.264 decoding software to decode and analyze the H.264 stream obtained by OSDK.
The following is the analysis result of the stream obtained by using Sample djiosdk-liveview-sample, the video is recorded in the room for 9~10 seconds.
Table1.The analysis result of the H.264
|M300 FPV||M210 V2 FPV||Z30||XTS||X7||H20T|
|M300 FPV||M210 V2 FPV||Z30||XTS||X7||H20T|
|Video Stream Type||AVC/H.264||AVC/H.264||AVC/H.264||AVC/H.264||AVC/H.264||AVC/H.264|
|Resolution||608 x 448||608 x 448||1280 x 720||640 x 512||1280 x 720||1920 x 1080|
|Aspect Ratio||4 x 3||4 x 3||16 x 9||5 x 4||16 x 9||16 x 9|
|File Size (Bytes)||867619||877819||4559469||5472872||8765755||17243162|
Restricted by the computing power of the computing platform, the applications developed based on OSDK may have the following problems when encoding and decoding:
Z30 adopts Main Profile technology and USES GDR encoding format, developers cannot decode the H.264 video stream of camera Z30 by FFMpeg decoding method by using the sample provided by OSDK, but according to the actual development environment, developers can choose other decoders such as hardware decoder or JM decoder to decode the H.264 stream of the Z30.
NOTE OSDK doesn't support obtain and switch the camera images of the camera which has multiple light sources. If you want to switch it, please switch it on the DJI Pilot.