Liveview

2022-06-21
No Rating

Overview

In order to help developers realize real-time image return and real-time processing during the automated flight process of drones, and realize functions such as target tracking and target detection, DJI has opened the drone Liveview system. The code stream data of the DJI drone Liveview system can be obtained, combined with image recognition algorithms or specific links, to develop applications that meet the needs of specific usage scenarios.

Description

  • Only applications developed using Linux support developers using DJI's Liveview system.
  • When using the advanced sensing function of DJI PSDK, you need to use a USB cable to receive the raw stream data of the drone Liveview system.

Basic Concepts

Camera stream

In order to satisfy the function of obtaining camera code stream for applications developed by developers using PSDK, PSDK provides the function of obtaining camera code stream, which supports obtaining FPV camera or obtaining H.264 code stream and RGB image of I gimbal camera.

Description

  • M300 RTK Drone supports getting H.264 stream from FPV camera and all gimbal cameras.
  • Since the callback functions for obtaining the FPV camera code stream and obtaining the camera code stream on the I gimbal run in separate threads, the imshow module of OpenCV only supports running in one thread, so it only supports developers to obtain FPV cameras or Acquire the H.264 code stream and RGB image of the I gimbal camera.
  • After obtaining the camera stream, please install a decoder such as FFmpeg to decode.
  • Please refer to H.264 standard code stream Streaming Standardsopen in new window.

Resolution and Frame Rate

PSDK supports developers to obtain the code stream of the camera on the No. I gimbal on the M300 RTK drone. Developers or users can mount different types of cameras according to actual use requirements, and specify the frame rate according to the model of the camera and the working mode of the camera. to obtain the desired code stream.

  • Photo mode:
  • FPV camera stream: support to obtain images with a resolution of 608x448.
  • Camera Mode: Supports acquiring images with resolutions of 1440x1080 (1080p) and 960x720 (720p).
  • Video mode: Support to obtain videos with resolutions of 1920x1080 (1080p) and 1280x720 (720p).

Note: The frame rate of the FPV camera and main camera stream is 30 FPS.

Camera H.264 stream

The process of obtaining the H.264 code stream of the camera on the M300 RTK drone is as follows:

  1. Before using the function of obtaining the camera H.264 code stream, **please implement the DjiLiveview_H264Callback function according to the actual use needs to obtain and process the camera H.264 code stream.
  2. Call the DjiLiveview_StartH264Stream() interface to specify the camera to obtain the stream, the callback function to receive the camera's H.264 stream, and user information;
  3. Turn on the drone and the onboard computer, and run the application developed based on PSDK. At this time, the drone will push the H.264 stream to the onboard computer;
  4. After the onboard computer receives the data of the H.264 code stream, it will trigger (passed into the callback function set by the developer as an input parameter) the application developed based on the PSDK;
  5. The function designed by the developer according to the actual needs DjiLiveview_H264Callback After obtaining the H.264 code stream of the camera, it will perform corresponding operations such as storage, decoding and forwarding of the obtained H.264 code stream.

Use the function to get the camera's H.264 stream

1. Initialize the Liveview module

Before using the camera stream processing function, you need to call the DjiLiveview_Init interface to initialize the liveview module.

returnCode = DjiLiveview_Init();
if (returnCode != DJI_ERROR_SYSTEM_MODULE_CODE_SUCCESS) {
USER_LOG_ERROR("Liveview init failed, error code: 0x%08X", returnCode);
goto out;
}

2. Start getting camera H.264 stream

Control the application to receive the H.264 stream of the specified camera.

returnCode = DjiLiveview_StartH264Stream(DJI_LIVEVIEW_CAMERA_POSITION_FPV, DJI_LIVEVIEW_CAMERA_SOURCE_DEFAULT,
DjiTest_FpvCameraStreamCallback);
if (returnCode != DJI_ERROR_SYSTEM_MODULE_CODE_SUCCESS) {
USER_LOG_ERROR("Request h264 of fpv failed, error code: 0x%08X", returnCode);
goto out;
}

3. Stop getting H.264 stream

Controls the application to stop receiving the camera's H.264 stream.

returnCode = DjiLiveview_StopH264Stream(DJI_LIVEVIEW_CAMERA_POSITION_FPV);
if (returnCode != DJI_ERROR_SYSTEM_MODULE_CODE_SUCCESS) {
USER_LOG_ERROR("Request to stop h264 of fpv failed, error code: 0x%08X", returnCode);
goto out;
}

4. Save or process H.264 stream

After an application developed based on PSDK acquires the H.264 code stream, the developer can perform the required operations on the acquired H.264 code stream.

typedef void (*DjiLiveview_H264Callback)(E_DjiLiveViewCameraPosition position, const uint8_t *buf, uint32_t len);

H.264 stream parsing

Developers can use Elecard StreamEye Tools, H264Visa and other H.264 decoding software to decode and analyze the H.264 code stream obtained with PSDK. The following is the analysis result of using the Camera stream view sample to obtain the camera code stream on DJI drones. All videos were recorded indoors and lasted 9-10 seconds.

Table 1. Camera H.264 code stream analysis results

M300 FPVZ30XTSH20T
Video Stream TypeAVC/H.264AVC/H.264AVC/H.264AVC/H.264
Resolution608x4481280 x 720640 x 5121920 x 1080
ProfileMain:4.1Main:4.1High:5.0High:4.0
Aspect Ratio4 x 316 x 95 x 416 x 9
InterlacedNoNoNoNo
File Size (Bytes)8676194559469547287217243162
Frames Count271300240299
Frame SizeMax

5095

348485336871881
Avg3201151982280357669
Min199041641102551506

I Frame

Max

4802

0517290
Avg42430370540

P Frame

Max

5095

348485336871881
Avg3177151982231257669

B Frame

Max

0

000
AVG0000

FrameRate

Declared

30.00

0.0025.0030.00
Real30.0030.0025.3730.00

BitRate (bit/s)

Max8469603988324536780814462654
Avg7682403647523462837913840574
Min7310403070083417940413307293

H.264 decoding delay

Note: The test environment for the following latency data is: H264 decoder: FFMpeg Decoding Type: Software Decoding Run the code: Camera stream view sample

ModelOperating PlatformLoadLatency
M300Manifold 2-GFPV250ms~300ms
M300Manifold 2-GH20T360ms~400ms
M300Manifold 2-CFPV170ms~200ms
M300Manifold 2-CH20T230ms~250ms
  • PSDK only provides code implementation for H264 pure software decoding
  • If there is a need to improve decoding performance, users can use a hardware decoder to improve decoding efficiency

common problem

There was an error decoding the decoder

Restricted by the computing power of the computing platform, the applications developed based on PSDK may encounter the following problems when encoding and decoding:

  • Slow decoding: it takes a while for the decoder to decode the first frame
  • Frame loss: insufficient computing power of the computing platform
  • An error occurs when using FFmpeg to decode: Please try decoding on Ubuntu 16.04, and make sure that the RNDIS, USB and network port drivers are correctly installed in the application running the decoder, and ensure that the application can correctly identify the M300 RTK drone.

Unable to parse H.264 stream data of camera Z30

Since the Z30 adopts the Main Profile technology and uses the GDR encoding format, the developer cannot decode the H.264 video stream of the camera Z30 through the FFMpeg decoding method using the Sample provided by the PSDK, but the developer can choose other decoders according to the actual development environment. Such as hardware decoder or JM decoder to decode the H.264 stream data of camera Z30.

I frame sometimes appears in H20/H20T H.264 stream in GDR encoding format

The H20/H20T is a multi-lens payload camera. When DJI Pilot switches camera images, the camera's H.264 stream will insert I-frames to help DJI Pilot's image transfer images construct the image. Therefore, I-frames will appear in the H.264 stream, which does not affect the decoding process of the GDR format.

Applicable Products

  • Matrice 300 RTK: support Z30, XTS, XT2, H20, H20T (developers can get video stream without using remote control)

Note: PSDK currently does not support simultaneous acquisition and switching of images from cameras that support multiple light sources. To switch, please set the switching camera image in DJI Pilot.