Overview

In order to help developers realize real-time image return and real-time processing during the automated flight process of drones, and realize functions such as target tracking and target detection, DJI has opened the drone Liveview system. The code stream data of the DJI drone Liveview system can be obtained, combined with image recognition algorithms or specific links, to develop applications that meet the needs of specific usage scenarios.

Note:

• Only applications developed using Linux and RTOS support developers using DJI's Liveview system.
• When using the advanced sensing function of DJI PSDK, you need to use a USB cable to receive the raw stream data of the drone Liveview system.

Basic Concepts

Camera Stream

In order to satisfy the function of obtaining camera code stream for applications developed by developers using PSDK, PSDK provides the function of obtaining camera code stream, which supports obtaining FPV camera or obtaining H.264 code stream and RGB image of I gimbal camera.

Note:

• M300 RTK Drone supports getting H.264 stream from FPV camera and all gimbal cameras.
• Since the callback functions for obtaining the FPV camera code stream and obtaining the camera code stream on the gimbal 1 run in separate threads, the imshow module of OpenCV only supports running in one thread, so it only supports developers to obtain FPV cameras or Acquire the H.264 code stream and RGB image of camera mounted on the gimbal 1.
• After obtaining the camera stream, please install a decoder such as FFmpeg to decode.
• Please refer to H.264 standard code stream Streaming Standards.

Resolution and Frame Rate

PSDK supports developers to obtain the code stream of the camera on the No. I gimbal on the M300 RTK drone. Developers or users can mount different types of cameras according to actual use requirements, and specify the frame rate according to the model of the camera and the working mode of the camera. to obtain the desired code stream.

• Photo mode:
• FPV camera stream: support to obtain images with a resolution of 608x448.
• Camera Mode: Supports acquiring images with resolutions of 1440x1080 (1080p) and 960x720 (720p).
• Video mode: Support to obtain videos with resolutions of 1920x1080 (1080p) and 1280x720 (720p).

Note: The frame rate of the FPV camera and main camera stream is 30 FPS.

Camera H.264 Stream

The process of obtaining the H.264 code stream of the camera on the M300 RTK drone is as follows:

1. Before using the function of obtaining the camera H.264 code stream, **please implement the DjiLiveview_H264Callback function according to the actual use needs to obtain and process the camera H.264 code stream.
2. Call the DjiLiveview_StartH264Stream() interface to specify the camera to obtain the stream, the callback function to receive the camera's H.264 stream, and user information;
3. Turn on the drone and the onboard computer, and run the application developed based on PSDK. At this time, the drone will push the H.264 stream to the onboard computer;
4. After the onboard computer receives the data of the H.264 code stream, it will trigger (passed into the callback function set by the developer as an input parameter) the application developed based on the PSDK;
5. The function designed by the developer according to the actual needs DjiLiveview_H264Callback After obtaining the H.264 code stream of the camera, it will perform corresponding operations such as storage, decoding and forwarding of the obtained H.264 code stream.

Use the Function to Get the Camera's H.264 Stream

1. Initialize the Liveview Module

Before using the camera stream processing function, you need to call the DjiLiveview_Init interface to initialize the liveview module.

returnCode = DjiLiveview_Init();
if (returnCode != DJI_ERROR_SYSTEM_MODULE_CODE_SUCCESS) {
USER_LOG_ERROR("Liveview init failed, error code: 0x%08X", returnCode);
goto out;
}

2. Start Getting Camera H.264 Stream

Control the application to receive the H.264 stream of the specified camera.

returnCode = DjiLiveview_StartH264Stream(DJI_LIVEVIEW_CAMERA_POSITION_FPV, DJI_LIVEVIEW_CAMERA_SOURCE_DEFAULT,
DjiTest_FpvCameraStreamCallback);
if (returnCode != DJI_ERROR_SYSTEM_MODULE_CODE_SUCCESS) {
USER_LOG_ERROR("Request h264 of fpv failed, error code: 0x%08X", returnCode);
goto out;
}

3. Stop Getting H.264 Stream

Controls the application to stop receiving the camera's H.264 stream.

returnCode = DjiLiveview_StopH264Stream(DJI_LIVEVIEW_CAMERA_POSITION_FPV);
if (returnCode != DJI_ERROR_SYSTEM_MODULE_CODE_SUCCESS) {
USER_LOG_ERROR("Request to stop h264 of fpv failed, error code: 0x%08X", returnCode);
goto out;
}

4. Save or Process H.264 Stream

After an application developed based on PSDK acquires the H.264 code stream, the developer can perform the required operations on the acquired H.264 code stream.

typedef void (*DjiLiveview_H264Callback)(E_DjiLiveViewCameraPosition position, const uint8_t *buf, uint32_t len);

H.264 Stream Parsing

Developers can use Elecard StreamEye Tools, H264Visa and other H.264 decoding software to decode and analyze the H.264 code stream obtained with PSDK. The following is the analysis result of using the Camera stream view sample to obtain the camera code stream on DJI drones. All videos were recorded indoors and lasted 9-10 seconds.

Table. Camera H.264 code stream analysis results

Model	M300 RTK / M350 RTK						M30		M30T		M3E	M3T	M3D	M3TD
	FPV	H30/H30T	H20/H20T/H20N	P1	L1	L2	FPV	Main camera	FPV	Main camera	Main camera	Main camera	Main camera	Main camera
Video Stream Type	AVC/H.264	AVC/H.264	AVC/H.264	AVC/H.264	AVC/H.264	AVC/H.264	AVC/H.264	AVC/H.264	AVC/H.264	AVC/H.264	AVC/H.264	AVC/H.264	AVC/H.264	AVC/H.264
Resolution	M300: 1280x720 M350: 1920x1080	1440x1080	1920x1080	1920x1080	1920x1080	1920x1080	1920x1080	1920x1080	1920x1080	1920x1080 IR:640x512/1280x1024	1920x1080	1920x1080 IR:640x512/1280x1024	1920x1080	1920x1080 IR:640x512/1280x1024
FramesRate	30	30	30	30	30	30	30	30	30	30	30	30	30	30
BitRate (bit/s)	16Mbps	16Mbps	16Mbps	8Mbps	8Mbps	32Mbps	16Mbps	16Mbps	16Mbps	16Mbps IR:8Mbps	16Mbps	16Mbps IR:8Mbps	16Mbps	16Mbps IR:8Mbps
YUV format	YUV420	YUV420	YUV420	YUV420	YUV420	YUV420	YUV420	YUV420	YUV420	YUV420	YUV420	YUV420	YUV420	YUV420

H.264 decoding delay

Note: The following latency data represents the average values from five tests, and the test environment is:

• PSDK version: PSDK 3.9
• Aircraft model: M350 RTK
• H264 decoder: FFMpeg
• Decoding type: Software decoding
• Run the code: Camera stream view sample

Operating Platform	Load	Latency
Manifold 2-G	FPV	256 ms
Manifold 2-G	H20T	372 ms
Manifold 2-G	H30T	354 ms

• PSDK only provides code implementation for H264 pure software decoding
• If there is a need to improve decoding performance, users can use a hardware decoder to improve decoding efficiency

Common Problem

There was an error decoding the decoder

Restricted by the computing power of the computing platform, the applications developed based on PSDK may encounter the following problems when encoding and decoding:

• Slow decoding: it takes a while for the decoder to decode the first frame
• Frame loss: insufficient computing power of the computing platform
• An error occurs when using FFmpeg to decode: Please try decoding on Ubuntu 16.04, and make sure that the RNDIS, USB and network port drivers are correctly installed in the application running the decoder, and ensure that the application can correctly identify the M300 RTK drone.

I frame sometimes appears in H20/H20T H.264 stream in GDR encoding format

The H20/H20T is a multi-lens payload camera. When DJI Pilot switches camera images, the camera's H.264 stream will insert I-frames to help DJI Pilot's image transfer images construct the image. Therefore, I-frames will appear in the H.264 stream, which does not affect the decoding process of the GDR format.

Applicable Products

• Matrice 300 RTK supports Z30, XTS, XT2, P1, L1, L2, H20/H20T, H30/H30T. Developers can get video stream without using the remote controller.
• Matrice 350 RTK supports P1, L1, L2, H20/H20T, H30/H30T. Developers can get video stream without using the remote controller.

Note: PSDK currently does not support simultaneous acquisition and switching of images from cameras that support multiple light sources. To switch, please set the switching camera image in DJI Pilot.