If you come across any mistakes or bugs in this tutorial, please let us know by sending emails to dev@dji.com. Please feel free to send us Github pull request and help us fix any issues.
In this tutorial, you will learn how to build a cool panorama app. With the help of the powerful DJI Mobile SDK and OpenCV libraries, it is actually easy. You will use the Waypoint feature of Intelligent Navigation and Virtual Stick to rotate the aircraft to take photos. The panorama application can be broken down into several simple tasks, all you need to do is to follow the tutorial step by step. Let's get started!
You can download the tutorial's final sample project from this Github Page.
Application Activation and Aircraft Binding in China
For DJI SDK mobile application used in China, it's required to activate the application and bind the aircraft to the user's DJI account.
If an application is not activated, the aircraft not bound (if required), or a legacy version of the SDK (< 4.1) is being used, all camera live streams will be disabled, and flight will be limited to a zone of 100m diameter and 30m height to ensure the aircraft stays within line of sight.
1. Now, let's create a new project in Xcode, choose Single View Application template for your project and press "Next", then enter "PanoDemo" in the Product Name field and keep the other default settings.
Once the project is created, let's import the DJISDK.framework to the project. If you are not familiar with the process of importing DJI SDK using Cocoapods, please check this tutorial: Importing and Activating DJI SDK in Xcode Project
For importing the DJIWidget to the project, you can check our previous tutorial Creating a Camera Application to learn how to download and import the DJIWidget into your Xcode project.
2. In the Main.storyboard, add a new View Controller called CaptureViewController and set it as the root View Controller for the new View Controller you just added in Main.storyboard:
3. Add a UIView inside the View Controller and set it as an IBOutlet called "fpvPreviewView" in the CaptureViewController.h:
4. In the viewDidLoad method, set fpvPreviewView instance as a view of DJIVideoPreviewer to show the Video Stream, then invoke the registerApp method to register the app:
- (void) registerApp { //Please enter the App Key in the info.plist file to register the App. [DJISDKManager registerAppWithDelegate:self]; }
Also, implement the DJISDKManagerDelegate methods to do initial setup after register app success. Moreover, in the productConnected: method, let's fetch a camera object and set its delegate and its playbackManager property's delegate as shown below:
if (![DJISDKManager product]) { returnnil; } return [DJISDKManager product].camera; }
#pragma mark DJISDKManagerDelegate Method
- (void)appRegisteredWithError:(NSError *)error {
NSString* message = @"Register App Successfully!"; if (error) { message = @"Register App Failed! Please enter your App Key and check the network."; }else{ NSLog(@"registerAppSuccess");
5. Build and run the project on your mobile device, then connect your device to the aircraft. Now you can see view the camera's video stream on your device.
Rotating with Virtual Stick
In order to generate a panorama, you need to take a series of photos from different angles while keeping the center of the drone still. This is easy to do with the gimbal on Inspire 1, but with the Phantom 4 you cannot control the gimbal's yaw angle using our SDK. Instead we will rotate the entire drone. Here's the instruction.
1. Add a Capture button in Capture View Controller in Main.storyboard. Name its IBOutlet as captureBtn respectively, and its IBAction as -(IBAction)onCaptureButtonClicked:(id)sender.
2. Next, let's define the angle for each rotation and invoke the rotateDroneWithJoystick method in -(IBAction)onCaptureButtonClicked:(id)sender:
Furthermore, implement the -(DJIFlightController*) fetchFlightController method and configure the DJIFlightController object in the following DJISDKManagerDelegate method:
As the code shown above, we configure the flightController's delegate, and yawControlMode properties. Then invoke the setVirtualStickModeEnabled:withCompletion: method to prepare for the virtual stick control.
3. Using the flightController virtual stick api is similar to sending commands using your remote controller. The virtual stick api can be used to directly specify the pitch, roll, yaw and throttle values of the drone and must be called with a certain frequency(Like 10 Hz) determined by the drone's flight controller, otherwise the flight controller will assume that the connection is lost, and the command may not be executed successfully. Hense, we should use a NSTimer to send virtual stick command in 10Hz as shown below:
- (void)rotateDroneWithJoystick {
for(int i = 0;i < PHOTO_NUMBER; i++){
float yawAngle = ROTATE_ANGLE*i;
if (yawAngle > 180) { //Filter the angle between -180 ~ 0, 0 ~ 180 yawAngle = yawAngle - 360; }
[flightController sendVirtualStickFlightControlData:vsFlightCtrlData withCompletion:^(NSError * _Nullable error) { if (error) { NSLog(@"Send FlightControl Data Failed %@", error.description); } }];
}
You can set up the virtual stick flight control data by setting a DJIVirtualStickFlightControlData structure. As the code shows above, it use a for loop to control the drone to rotate 45 degrees for 8 times, each time the yawAngle will be updated, and assign its value to the corresponding yaw value of DJIVirtualStickFlightControlData:
Also, for DJI Products which have collision avoidance feature, like Phantom 4, Mavic Pro, Spark, etc, we can enable the collision avoidance for virtual stick control by setting YES to the isVirtualStickAdvancedModeEnabled property of DJIFlightController as shown below:
4. Now try to build and run the app, connect your mobile device to the remote controller, switch the flight mode to F (Function) mode, take off the drone and press the Capture button. You should see the drone rotates in 360 degrees, 45 degrees each time.
Here is the gif animation example of using DJI Assistant 2 Simulator to test this feature:
In the code above, we implement the following things:
We create the rotateDroneWithJoystick and setCameraModeToShootPhoto methods to make sure the current work mode of the camera is DJICameraModeShootPhoto in case it's set to other values before we start the 360 degrees panorama shoot photo action using virtual stick. Also, in the rotateDroneWithJoystick method, we firstly check if the current DJI Product is DJI Spark, if so, we invoke the setGestureModeEnabled:withCompletion: method of DJIActiveTrackMissionOperator to disable gesture mode since DJI Spark will enable it by default and the virtual stick control may not be available under gesture mode.
Then we refactor the productConnected: delegate method and create the enableVirtualStick method to configure the yawControlMode, rollPitchCoordinateSystem properties and enable the virtual stick mode of DJIFlightController.
Create the executeVirtualStickControl method and dispatch a new asynchronous thread to execute it in the completion block of setVirtualStickModeEnabled:withCompletion: method. In the executeVirtualStickControl method, we use sleep(2) to sleep for 2 seconds between rotating the drone and shooting photo actions here to ensure each action is executed ordered(Shoot photo action may take around 2 seconds to finish). When the for loop finished, we disable the virtual stick mode and show an alert view in the main thread to notify the user "Capture finished".
Rotating Gimbal to Shoot Photos
If you have an Inspire 1, you will benefit from being able to shoot photos without moving the drone, taking advantage of the gimbal's 360 rotation capabilities. Rotating the gimbal while keeping the drone still in the air means you will have a stable photography platform, which helps you to create a wonderful panorama. Let's take a look at the implementation.
1. let's implement the - (DJIGimbal*) fetchGimbal method to fetch the gimbal component:
2. Next, implement the rotateGimbal method as shown below to rotate the gimbal clockwise from the origin position to 360 degrees(45 degrees each time), then take photos between each rotation:
#pragma mark - Rotate Gimbal Methods - (void)rotateGimbal {
In the code above, we implement the following features:
Set the DJICamera's cameraMode to DJICameraModeShootPhoto, then dispatch a new asynchronous thread to execute the executeRotateGimbal method.
In the executeRotateGimbal method, reset Gimbal's position. It is necessary to reset the gimbal's position before rotating and shooting photos, otherwise the gimbal may reach the maximum angle of yaw axis when rotating 360 degrees.
Use a for loop to shoot photo and rotate gimbal 45 degrees each time. When the for loop finished, we can show an alert view in the main thread to notify the user "Capture Photos finished".
3. Rewrite the onCaptureButtonClicked method as shown below:
Build and run the app, and try the capture button function of the app without taking off the Inspire 1. You should see the gimbal of Inspire 1 rotating 360 degrees and shoot photos smoothly.
Shooting Photos with DJIMutableWaypointMission
Note: Please make sure the drone's battery energy percentage is more than 30% when you use DJIMutableWaypointMission
It seems a bit inconvenient and odd to use sleep(2) between rotating the drone and shooting photos. Shouldn't there be an easier, more efficient way to implement this? Yes, the DJIMutableWaypointMission is designed for executing series of actions within different waypoints, which is perfect for our application!
1. To use the DJIMutableWaypointMission, firstly we should implement the DJIFlightControllerDelegate protocol in the class extension of CaptureViewController.m as shown below:
Here we create an uploadMissionProgressAlert to show the upload mission progress message. The "aircraftLocation", "aircraftAltitude", "gpsSignalLevel" and "aircraftYaw" properties will be updated as the latest flight controller system status of the drone.
Moreover, initialize the aircraftLocation property in the ViewDidLoad method:
As the code shown above, we update the aircraftLocation, gpsSignalLevel, aircraftAltitude and aircraftYaw property values in the DJIFlightControllerDelegate method.
3. Now let's create a new method named missionOperator method to fetch the DJIWaypointMissionOperator object and initialize the DJIMutableWaypointMission by creating a new method named initializeMission as shown below:
[mission addWaypoint:wp1]; [mission addWaypoint:wp2]; [mission setFinishedAction:DJIWaypointMissionFinishedNoAction]; //Change the default action of Go Home to None
In the code above, we create a DJIMutableWaypointMission object firstly and set its maxFlightSpeed and autoFlightSpeed properties. Then we use a for loop to create 16 DJIWaypointAction objects. Each step in the for loop, we create two DJIWaypointActions. Set the first waypoint action type as DJIWaypointActionTypeShootPhoto, the other waypoint action type as DJIWaypointActionTypeRotateAircraft with the appropriate rotate angles. Then add these two waypoint actions to the first DJIWaypoint.
DJIWaypointMission requires at least two waypoints, and each waypoint must have different physical location, so we create another DJIWaypoint and control the drone to fly upwards 1 meter and take no action.
Next, we add the two DJIWaypoint to the DJIWaypointMision object and set its finishAction type as DJIWaypointMissionFinishedNoAction. If we don't set the finishAction property, when the drone finish the waypoint mission, it will execute the go home command automatically instead of staying at the original position.
Furthermore, we invoke the loadMission: method of DJIWaypointMissionOperator to load the waypoint mission into the operator.
Lastly, invoke the addListenerToUploadEvent:withQueue:andBlock: and addListenerToFinished:withQueue:andBlock: methods of DJIWaypointMissionOperator to track the upload and finish events of the waypoint mission and show alert view to inform the user about the mission execution status and progress.
4. Once it's finished, let's create two new methods called - (void)uploadWaypointMission and - (void)startWaypointMission to upload waypoint mission to the drone and start the mission. Here is the code:
In the uploadWaypointMission method, we firstly call the initializeMission method to initialize the DJIMutableWaypointMission. Then we invoke DJIWaypointMissionOperator's following method to upload waypoint mission task to the drone:
In the completion block, we notify users the start mission result by showing an alertView.
5. Since the DJIWaypointMission relies on good GPS signal quality, you should check the GPS signal status before executing the waypoint mission. At the same time, you should also check whether the aircraftLocation is valid. Let's implement the rotateDroneWithWaypointMission method as shown below:
Build and run your code, take off the drone and fly to an appropriate altitude and press the capture button to execute the waypoint mission. You should see the drone start to rotate and shoot photos automatically.
So far we have three methods to rotate the drone and shoot photos, we had better create an alert view to ask the user to choose which method to control your drone.
Let's update the onCaptureButtonClicked method as shown below:
Now the user can choose their preferred methods to take 360 degrees Panorama photos.
Selecting and Downloading Photos
Using Playback Mode
In order to download multiple photos, you should go through a series of playback modes. Firstly, enter Playback mode, then enter Multiple Preview mode, furthermore enter Multiple Edit mode, lastly, select all the photos you just capture, and download them.
1. Add a new Download button in the CaptureViewController of Main.storyboard, then create its IBOutlet named downloadBtn and IBAction as -(IBAction)onDownloadButtonClicked:(id)sender in the CaptureViewController.h file as shown below:
2. Add a new variable selectedPhotoNumber to record the number of photos selected in the class extension of "CaptureViewController.m" file:
@property (nonatomic, assign) __block int selectedPhotoNumber;
Now, let's implement the DJIPlaybackDelegate method as shown below to update the selected photo num:
Here we invoke the setMode:withCompletion: method to set the camera mode to DJICameraModePlayback . If it succeeded, we can invoke the selectPhotosForPlaybackMode method to select photos.
4. Once it's done, let's implement the selectPhotosForPlaybackMode method to select the latest photos you have captured for the panorama:
It takes a few seconds for the drone to respond to commands, so you should dispatch an asynchronous thread to send commands and call the sleep() between them in case you freeze the UI interaction of in main thread. As shown in the flow diagram above, you should enter MultiplePreviewMode and MultipleEditMode before selecting photos. But how do we select the lastest 8 photos for panorama? Here is our strategy:
The default selected photo is the last photo. Select all photos in the last page. If you have selected more photos than you need, you can deselect the extra photos starting from the index 0. If you do not have enough photos selected, just go to the previous page and repeat the procedure again until you select the exact number of photos you needed. Here is the gif animation for selecting photos:
5. After you select all the photos, invoke the downloadPhotosForPlaybackMode method to download all the photos you need.
Create and implement the downloadPhotosForPlaybackMode method as shown below:
-(void)downloadPhotosForPlaybackMode { __block int finishedFileCount = 0; __block NSMutableData* downloadedFileData; __block long totalFileSize; __block NSString* targetFileName;
self.imageArray=[NSMutableArray new];
DJICamera *camera = [self fetchCamera]; if (camera == nil) return;
In the code above, we firstly add several variables and init the imageArray object. Then call the DJIPlaybackManager's following method to download the selected photos:
In the DJIFileDownloadPreparingBlock, we initialize the "downloadedFileData" and dispatch a main thread to show alertView to user for notifying the current download progress. Then we append the "downloadedFileData" in the DJIFileDownloadingBlock and update the "downloadProgressAlert" message with the current process in the main thread.
In the DJIFileDownloadCompletionBlock, we increase the "finishedFileCount" property value and save the downloaded photo image in the imageArray. in a local album in the DJIFileDownloadCompletionBlock.
Finally, let's notify the users the download result by showing UIAlertView in the main thread in the overallCompletionBlock. Moreover, set the camera mode back to "DJICameraModeShootPhoto" after the photodownload finished.
Build and run the code, press the capture button to rotate the drone and shoot photos, after it finished, press download button to download all the photos shoot. If everything goes well, you should see the following animation:
Using Media Download Mode
In order to download multiple photos using Media Download Mode, you should make good use of the DJIMediaManager and DJIFetchMediaTaskScheduler to refresh media file list and schedule the tasks.
Firstly, enter Media Download mode, then refresh the media file list from the SD card, furthermore resume the DJIFetchMediaTaskScheduler, moreover, create multiple DJIFetchMediaTask for the photos to download, lastly, enqueue the tasks to the scheduler to execute and download photos.
Now, let's improve the onDownloadButtonClicked: method as shown below:
Here we firstly check if the DJICamera support media download mode, and invoke the setMode:withCompletion: method to set the camera mode to DJICameraModeMediaDownload. If it succeeded, we can invoke the loadMediaListsForMediaDownloadMode method to select photos.
Next, let's implement the loadMediaListsForMediaDownloadMode method as shown below:
In the code above, we invoke the refreshFileListOfStorageLocation: method of DJIMediaManager to refresh the file list from the SD card. If there is no error, invoke the downloadPhotosForMediaDownloadMode method to download photos.
Once you finish the steps above, let's implement the downloadPhotosForMediaDownloadMode method as shown below to download photos:
-(void)downloadPhotosForMediaDownloadMode { __block int finishedFileCount = 0;
In the code above, we implement the following features:
Firstly, we initialize an NSMutableArray imageArray, which will be used to stored the downloaded images. And then invoke the sdCardFileListSnapshot method of DJIMediaManager to get the current DJIMediaFile file list, and store them in the files array object. After that, we check if the current media files' count is less than PHOTO_NUMBER, which is the photo count to create a panorama. If so, inform the user that not enough photos are taken using a UIAlertView and return.
Moreover, we invoke the resumeWithCompletion: method to resume the DJIFetchMediaTaskScheduler and inform the user that resume file task scheduler failed using a UIAlertView.
Lastly, we create a for loop and create 8(The value of PHOTO_NUMBER) DJIFetchMediaTask objects by invoking the taskWithFile:content:andCompletion: method of DJIFetchMediaTask class. Inside the completion block of the method, we firstly check if any error exists and show a UIAlertView to inform users. If not, access the preview property of DJIMediaFile to get the preview image for this media and add it to the imageArray array. Next, increase the value of finishedFileCount by 1 and show the media file download status message by using the downloadProgressAlert. If the value of finishedFileCount reach 8, we show a UIAlertView to inform the user that the download complete and set the camera mode back to DJICameraModeShootPhoto. Lastly, we invoke the moveTaskToEnd: method of DJIFetchMediaTaskScheduler to push the newly created task to the back of the queue for executing.
Build and run the code, press the capture button to rotate the drone and shoot photos, here we use Spark for testing. After it finished, press download button to download all the photos shoot. If everything goes well, you should see the following animation:
Using OpenCV to Create Panorama
Creating Stitching View
Since we don't need the FPV view when stitching photos, we can create a new view controller called StitchingViewController to generate and present the panorama photo.
1. Embed the CaptureViewController in a navigation controller and add a new button called Stitch on it.
Then create a new view controller called StitchingViewController and add it in the Main.storyboard correspondingly. Then add a button named Stitch in the CaptureViewController and set its action segue to show the StitchingViewController. Set the segue's identifier to Stitching.
2. Let's add an instance variable imageArray in the StitchingViewController.h:
Then add the prepareForSegue method to pass the downloaded photos to the next view controller in CaptureViewController.m:
//Pass the downloaded photos to StitchingViewController -(void)prepareForSegue:(UIStoryboardSegue *)segue sender:(id)sender { if([segue.identifier isEqualToString:@"Stitching"]) { [segue.destinationViewController setValue:self.imageArray forKey:@"imageArray"]; } }
3. Add an activity indicator in the StitchingViewController in Main.storyboard, set its behavior to Hides When Stopped, then set its outlet as activityIndicator in the StitchingViewController.h file:
Importing OpenCV
1. Build the opencv2.framework for iOS. You can check for more details in Installation OpenCV in iOS. If you have any difficulties compiling the source code, you can just download this project and copy the opencv2.framework.
2. Select the project target, go to Build Phases -> Link Binary With Libraries. Click the "+" button at the bottom and add theAssetsLibrary.framework library to your project. Click the "+" button, click on Add others and navigate to the directory where opencv2.framework is located and click open.
3. Add a PCH file into your project. Go to Build Setting -> Precompile Prefix Header, set its value YES, and set Prefix Header to be the location of the PCH file.
Open your PCH file and add the following lines of code:
4. Build your project. If nothing wrong happens, you may successfully setup the OpenCV library in your Xcode project.
Stitching Photos
1. In the OpenCV, all the image processing operations are usually carried out on the Mat structure. However, on iOS, to render an image on screen it has to be an instance of the UIImage class. In order to convert an OpenCV Mat to an UIImage we can use the Core Graphics framework in iOS. Find more details, please check Convert UIImage to cv::Mat. You can define a OpenCVConversion class to deal with the conversion issue.
Let's implement the OpenCVConversion.h file shown as below:
@interfaceOpenCVConversion : NSObject
+ (cv::Mat)cvMatFromUIImage:(UIImage *)image;//convert UIImage to cv::Mat + (cv::Mat)cvMatGrayFromUIImage:(UIImage *)image;//convert UIImage to gray cv::Mat + (cv::Mat)cvMat3FromUIImage:(UIImage *)image; //convert UIImage to cv::Mat without alpha channel +(UIImage *)UIImageFromCVMat:(cv::Mat)cvMat;//convert cv::Mat to UIImage
2. In order to call OpenCV functions in objective-C, you should create a header file and a C++ file as the stitching wrapper file. Input a vector of cv::Mat and the reference of cv::Mat. This function will tell whether the stitching is successful and return the stitched cv::Mat through a result reference. The code in StitchingWrapper.h is shown below.
if (status != Stitcher::OK) { returnfalse; } returntrue; }
3. Now you can customize your stitching method in the new class called Stitching. Here is the class method declaration for stitching implemented in the Stitching.h file. Users can input an image array and a reference of cv::Mat, it will return the stitching result:
Because of the limited computing ability and memory storage of mobile devices, we don't suggest you to process full resolution images, because the stitching process may take a long time or cause run out of memory problem. We call the + (UIImage *)compressedToRatio:(UIImage *)img ratio:(float)ratio method to reduce the photo width and height by a ratio of HIGHT_COMPRESS_RATIO or LOW_COMPRESS_RATIO, which reduces the resolution dramatically(Like HIGHT_COMPRESS_RATIO2). If the image size is not small(The height is less than 1000), we use LOW_COMPRESS_RATIO, otherwise, use HIGHT_COMPRESS_RATIO.
Then we convert the images to cv::Mat and push them into cv::vector. Finally, we can call the stitch function in the StitchingWrapper.cpp file to do the actual stitching work.
4. Now add an UIImageView in the Main.storyboard, set its Mode to Aspect Fit and create its outlet, called imageView in the "StitchingViewController.h" file.
Replace the StitchingViewController.mm with the following code:
if(isTopNotBlack && isLeftNotBlack && isBottomNotBlack && isRightNotBlack) { cv::Mat imageReference = src(roiRect); imageReference.copyTo(dest); returntrue; } // If not, scale ROI down // if x is increased, width has to be decreased to compensate if(!isLeftNotBlack) { roiRect.x++; roiRect.width--; } // same is valid for y if(!isTopNotBlack) { roiRect.y++; roiRect.height--; } if(!isRightNotBlack) { roiRect.width--; } if(!isBottomNotBlack) { roiRect.height--; } if(roiRect.width <= 0 || roiRect.height <= 0) { returnfalse; } } }
The bool checkBlackRow(const cv::Mat& roi, int y) function checks whether the number of black pixels in row y is more than CUTBLACKTHREASHOLD, while the bool checkBlackColumn(const cv::Mat& roi, int x) method checks the same for column x. These two functions indicate whether the row or column is part of a black edge that we don't need.
+ (bool) cropWithMat: (const cv::Mat &)src andResult: (cv::Mat &)dest converts the src cv::Mat to gray cv::Mat and initializes the roiRect to be the frame of src cv::Mat. Then it checks each of the four edges of the src Mat in roiRect. If an edge is black, it narrows the corresponding edge of roiRect and repeats checking and narrowing until none of the edges are black. Finally, it will copy the pixels in the roiRect of src to dest to complete the cropping process shown below.
Build and run the app, shoot a series of photos, download them and stitch them together. You should generat a perfect panorama now! Here is an example:
Summary
In this tutorial, you’ve learned how to use the Virtual Stick feature and the WaypointMission feature of Intelligent Navigation to control the aircraft to rotate and take photos. Also, you learn how to rotate the gimbal to take panorama photos too. Lastly, we used the OpenCV's features to stitch and crop photos into a cool panorama!
Congratulations! Now that you've finished the demo project, you can use what you have learnt to start building your own panorama applications. You can improve the project by showing the aircraft's flight mode type, current GPS satellite count, vertical and horizontal flight speed and the flight altitude, etc. In order to make an amazing Panorama Application, you still have a long way to go! Good luck, and hope you enjoyed this tutorial!
