The Advanced Sensing - Stereo depth perception sample demonstrates how to rectify front-facing stereo images, calculate disparity map, and unproject 3D point cloud using OpenCV and CUDA. The CMakeLists of this sample will detect if developers have OpenCV or CUDA installed in their system. OpenCV is required for image processing. CUDA is optional and used for accelerating the computation. To have the best user experience, this sample also make use of the ximgproc module from OpenCV contrib module to post-filter the disparity map.
To avoid compatibility issue between three third-party libraries: OpenCV, OpenCV Contrib, and CUDA, we suggest developers to use OpenCV and OpenCV Contrib version 3.3.0+ and CUDA 8+. Please see appendix below for instructions on installing these libraries.
To build the samples, follow the steps in Sample Setup to build with Advanced Sensing support.
In ROS, an extended sample with object detection is provided, please visit here for more information.
Before using this sample, please calibrate your front-facing stereo cameras.
See appendix below for more information. With the calibrated parameters,
please modify the
m210_stereo_param.yaml file located inside the sample folder accordingly. These parameters
will be parsed by the sample and be used for image rectification, stereo matching
and point cloud projection.
This sample is built on top of the multi-thread sample.
StereoProcessContainer inherited from
ImageProcessContainer is provided to store front-facing
VGA images on the reading thread and to process them on processing thread. Several C++ classes are provided for image
To run this sample, please execute the following command:
This sample provides four incremental steps that will ultimately lead to point cloud projection:
As shown below, images from left and right cameras are concatenated side by side and epipolar lines are drawn on the image to visualize the result of rectification.
With rectified stereo images, disparity map can be generated. Here we use OpenCV StereoBM API to compute the stereo correspondence. The default disparity range is 64 and the block size is 13.
The disparity map generated from block matching algorithm might perform poorly in scene that has texture-less areas, object occlusions, and depth discontinuities. Here we include the ximgproc module in OpenCV Contrib to post-filter the disparity map. To learn more information about this API, please visit this page.
Finally, with disparity map and camera parameters, we can obtain 3D information of each pixel to generate a point cloud. Here we use Viz3d module in OpenCV to visualize the point cloud.
With the point cloud, we have 3D information of each pixel. Combine this information with object detection algorithm, we can obtain the 3D location of the object in the image relative to the optical center of the camera. Please visit object depth perception sample for more information.
This sample utilizes three libraries to perform image processing.
Different versions might not be compatible with each other.
We recommend using
OpenCV Contrib 3.3.1, and
Please follow this page for detailed instructions.
OpenCV Contrib 3.3.1
We first checkout 3.3.1 version, OpenCV core library will include and build it.
An example cmake command is shown below, please make according changes.
cmake -D CMAKE_BUILD_TYPE=RELEASE \
There are several libraries online that provide source code and tutorial on how to calibrate stereo cameras. Here we lists some of them for developers.
Camera_calibration package from ROS
OpenCV calib3d API (Example source code)