In my previous blog, I talked about how I modified AirSim to capture full-view panorama RGB and depth images from Unreal Engine. Later, I found out that the depth images have some artifacts near the boundaries of the objects. It turns out that when Unreal Engine is unwarping the cube rendered textures to a panorama image, there are interpolations. It is OK, or the desired behavior, for RGB images because interpolation makes the values of pixels smooth and visually appealing. However, it is an undesired feature for depth images since interpolation brings about “ghost” points floating between the foreground and background objects.
To reduce this effect, I thought nearest-neighbor interpolation may be a better choice when generating the full-view panorama from the cube rendered textures. But this means to perform the unwarping operation by myself and interact with the render targets directly.
Again, it took me some time to accomplish it and I learned new stuff during the adventure of, again, hacking the source code of AirSim.
2 Modification overview
The first thing that came to my mind is using OpenCV’s
remap() function to perform a nearest-neighbor sampling from the 6 faces of the cube render target. So I implemented and tested a standalone program that takes 6 images and generates a full-view panorama.
The next thing is making Unreal Engine and AirSim work with OpenCV. This is much harder than I originally thought. It gave me lots of pain just making Unreal Engine happy with linking against OpenCV’s libraries.
The rest is adding new routines to AirSim and make it use the new nearest-neighbor interpolation when creating panorama depth images.
Note for Windows and macOS users, the following instructions make the AirSim work only on Linux.
3 The standalone unwarp program
To test the basic idea of unwarping the 6 faces of the cube render target into a panorama, I first write a sample code and build a standalone program. The program copies 6 floating-point value images into an internal image cross representation. Then an output panorama is generated based on the image shape specification. The internal image cross representation and the resulting panorama are shown in the following two figures.
At this time, the OpenCV used is the one installed in the system by
apt install. OpenCV compatibility will be handled in the next section. Later, a static library is separated from this standalone program. The code for the static library is copied into the AirSim source folder under a new directory called “CustomizedInterpolation”.
4 Add the unwarp program to AirSim
4.1 Prepare OpenCV
As I said, this is really, much harder than I thought. There is a blog talking about how to use OpenCV with Unreal Engine. But on my computer, the OpenCV’s
check() function conflicts with the one defined by Unreal Engine. The blog author suggests commenting out OpenCV’s
check(), however, it makes other parts of OpenCV unhappy. Later I found out that there are trickier issues associated with the build toolchain. The OpenCV packages installed by
apt install are built by linking against the
libstdc++ libraray, which uses a different
std::string implementation with that adopted by Unreal Engine’s toolchain, in this case, clang-8.
Then, I decided to compile OpenCV by myself. This costs me a long time to finally figure out all the essential steps to make things work. I’ll just summarize the procedures.
First, make sure we have the clang-8 toolchain set up properly.
sudo apt-get install -y clang-8 clang++-8 libc++-8-dev libc++abi-8-dev
Actually, the above command is already executed by the
setup.sh script of AirSim.
Then, clone the OpenCV source code and pick an appropriate tag. I choose 4.2.0 because it is the version my current system uses. Create a build directory and an install directory as usual. We only need an OpenCV package that delivers the essential functionalities. To reduce third-party dependency as much as possible, we will disable lots of irrelative features of OpenCV. The CMake configuration parameters are listed as follows. (Go to OpenCV’s offical documentation for more details.)
cmake -DCMAKE_C_COMPILER=/usr/bin/clang-8 -DCMAKE_CXX_COMPILER=/usr/bin/clang++-8 \ -DCMAKE_CXX_FLAGS="-stdlib=libc++" \ -DBUILD_TESTS=OFF \ -DBUILD_PERF_TESTS=OFF \ -DBUILD_EXAMPLES=OFF \ -DBUILD_opencv_apps=OFF \ -DBUILD_SHARED_LIBS=OFF \ -DBUILD_ZLIB=ON \ -DWITH_1394=OFF \ -DWITH_ADE=OFF \ -DWITH_FFMPEG=OFF \ -DWITH_GSTREAMER=OFF \ -DWITH_GTK=OFF \ -DWITH_IMGCODEC_HDR=OFF \ -DWITH_IMGCODEC_PFM=OFF \ -DWITH_IMGCODEC_PXM=OFF \ -DWITH_IMGCODEC_SUNRASTER=OFF \ -DWITH_ITT=OFF \ -DWITH_JASPER=OFF \ -DWITH_JPEG=OFF \ -DWITH_LAPACK=OFF \ -DWITH_OPENCL=OFF \ -DWITH_OPENCLAMDBLAS=OFF \ -DWITH_OPENCLAMDFFT=OFF \ -DWITH_OPENEXR=OFF \ -DWITH_PNG=OFF \ -DWITH_PROTOBUF=OFF \ -DWITH_QUIRC=OFF \ -DWITH_TIFF=OFF \ -DWITH_VTK=OFF \ -DWITH_V4L=OFF \ -DWITH_WEBP=OFF \ -DOPENCV_DNN_OPENCL=OFF \ -DCMAKE_BUILD_TYPE=Release \ -DBUILD_LIST=core,highgui,imgproc,imgcodecs \ -DCMAKE_INSTALL_PREFIX=install_dir \ opencv_dir
After configuring OpenCV, build it as usual
make -j4 make install
The installed libraries are saved at
install_dir/lib, some thirdparty libraries are built and saved at
~/.zshrc if you’re using zsh) to add a new environment variable
~/.bashrc to make the new environment variable available to the current terminal (or open a new terminal).
4.2 Compile the static library
Now we could do a
git pull in the directory of the modified AirSim repository and checkout the
CubeNearest branch. In this branch, I added the unwarp codes to a separate directory called
CustomizedInterpolation and added the associated CMakeLists.txt file to the
cmake directory. Some modifications are added to the
build.sh script to copy and synchronize the files of
CustomizedInterpolation. Just run
build.sh as usual.
build.sh will try to build the new target we have just added. Make sure that the OpenCV is found in the install directory that is specified in the previous section. Here is the output of
build.sh on my current computer.
Upon successful execution of
build.sh, header files and a static library will be copied to the
Unreal/Environments/Blocks/Plugins/AirSim/Sources directories, just as the rest of AirSim components do. However, an additional operation is needed here to bring all the static libraries of OpenCV to
Unreal/Environments/Blocks/Plugins/AirSim/Source/CustomizedInterpolation/lib. This step is performed manually at the moment. When copying the OpenCV libraries, remember to also copy the third-party ones. On my computer, the final list of static libraries in
4.3 Build the AirSim Unreal Plugin
Now, as usual, we could use Visual Studio code to open the Blocks project and perform a build.
A new parameter is added to the capture settings. Now in the settings.json file, for CubeDepth image type, we could use
CubeNearest to turn on or off the customized interpolation for depth panorama images.
Previously, because Unreal Engine applies interpolation among the floating-point values, the final depth panorama image contains artifacts. These artifacts are best visualized by converting the depth image into a 3D point cloud. The following image is the result. We could clearly see that there are ghost points floating between the foreground (e.g. trees) and the background objects (e.g. far-away buildings and the sky).
By using the nearest neighbor interpolation, the depth panorama is more cleaner and the associated point cloud shows no ghost point anymore.