Lidar Detection of Images

The Problem

The client was a company that had invented a unique system to scan houses for possible energy remediation. It created a device that was mounted on cars and took thousands of pictures of city streets, much like Google Streetview. Unlike StreetView, though, this device took near-infrared (NIR) and LiDAR (a pulsed laser technology) images. The purpose was to scan houses to determine which would benefit the most from energy remediation services. The client was positioning itself to sell these images to local energy companies, which would then be able to optimize their remediation efforts to the houses that would benefit the most.  

The client’s vehicle-based data collection platform records data from thousands of buildings per evening. Determining which portions of the data localize individual buildings was a challenging computer vision and geographic information system effort. The challenge required:

• Using pre-existing training data to automatically determine the boundaries of buildings – specifically houses
• Determining depth and dimensions of such buildings using calibrated LiDAR system
• Isolating pixels of the structure

Long-wave infrared and near-infrared cameras record nearly one million images of energy loss scenes nightly. These co-collected images differ by scene coverage, but this is reconciled by time of collection and by overlapping features in 3D point clouds. Each building has a limited set of features (less than 20) that make up the most interesting parts of the energy analysis computations. Features include windows, doors, walls, soffits, roofs, etc. Finding such features can be a challenge, depending on the type of construction, geography, image condition, and obstructions. All available data including images and point clouds may be needed to determine these features. Tens of thousands of features have been tagged for training data. The challenge included mapping pixels between cameras using 3D features obtained from LiDAR data.

The Approach

Our expert was asked to create a neural network to establish baseline machine learning performance for recognizing buildings in Near-infrared (NIR) data, as a starting point for more advanced image recognition. To perform this baseline, she normalized image data and trained a variety of quickprop, resilient backprop, conjugate gradient, and standard back-propagation networks. She trained on the NIR data in various formats, in both normalized and un-normalized forms, as well as with image subsampling. The expert began by benchmarking in open source R. The benchmark indicated that the full training would require several hundred hours to train. To address this performance issue, she switched to using the Emgu framework – a .NET wrapper to the OpenCV image processing library – to process image data into a binary format. She then used this format as an input to the Encog machine learning framework, where she built a small menu-based system that used trained networks with a variety of parameters. She conducted several dozen training runs with various parameters and network sizes, and comparing a variety of algorithms. The resulting network can now be used as a baseline for performance (both in terms of training time as well as network performance) with more advanced algorithms, such as convolutional networks and Deep Learning.