Implement split-merge processing

[spwoodcock]

Original plan (TBC):

• Accept optional images.json containing EXIF data, to speed up workflow.
• Do splitting based on images.json metadata.
• Spawn multiple standard ContainerSet workflows, but using an artifact JSON containing the URLs to download (s3 paths).
• Run the many ContainerSets for processing each stage.
• Final stage depends on all returned results --> merge dataset, then upload.

[spwoodcock]

More details here: hotosm/drone-tm#640

[spwoodcock]

• In Freetown we determined the z-values can be really far off in the raw drone data. If altitude / height data is poor, the vertical dimension of the reconstruction is inaccurate.
• So the current process determined from processing 80km2 in Freetown is to process each flight point cloud individually, snap those to the global DEM, then run the ODM split-merge process on the corrected data.

Processing using cascading point cloud alignment

The proposed plan using EC2 instance, before this is put into ScaleODM.

1. Select a first task / flight to process, ideally central to the whole project and with accurate georeferencing (++ for GCP). Process point cloud + DEM for the first flight, approx 500 photos.
2. Process 4 adjacent flights in N-S-E-W directions, using the point cloud from flight 1 and the align flag (--align reference_point_cloud.laz) in ODM (to align on the first point cloud).
3. Continue this as a transect horizontal and vertical cascading out.
4. 'Fatten' the transects out by processing outwards, to capture all the tasks.

For ScaleODM essentially EC2 in the workflow would be replace by a container, which is scheduled on whatever node is available to process it.

Note

For now we need to download the point cloud from the previous task to do alignment.
Ideally in future ODM + ScaleODM would support Cloud Optimised Point Cloud, simply querying the S3 data already pushed.

Potential issue: alignment drift

• As we use a central 'anchor' point, a lot depends on the accuracy of the point.
• Errors can also cascade out as we process.
• An approach to mitigate this for a certain size project could be considering adding multiple 'anchors' throughout the project area, having them all converge together and do addition correction somehow?
• Mitigations:
  1. Periodically align to the global DEM.
  2. Keep overlaps large enough to not introduce too much error.

Automating the process

1. Process each task individually, using point cloud correction from adjacent tasks.
2. Run the merge portion of ODM split-merge (as DroneTM essentially takes care of the 'split' portion.

• The 'split' part of split merge typically divides up a larger area into overlapping spatial clusters, giving us multiple ODM tasks to work on.
• DroneTM essentially does this with it's 'tasking' model, giving us a naturally split project.
• So we can odm_split_merge --merge --> products across entire project.
  • Dense point clouds: merged using PDAL + filtering.
  • DEMs: merged using GDAL warp + mosaic
  • Orthos: merged using GDAL merge / VRT

Summary:
STEP 1: Process central flight
STEP 2: Process adjacent flights with --align
STEP 3: Cascade outward
STEP 4: DEM snap correction
STEP 5: Run ODM merge

[spwoodcock]

Updated notes from Steve on how this was achieved for Freetown v1

LAZ Alignment Approach

The approach described above with aligning against the overlapping flight point cloud works well, except error in alignment propagates along whatever axis you are aligning. Compounded overtime, you get an elevation model that looks like this:

A smarter approach would be to calculate the alignment matrices in some aggregate way between all the flights, and choose a compromise, maybe least squares or least error of some sort, but basically find a global solution with the errors distributed evenly across the landscape.

Simpler DEM Correction Approach

In aggregate, across hundreds of images, across flights across multiple days, the aggregate error in hundreds or thousands of images is fairly low.

However, as we use low cost drones, this is typically only true in XY, not in Z (due to the instruments on board).

So an approach we can use is:

1. Run all the datasets per flight.
2. Calculate differences with JAXA/SRTM or better yet MERIT DEM.

• As the gloal DEM is fairy low resolution, we only have so many sampling points within a flight.
• We could just choose one value towards the middle of the flight.
• If multiple sampling points are possible, we could try an approach to average differences across a few sampling points.

3. Then use that difference with MERIT to apply a single Z correction factor to each flight.

• Here it's probably a good idea to verify some pixel values in images against the XY values of the global DEM.
• If the XY coord difference isn't close to being zero relative to the 90m MERIT pixels, then there is probably an issue with the flight & it needs to be re-flown - validation step & user feedback on failure!
• Also, "one disadvantage to MERIT is it is a heavily synthesized dataset with hydrologic flow assumptions built in... so comarison against raw datasets such as SRTM/JAXA/TanDEM-X etc could be more useful".

4. Correct EXIF metadata tags for elevation in all source images (a bit of a pain - requiring re-upload and overwrite of the original source). Perhaps we can encode a custom EXIF value containing the original Z value as a backup?
5. Reprocess data, but in larger blocks a few thousand images or less. Merge the larger orthos together into a single stitched ortho.

Note

Up to 13,000 images in a single block is possible, but anything above maybe 5-7000 images starts to get very slow.
After updates to OpenSfM this might change, but for now we can target about 5000 images per block?

The smaller the chunk, the more seams on the final image, the more possible alignment issues, and possible higher aggregate XY error (error that seems to wash out with larger datasets).

Also we are proposing doing compact groups of processing to better minimize seams, distribute x and y errors, ensure robustness to rotations. But we have never tested "making each block a strip that goes the entire length of the region (e.g. one east-west row of tasks), start at the top (north) and align each successive row to the south of the previous one".
Doing this might only matter for more seams and definitely produce more predictable seam locations. It might not matter for error at all, but to be tested.
Also note: seam locations and lengths don't matter if alignment, cutlines, and blending are good.

Note

To merge point clouds we can use PDAL: https://github.com/PDAL/wrench
DEMs can be blended with: https://github.com/OpenDroneMap/ODM/tree/master/contrib/dem-blend
Which extracts the equivalent step for elevation models.
It's a fully ported (and optimized) version of the dem-blend step in this dedicated tool: https://github.com/localdevices/dem_blender

Summary

• It's pretty easy to do a big chunk of the processing via the discrete dronetm task areas - distributed parallel processing across the cluster.
• When it comes to processing the bigger chunks, we need bigger machines and this becomes more complex. It's recommended to have 256GB RAM for processing 5000 images. An example machine in AWS would be r7i.8xlarge, 256GB & 32 vCPUs, costing about $2/hr. Best case, we automate this step too. Worst case, we have a bunch of inputs for a smaller manual processing step on a big machine. Assessment of spec requirements link.