AI or algorithmic ways to modify vegetation in a 3D model while keeping it photoreal?
Hey everyone,
I'm flying a DJI Matrice 4D and processing the data in DJI Terra (orthophoto, DSM, textured mesh / B3DM). For some projects I'd like to change the vegetation in the resulting 3D model while keeping the rest of the scene as photorealistic as possible.
Example: a site is captured in summer with full foliage, but I want to show the same area in winter (snow) colors, with denser forest, different tree species, or partially cleared. Basically swap or modify the green stuff without the result looking like an obvious CGI paste-in.
Another problem is, that the trees are not looking good when zooming in (video), for the projects I'm doing, it would rather keep the flight height at about 120m AGL, otherwise it takes ages in flight-time.
A few things I'm wondering about:
- Any AI-based workflows that can isolate vegetation in a textured mesh (segmentation + inpainting, diffusion-based texture replacement, etc.) and replace it convincingly?
- Algorithmic approaches that detect vegetation first (NDVI on the ortho, height filtering via DSM, mesh classification) and then substitute it with library assets like SpeedTree or Quixel that actually blend with the surrounding terrain?
- Has anyone found a pipeline that handles the transition between modified vegetation and untouched ground without visible seams?
I've been experimenting in Blender but the moment I drop in standard 3D tree assets it immediately looks fake next to the photogrammetry texture. Curious if anyone has cracked this, either with commercial tools, research projects, or some clever Blender/Unreal hack.
Cheers