Differentiable Rendering

Differentiable rendering can be used to optimize the underlying 3D properties, like geometry and lighting, by back-propagating gradients from the loss in the image space. Kaolin Library integrates techniques from DIB-R and DIB-R++ published techniques, as well as follow up improvements. Many Kaolin utilities also integrate with an alternative nvdiffrast differentiable rendering utility, also from NVIDIA.

We provide an end-to-end basic tutorial using the kaolin.render.mesh functionality for mesh optimization in examples/tutorial/dibr_tutorial.ipynb. See also Differentiable Camera, Differentiable Lighting, and Easy PBR Shader.

References

• DIB-R: “Learning to predict 3d objects with an interpolation-based differentiable renderer.” Chen, Wenzheng, Huan Ling, Jun Gao, Edward Smith, Jaakko Lehtinen, Alec Jacobson, and Sanja Fidler. NeurIPS 2019.

• “DIB-R++: learning to predict lighting and material with a hybrid differentiable renderer.” Chen, Wenzheng, Joey Litalien, Jun Gao, Zian Wang, Clement Fuji Tsang, Sameh Khamis, Or Litany, and Sanja Fidler. NeurIPS 2021.

• Nvdiffrast: “Modular primitives for high-performance differentiable rendering.” Laine, Samuli, Janne Hellsten, Tero Karras, Yeongho Seol, Jaakko Lehtinen, and Timo Aila. SIGGRAPH (TOG) 2020.