Easy PBR Shader (USD, gltf, obj)¶
It can be challenging to compose custom differentiable rendering pipelines, especially given the wide
range of conventions, and this step can take some ramp-up time for research projects.
With Kaolin Library v0.16.0, we are piloting a standard differentiable
shader kaolin.render.easy_render.render_mesh that works out-of-the-box for many meshes,
supporting Spherical Gaussian lighting, and a partial set of physics based rendering (PBR) material maps, including albedo
textures, roughness and specular workflows and normal maps.
See End-to-end Tutorial: examples/tutorial/easy_mesh_render.ipynb.
Caution
There are many differentiable rendering variants, and this particular module has not yet been research-tested for specific applications. We did find it robust across many in-the-wild meshes any welcome your feedback on our GitHub. Please cite Kaolin if this is useful in your research (citation).
Consistent Convensions for USD, GLTF, GLB, OBJ¶
One of the issues with differentiably rendering existing meshes is that input file formats as well as readers into tensors used in deep learning frameworks like PyTorch have a wide range of
convensions. Since v0.16.0, Kaolin has started to make the effort to make meshes imported from .USD, .GLTF, .obj using
kaolin.io.mesh.import_mesh follow consistent
convensions, both for imported geometry, represented as kaolin.rep.SurfaceMesh and materials, represented as kaolin.render.materials.PBRMaterial.
Given staggering number of subtleties, we cannot guarantee consistency or full support for many aspects of these formats, but
latest release is a step toward enabling 3D DL researchers to more easily integrate real meshes into
their research, rather than creating them from scratch. For example, we would like to enable research into iterative workflows for artists, allowing use of
traditional authoring tools in conjunction with new AI algorithms. To load a mesh into PyTorch (see also Working With Surface Meshes):
mesh = kaolin.io.import_mesh('usd_gltf_or_obj_path', triangulate=True).cuda()
Standard Differentiable PBR Shader¶
To render imported (or constructed) triangular meshes (note that you can specify triangluate=True on import), simply call kaolin.render.easy_render.render_mesh
to output full composited pass, as well as normals, albedo, specular, alpha and face_id passes. You will first need to create kaolin.render.camera.Camera
(see Differentiable Camera) or use a default:
# Use default camera, lighting or construct your own
camera = kaolin.render.easy_render.default_camera(512).cuda()
lighting = kaolinl.render.easy_render.default_lighting().cuda()
# Render
render_res = kal.render.easy_render.render_mesh(in_cam, mesh, lighting=lighting, **kwargs)
Tip
Both DIB-R and nvdiffrast pipelines are supported, by passing in backend="cuda" and backend="nvdiffrast", respectively.
See documentation for all available options.
A Note on the Backends¶
See Differentiable Rendering for background on different backends. As noted above, the easy_render.render_mesh
supports two back ends. We recommend using nvdiffrast. If this library is installed, a default CUDA based context will be created and reused on the provided GPU device.
Note that not all nvdiffrast capabilities (like anti-aliasing) are currently available through the high-level function.
Interactive Jupyter Viewing¶
The rendering function is fully compatible with Kaolin Interactive 3D Viewer, enabling easy inspection of rendering passes right in your debugging notebook:
(here showing differentiable rendering of a Sea Urchin Shell by Drakery from Sketchfab).