SurfaceMesh

Tutorial

For a walk-through of kaolin.rep.SurfaceMesh features, see working_with_meshes.ipynb.

API

class kaolin.rep.SurfaceMesh(vertices: Union[FloatTensor, list], faces: Union[LongTensor, list], normals: Optional[Union[FloatTensor, list]] = None, uvs: Optional[Union[FloatTensor, list]] = None, face_uvs_idx: Optional[Union[LongTensor, list]] = None, face_normals_idx: Optional[Union[LongTensor, list]] = None, material_assignments: Optional[Union[Tensor, list]] = None, materials: Optional[list] = None, vertex_normals: Optional[Union[FloatTensor, list]] = None, vertex_tangents: Optional[Union[FloatTensor, list]] = None, vertex_colors: Optional[Union[FloatTensor, list]] = None, vertex_features: Optional[Union[FloatTensor, list]] = None, face_normals: Optional[Union[FloatTensor, list]] = None, face_uvs: Optional[Union[FloatTensor, list]] = None, face_vertices: Optional[Union[FloatTensor, list]] = None, face_tangents: Optional[Union[FloatTensor, list]] = None, face_colors: Optional[Union[FloatTensor, list]] = None, face_features: Optional[Union[FloatTensor, list]] = None, transform: Optional[FloatTensor] = None, strict_checks: bool = True, unset_attributes_return_none: bool = True, allow_auto_compute: bool = True)

This container class manages data attributes (pytorch tensors) of a homogeneous surface mesh (i.e. with all faces of an equal number of vertices, such as triangle mesh), or a batch of meshes following three Batching strategies. SurfaceMesh allows converting between these batching strategies, and automatically computes some attributes (such as face normals) on access (see supported attributes). This data type does not extend to volumetric tetrahedral meshes at this time and has limited support for materials.

Overview

To construct a SurfaceMesh object, pass vertices and faces (can be 0-length) tensors and any other supported attributes; batching strategy will be automatically determined from the inputs:

vertices = torch.rand((B, V, 3), dtype=torch.float32, device=device)
faces = torch.randint(0, V - 1, (F, 3), dtype=torch.long)
mesh = SurfaceMesh(faces, vertices)

To load a SurfaceMesh object:

from kaolin.io import obj, usd
mesh = obj.load_mesh(path)
mesh2 = usd.load_mesh(path2)

Examine mesh properties:

>>> print(mesh)  # Note auto-computable attributes
SurfaceMesh object with batching strategy NONE
            vertices: [42, 3] (torch.float32)[cpu]
               faces: [80, 3] (torch.int64)[cpu]
       face_vertices: if possible, computed on access from: (faces, vertices)
        face_normals: if possible, computed on access from: (normals, face_normals_idx) or (vertices, faces)
            face_uvs: if possible, computed on access from: (uvs, face_uvs_idx)
      vertex_normals: if possible, computed on access from: (faces, face_normals)

>>> mesh.face_normals  # Causes attribute to be computed
>>> print(mesh.describe_attribute("face_normals"))
        face_normals: [80, 3, 3] (torch.float32)[cpu]

To get a sense for what tensors the mesh can contain for different batching strategies see table below, or run:

>>> print(SurfaceMesh.attribute_info_string(SurfaceMesh.Batching.FIXED))
Expected SurfaceMesh contents for batching strategy FIXED
            vertices: (torch.FloatTensor) of shape ['B', 'V', 3]
               faces: (torch.IntTensor)   of shape ['F', 'FSz']
       face_vertices: (torch.FloatTensor) of shape ['B', 'F', 'FSz', 3]
             normals: (torch.FloatTensor) of shape ['B', 'VN', 3]
    face_normals_idx: (torch.IntTensor)   of shape ['B', 'F', 'FSz']
        face_normals: (torch.FloatTensor) of shape ['B', 'F', 'FSz', 3]
                 uvs: (torch.FloatTensor) of shape ['B', 'U', 2]
        face_uvs_idx: (torch.IntTensor)   of shape ['B', 'F', 'FSz']
            face_uvs: (torch.FloatTensor) of shape ['B', 'F', 'FSz', 2]
      vertex_normals: (torch.FloatTensor) of shape ['B', 'V', 3]
     vertex_tangents: (torch.FloatTensor) of shape ['B', 'V', 3]
       vertex_colors: (torch.FloatTensor) of shape ['B', 'V', None]
     vertex_features: (torch.FloatTensor) of shape ['B', 'V', None]
       face_tangents: (torch.FloatTensor) of shape ['B', 'F', 'FSz', 3]
         face_colors: (torch.FloatTensor) of shape ['B', 'F', 'FSz', None]
       face_features: (torch.FloatTensor) of shape ['B', 'F', 'FSz', None]
material_assignments: (torch.IntTensor)   of shape ['B', 'F']
           transform: (torch.FloatTensor) of shape ['B', 4, 4]
           materials: non-tensor attribute

Note

This class is using python logging, so set up logging to get diagnostics:

import logging
import sys
logging.basicConfig(level=logging.INFO, stream=sys.stdout)

Supported Attributes:

SurfaceMesh supports the following attributes, which can be provided to the constructor or set on the object. See supported batching strategies.

Attribute

Batching.NONE

Batching.FIXED

Batching.LIST

Computable?

vertices

V x 3

B x V x 3

[V_i x 3]

N

faces

F x FSz

F x FSz

[F_i x FSize]

N

face_vertices

F x FSz x 3

B x F x FSz x 3

[F_i x FSz_i x 3]

Y

normals

VN x 3

B x VN x 3

[VN_i x 3]

N

face_normals_idx

F x FSz

B x F x FSz

[F_i x FSz]

N

face_normals

F x FSz x 3

B x F x FSz x 3

[F_i x FSz_i x 3]

Y

uvs

U x 2

B x U x 2

[U_i x 2]

N

face_uvs_idx

F x FSz

B x F x FSz

[F_i x FSz]

N

face_uvs

F x FSz x 2

B x F x FSz x 2

[F_i x FSz_i x 2]

Y

vertex_normals

V x 3

B x V x 3

[V_i x 3]

Y

vertex_tangents

V x 3

B x V x 3

[V_i x 3]

Y

vertex_colors

V x 3

B x V x 3

[V_i x 3]

Y

vertex_features

V x Nfeat

B x V x Nfeat

[V_i x Nfeat_i]

N

face_tangents

F x FSz x 3

B x FSz x 3

[F_i x FSz_i x 3]

Y

face_colors

F x FSz x 3

B x F x FSz x 3

[F_i x FSz_i x 3]

Y

face_features

F x FSz x Nfeat

B x FSz x Nfeat

[F_i x FSz_i x Nfeat_i]

Y

material_assignments

F

B x F

[F_i]

N

transform

4 x 4

4 x 4 or B x 4 x 4

4 x 4 or B x 4 x 4

N

materials (non-tensor)

list

list of lists

list of lists

N

Legend: B - batch size, V - number of vertices, VN - number of vertex normals, U - number of UV coordinates, F - number of faces, FSz - number of vertices per face, Nfeat - number of custom feature channels, {?}_i - count for the ith mesh, […] - list of tensors of shapes.

Note

SurfaceMesh will not sanity check consistency of manually set attributes.

Supported Batching Strategies

SurfaceMesh can be instantiated with any of the following batching strategies, and supports conversions between batching strategies. Current batching strategy of a mesh object can be read from mesh.batching or by running print(mesh).

For example:

mesh = kaolin.io.obj.load_mesh(path)
print(mesh)
mesh.to_batched()
print(mesh)
class Batching(value, names=<not given>, *values, module=None, qualname=None, type=None, start=1, boundary=None)

Batching strategies supported by the SurfaceMesh.

Batching.FIXED = 'FIXED'

a batch of meshes with fixed topology (i.e. same faces array)

Batching.LIST = 'LIST'

a list of meshes of any topologies

Batching.NONE = 'NONE'

a single unbatched mesh

classmethod attribute_info_string(batching: Batching)

Outputs information about expected mesh contents and tensor sizes, given a batching strategy. Only includes tensor and material attributes.

Parameters

batching (SurfaceMesh.Batching) – batching strategy

Returns

multi-line string of attributes and their shapes

Return type

(str)

check_sanity()

Checks that tensor attribute sizes are consistent for the current batching strategy. Will log any inconsistencies.

Returns

true if sane, false if not

Return type

(bool)

set_batching(batching: Batching, skip_errors=False)

Converts a mesh to a different batching strategy. Modifies the mesh in place and returns self.

All conversions are supported, except the following:
  • to NONE from FIXED or LIST batch with more than one mesh

  • to FIXED from LIST batch where fixed topology items are different

Parameters
  • batching (SurfaceMesh.Batching) – desirable batching strategy.

  • skip_errors – if true, will simply unset attributes that cannot be converted (useful if e.g. vertices are of fixed topology, but meshes have variable number of normals that cannot be stacked)

Returns

(self)

to_batched()

Convenience shorthand to convert unbatched mesh to FIXED topology batched mesh. Modifies the mesh in place and returns self.

Returns

(self)

getattr_batched(attr: str, batching: Batching)

Same as getattr or mesh.attr, but converts the attribute value to desired batching strategy before returning.

All conversions are supported, except the following:
  • to NONE from FIXED or LIST batch with more than one mesh

  • to FIXED from LIST batch where fixed topology items are different

Parameters
Returns

attribute value

classmethod cat(meshes: Sequence[SurfaceMesh], fixed_topology: bool = True, skip_errors: bool = False)

Concatenates meshes or batches of meshes to create a FIXED (if fixed_topology) or LIST batched mesh. Only attributes present in all the meshes will be preserved, with special treatment for auto-computable attributes. For example, if one mesh has face_normals, and another mesh allows auto-computation of face_normals, then it will be auto-computed.

transform matrices are batched like any other tensor attribute and are never applied. Use flatten() to merge meshes into a single world-space mesh with transforms applied.

Parameters
  • meshes – meshes to concatenate; any batching is supported

  • fixed_topology – if to create a FIXED batched mesh (input must comply to assumptions)

  • skip_errors – if True, will not fail if some attributes fail to convert to target batching

Returns

new mesh

Return type

(SurfaceMesh)

static convert_attribute_batching(val: Union[Tensor, list], from_batching: Batching, to_batching: Batching, is_tensor: bool = True, fixed_topology: bool = False, batch_size: int = None)

Converts tensors between different SurfaceMesh.Batching strategies. The input value is expected to respect the provided from_batching. Will fail if conversion cannot be done

Approximate summary of conversions for tensor values:
  • NONE -> LIST: return [val]

  • NONE -> FIXED: return val.unsqueeze(0) unless fixed_topology

  • LIST -> NONE: return val[0], fails if list longer than 1

  • LIST -> FIXED: return torch.stack(val) (or val[0] if fixed_topology)

  • FIXED -> NONE: return val.squeeze(0) unless fixed_topology, fails if list longer than 1

  • FIXED -> LIST: return [val[i, ...] for i ...] (or [val for i ...] if fixed_topology)

Non-tensor values are stored as lists for FIXED and LIST batching.

Note

This method is only useful for converting batching of custom attributes and is not needed if only working with attributes natively supported by the SurfaceMesh.

Parameters
  • val – value to convert, must be consistent with from_batching

  • from_batching – batching type to convert from

  • to_batching – batching type to convert to

  • is_tensor – if the converted value is a tensor attribute (and not e.g. unstructured value to store in lists)

  • fixed_topology – if the attribute should be the same across items in a FIXED batching

  • batch_size – desirable batch size; must be consistent with val (will be guessed in most cases, but when converting fixed_topology items to e.g. LIST batching, this value is needed)

Attribute Access

By default, SurfaceMesh will attempt to auto-compute missing attributes on access. These attributes will be cached, unless their ancestors have requires_grad == True. This behavior of the mesh object can be changed at construction time (allow_auto_compute=False) or by setting mesh.allow_auto_compute later. In addition to this convenience API, explicit methods for attribute access are also supported.

For example, using convenience API:

# Caching is enabled by default
mesh = kaolin.io.obj.load_mesh(path, with_normals=False)
print(mesh)
print(mesh.has_attribute('face_normals'))  # False
fnorm = mesh.face_normals  # Auto-computed
print(mesh.has_attribute('face_normals'))  # True (cached)

# Caching is disabled when gradients need to flow
mesh = kaolin.io.obj.load_mesh(path, with_normals=False)
mesh.vertices.requires_grad = True   # causes caching to be off
print(mesh.has_attribute('face_normals'))  # False
fnorm = mesh.face_normals  # Auto-computed
print(mesh.has_attribute('face_normals'))  # False (caching disabled)

For example, using explicit API:

mesh = kaolin.io.obj.load_mesh(path, with_normals=False)
print(mesh.has_attribute('face_normals'))  # False
fnorm = mesh.get_or_compute_attribute('face_normals', should_cache=False)
print(mesh.has_attribute('face_normals'))  # False
get_attributes(only_tensors=False)

Returns names of all attributes that are currently set.

Parameters

only_tensors – if true, will only include tensor attributes

Returns

list of string names

Return type

(list)

has_attribute(attr: str)

Checks if a given attribute is present without trying to compute it, if not.

Parameters

attr – attribute name

Returns

True if attribute is set and is not None

Return type

(bool)

has_or_can_compute_attribute(attr: str)

Returns true if this attribute is set or has all the requirements to be computed. Note that actual computation may still fail at run time.

Parameters

attr – attribute name to check

Returns

True if exists or likely to be computable.

Return type

(bool)

probably_can_compute_attribute(attr: str)

Checks that the attributes required for computing attribute exist and returns true if the attribute is likely to be computable (not that it is not possible to determine this for sure without actually computing the attribute, as there could be runtime errors that occur during computation).

Parameters

attr – attribute name to check

Returns

(bool) True if likely to be computable

get_attribute(attr: str)

Gets attribute without any auto-computation magic. If attribute is not set will either return None if mesh.unset_attributes_return_none or raise an exception.

Parameters

attr – attribute name, see attributes

Returns

attribute value

Raises

AttributeError – if attribute name is not supported, or if attribute is not set and not mesh.unset_attributes_return_none

get_or_compute_attribute(attr: str, should_cache: Optional[bool] = None)

Gets or computes an attribute, while allowing explicit control of caching of the computed value. If attribute is not set and cannot be computed will either return None if mesh.unset_attributes_return_none or raise an exception.

Note that in the event that mesh contains face_normals, but normals and face_normals_idx are needed instead call mesh.ensure_indexed_attribute(“normals”) and similarly for uvs. # TODO: proper doc ref

Parameters
  • attr – attribute name, see attributes

  • should_cache – if True, will cache attribute if it was computed; if False, will not cache; by default will decide if to cache based on requires_grad of variables used in computation (will not cache if any has requires_grad is True).

Returns

attribute value

Inspecting and Copying Meshes

To make it easier to work with, SurfaceMesh supports detailed print statements, as well as len(), copy(), deepcopy() and can be converted to a dictionary.

Supported operations:

import copy
mesh_copy = copy.copy(mesh)
mesh_copy = copy.deepcopy(mesh)
batch_size = len(mesh)

# Print default attributes
print(mesh)

# Print more detailed attributes
print(mesh.to_string(detailed=True, print_stats=True))

# Print specific attribute
print(mesh.describe_attribute('vertices'))
to_string(print_stats=False, detailed=False)

Returns information about tensor attributes currently contained in the mesh as a multi-line string.

Parameters
  • print_stats (bool) – if to print statistics about values in each tensor

  • detailed (bool) – if to include additional information about each tensor

Returns

multi-line string with attribute information

Return type

(str)

describe_attribute(attr, print_stats=False, detailed=False)

Outputs an informative string about an attribute; the same method used for all attributes in to_string.

Args:

print_stats (bool): if to print statistics about values in each tensor detailed (bool): if to include additional information about each tensor

Returns

multi-line string with attribute information

Return type

(str)

as_dict(only_tensors=False)

Returns currently set items as a dictionary. Does not auto-compute any items, but returns raw values.

Parameters

only_tensors (bool) – if true, will only include tensor attributes

Returns

currently set attributes as a dictionary

Return type

(dict)

Tensor Operations

Convenience operations for device and type conversions of some or all member tensors.

cuda(device=None, attributes=None)

Calls cuda on all or only on select tensor attributes, returns a copy of self.

Parameters
  • device – device to set

  • attributes (list of str) – if set, will only call cuda() on select attributes

Returns

(SurfaceMesh) shallow copy, with the exception of attributes that were converted

cpu(attributes=None)

Calls cpu() on all or only on select tensor attributes, returns a copy of self.

Parameters

attributes (list of str) – if set, will only call cpu() on select attributes

Returns

(SurfaceMesh) shallow copy, with the exception of attributes that were converted

to(device, attributes=None)

Converts all or select tensor attributes to provided device; returns copy of self.

Parameters
  • device (str, torch.device) – device to call torch tensors’ to method with

  • attributes (list of str) – if set, will only convert select attributes

Returns

(SurfaceMesh) shallow copy, with the exception of attributes that were converted

float_tensors_to(float_dtype)

Converts all floating point tensors to the provided type; returns shallow copy.

Parameters

float_dtype – torch dtype such as torch.float16, torch.float32

Returns

(SurfaceMesh) shallow copy, with the exception of attributes that were converted

detach(attributes=None)

Detaches all or select attributes in a shallow copy of self.

Parameters

attributes (list of str) – if set, will only call cuda on select attributes

Returns

(SurfaceMesh) shallow copy, with the exception of attributes that were converted

Other

static supported_tensor_attributes()
static computable_attribute_requirements()
static assert_supported(attr)
__init__(vertices: Union[FloatTensor, list], faces: Union[LongTensor, list], normals: Optional[Union[FloatTensor, list]] = None, uvs: Optional[Union[FloatTensor, list]] = None, face_uvs_idx: Optional[Union[LongTensor, list]] = None, face_normals_idx: Optional[Union[LongTensor, list]] = None, material_assignments: Optional[Union[Tensor, list]] = None, materials: Optional[list] = None, vertex_normals: Optional[Union[FloatTensor, list]] = None, vertex_tangents: Optional[Union[FloatTensor, list]] = None, vertex_colors: Optional[Union[FloatTensor, list]] = None, vertex_features: Optional[Union[FloatTensor, list]] = None, face_normals: Optional[Union[FloatTensor, list]] = None, face_uvs: Optional[Union[FloatTensor, list]] = None, face_vertices: Optional[Union[FloatTensor, list]] = None, face_tangents: Optional[Union[FloatTensor, list]] = None, face_colors: Optional[Union[FloatTensor, list]] = None, face_features: Optional[Union[FloatTensor, list]] = None, transform: Optional[FloatTensor] = None, strict_checks: bool = True, unset_attributes_return_none: bool = True, allow_auto_compute: bool = True)

Initializes the surface mesh object, while automatically detecting a batching strategy (see supported attributes for expected tensor dimensions). The vertices and faces tensors are required, but the number of faces/vertices can be 0. Any or all of the other attributes can be also provided or set later.

Parameters
  • vertices – xyz locations of vertices.

  • faces – indices into vertices array for each vertex of each face; this is the only fixed topology item for Batching.FIXED.

  • face_vertices – xyz locations for each vertex of each face; can be set directly or is auto-computable by indexing vertices with faces.

  • normals – xyz normal values, indexed by face_normals_idx.

  • face_normals_idx – indices into normals for each vertex in each face.

  • face_normals – xyz normal values for each face; can be set directly or is auto-computable by 1) indexing normals with face_normals_idx, or (if either is missing and mesh is triangular) by 2) using vertex locations.

  • uvs – uv texture coordinates, indexed by face_uvs_idx.

  • face_uvs_idx – indices into uvs for each vertex of each face.

  • face_uvs – uv coordinate values for each vertex of each face; can be set directly or is auto-computable by indexing uvs with face_uvs_idx.

  • vertex_normals – xyz normal values, corresponding to vertices; can be set directly or is auto-computable by averaging face_normals of faces incident to a vertex.

  • vertex_tangents – tangents values used to compute the orientation of normal perturbation, corresponding to vertices; can be set directly or is auto-computable, from vertices and face_uvs.

  • vertex_colors – color values associated with every vertex, expected RGB in range is 0..1 float tensor, but this will not be enforced; if face_colors are set instead this value will auto-compute to average across all faces incident to each vertex.

  • vertex_features – custom features of any channel number associated to every vertex; if face_features are set instead this value will auto-compute to average across all faces incident to each vertex.

  • face_tangents – the vertex_tangents value mapped to every vertex of every face, typically this and vertex_tangents would be auto-computed.

  • face_colors – per-face per-vertex color values; if only one color is available per face, tile the tensor to provide one color for each face vertex; if vertex_colors are set instead, this will auto-compute, but setting face_colors directly allows the same vertex to have different color value within distinct faces (same property is true for face_uvs and face_normals across several 3D formats). Expected color value is RGB in range is 0..1 float, but this will not be enforced.

  • face_features – custom per-face per-vertex features of any channel; if vertex_features are set instead, this will auto-compute, but setting face_features directly allows the same vertex to have different features within distinct faces.

  • material_assignments – indices into materials list for each face.

  • materials – raw materials as output by the io reader.

  • transform – optional 4x4 affine transformation matrix from local space to world space, stored on the mesh and not applied by default. Apply to the mesh geometry via as_transformed(). Shape is (4, 4) for Batching.NONE. For Batching.FIXED and Batching.LIST it may be a single (4, 4) tensor (broadcast to every element) or a (B, 4, 4) tensor (one matrix per element).

  • strict_checks – if True, will raise exception if any tensors passed to the construcor have unexpected shapes (see shapes matrix above); note that checks are less strict for Batching.LIST batching (default: True).

  • unset_attributes_return_none – if set to False exception will be raised when accessing attributes that are not set (or cannot be computed), if True will simply return None (default: True).

  • allow_auto_compute

    whether to allow auto-computation of attributes on mesh

    attribute access; see supported attributes (default: True).

    Note

    SurfaceMesh will not sanity check consistency of manually set attributes. That is left to the user.

vertex_tangents
vertex_colors
vertex_features
face_tangents
face_colors
face_features
transform
is_triangular()

Returns True only if all the meshes in the batch are triangular.

Returns: (bool)

classmethod flatten(meshes: Sequence[SurfaceMesh], skip_errors: bool = False, group_materials_by_name: bool = False)

Flattens a group of meshes into a single world-space mesh. Any transform present on input meshes is applied to their vertices (via as_transformed()) before concatenation, so the result is always in world space and has no transform. For indexed attributes such as normals and face_normals_idx, indices are properly book-kept. For all other attributes, concatenation is attempted if the attribute is present in all meshes. To ignore concatenation incompatibilities, set skip_errors=True.

Args:

meshes (list of SurfaceMesh): meshes to concatenate; any mix of batching strategies is supported. skip_errors (bool): if True, will not fail if some attributes cannot be flattened. group_materials_by_name (bool): if True, will group materials by name and assume they are the same.

Returns

new mesh in world space with no transform

Return type

(SurfaceMesh)

ensure_indexed_attribute(attr: str, should_cache: Optional[bool] = None)

Sometimes the mesh contains face_normals or face_uvs with values already mapped to every vertex of every face in a single array, and does not have the corresponding indexed values. For example, USD format commonly loads with face_normals, but not normals or face_normals_idx. Or, mesh flattening or concatenation can also cause this to happen. In the event that the indexed attribute is needed, call this function for normals or uvs. Note: behavior is undefined if mesh contains either values or the index as well as face values, e.g. if mesh has face_normals and ‘face_normals_idx’, but not normals, this function will return None, None.

Example

normals, face_normals_idx = mesh.ensure_indexed_attribute(“normals”)

Parameters
  • attr – attribute name, only ‘normals’ and ‘uvs’ are supported

  • should_cache – if True, will cache attribute if it was computed; if False, will not cache; by default will decide if to cache based on requires_grad of variables used in computation (will not cache if any has requires_grad is True).

Returns

(values, index) - e.g. ‘normals’ and ‘face_normals_idx’ with sizes consistent with the current

batching strategy, or None, None if

as_transformed()

Creates a new SurfaceMesh with the stored transform applied.

Transforms vertices and direction attributes using the stored transform matrix. Normals (normals, vertex_normals, face_normals) transform by \((M^{-1})^T\); tangents (vertex_tangents, face_tangents) transform by the linear part \(M\). All transformed direction vectors are re-normalized. face_vertices is cleared so it will be auto-recomputed on demand. Returns a new SurfaceMesh with transform unset.

transform may be:

  • Batching.NONE: a (4, 4) tensor.

  • Batching.FIXED or Batching.LIST: a (4, 4) tensor (applied to all batch elements) or a (B, 4, 4) tensor (one matrix per batch element).

Returns

new mesh in world space with transform cleared, or self if transform is not set.

Return type

(SurfaceMesh)