SurfaceMesh¶
Tutorial¶
For a walk-through of kaolin.rep.SurfaceMesh features,
see working_with_meshes.ipynb.
API¶
- class kaolin.rep.SurfaceMesh(vertices: Union[FloatTensor, list], faces: Union[LongTensor, list], normals: Optional[Union[FloatTensor, list]] = None, uvs: Optional[Union[FloatTensor, list]] = None, face_uvs_idx: Optional[Union[LongTensor, list]] = None, face_normals_idx: Optional[Union[LongTensor, list]] = None, material_assignments: Optional[Union[Tensor, list]] = None, materials: Optional[list] = None, vertex_normals: Optional[Union[FloatTensor, list]] = None, vertex_tangents: Optional[Union[FloatTensor, list]] = None, vertex_colors: Optional[Union[FloatTensor, list]] = None, vertex_features: Optional[Union[FloatTensor, list]] = None, face_normals: Optional[Union[FloatTensor, list]] = None, face_uvs: Optional[Union[FloatTensor, list]] = None, face_vertices: Optional[Union[FloatTensor, list]] = None, face_tangents: Optional[Union[FloatTensor, list]] = None, face_colors: Optional[Union[FloatTensor, list]] = None, face_features: Optional[Union[FloatTensor, list]] = None, transform: Optional[FloatTensor] = None, strict_checks: bool = True, unset_attributes_return_none: bool = True, allow_auto_compute: bool = True)¶
This container class manages data attributes (pytorch tensors) of a homogeneous surface mesh (i.e. with all faces of an equal number of vertices, such as triangle mesh), or a batch of meshes following three
Batchingstrategies.SurfaceMeshallows converting between these batching strategies, and automatically computes some attributes (such as face normals) on access (see supported attributes). This data type does not extend to volumetric tetrahedral meshes at this time and has limited support for materials.Overview
To construct a
SurfaceMeshobject, passverticesandfaces(can be 0-length) tensors and any other supported attributes; batching strategy will be automatically determined from the inputs:vertices = torch.rand((B, V, 3), dtype=torch.float32, device=device) faces = torch.randint(0, V - 1, (F, 3), dtype=torch.long) mesh = SurfaceMesh(faces, vertices)
To load a
SurfaceMeshobject:from kaolin.io import obj, usd mesh = obj.load_mesh(path) mesh2 = usd.load_mesh(path2)
Examine mesh properties:
>>> print(mesh) # Note auto-computable attributes SurfaceMesh object with batching strategy NONE vertices: [42, 3] (torch.float32)[cpu] faces: [80, 3] (torch.int64)[cpu] face_vertices: if possible, computed on access from: (faces, vertices) face_normals: if possible, computed on access from: (normals, face_normals_idx) or (vertices, faces) face_uvs: if possible, computed on access from: (uvs, face_uvs_idx) vertex_normals: if possible, computed on access from: (faces, face_normals) >>> mesh.face_normals # Causes attribute to be computed >>> print(mesh.describe_attribute("face_normals")) face_normals: [80, 3, 3] (torch.float32)[cpu]
To get a sense for what tensors the mesh can contain for different batching strategies see table below, or run:
>>> print(SurfaceMesh.attribute_info_string(SurfaceMesh.Batching.FIXED)) Expected SurfaceMesh contents for batching strategy FIXED vertices: (torch.FloatTensor) of shape ['B', 'V', 3] faces: (torch.IntTensor) of shape ['F', 'FSz'] face_vertices: (torch.FloatTensor) of shape ['B', 'F', 'FSz', 3] normals: (torch.FloatTensor) of shape ['B', 'VN', 3] face_normals_idx: (torch.IntTensor) of shape ['B', 'F', 'FSz'] face_normals: (torch.FloatTensor) of shape ['B', 'F', 'FSz', 3] uvs: (torch.FloatTensor) of shape ['B', 'U', 2] face_uvs_idx: (torch.IntTensor) of shape ['B', 'F', 'FSz'] face_uvs: (torch.FloatTensor) of shape ['B', 'F', 'FSz', 2] vertex_normals: (torch.FloatTensor) of shape ['B', 'V', 3] vertex_tangents: (torch.FloatTensor) of shape ['B', 'V', 3] vertex_colors: (torch.FloatTensor) of shape ['B', 'V', None] vertex_features: (torch.FloatTensor) of shape ['B', 'V', None] face_tangents: (torch.FloatTensor) of shape ['B', 'F', 'FSz', 3] face_colors: (torch.FloatTensor) of shape ['B', 'F', 'FSz', None] face_features: (torch.FloatTensor) of shape ['B', 'F', 'FSz', None] material_assignments: (torch.IntTensor) of shape ['B', 'F'] transform: (torch.FloatTensor) of shape ['B', 4, 4] materials: non-tensor attribute
Note
This class is using python logging, so set up logging to get diagnostics:
import logging import sys logging.basicConfig(level=logging.INFO, stream=sys.stdout)
Supported Attributes:
SurfaceMeshsupports the following attributes, which can be provided to the constructor or set on the object. See supported batching strategies.Attribute
Batching.NONEBatching.FIXEDBatching.LISTComputable?
vertices
V x 3
B x V x 3
[V_i x 3]
N
faces
F x FSz
F x FSz
[F_i x FSize]
N
face_vertices
F x FSz x 3
B x F x FSz x 3
[F_i x FSz_i x 3]
Y
normals
VN x 3
B x VN x 3
[VN_i x 3]
N
face_normals_idx
F x FSz
B x F x FSz
[F_i x FSz]
N
face_normals
F x FSz x 3
B x F x FSz x 3
[F_i x FSz_i x 3]
Y
uvs
U x 2
B x U x 2
[U_i x 2]
N
face_uvs_idx
F x FSz
B x F x FSz
[F_i x FSz]
N
face_uvs
F x FSz x 2
B x F x FSz x 2
[F_i x FSz_i x 2]
Y
vertex_normals
V x 3
B x V x 3
[V_i x 3]
Y
vertex_tangents
V x 3
B x V x 3
[V_i x 3]
Y
vertex_colors
V x 3
B x V x 3
[V_i x 3]
Y
vertex_features
V x Nfeat
B x V x Nfeat
[V_i x Nfeat_i]
N
face_tangents
F x FSz x 3
B x FSz x 3
[F_i x FSz_i x 3]
Y
face_colors
F x FSz x 3
B x F x FSz x 3
[F_i x FSz_i x 3]
Y
face_features
F x FSz x Nfeat
B x FSz x Nfeat
[F_i x FSz_i x Nfeat_i]
Y
material_assignments
F
B x F
[F_i]
N
transform
4 x 4
4 x 4 or B x 4 x 4
4 x 4 or B x 4 x 4
N
materials (non-tensor)
list
list of lists
list of lists
N
Legend: B - batch size, V - number of vertices, VN - number of vertex normals, U - number of UV coordinates, F - number of faces, FSz - number of vertices per face, Nfeat - number of custom feature channels, {?}_i - count for the ith mesh, […] - list of tensors of shapes.
Note
SurfaceMeshwill not sanity check consistency of manually set attributes.Supported Batching Strategies
SurfaceMeshcan be instantiated with any of the following batching strategies, and supports conversions between batching strategies. Current batching strategy of ameshobject can be read frommesh.batchingor by runningprint(mesh).For example:
mesh = kaolin.io.obj.load_mesh(path) print(mesh) mesh.to_batched() print(mesh)
- class Batching(value, names=<not given>, *values, module=None, qualname=None, type=None, start=1, boundary=None)¶
Batching strategies supported by the
SurfaceMesh.- Batching.FIXED = 'FIXED'¶
a batch of meshes with fixed topology (i.e. same faces array)
- Batching.LIST = 'LIST'¶
a list of meshes of any topologies
- Batching.NONE = 'NONE'¶
a single unbatched mesh
- classmethod attribute_info_string(batching: Batching)¶
Outputs information about expected mesh contents and tensor sizes, given a batching strategy. Only includes tensor and material attributes.
- Parameters
batching (SurfaceMesh.Batching) – batching strategy
- Returns
multi-line string of attributes and their shapes
- Return type
(str)
- check_sanity()¶
Checks that tensor attribute sizes are consistent for the current batching strategy. Will log any inconsistencies.
- Returns
true if sane, false if not
- Return type
(bool)
- set_batching(batching: Batching, skip_errors=False)¶
Converts a mesh to a different batching strategy. Modifies the mesh in place and returns self.
- All conversions are supported, except the following:
to NONE from FIXED or LIST batch with more than one mesh
to FIXED from LIST batch where fixed topology items are different
- Parameters
batching (SurfaceMesh.Batching) – desirable batching strategy.
skip_errors – if true, will simply unset attributes that cannot be converted (useful if e.g. vertices are of fixed topology, but meshes have variable number of normals that cannot be stacked)
- Returns
(self)
- to_batched()¶
Convenience shorthand to convert unbatched mesh to FIXED topology batched mesh. Modifies the mesh in place and returns self.
- Returns
(self)
- getattr_batched(attr: str, batching: Batching)¶
Same as getattr or mesh.attr, but converts the attribute value to desired batching strategy before returning.
- All conversions are supported, except the following:
to NONE from FIXED or LIST batch with more than one mesh
to FIXED from LIST batch where fixed topology items are different
- Parameters
attr (str) – attribute name
batching (SurfaceMesh.Batching) – desirable batching strategy.
- Returns
attribute value
- classmethod cat(meshes: Sequence[SurfaceMesh], fixed_topology: bool = True, skip_errors: bool = False)¶
Concatenates meshes or batches of meshes to create a FIXED (if fixed_topology) or LIST batched mesh. Only attributes present in all the meshes will be preserved, with special treatment for auto-computable attributes. For example, if one mesh has face_normals, and another mesh allows auto-computation of face_normals, then it will be auto-computed.
transformmatrices are batched like any other tensor attribute and are never applied. Useflatten()to merge meshes into a single world-space mesh with transforms applied.- Parameters
meshes – meshes to concatenate; any batching is supported
fixed_topology – if to create a FIXED batched mesh (input must comply to assumptions)
skip_errors – if True, will not fail if some attributes fail to convert to target batching
- Returns
new mesh
- Return type
- static convert_attribute_batching(val: Union[Tensor, list], from_batching: Batching, to_batching: Batching, is_tensor: bool = True, fixed_topology: bool = False, batch_size: int = None)¶
Converts tensors between different
SurfaceMesh.Batchingstrategies. The input value is expected to respect the providedfrom_batching. Will fail if conversion cannot be done- Approximate summary of conversions for tensor values:
NONE->LIST: return[val]NONE->FIXED: returnval.unsqueeze(0)unless fixed_topologyLIST->NONE: returnval[0], fails if list longer than 1LIST->FIXED: returntorch.stack(val)(orval[0]if fixed_topology)FIXED->NONE: returnval.squeeze(0)unless fixed_topology, fails if list longer than 1FIXED->LIST: return[val[i, ...] for i ...](or[val for i ...]if fixed_topology)
Non-tensor values are stored as lists for
FIXEDandLISTbatching.Note
This method is only useful for converting batching of custom attributes and is not needed if only working with attributes natively supported by the
SurfaceMesh.- Parameters
val – value to convert, must be consistent with
from_batchingfrom_batching – batching type to convert from
to_batching – batching type to convert to
is_tensor – if the converted value is a tensor attribute (and not e.g. unstructured value to store in lists)
fixed_topology – if the attribute should be the same across items in a
FIXEDbatchingbatch_size – desirable batch size; must be consistent with
val(will be guessed in most cases, but when convertingfixed_topologyitems to e.g.LISTbatching, this value is needed)
Attribute Access
By default,
SurfaceMeshwill attempt to auto-compute missing attributes on access. These attributes will be cached, unless their ancestors haverequires_grad == True. This behavior of themeshobject can be changed at construction time (allow_auto_compute=False) or by settingmesh.allow_auto_computelater. In addition to this convenience API, explicit methods for attribute access are also supported.For example, using convenience API:
# Caching is enabled by default mesh = kaolin.io.obj.load_mesh(path, with_normals=False) print(mesh) print(mesh.has_attribute('face_normals')) # False fnorm = mesh.face_normals # Auto-computed print(mesh.has_attribute('face_normals')) # True (cached) # Caching is disabled when gradients need to flow mesh = kaolin.io.obj.load_mesh(path, with_normals=False) mesh.vertices.requires_grad = True # causes caching to be off print(mesh.has_attribute('face_normals')) # False fnorm = mesh.face_normals # Auto-computed print(mesh.has_attribute('face_normals')) # False (caching disabled)
For example, using explicit API:
mesh = kaolin.io.obj.load_mesh(path, with_normals=False) print(mesh.has_attribute('face_normals')) # False fnorm = mesh.get_or_compute_attribute('face_normals', should_cache=False) print(mesh.has_attribute('face_normals')) # False
- get_attributes(only_tensors=False)¶
Returns names of all attributes that are currently set.
- Parameters
only_tensors – if true, will only include tensor attributes
- Returns
list of string names
- Return type
(list)
- has_attribute(attr: str)¶
Checks if a given attribute is present without trying to compute it, if not.
- Parameters
attr – attribute name
- Returns
True if attribute is set and is not None
- Return type
(bool)
- has_or_can_compute_attribute(attr: str)¶
Returns true if this attribute is set or has all the requirements to be computed. Note that actual computation may still fail at run time.
- Parameters
attr – attribute name to check
- Returns
True if exists or likely to be computable.
- Return type
(bool)
- probably_can_compute_attribute(attr: str)¶
Checks that the attributes required for computing attribute exist and returns true if the attribute is likely to be computable (not that it is not possible to determine this for sure without actually computing the attribute, as there could be runtime errors that occur during computation).
- Parameters
attr – attribute name to check
- Returns
(bool) True if likely to be computable
- get_attribute(attr: str)¶
Gets attribute without any auto-computation magic. If attribute is not set will either return
Noneifmesh.unset_attributes_return_noneor raise an exception.- Parameters
attr – attribute name, see attributes
- Returns
attribute value
- Raises
AttributeError – if attribute name is not supported, or if attribute is not set and
not mesh.unset_attributes_return_none
- get_or_compute_attribute(attr: str, should_cache: Optional[bool] = None)¶
Gets or computes an attribute, while allowing explicit control of caching of the computed value. If attribute is not set and cannot be computed will either return
Noneifmesh.unset_attributes_return_noneor raise an exception.Note that in the event that mesh contains face_normals, but normals and face_normals_idx are needed instead call mesh.ensure_indexed_attribute(“normals”) and similarly for uvs. # TODO: proper doc ref
- Parameters
attr – attribute name, see attributes
should_cache – if
True, will cache attribute if it was computed; ifFalse, will not cache; by default will decide if to cache based onrequires_gradof variables used in computation (will not cache if any hasrequires_grad is True).
- Returns
attribute value
Inspecting and Copying Meshes
To make it easier to work with,
SurfaceMeshsupports detailed print statements, as well aslen(),copy(),deepcopy()and can be converted to a dictionary.Supported operations:
import copy mesh_copy = copy.copy(mesh) mesh_copy = copy.deepcopy(mesh) batch_size = len(mesh) # Print default attributes print(mesh) # Print more detailed attributes print(mesh.to_string(detailed=True, print_stats=True)) # Print specific attribute print(mesh.describe_attribute('vertices'))
- to_string(print_stats=False, detailed=False)¶
Returns information about tensor attributes currently contained in the mesh as a multi-line string.
- describe_attribute(attr, print_stats=False, detailed=False)¶
Outputs an informative string about an attribute; the same method used for all attributes in
to_string.- Args:
print_stats (bool): if to print statistics about values in each tensor detailed (bool): if to include additional information about each tensor
- Returns
multi-line string with attribute information
- Return type
(str)
- as_dict(only_tensors=False)¶
Returns currently set items as a dictionary. Does not auto-compute any items, but returns raw values.
Tensor Operations
Convenience operations for device and type conversions of some or all member tensors.
- cuda(device=None, attributes=None)¶
Calls cuda on all or only on select tensor attributes, returns a copy of self.
- Parameters
device – device to set
attributes (list of str) – if set, will only call cuda() on select attributes
- Returns
(SurfaceMesh) shallow copy, with the exception of attributes that were converted
- cpu(attributes=None)¶
Calls cpu() on all or only on select tensor attributes, returns a copy of self.
- Parameters
attributes (list of str) – if set, will only call cpu() on select attributes
- Returns
(SurfaceMesh) shallow copy, with the exception of attributes that were converted
- to(device, attributes=None)¶
Converts all or select tensor attributes to provided device; returns copy of self.
- Parameters
device (str, torch.device) – device to call torch tensors’
tomethod withattributes (list of str) – if set, will only convert select attributes
- Returns
(SurfaceMesh) shallow copy, with the exception of attributes that were converted
- float_tensors_to(float_dtype)¶
Converts all floating point tensors to the provided type; returns shallow copy.
- Parameters
float_dtype – torch dtype such as torch.float16, torch.float32
- Returns
(SurfaceMesh) shallow copy, with the exception of attributes that were converted
- detach(attributes=None)¶
Detaches all or select attributes in a shallow copy of self.
- Parameters
attributes (list of str) – if set, will only call cuda on select attributes
- Returns
(SurfaceMesh) shallow copy, with the exception of attributes that were converted
Other
- static supported_tensor_attributes()¶
- static computable_attribute_requirements()¶
- static assert_supported(attr)¶
- __init__(vertices: Union[FloatTensor, list], faces: Union[LongTensor, list], normals: Optional[Union[FloatTensor, list]] = None, uvs: Optional[Union[FloatTensor, list]] = None, face_uvs_idx: Optional[Union[LongTensor, list]] = None, face_normals_idx: Optional[Union[LongTensor, list]] = None, material_assignments: Optional[Union[Tensor, list]] = None, materials: Optional[list] = None, vertex_normals: Optional[Union[FloatTensor, list]] = None, vertex_tangents: Optional[Union[FloatTensor, list]] = None, vertex_colors: Optional[Union[FloatTensor, list]] = None, vertex_features: Optional[Union[FloatTensor, list]] = None, face_normals: Optional[Union[FloatTensor, list]] = None, face_uvs: Optional[Union[FloatTensor, list]] = None, face_vertices: Optional[Union[FloatTensor, list]] = None, face_tangents: Optional[Union[FloatTensor, list]] = None, face_colors: Optional[Union[FloatTensor, list]] = None, face_features: Optional[Union[FloatTensor, list]] = None, transform: Optional[FloatTensor] = None, strict_checks: bool = True, unset_attributes_return_none: bool = True, allow_auto_compute: bool = True)¶
Initializes the surface mesh object, while automatically detecting a batching strategy (see supported attributes for expected tensor dimensions). The vertices and faces tensors are required, but the number of faces/vertices can be 0. Any or all of the other attributes can be also provided or set later.
- Parameters
vertices – xyz locations of vertices.
faces – indices into
verticesarray for each vertex of each face; this is the only fixed topology item forBatching.FIXED.face_vertices – xyz locations for each vertex of each face; can be set directly or is auto-computable by indexing
verticeswithfaces.normals – xyz normal values, indexed by
face_normals_idx.face_normals_idx – indices into
normalsfor each vertex in each face.face_normals – xyz normal values for each face; can be set directly or is auto-computable by 1) indexing
normalswithface_normals_idx, or (if either is missing and mesh is triangular) by 2) using vertex locations.uvs – uv texture coordinates, indexed by
face_uvs_idx.face_uvs_idx – indices into
uvsfor each vertex of each face.face_uvs – uv coordinate values for each vertex of each face; can be set directly or is auto-computable by indexing
uvswithface_uvs_idx.vertex_normals – xyz normal values, corresponding to vertices; can be set directly or is auto-computable by averaging
face_normalsof faces incident to a vertex.vertex_tangents – tangents values used to compute the orientation of normal perturbation, corresponding to vertices; can be set directly or is auto-computable, from
verticesandface_uvs.vertex_colors – color values associated with every vertex, expected RGB in range is 0..1 float tensor, but this will not be enforced; if
face_colorsare set instead this value will auto-compute to average across all faces incident to each vertex.vertex_features – custom features of any channel number associated to every vertex; if
face_featuresare set instead this value will auto-compute to average across all faces incident to each vertex.face_tangents – the vertex_tangents value mapped to every vertex of every face, typically this and
vertex_tangentswould be auto-computed.face_colors – per-face per-vertex color values; if only one color is available per face, tile the tensor to provide one color for each face vertex; if
vertex_colorsare set instead, this will auto-compute, but settingface_colorsdirectly allows the same vertex to have different color value within distinct faces (same property is true forface_uvsandface_normalsacross several 3D formats). Expected color value is RGB in range is 0..1 float, but this will not be enforced.face_features – custom per-face per-vertex features of any channel; if
vertex_featuresare set instead, this will auto-compute, but settingface_featuresdirectly allows the same vertex to have different features within distinct faces.material_assignments – indices into
materialslist for each face.materials – raw materials as output by the io reader.
transform – optional 4x4 affine transformation matrix from local space to world space, stored on the mesh and not applied by default. Apply to the mesh geometry via
as_transformed(). Shape is(4, 4)forBatching.NONE. ForBatching.FIXEDandBatching.LISTit may be a single(4, 4)tensor (broadcast to every element) or a(B, 4, 4)tensor (one matrix per element).strict_checks – if
True, will raise exception if any tensors passed to the construcor have unexpected shapes (see shapes matrix above); note that checks are less strict forBatching.LISTbatching (default:True).unset_attributes_return_none – if set to
Falseexception will be raised when accessing attributes that are not set (or cannot be computed), ifTruewill simply returnNone(default:True).allow_auto_compute –
- whether to allow auto-computation of attributes on mesh
attribute access; see supported attributes (default:
True).
Note
SurfaceMeshwill not sanity check consistency of manually set attributes. That is left to the user.
- vertex_tangents¶
- vertex_colors¶
- vertex_features¶
- face_tangents¶
- face_colors¶
- face_features¶
- transform¶
- is_triangular()¶
Returns True only if all the meshes in the batch are triangular.
Returns: (bool)
- classmethod flatten(meshes: Sequence[SurfaceMesh], skip_errors: bool = False, group_materials_by_name: bool = False)¶
Flattens a group of meshes into a single world-space mesh. Any
transformpresent on input meshes is applied to their vertices (viaas_transformed()) before concatenation, so the result is always in world space and has notransform. For indexed attributes such as normals and face_normals_idx, indices are properly book-kept. For all other attributes, concatenation is attempted if the attribute is present in all meshes. To ignore concatenation incompatibilities, set skip_errors=True.- Args:
meshes (list of SurfaceMesh): meshes to concatenate; any mix of batching strategies is supported. skip_errors (bool): if True, will not fail if some attributes cannot be flattened. group_materials_by_name (bool): if True, will group materials by name and assume they are the same.
- Returns
new mesh in world space with no
transform- Return type
- ensure_indexed_attribute(attr: str, should_cache: Optional[bool] = None)¶
Sometimes the mesh contains face_normals or face_uvs with values already mapped to every vertex of every face in a single array, and does not have the corresponding indexed values. For example, USD format commonly loads with face_normals, but not normals or face_normals_idx. Or, mesh flattening or concatenation can also cause this to happen. In the event that the indexed attribute is needed, call this function for normals or uvs. Note: behavior is undefined if mesh contains either values or the index as well as face values, e.g. if mesh has face_normals and ‘face_normals_idx’, but not normals, this function will return None, None.
Example
normals, face_normals_idx = mesh.ensure_indexed_attribute(“normals”)
- Parameters
attr – attribute name, only ‘normals’ and ‘uvs’ are supported
should_cache – if
True, will cache attribute if it was computed; ifFalse, will not cache; by default will decide if to cache based onrequires_gradof variables used in computation (will not cache if any hasrequires_grad is True).
- Returns
- (values, index) - e.g. ‘normals’ and ‘face_normals_idx’ with sizes consistent with the current
batching strategy, or None, None if
- as_transformed()¶
Creates a new SurfaceMesh with the stored
transformapplied.Transforms
verticesand direction attributes using the storedtransformmatrix. Normals (normals,vertex_normals,face_normals) transform by \((M^{-1})^T\); tangents (vertex_tangents,face_tangents) transform by the linear part \(M\). All transformed direction vectors are re-normalized.face_verticesis cleared so it will be auto-recomputed on demand. Returns a new SurfaceMesh withtransformunset.transformmay be:Batching.NONE: a(4, 4)tensor.Batching.FIXEDorBatching.LIST: a(4, 4)tensor (applied to all batch elements) or a(B, 4, 4)tensor (one matrix per batch element).