kaolin.visualize

API

class kaolin.visualize.BaseIpyVisualizer(height, width, camera, render, fast_render=None, watched_events=None, max_fps=None, canvas=None, event_canvas=None)

Bases: object

Base class for ipython visualizer.

To create a visualizer one must define the class attribute _WATCHED_EVENTS and the method _handle_event().

the method _handle_event() must use the methods self.render() or self.fast_render() to update the canvas

You can overload the constructor (make sure to reuse the base class one so that ipycanvas and ipyevents are properly used)

Parameters

• height (int) – Height of the canvas.

• width (int) – Width of the canvas.

• camera (kaolin.render.camera.Camera) – Camera used for the visualization.

• render (Callable) – render function that take a kal.render.camera.Camera as input. Must return a torch.ByteTensor as output, or a dictionary where the element ‘img’ is a torch.ByteTensor to be displayed, of shape \((\text{output_height}, \text{output_width}, 3)\), height and width don’t have to match canvas dimension.

• fast_render (optional, Callable) – A faster rendering function that may be used when doing high frequency manipulation such as moving the camera with a mouse. Default: same than render.

• watched_events (list of str) – Events to be watched by the visualizer (see ipyevents main documentation).

• max_fps (optional, float) – maximum framerate for handling consecutive events, this is useful when the rendering is slow to avoid freezes. Typically 24 fps is great when working on a local machine with render() close to real-time, and lower to 10 fps with slower rendering or network latency.

• canvas (optional, ipycanvas.Canvas) – If you don’t want the visualizer to create a canvas automatically, pass the canvas object to be drawn on to this function. By default, this canvas will also be used for processing events. Note that in case of ipycanvas.MultiCanvas, only the parent MultiCanvas object can process events, so if you are drawing of a sub-canvas of a MultiCanvas, pass the MultiCanvas object as event_canvas=. Dimensions must be matching height and width.

• event_canvas (optional, ipywidgets.DOMWidget) – If you want visualizer to receive events from a different canvas object from the one that the rendering shows on, pass this object here. Note that in case of ipycanvas.MultiCanvas, only the parent MultiCanvas object can process events. By default, the same canvas is used for events and drawing.

camera

The camera used for rendering.

Type

(kaolin.render.camera.Camera)

canvas

The canvas on which the rendering is displayed.

Type

(ipycanvas.Canvas)

current_output

The current output of the rendering function.

Type

(torch.Tensor)

event

the event handler.

Type

(ipyevents.Event)

event_canvas

The widget used to handle the events.

Type

(ipywidgets.DOMWidget)

fast_render

The fast rendering function.

Type

(Callable)

fast_render_update()

Update the Canvas with fast_render()

height

The Canvas height.

Type

(int)

property max_fps

maximum fps for handling consecutive events

out

An output where error and prints are displayed.

Type

(ipywidgets.Output)

render

The rendering function.

Type

render (Callable)

render_update()

Update the Canvas with render()

show()

display the Canvas with interactive features

width

The Canvas width.

Type

(int)

class kaolin.visualize.IpyFirstPersonVisualizer(height, width, camera, render, fast_render=None, world_up=None, zoom_sensitivity=0.001, rotation_sensitivity=0.4, translation_sensitivity=1.0, key_move_sensitivity=0.05, max_fps=24.0, up_key='i', down_key='k', left_key='j', right_key='l', forward_key='o', backward_key='u', update_only_on_release=False, additional_watched_events=None, additional_event_handler=None, canvas=None, event_canvas=None)

Bases: BaseIpyVisualizer

An interactive first person visualizer that can display on jupyter notebook.

You can move the orientation with the left button of the mouse, move the position of the camera with the right button of the mouse or the associated key, and zoom with the wheel.

Parameters

• height (int) – Height of the canvas.

• width (int) – Width of the canvas.

• camera (kal.render.camera.Camera) – Camera used for the visualization.

• render (Callable) – render function that take a kal.render.camera.Camera as input. Must return a torch.ByteTensor as output, or a dictionary where the element ‘img’ is a torch.ByteTensor to be displayed, of shape \((\text{output_height}, \text{output_width})\), height and width don’t have to match canvas dimension.

• fast_render (optional, Callable) – A faster rendering function that may be used when doing high frequency manipulation such as moving the camera with a mouse. Default: same than render.

• world_up (optional, torch.Tensor) – World up axis, of shape \((3,)\). If provided the camera will be reoriented to avoid roll. Default: camera.cam_up().

• zoom_sensitivity (float) – Sensitivity of the wheel on zoom. Default: 1e-3.

• rotation_sensitivity (float) – Sensitivity of the mouse on rotations. Default: 0.4.

• translation_sensitivity (float) – Sensitivity of the mouse on camera translation. Default: 1.

• key_move_sensitivity (float) – Amount of camera movement on key press. Default 0.05.

• max_fps (optional, float) – maximum framerate for handling consecutive events, this is useful when the rendering is slow to avoid freezes. Typically 24 fps is great when working on a local machine with render() close to real-time. And you lower to 10 fps with slower rendering or network latency. Default: 24 fps.

• up_key (str) – key associated to moving up. Default ‘i’.

• down_key (str) – key associated to moving up. Default ‘k’.

• left_key (str) – key associated to moving up. Default ‘j’.

• right_key (str) – key associated to moving up. Default ‘l’.

• forward_key (str) – key associated to moving up. Default ‘o’.

• backward_key (str) – key associated to moving up. Default ‘u’.

• update_only_on_release (bool) – If true, the canvas won’t be updated while the mouse button is pressed and only when it’s released. To avoid freezes with very slow rendering functions. Default: False.

• additional_watched_events (optional, list of str) – Additional events to be watched by the visualizer (see ipyevents main documentation). To be used for customed events such as enabling / disabling a feature on a key press. [‘wheel’, ‘mousedown’, ‘mouseup’, ‘mousemove’, ‘mouseleave’] are already watched. Default: None.

• additional_event_handler (optional, Callable) – Additional event handler to be used for customed events such as enabling / disabling a feature on a key press. The Callable must take as input a tuple of (this visualizer object, the event). (see ipyevents main documentation).

• res (optional, tuple of ints) – height and width of the canvas, if not defined will get inferred from canvas if defined, otherwise from the output of render.

• canvas (optional, ipycanvas.Canvas) – If you don’t want the visualizer to create a canvas automatically, pass the canvas object to be drawn on to this function. By default, this canvas will also be used for processing events. Note that in case of ipycanvas.MultiCanvas, only the parent MultiCanvas object can process events, so if you are drawing of a sub-canvas of a MultiCanvas, pass the MultiCanvas object as event_canvas=. Dimensions must be matching height and width.

• event_canvas (optional, ipywidgets.DOMWidget) – If you want visualizer to receive events from a different canvas object from the one that the rendering shows on, pass this object here. Note that in case of ipycanvas.MultiCanvas, only the parent MultiCanvas object can process events. By default, the same canvas is used for events and drawing.

world_up

Type

torch.Tensor

zoom_sensitivity

Type

float

rotation_sensitivity

Type

float

translation_sensitivity

Type

float

key_move_sensitivity

Type

float

update_only_on_release

Type

bool

camera

The camera used for rendering.

Type

(kaolin.render.camera.Camera)

canvas

The canvas on which the rendering is displayed.

Type

(ipycanvas.Canvas)

current_output

The current output of the rendering function.

Type

(torch.Tensor)

event

the event handler.

Type

(ipyevents.Event)

event_canvas

The widget used to handle the events.

Type

(ipywidgets.DOMWidget)

fast_render

The fast rendering function.

Type

(Callable)

fast_render_update()

Update the Canvas with fast_render()

height

The Canvas height.

Type

(int)

property max_fps

maximum fps for handling consecutive events

out

An output where error and prints are displayed.

Type

(ipywidgets.Output)

render

The rendering function.

Type

render (Callable)

render_update()

Update the Canvas with render()

show()

display the Canvas with interactive features

width

The Canvas width.

Type

(int)

class kaolin.visualize.IpyTurntableVisualizer(height, width, camera, render, fast_render=None, focus_at=None, world_up_axis=1, zoom_sensitivity=0.001, forward_sensitivity=0.001, rotation_sensitivity=1.5, translation_sensitivity=1.0, max_fps=24.0, update_only_on_release=False, additional_watched_events=None, additional_event_handler=None, canvas=None, event_canvas=None)

Bases: BaseIpyVisualizer

An interactive turntable visualizer that can display on jupyter notebook.

You can move around with the mouse (using the left button), zoom with the wheel and get closer to the center with the wheel + control key.

Parameters

• height (int) – Height of the canvas.

• width (int) – Width of the canvas.

• camera (kal.render.camera.Camera) – Camera used for the visualization. Note: The camera will be reoriented to look at focus_at and with respect to world_up.

• render (Callable) – render function that take a kal.render.camera.Camera as input. Must return a torch.ByteTensor as output, or a dictionary where the element ‘img’ is a torch.ByteTensor to be displayed, of shape \((\text{output_height}, \text{output_width})\), height and width don’t have to match canvas dimension.

• fast_render (optional, Callable) – A faster rendering function that may be used when doing high frequency manipulation such as moving the camera with a mouse. Default: same than render.

• focus_at (optional, torch.Tensor) – The center of the turntable on which the camera is focusing on. Default: (0, 0, 0).

• world_up_axis (optional, int) – The up axis of the world, in the coordinate system. Default: 1.

• zoom_sensitivity (float) – Sensitivity of the wheel on zoom. Default: 1e-3.

• forward_sensitivity (float) – Sensitivity of the wheel on forward. Default: 1e-3.

• rotation_sensitivity (float) – Sensitivity of the mouse on left click movements. Default: 1.5.

• translation_sensitivity (float) – Sensitivity of the mouse on right click movements. Default: 1.

• max_fps (optional, float) – maximum framerate for handling consecutive events, this is useful when the rendering is slow to avoid freezes. Typically 24 fps is great when working on a local machine with render() close to real-time. And you lower to 10 fps with slower rendering or network latency. Default: 24 fps.

• update_only_on_release (bool) – If true, the canvas won’t be updated while the mouse button is pressed and only when it’s released. To avoid freezes with very slow rendering functions. Default: False.

• additional_watched_events (optional, list of str) – Additional events to be watched by the visualizer (see ipyevents main documentation). To be used for customed events such as enabling / disabling a feature on a key press. [‘wheel’, ‘mousedown’, ‘mouseup’, ‘mousemove’, ‘mouseleave’] are already watched. Default: None.

• additional_event_handler (optional, Callable) – Additional event handler to be used for customed events such as enabling / disabling a feature on a key press. The Callable must take as input a tuple of (this visualizer object, the event). (see ipyevents main documentation).

• canvas (optional, ipycanvas.Canvas) – If you don’t want the visualizer to create a canvas automatically, pass the canvas object to be drawn on to this function. By default, this canvas will also be used for processing events. Note that in case of ipycanvas.MultiCanvas, only the parent MultiCanvas object can process events, so if you are drawing of a sub-canvas of a MultiCanvas, pass the MultiCanvas object as event_canvas=. Dimensions must be matching height and width.

• event_canvas (optional, ipywidgets.DOMWidget) – If you want visualizer to receive events from a different canvas object from the one that the rendering shows on, pass this object here. Note that in case of ipycanvas.MultiCanvas, only the parent MultiCanvas object can process events. By default, the same canvas is used for events and drawing.

focus_at

Type

torch.Tensor

world_up_axis

Type

int

zoom_sensitivity

Type

float

rotation_sensitivity

Type

float

mouse_sensitivity

Type

float

update_only_on_release

Type

bool

camera

The camera used for rendering.

Type

(kaolin.render.camera.Camera)

canvas

The canvas on which the rendering is displayed.

Type

(ipycanvas.Canvas)

current_output

The current output of the rendering function.

Type

(torch.Tensor)

event

the event handler.

Type

(ipyevents.Event)

event_canvas

The widget used to handle the events.

Type

(ipywidgets.DOMWidget)

fast_render

The fast rendering function.

Type

(Callable)

fast_render_update()

Update the Canvas with fast_render()

height

The Canvas height.

Type

(int)

property max_fps

maximum fps for handling consecutive events

out

An output where error and prints are displayed.

Type

(ipywidgets.Output)

render

The rendering function.

Type

render (Callable)

render_update()

Update the Canvas with render()

show()

display the Canvas with interactive features

width

The Canvas width.

Type

(int)

class kaolin.visualize.Timelapse(log_dir, up_axis='Y')

Bases: object

add_mesh_batch(iteration=0, category='', vertices_list=None, faces_list=None, uvs_list=None, face_uvs_idx_list=None, face_normals_list=None, materials_list=None)

Add meshes to visualizer output.

Parameters

• iteration (int) – Positive integer identifying the iteration the supplied meshes belong to.

• category (str, optional) – Batch name.

• vertices_list (list of tensors, optional) – Vertices for N meshes of shape (num_vertices, 3).

• faces_list (list of tensors, optional) – Faces for N meshes of shape (num_faces, face_size).

• uvs_list (list of tensors, optional) – UV coordinates for N meshes of shape (num_uvs, 2).

• face_uvs_idx_list (list of tensors, optional) – Index of UV coordinates for N meshes of shape (num_faces, face_size).

• face_normals_list (list of tensors, optional) – Face normals for N meshes of shape (num_faces, face_size, 3).

• materials_list (list, optional) – List of materials for N meshes. For each mesh, if a list of io.Materials is supplied, each material is applied to the mesh as a ShadingVariant. A name for each material can be defined by supplying a dictionary in the form of {‘material_name’: material}.

add_pointcloud_batch(iteration=0, category='', pointcloud_list=None, colors=None, points_type='point_instancer')

Add pointclouds to visualizer output.

Parameters

• iteration (int) – Positive integer identifying the iteration the supplied pointclouds belong to.

• category (str, optional) – Batch name.

• pointcloud_list (list of tensors, optional) – Batch of point clouds as (B x N x 3) tensor or list of variable length point cloud tensors, each (N_i x 3).

• colors (list of tensors, optional) – Batch of RGB colors of length N.

• points_type (str) – String that indicates whether to save pointcloud as UsdGeomPoints or PointInstancer. “usd_geom_points” indicates UsdGeomPoints and “point_instancer” indicates PointInstancer. Please refer here for UsdGeomPoints: https://graphics.pixar.com/usd/docs/api/class_usd_geom_points.html and here for PointInstancer https://graphics.pixar.com/usd/docs/api/class_usd_geom_point_instancer.html. Default: “point_instancer”.

add_voxelgrid_batch(iteration=0, category='', voxelgrid_list=None, colors=None, semantic_ids=None)

Add voxelgrids to visualizer output.

Parameters

• iteration (int) – Positive integer identifying the iteration the supplied voxelgrids belong to.

• category (str, optional) – Batch name.

• voxelgrid_list (list of tensors, optional) – Batch of points of length N defining N pointclouds.

• colors (list of tensors, optional) – Batch of RGB colors of length N.

• semantic_ids (list of int, optional) – Batch of semantic IDs.

class kaolin.visualize.TimelapseParser(log_dir)

Bases: object

Utility class for working with log directories created using the Timelapse interface. For example, this class can be used to extract raw data from the written checkpoints as follows:

Example

# During training timelapse = Timelapse(log_dir) timelapse.add_pointcloud_batch(iteration=idx, category=”prediction”, pointcloud_list=[predictions])

# Examining training run parser = TimelapseParser(log_dir) path = parser.get_file_path(“pointcloud”, “prediction”, 0) cloud = kaolin.io.usd.import_pointclouds(path, time=iteration_number) # time should be iteration number # Visualize, save or analyze as desired

class CategoryInfo(category, ids=None, end_time=0)

Bases: object

Corresponds to a “category” specified in Timelapse for a specific type like “mesh”. The ids corresponds to the number of objects saved in calls like add_mesh_batch, and end_time is the largest end time in the group.

add_instance(new_id, end_timecode)

serializable()

check_for_updates()

Updates parse information if it has changed in the logdir.

Returns

(bool) - True if updates exist, False if not

get_category_info(type, category)

get_category_names_by_type(type)

get_file_path(type, category, id)

Gets file path by keys. :param type: one of “mesh”, “pointcloud”, “voxelgrid” :type type: str :param category: category passed to Timelapse when writing checkpoints :type category: str :param id: id of the item within its batch :type id: int

Returns

(str) or None

static get_filepaths(logdir)

Get all USD file paths within a directory that match naming conventions imposed by Timelapse.

Parameters

logdir (str) – root directory where USD timelapse files are written

Returns

keyed by tuples (typestr, category, id_within_batch) with values

containing full USD file paths

Return type

dict

num_mesh_categories()

num_mesh_items()

num_pointcloud_categories()

num_pointcloud_items()

num_voxelgrid_categories()

num_voxelgrid_items()

static parse_filepath_info(filepaths)

Parses a directory of checkpoints written by Timelapse module into a summary format.

Parameters

filepaths – dictionary output by get_filepaths

Returns

dictionary keyed by checkpoint type (“mesh”, “pointcloud”, “voxelgrid”)

with each value a list of (serializable) CategoryInfo

kaolin.visualize.update_canvas(canvas, image)