Kaolin Viewer: Layers and Behaviors

This page covers the interactive viewer itself, the catalog of built-in behaviors it understands, how to write your own behaviors, and the typed option schemas that tie Python and the browser together.

Overview & design

The KaolinViewer is a React component, exposed to Python, that provides an interactive 2D/3D surface inside a Plotly Dash app. Its design has two core concepts:

  • Layers – the viewer is composed of stacked, pixel-aligned layers (canvas, svg, …). Layers share the same coordinate space and are composited with opacity, so you can, for example, draw a server-rendered image on one canvas and let the user annotate on another above it.

  • Behaviors – named units of client-side logic that map user interaction (pointer / touch / keyboard) and incoming messages to actions. A behavior may paint on a layer, drive the camera, or stream data to/from the server.

Everything – layers and behaviors alike – runs client-side in the browser. Behaviors are what make rich client-server workflows possible: a behavior can serialize the camera or a custom binary message and send it over a WebSocket, and another behavior can receive server frames and draw them. Heavy work (rendering, inference) stays on the server while interaction stays fluid in the browser. The browser-side helpers used to build behaviors live in the JavaScript API.

Important

Never construct KaolinViewer directly. Always assemble a viewer with ViewerBuilder and call build(). The builder allocates globally-unique layer/behavior ids, validates behavior options against their schemas, and wires the clientside callbacks that a hand-built component would be missing.

Getting started

A typical flow is: create a ViewerBuilder, add one or more layers, attach behaviors by name, optionally generate UI controls for those behaviors, then build() the Dash component:

from kaolin.visualize.dash import ViewerBuilder
from kaolin.render.easy_render import default_camera

builder = ViewerBuilder(camera=default_camera(500))

# 1. Stack a layer (canvas/svg) inside the viewer.
draw_layer = builder.add_layer('canvas')

# 2. Attach a behavior by its registered name (see Built-in Behaviors).
draw_id = builder.add_behavior('drawing', active_layer_id=draw_layer,
                               options={'color': '#ffcc00'})

# 3. Auto-generate UI controls for that behavior's options. Returns Dash
#    components you insert into your app layout.
option_controls = builder.add_user_behavior_options(draw_id)

# 4. Build the Dash component. NEVER instantiate KaolinViewer yourself.
viewer = builder.build()

From here, dive into the detailed sections:

Configuring the viewer

ViewerBuilder is the single entry point for assembling a viewer. The most common methods are:

  • add_layer() – add a canvas / svg layer; returns a builder-level layer id.

  • add_behavior() – attach a behavior by registered name, optionally bound to a layer and configured with options.

  • add_user_behavior_options() – auto-generate UI controls for a behavior’s options (see Built-in behaviors).

  • add_remote_rendering() – one-call setup for server-side rendering: streams the camera to the server and draws the frames it pushes back.

  • build() – produce the KaolinViewer Dash component.

Server-side rendering in a few lines:

builder = ViewerBuilder(camera=default_camera(800))
render_layer = builder.add_layer('canvas')
# Stream the camera to the server and draw the frames it returns.
builder.add_remote_rendering(active_layer_id=render_layer,
                             connection_id='main-ws')
viewer = builder.build()
class kaolin.visualize.dash.viewer.KaolinViewer(*args, **kwargs)

Bases: KaolinViewerInternal

Built-in behaviors

Kaolin ships a library of ready-made behaviors (drawing, SVG annotation, camera sending, remote-image drawing, …). Use them by passing their name to add_behavior(). BehaviorLibrary is the single Python entry point to discover what is available – it merges the build-time library manifest with any behaviors found in your own asset directories:

from kaolin.visualize.dash import BehaviorLibrary

print(BehaviorLibrary)                  # human-readable summary
names = BehaviorLibrary.names()         # available behavior names
meta = BehaviorLibrary.meta('drawing')  # BehaviorMeta (incl. option schema)

A behavior’s option schema is also what powers auto-generated UI. Call add_user_behavior_options() with a behavior id to get Dash controls (plus a hidden store for per-tab persistence); each control edit is pushed to the live behavior in the browser without a server roundtrip:

draw_id = builder.add_behavior('drawing', active_layer_id=draw_layer)

# All uiBound options, or a subset by name / overriding OptionSpec.
controls = builder.add_user_behavior_options(draw_id)
controls = builder.add_user_behavior_options(draw_id, options=['thickness'])

# ...insert `controls` into your Dash layout (e.g. a sidebar).

Built-in / known behavior discovery for the Dash viewer.

BehaviorLibrary is the single Python entry point downstream code (auto-UI, builder, sidebar generators) should use to enumerate behaviors. It merges:

Both sources expose kaolin.visualize.dash.option.BehaviorMeta instances, so downstream code does not need to care which origin a behavior came from.

Singleton style mirrors kaolin.visualize.dash.builder.SessionRegistry: all public methods are class methods working on class-level state, and there is no module-level instance.

Conflict policy: a user-registered name that collides with a library name (or with another user dir) raises ValueError. This makes silent shadowing impossible — rename the conflicting user behavior instead.

class kaolin.visualize.dash.builtins.BehaviorLibrary

Bases: object

Process-wide registry of behavior names and their metadata.

Library behaviors are loaded once from the build-time manifest. User directories are opt-in: callers (e.g. WebappBuilder / ViewerBuilder) call register_user_directory() once per directory; subsequent names() / meta() / is_builtin_behavior() calls reflect the merged view.

All state is class-level; do not instantiate. The class is intended to be used directly:

from kaolin.visualize.dash.builtins import BehaviorLibrary

BehaviorLibrary.register_user_directory('kaolin/app/segment/assets')
print(BehaviorLibrary)                          # summary
print(BehaviorLibrary.to_string(True))          # detailed
print(BehaviorLibrary.behavior_to_string('drawing', True))
BehaviorLibrary.is_builtin_behavior('drawing')  # True
BehaviorLibrary.is_user_behavior('drawing')     # False
BehaviorLibrary.meta('drawing')

Not thread-safe; assume single-threaded build/server initialization (matches Dash’s threading model for layout assembly).

classmethod all_meta() → dict[str, kaolin.visualize.dash.option.BehaviorMeta]

Return a merged {name: BehaviorMeta} dict (library + user).

classmethod behavior_to_string(name: str, detailed: bool = False) → str

Same semantics as to_string(), but for a single behavior.

Parameters

  • name – behavior name to render.

  • detailed – if True, include description, element binding and option schema (matching to_string() detailed mode).

Raises

ValueError – if name is neither a library nor a user behavior.

classmethod invalidate_user_cache() → None

Force a re-scan of user directories on the next lookup.

classmethod is_builtin_behavior(name: str) → bool

True iff name appears in the build-time library manifest.

classmethod is_user_behavior(name: str) → bool

True iff name was discovered in a registered user directory.

classmethod meta(name: str) → kaolin.visualize.dash.option.BehaviorMeta | None

Return the BehaviorMeta for name, or None if unknown.

Name collisions are impossible by construction (they raise at register_user_directory() time), so this is an unambiguous lookup.

classmethod names() → list[str]

All known behavior names (library + user), sorted.

classmethod register_user_directory(path: str | pathlib.Path) → None

Register a directory whose .js/.ts/.tsx files should be scanned for BehaviorRegister.register(...) calls.

Duplicate registrations are ignored. The clash check (against the library and against previously registered user dirs) runs eagerly here so the offending call site is the one that raises.

classmethod to_string(detailed: bool = False) → str

Human-readable summary of the registry, one behavior per line.

Parameters

detailed – if True, include the per-behavior element binding and its options schema (a behavior’s effective input surface).

Built-in and user behaviors are listed under separate headers.

Custom behaviors

Behaviors are resolved by name in the browser, so a custom behavior is just a class registered under a name – plain JavaScript is enough, no TypeScript or build step required. Drop a script (e.g. custom.js) into your app’s served assets; the kaolin namespace is loaded automatically (see the JavaScript API).

// custom.js -- loaded in the browser; `kaolin` is available globally.
const { CanvasBehavior, BehaviorRegister, OptionKind } = kaolin.core.behavior;

class DotBehavior extends CanvasBehavior {
    // Schema drives Python-side validation and auto-UI (see Behavior Schemas).
    static schema = { radius: { kind: OptionKind.INT, default: 6, min: 1, max: 40 } };

    // Re-render whenever an option is edited from Python.
    updateForOptions() { this.redraw(); }

    // CanvasBehavior provides `this.element` / `this.ctx` and pointer handlers.
    onPointerDown(event) { this.x = event.offsetX; this.y = event.offsetY; this.redraw(); }

    redraw() {
        if (!this.ctx || this.x === undefined) return;
        this.ctx.beginPath();
        this.ctx.arc(this.x, this.y, this.options.radius, 0, 2 * Math.PI);
        this.ctx.fill();
    }
}

// 'dot' is the name you pass to add_behavior(...) from Python.
BehaviorRegister.register('dot', DotBehavior, 'Draws a dot where you click.');

Then use it from Python exactly like a built-in:

dot_layer = builder.add_layer('canvas')
dot_id = builder.add_behavior('dot', active_layer_id=dot_layer,
                              options={'radius': 10})

See the JavaScript API for the base classes (Behavior, InteractiveBehavior, CanvasBehavior, CameraControllerBase, MessageHandlerBase), the event interfaces, and the binary message I/O utilities used for custom client-server communication.

Behavior schemas

Each behavior declares an option schema: the set of configurable parameters, their types, bounds, and defaults. On the browser side this is a Zod schema (or, in plain JS, the static schema object shown above); at build time it is projected to JSON and read back on the Python side as OptionSpec objects.

A single OptionSpec describes one option and is the unit that flows through the whole system:

  • it round-trips through JSON (Python ⇄ TypeScript) via as_dict / from_dict;

  • BehaviorLibrary exposes the specs of every known behavior;

  • add_user_behavior_options() turns specs into Dash controls and validates option values against them.

You rarely construct one by hand, but you can – for example to override a bound or define an option for a custom behavior that has no manifest:

from kaolin.visualize.dash.option import OptionSpec, OptionKind

thickness = OptionSpec(
    name='thickness', kind=OptionKind.INT, default=4, minimum=1, maximum=20)

# Expose just this option, with our overridden bounds, as UI for a behavior.
controls = builder.add_user_behavior_options(draw_id, options=[thickness])

Typed description of a single behavior / UI option.

Defines the OptionKind enum and OptionSpec, the Python-side mirror of the TypeScript OptionKind / OptionSpec in kaolin/visualize/dash/components/src/ts/core/behavior/option.ts. Field names are intentionally kept identical across the two sides so specs round-trip through JSON (see OptionSpec.as_dict() / OptionSpec.from_dict()).

An OptionSpec can also be derived from a typed Python source via OptionSpec.from_annotated_field(), which reads numeric bounds, step, and allowed-value sets from PEP 593 Annotated[...] metadata.

class kaolin.visualize.dash.option.BehaviorMeta(description: str | None = None, options: dict[str, kaolin.visualize.dash.option.OptionSpec] | None = None, source_path: str | None = None)

Bases: object

Metadata for a single registered behavior: a description plus its option specs.

Python-side mirror of the TypeScript BehaviorMeta in kaolin/visualize/dash/components/src/ts/core/behavior/option.ts, and the decoded form of one entry in the behavior manifest emitted by dump_behavior_manifest.ts. Use from_dict() to read a manifest entry into typed OptionSpec instances and as_dict() for the inverse (the JSON round-trip is symmetric with the TS toJSON / fromJson).

source_path is Python-only provenance (the user-asset file a behavior was discovered in); it has no TypeScript counterpart and is omitted from as_dict() when unset.

TODO: extend both this class and the TypeScript BehaviorMeta with the behavior’s interface flags once the dumper can emit them, e.g. is_element_bound / is_message_handler / is_camera_controller (camelCase isElementBound / isMessageHandler / isCameraController on the wire).

__init__(description: str | None = None, options: dict[str, kaolin.visualize.dash.option.OptionSpec] | None = None, source_path: str | None = None)

Create behavior metadata.

Parameters

  • description (str | None) – human-readable behavior description, or None if none.

  • options (dict[str, OptionSpec] | None) – option specs keyed by name; None is treated as an empty mapping.

  • source_path (str | None) – provenance for a user-scanned behavior (the file it was registered in), or None for library behaviors.

as_dict() → dict

Serialize to a JSON-friendly dict (manifest entry form).

Each option is serialized via OptionSpec.as_dict(), so the result round-trips through from_dict(). sourcePath is included only when set.

Returns

(dict) a dict with description and an options mapping of name -> OptionSpec.as_dict() (plus sourcePath if present).

as_string(name: str, detailed: bool = False) → str

Render this behavior as human-readable, indented text.

Parameters

  • name (str) – the behavior’s registered name (kept as the registry key, not stored on the instance).

  • detailed (bool) – if True, include description, provenance, and the per-option summary; otherwise a single - name line.

Returns

(str) the rendered (possibly multi-line) summary.

classmethod from_dict(data: dict) → BehaviorMeta

Build a BehaviorMeta from a manifest entry dict.

Inverse of as_dict(). Each option dict is rebuilt via OptionSpec.from_dict() (falling back to the mapping key for a spec’s name if absent). sourcePath (or source_path) is read as provenance. Unknown top-level keys raise ValueError.

Parameters

data (dict) – a manifest entry with optional description, options, and sourcePath keys.

Returns

(BehaviorMeta) the decoded metadata.

class kaolin.visualize.dash.option.OptionKind(value, names=<not given>, *values, module=None, qualname=None, type=None, start=1, boundary=None)

Bases: str, Enum

Supported behavior-option value kinds.

Mirrors the TS-side OptionKind enum in kaolin/visualize/dash/components/src/ts/core/behavior/option.ts. The enum is str-mixed so members compare equal to their string values (OptionKind.INT == 'int') and round-trip cleanly through JSON / dict-shaped option specs.

class kaolin.visualize.dash.option.OptionSpec(name: str, kind: OptionKind, default: Any = None, description: str | None = None, min: int | float | None = None, max: int | float | None = None, step: int | float | None = None, values: list | None = None, ui_bound: bool | None = None)

Bases: object

Declarative description of a single behavior / UI option.

Mirrors the TS-side OptionSpec interface in kaolin/visualize/dash/components/src/ts/core/behavior/option.ts. Field names are intentionally kept identical to the TS side (including the camelCase uiBound emitted by as_dict()) so the two definitions stay trivially comparable.

A spec is typically flat (one scalar value), but a OptionKind.GROUP spec may hold ordered children to describe nested option groups.

Parameters

  • name (str) – option name; matches its key within the behavior schema.

  • kind (OptionKind) – value kind, drives which auto-UI control is generated.

  • default (Any) – default value used when the caller omits this key. Stored in its JSON-serializable form, not the rich Python form: for an enum option pass the member’s value (e.g. 'a'), not the member itself. Use add_type_cast() to register Python<->JSON casters.

  • description (Optional[str]) – human-readable label / tooltip for the auto-UI.

  • min (Optional[int | float]) – numeric lower bound (numeric kinds only).

  • max (Optional[int | float]) – numeric upper bound (numeric kinds only).

  • step (Optional[int | float]) – numeric step (numeric kinds only).

  • values (Optional[list]) – allowed value set (enum kind only).

  • ui_bound (Optional[bool]) – whether the option is user-accessible through UI.

add_child(child: OptionSpec) → None

Append a child spec; only valid when this spec is a group kind.

Parameters

child (OptionSpec) – the nested option to add.

Raises

ValueError – if this spec’s kind is not OptionKind.GROUP.

add_type_cast(py_to_json: Callable[[Any], Any], json_to_py: Callable[[Any], Any]) → None

Register custom JSON<->Python value casters for this option.

Casters convert between an option’s rich Python value and its JSON-serializable form – e.g. an enum member and its .value. They are intentionally not constructor arguments and are private to the instance: they are never serialized, and as_dict() warns if any custom caster is set (since it cannot round-trip a callable).

Parameters

  • py_to_json (Callable) – converts a Python value to its JSON form.

  • json_to_py (Callable) – converts a JSON value back to its Python form.

as_dict() → dict

Serialize this spec to a JSON-friendly dict (TS-compatible keys).

kind is emitted as its string value and None fields are omitted, so the result round-trips through from_dict(). Custom casters set via add_type_cast() are not serialized; a warning is logged if any are set.

Returns

(dict) a dict with name and kind always present, plus any set optional fields (default, description, min, max, step, values, uiBound) and, for group kinds, a recursively serialized children list.

as_string() → str

One-line, human-readable summary of this option (kind + range/values + default + description), suitable for listing under a behavior.

Returns

(str) the summary line.

classmethod from_annotated_field(name, annotation: Any, default: Any = None) → OptionSpec

Build an OptionSpec from a (name, annotation, default) triple.

Reads numeric bounds / step / allowed-values out of PEP 593 metadata via _min_max_from_annotation_metadata(), _step_from_annotation_metadata(), and _values_from_annotation_metadata(), then maps the base type to an OptionKind. When the base type is an enum.Enum subclass a member <-> value caster is registered via add_type_cast().

Examples

Supported annotation forms:

from typing import Annotated, Literal
from annotated_types import Ge, Gt, Interval, MultipleOf

# Bare scalar types -> int / float / bool / string options
OptionSpec.from_annotated_field('count', int, default=1)
OptionSpec.from_annotated_field('name', str, default='cube')
OptionSpec.from_annotated_field('visible', bool, default=True)

# Numeric bounds / step, via annotated_types or a plain dict
OptionSpec.from_annotated_field('age', Annotated[int, Gt(0)], default=1)
OptionSpec.from_annotated_field('ratio', Annotated[float, Interval(ge=0, le=1)], default=0.5)
OptionSpec.from_annotated_field('size', Annotated[int, MultipleOf(2)], default=2)
OptionSpec.from_annotated_field('size', Annotated[int, {'min': 0, 'max': 10, 'step': 2}], default=2)

# Enum options: a Literal, an explicit value list, or an enum.Enum subclass
OptionSpec.from_annotated_field('mode', Literal['a', 'b', 'c'], default='a')
OptionSpec.from_annotated_field('mode', Annotated[str, {'values': ['a', 'b']}], default='a')
OptionSpec.from_annotated_field('mode', Annotated[str, ['a', 'b']], default='a')
OptionSpec.from_annotated_field('mode', Color, default=Color.RED)  # Color(enum.Enum)

# Mark an option as UI-bound via a UIBound marker or a dict
OptionSpec.from_annotated_field('age', Annotated[int, Gt(0), UIBound(True)], default=1)
OptionSpec.from_annotated_field('age', Annotated[int, Gt(0), {'uiBound': True}], default=1)
Parameters

  • name (str) – option name.

  • annotation (Annotated) – the (possibly Annotated[...]) type annotation.

  • default (Any) – default value for the option.

Returns

(OptionSpec) the constructed (and validated) spec.

classmethod from_dict(data: dict) → OptionSpec

Build an OptionSpec from a JSON-friendly dict.

Inverse of as_dict(); kind may be an OptionKind or its string value, uiBound is accepted as an alias for ui_bound, and a children list is reconstructed recursively (only valid for group kinds, enforced by add_child()).

Parameters

data (dict) – dict with at least name and kind keys.

Returns

(OptionSpec) the constructed (and validated) spec.

property json_to_py: Callable[[Any], Any]

Caster turning this option’s JSON-friendly value into its Python value.

merged(overrides: dict | None = None) → OptionSpec

Return a new spec copied from this one with overrides applied.

Implemented on top of as_dict() / from_dict() so the set of serializable fields (and the children handling) lives in exactly one place. overrides keys are therefore the as_dict() field names (default, min, max, step, values, description, name, kind, uiBound); ui_bound is accepted as an alias for uiBound and an OptionKind is accepted for kind. The copy is re-validated and custom casters are carried over. Passing no overrides returns a validated copy.

Parameters

overrides (Optional[dict]) – field overrides to apply over this spec.

Returns

(OptionSpec) the merged, validated copy.

Raises
property py_to_json: Callable[[Any], Any]

Caster turning this option’s Python value into its JSON-friendly form.

validate() → None

Check field consistency, raising ValueError on a bad spec.

Enforces the option contract: numeric bounds (min / max / step) only apply to OptionKind.INT / OptionKind.FLOAT, values only applies to OptionKind.ENUM, and min <= max when both given. Also runs a loose check on default (when set): an enum default must be one of values, and any other default must survive the kind’s caster.

Raises

ValueError – if any field is inconsistent with kind.

class kaolin.visualize.dash.option.UIBound(is_bound: bool = True)

Bases: object

PEP 593 annotation marking whether an option is user-accessible through UI.

Use inside an Annotated[...] to set OptionSpec.ui_bound from a typed Python source, e.g. Annotated[int, Gt(0), UIBound(True)]. The plain dict forms {'ui_bound': ...} / {'uiBound': ...} are accepted as alternatives.

Parameters

is_bound (bool) – whether the option is exposed in the auto-generated UI.

kaolin.visualize.dash.option.specs_from_dataclass(instance: Any, names: Optional[Iterable[str]] = None, annotations: dict[str, Any] | None = None) → list[kaolin.visualize.dash.option.OptionSpec]

Build an OptionSpec list from a dataclass instance or class.

Each field is mapped to a spec via OptionSpec.from_annotated_field(), reading its default from the instance’s current attribute value. A nested dataclass field becomes an OptionKind.GROUP spec whose children are the nested fields (recursively); a nested dataclass without a default instance to recurse into is skipped with a warning.

A dataclass class is also accepted, provided it can be instantiated with no arguments (i.e. every field has a default); the defaults of that throwaway instance are used.

Parameters

  • instance (Any) – a dataclass instance, or a dataclass class instantiable with defaults.

  • names (Optional[Iterable[str]]) – field names to include and order; defaults to all fields in declaration order.

  • annotations (Optional[dict]) – {field_name: annotation} overrides that supplement or replace individual field annotations.

Returns

(list) the constructed OptionSpec objects, in the resolved field order.

Raises

  • TypeError – if instance is not a dataclass, or is a dataclass class that cannot be instantiated with defaults.

  • ValueError – if names contains a field not present on the dataclass.

kaolin.visualize.dash.option.specs_from_function(fnc: Callable, names: Optional[Iterable[str]] = None, annotations: dict[str, Any] | None = None) → list[kaolin.visualize.dash.option.OptionSpec]

Build an OptionSpec list from a callable’s signature.

Behaves like specs_from_dataclass() but introspects a function signature: *args / **kwargs are skipped, a parameter without a default is treated as default=None, and parameters lacking a type annotation are skipped with a warning.

Parameters

  • fnc (Callable) – callable to introspect (bound methods are supported).

  • names (Optional[Iterable[str]]) – parameter names to include and order; defaults to all accepted parameters in declaration order.

  • annotations (Optional[dict]) – {param_name: annotation} overrides that supplement or replace individual parameter annotations.

Returns

(list) the constructed OptionSpec objects, in the resolved param order.

Raises

ValueError – if names contains a parameter not present on fnc.