Reference

nerfvis.scene

class Scene(title: str = 'Scene', default_opencv: bool = False)

Bases: object

Holds radiance field/volume/mesh/point cloud/lines objects for 3D visualization. Add objects using add_* as seen below, then use export()/display()/embed() to create a standalone web viewer you can open in a browser or embed in a notebook.

  • Single scene quick import: from nerfvis import scene

  • Multiple scenes: from nerfvis import Scene then scene = Scene(‘title’)

Parameters:

  • title – title to show when saving, default ‘Scene’. You can change it later by setting scene.title = ‘…’.

  • default_opencv – whether to use OpenCV camera space (instead of OpenGL). This can be changed by scene.set_opencv() and scene.set_opengl().

add_cube(name: str = 'cube', **kwargs)

Add a cube with side length 1 (verts {-0.5, 0.5}^3).

Parameters:

  • name – an identifier for this object

  • color – (3,) color, default is orange (common param)

  • vert_color – (36, 3) vertex color, optional advanced (common param)

  • translation – (3,), model translation (common param)

  • rotation – (3,), model rotation in axis-angle (common param)

  • scale – float, scale, default 1.0 (common param)

  • visible – bool, whether mesh should be visible on init, default true (depends on GET parameter in web version) (common param)

  • unlit – bool, whether mesh should be rendered unlit (common param)

  • time – int, time at which the mesh should be displayed; -1=always display (default) (common param)

set_world_up(world_up: ndarray)

Set world up vector (3D)

set_opencv()

Use OpenCV camera space (x,y,z). Affects all camera frustums and images added after

set_opengl()

Use OpenGL camera space (x,-y,-z). Affects all camera frustums and images added after

set_opencv_world()

Use OpenCV world space (y down)

set_opengl_world()

Use OpenGL world space (y up)

set_blender_world()

Use Blender world space (z up)

set_title(title: str)

Set title of output page

add_wireframe_cube(name: str, update_bb: bool = False, **kwargs)

Add a wireframe cube with side length 1 (verts {-0.5, 0.5}^3). Uses add_lines.

Parameters:

  • name – an identifier for this object

  • update_bb – bool, whether to update the bounding box, default False (since these cubes are usually used as reference only)

  • color – (3,) color, default is orange (common param)

  • vert_color – (36, 3) vertex color, optional advanced (common param)

  • translation – (3,), model translation (common param)

  • rotation – (3,), model rotation in axis-angle (common param)

  • scale – float, scale, default 1.0 (common param)

  • visible – bool, whether mesh should be visible on init, default true (depends on GET parameter in web version) (common param)

  • unlit – bool, whether mesh should be rendered unlit. Default True (common param)

  • time – int, time at which the mesh should be displayed; -1=always display (default) (common param)

add_arrow(name: str, a: ndarray | List[float], b: ndarray | List[float], arrow_size: float = 0.12, update_bb: bool = False, **kwargs)

Add a basic arrow (currently made of lines)

Parameters:

  • name – an identifier for this object

  • a – (3,) start point

  • b – (3,) end point

  • arrow_size – float, size of arrow head

  • update_bb – bool, whether to update the bounding box, default False

  • color – (3,) color, default is orange (common param)

  • vert_color – (rings*sectors, 3) vertex color, optional advanced (common param)

  • translation – (3,), model translation (common param)

  • rotation – (3,), model rotation in axis-angle (common param)

  • scale – float, scale, default 1.0 (common param)

  • visible – bool, whether mesh should be visible on init, default true (depends on GET parameter in web version) (common param)

  • unlit – bool, whether mesh should be rendered unlit (common param)

  • time – int, time at which the mesh should be displayed; -1=always display (default) (common param)

add_sphere(name: str = 'sphere', rings: int | None = None, sectors: int | None = None, update_bb: bool = True, **kwargs)

Add a UV sphere with radius 1

Parameters:

  • name – an identifier for this object

  • rings – int, number of lateral rings in UV sphere generation, default 15

  • sectors – int, number of sectors in UV sphere generation, default 30

  • update_bb – bool, whether to update the bounding box, default True

  • color – (3,) color, default is orange (common param)

  • vert_color – (rings*sectors, 3) vertex color, optional advanced (common param)

  • translation – (3,), model translation (common param)

  • rotation – (3,), model rotation in axis-angle (common param)

  • scale – float, scale, default 1.0 (common param)

  • visible – bool, whether mesh should be visible on init, default true (depends on GET parameter in web version) (common param)

  • unlit – bool, whether mesh should be rendered unlit (common param)

  • time – int, time at which the mesh should be displayed; -1=always display (default) (common param)

add_line(name: str, a: ndarray | List[float], b: ndarray | List[float], update_bb: bool = True, **kwargs)

Add a single line segment from a to b

Parameters:

  • name – an identifier for this object

  • a – (3,), first point

  • b – (3,), second point

  • update_bb – bool, whether to update the bounding box, default True

  • color – (3,) color, default is orange (common param)

  • vert_color – (2, 3) vertex color, optional (overrides color) (common param)

  • translation – (3,), model translation (common param)

  • rotation – (3,), model rotation in axis-angle (common param)

  • scale – float, scale, default 1.0 (common param)

  • visible – bool, whether mesh should be visible on init, default true (depends on GET parameter in web version) (common param)

  • unlit – bool, whether mesh should be rendered unlit. Default true (common param)

  • time – int, time at which the mesh should be displayed; -1=always display (default) (common param)

add_lines(name: str, points: ndarray, segs: ndarray | None = None, update_bb: bool = True, **kwargs)

Add a series of line segments (in browser, lines are always size 1 right now)

Parameters:

  • name – an identifier for this object

  • points – (N, 3) float, list of points

  • segs – (N, 2) int, optionally, indices between points for which to draw the segments. If not given, draws 1-2-3-4…

  • color – (3,) color, default is orange (common param)

  • vert_color – (N, 3) vertex color, optional (overrides color) (common param)

  • translation – (3,), model translation (common param)

  • rotation – (3,), model rotation in axis-angle (common param)

  • scale – float, scale, default 1.0 (common param)

  • visible – bool, whether mesh should be visible on init, default true (depends on GET parameter in web version) (common param)

  • unlit – bool, whether mesh should be rendered unlit. Default true (common param)

  • time – int, time at which the mesh should be displayed; -1=always display (default) (common param)

  • update_bb – bool, whether to update the bounding box. Default True

add_points(name: str, points: ndarray, point_size: float = 1.0, update_bb: bool = True, **kwargs)

Add a point cloud (in browser, points are always size 1 right now)

Parameters:

  • name – an identifier for this object

  • points – (N, 3) float, list of points

  • point_size – float, point size

  • update_bb – bool, whether to update the bounding box (default True)

  • color – (3,) color, default is orange (common param, can be 3 item list)

  • vert_color – (N, 3) vertex color, optional (overrides color) (common param)

  • translation – (3,), model translation (common param)

  • rotation – (3,), model rotation in axis-angle (common param)

  • scale – float, scale, default 1.0 (common param)

  • visible – bool, whether mesh should be visible on init, default true (depends on GET parameter in web version) (common param)

  • unlit – bool, whether mesh should be rendered unlit. Default true (common param)

  • time – int, time at which the mesh should be displayed; -1=always display (default) (common param)

add_mesh(name: str, points: ndarray, faces: ndarray | None = None, face_size: int | None = None, update_bb: bool = True, **kwargs)

Add a general mesh

Parameters:

  • name – an identifier for this object

  • points – (N, 3) float, list of points

  • faces – (N, face_size) int, list of faces; if not given, faces will be 0-1-2, 3-4-5, 6-7-8, etc (glDrawArrays)

  • face_size – int, one of 1,2,3. 3 means triangle mesh, 1 means point cloud, and 2 means lines. By default is determined from faces (usually 3).

  • update_bb – bool, whether to update the bounding box (default True)

  • color – (3,) color, default is orange (common param)

  • vert_color – (N, 3) vertex color, optional (overrides color) (common param)

  • translation – (3,), model translation (common param)

  • rotation – (3,), model rotation in axis-angle (common param)

  • scale – float, scale, default 1.0 (common param)

  • visible – bool, whether mesh should be visible on init, default true (depends on GET parameter in web version) (common param)

  • unlit – bool, whether mesh should be rendered unlit. Use this if you want to render vertex colors directly without lighting. Default false. (common param)

  • time – int, time at which the mesh should be displayed; -1=always display (default) (common param)

add_image(name: str, image: str | ndarray, r: ndarray | None = None, t: ndarray | None = None, focal_length: float = 1111.11, z: float = 0.3, image_size: int = 512, jpeg_quality: int = 80, update_bb: bool = True, **kwargs)

Add an image (as a textured plane mesh, using JPEG compression). Affected by set_opencv and set_opengl (run before!). pillow required

Parameters:

  • name – an identifier for this object

  • path – path to the image

  • rotation – (3,) or (4,) or (3, 3) C2W rotations for each camera, either as axis-angle, xyzw quaternion, or rotation matrix (similar to “rotation”)

  • translation – (3,) C2W translations for each camera applied after rotation

  • r – alias for rotation (overrides, for similarity with add_camera_frustum)

  • t – alias for translation (overrides, for similarity with add_camera_frustum)

  • z – the size of the camera frustum (distance of image). Required

  • image_size – max size of image for display. This is NOT the size of the input image, but the size to be displayed! NOTE: do not make this too large to save memory

  • jpeg_quality – JPEG quality for compression (0-100)

  • update_bb – bool, whether to update the bounding box (default True)

  • time – int, time at which the mesh should be displayed; -1=always display (default)

add_images(name_prefix: str, images: List[str] | List[ndarray] | ndarray, r: ndarray, t: ndarray, focal_length: float = 1111.11, z: float | None = None, n_jobs: int = 16, update_view: bool = True, with_camera_frustum: bool = True, connect: bool = False, camera_color: List[float] | ndarray = [1.0, 0.0, 0.0], **kwargs)

Add multiple images using add_image multiple times. Affected by set_opencv and set_opengl (run before!). pillow required

Parameters:

  • name_prefix – prefix for the name of each image. All images will be in this folder

  • images – list of paths to images or list of images. If paths, joblib is required for parallel loading.

  • r – (N, 3) or (N, 4) or (N, 3, 3) or (N, 9)

  • t – (N, 3), translation of cameras

  • focal_length – float, focal length

  • z – float, the size of the camera frustum (distance of image). If not given, tries to infer from t. If t is size only 1, then z=0.3 is used

  • n_jobs – int, number of parallel jobs to use to load images

  • with_camera_frustum – bool, whether to add camera frustum (lines) as well. name will be <name_prefix>_frustums

  • connect – bool, whether to connect the camera frustum to the image (only if with_camera_frustum=True)

  • camera_color – color of the camera frustum (only if with_camera_frustum=True)

  • update_bb – bool, whether to update the bounding box (default True)

  • update_view – bool, if true then updates the camera position, scene origin etc using these cameras

  • translation – (3,), model translation (common param)

  • rotation – (3,), model rotation in axis-angle (common param)

  • scale – float, scale, default 1.0 (common param)

  • visible – bool, whether mesh should be visible on init, default true (depends on GET parameter in web version) (common param)

  • unlit – bool, whether mesh should be rendered unlit (common param)

  • time – int, time at which the mesh should be displayed; -1=always display (default) (common param)

add_camera_frustum(name: str, focal_length: float | None = None, image_width: float | None = None, image_height: float | None = None, z: float | None = None, r: ndarray | None = None, t: ndarray | None = None, connect: bool = False, update_view: bool = True, update_bb: bool = True, **kwargs)

Add one or more ideal perspective camera frustums. Affected by set_opencv and set_opengl (run before!)

Parameters:

  • name – an identifier for this object

  • focal_length – the focal length of the camera (unnormalized), default 1111

  • image_width – the width of the image/sensor, default 800

  • image_height – the height of the image/sensor, default 800

  • z – the size of the camera frustum to draw. If not given, tries to infer a good value from t. Else if t is not available, defaults to 0.3

  • r – (N, 3) or (N, 4) or (N, 3, 3) or None, optional C2W rotations for each camera, either as axis-angle, xyzw quaternion, or rotation matrix; if not given, only one camera is added at identity.

  • t – (N, 3) or None, optional C2W translations for each camera applied after rotation; if not given, only one camera is added at identity.

  • connect – bool, if true then draws lines through the camera centers, default false

  • update_bb – bool, whether to update the bounding box (default True)

  • color – (3,) color, default is orange (common param)

  • vert_color – (N * 5, 3) vertex color, optional advanced (common param)

  • translation – (3,), model translation (common param)

  • rotation – (3,), model rotation in axis-angle (common param)

  • scale – float, scale, default 1.0 (common param)

  • visible – bool, whether mesh should be visible on init, default true (depends on GET parameter in web version) (common param)

  • unlit – bool, whether mesh should be rendered unlit (common param)

  • time – int, time at which the mesh should be displayed; -1=always display (default) (common param)

  • update_view – bool, if true then updates the camera position, scene origin etc using these cameras

add_mesh_from_file(path: str, name_suffix: str = '', center: bool = False, **kwargs)

Add a mesh from path using trimesh

Parameters:

  • path – the path to the mesh file

  • name_suffix – object name will be basename(path) + name_suffix

  • center – if true, centers object to mean

  • time – int, time at which the mesh should be displayed; -1=always display (default) (common param)

Rest of keyword arguments passed to add_mesh

add_axes(name: str = 'axes', length: float = 1.0, update_bb: bool = False, **kwargs)

Add RGB-XYZ axes at [0, 0, 0] (as hardcoded lines)

Parameters:

  • name – identifier, default “axes”

  • length – float, length of axes

  • time – int, time at which the mesh should be displayed; -1=always display (default) (common param)

  • update_bb – bool, whether to update the bounding box (default False) (since axes are used as reference only)

Rest of keyword arguments passed to add_lines

remove(name: str)

Remove an object with given name. Note: if you overwrite an object with the same name type and arguments, it will overwrite the object in the scene.

Parameters:

name – the name given to add_*

remove_all(names: List[str])

Remove object with given names

Parameters:

names – list of names as given to add_*

clear()

Clear all objects from the scene.

add_volume(name: str, density: ndarray, colors: ndarray, radius: float = 1.0, density_threshold: float = 1.0, data_format: str = 'RGBA', update_bb: bool = True, **kwargs)

Add a 3D volume using the PlenOctree renderer

Parameters:

  • name – an identifier for this object

  • density – (Dx, Dy, Dz); dimensions need not be powers of 2 nor equal

  • colors – (Dx, Dy, Dz, (3, channel_size)); color data, last dim is size 3 * channel_size

  • radius – 1/2 side length of volume

  • density_threshold – threshold below which density is ignored

  • data_format – standard PlenOctree data format string, one of RGBA | SH1 | SH4 | SH9 | SH16. The channel_size should be respectively 1 | 1 | 4 | 9 | 16``.

  • update_bb – bool, whether to update the bounding box (default True)

  • translation – (3,), model translation (common param)

  • rotation – (3,), model rotation in axis-angle (common param)

  • scale – float, scale, default 1.0 (common param)

  • visible – bool, whether mesh should be visible on init, default true (depends on GET parameter in web version) (common param)

  • time – int, time at which the mesh should be displayed; -1=always display (default) (common param)

add_volume_from_npz(name: str, file: str, update_bb: bool = True, **kwargs)

Add a volume already saved as npz (for example, PlenOctree checkpoints downloaded from the website)

Parameters:

  • name – an identifier for this object

  • file – path to npz file

  • translation – (3,), model translation (common param)

  • rotation – (3,), model rotation in axis-angle (common param)

  • scale – float, scale, default 1.0 (common param)

  • visible – bool, whether mesh should be visible on init, default true (depends on GET parameter in web version) (common param)

  • time – int, time at which the mesh should be displayed; -1=always display (default) (common param)

  • update_bb – bool, whether to update the bounding box (default True)

set_nerf(*args, **kwargs)

Deprecated, please use add_nerf

add_nerf(name: str, eval_fn: Callable[[...], Tuple[Any, Any]], center: Tuple[float, float, float] | List[float] | float | ndarray | None = None, radius: Tuple[float, float, float] | List[float] | float | ndarray | None = None, scale: float = 1.0, reso: int = 256, use_dirs: bool = False, sh_deg: int = 1, sh_proj_sample_count: int = 15, sh_proj_use_sparse: bool = True, sigma_thresh: float = 3.0, weight_thresh: float = 0.001, r: Any | None = None, t: Any | None = None, focal_length: float | Tuple[float, float] | None = None, image_width: float | None = None, image_height: float | None = None, sigma_multiplier: float = 1.0, chunk: int = 720720, device: str = 'cuda:0', update_bb: bool = True, **kwargs)

Discretize and display a NeRF (low quality, for visualization purposes only). Currently only supports PyTorch NeRFs. Requires tqdm, torch, svox, scipy

Parameters:

  • eval_fn –

    • If use_dirs=False: NeRF function taking a batch of points (B, 3) and returning (rgb (B, 3), sigma (B, 1)) after activation applied.

    • If use_dirs=True then this function should take points (B, 1, 3) and kwarg ‘dirs’ (1, sh_proj_sample_count, 3); it should return (rgb (B, sh_proj_sample_count, 3), sigma (B, sh_proj_sample_count, 1)) sigma activation

      should be applied but rgb must NOT have activation applied for SH projection to work correctly.

  • center – float or (3,), xyz center of volume to discretize (will try to infer from cameras from add_camera_frustum if not given)

  • radius – float or (3,), xyz half edge length of volume to discretize (will try to infer from cameras from add_camera_frustum if not given)

  • scale – float, multiples radius by this before using it (this is provided for convenience, for manually scaling the scene boundaries which is often needed to get the right bounds if using automatic radius from camera frustums i.e. not specifying center and radius)

  • reso – int, resolution of tree in all dimensions (must be power of 2)

  • use_dirs – bool, if true, assumes normal NeRF with viewdirs; uses SH projection to recover SH at each point with degree sh_deg. Will view directions as kwarg ‘dirs’ of eval_fn and expect pre-activation RGB output in addition to density. See the description for the eval_fn param above for more info on the function spec.

  • sh_deg – int, SH degree if use_dirs, must be between 0-4

  • sh_proj_sample_count – SH projection samples if use_dirs

  • sh_proj_use_sparse – Use sparse SH projection via least-squares rather than monte carlo inner product

  • sigma_thresh – float, simple density threshold (used if r, t not given)

  • weight_thresh – float, weight threshold as in PlenOctrees (used if r, t given)

  • r – (N, 3) or (N, 4) or (N, 3, 3) or None, optional C2W rotations for each camera, either as axis-angle, xyzw quaternion, or rotation matrix; if not given, only one camera is added at identity.

  • t – (N, 3) or None, optional C2W translations for each camera applied after rotation; if not given, only one camera is added at identity.

  • focal_length – float or Tuple (fx, fy), optional, focal length for weight thresholding

  • image_width – float, optional, image width for weight thresholding

  • image_height – float, optional, image height for weight thresholding

  • update_bb – bool, whether to update the bounding box (default True)

  • translation – (3,), model translation (common param)

  • rotation – (3,), model rotation in axis-angle (common param)

  • nerf_scale – float, scale, default 1.0; note this has a different name due to legacy conflict (common param)

  • visible – bool, whether mesh should be visible on init, default true (depends on GET parameter in web version) (common param)

  • time – int, time at which the mesh should be displayed; -1=always display (default) (common param)

write(path: str, compress: bool = True)

Write to drawlist npz which you can open with volrend (volrend --draw <output.npz>; nerfvis_base branch recommended for more up-to-date experience) as well as in the web viewer. Usually, it’s easier to use one of Scene.export(), Scene.display(), or Scene.emebd()

Parameters:

path – output npz path

export(dirname: str | None = None, display: bool = False, world_up: ndarray | None = None, cam_center: ndarray | None = None, cam_forward: ndarray | None = None, cam_origin: ndarray | None = None, compress: bool = True, instructions: List[str] = [], css: str = '', url: str = 'localhost', port: int = 8888, open_browser: bool = False, output_html_name: str = 'index.html', embed_output: bool = False, serve_nonblocking: bool = False) → Tuple[str, str]

Write to a standalone web viewer

Parameters:

  • dirname – output folder path, if not given then uses ./nerfvis_scenes/(0-9a-zA-Z_ from self.title)

  • display – if true, serves the output using http.server (default false)

  • world_up – (3,), optionally, world up unit vector for mouse orbiting (will try to infer from cameras from add_camera_frustum if not given)

  • cam_center – (3,), optionally, camera center point (will try to infer from cameras from add_camera_frustum if not given)

  • cam_forward – (3,), optionally, camera forward-pointing vector (will try to infer from cameras from add_camera_frustum if not given)

  • cam_origin – (3,), optionally, camera center of rotation point (will try to infer from cameras from add_camera_frustum if not given)

  • compress – whether to compress the output npz file (slower but smaller)

  • instructions – list of additional javascript instructions to execute (advanced)

  • css – additional CSS to inject

  • url – str, URL for server (if display=True) default localhost

  • port – int, port for server (if display=True) default 8888 (if not available, tries next up to 32)

  • open_browser – bool, if true then opens the web browser, if possible (default False)

  • embed_output – bool, if true, embeds the output in the html as self-loading base64/blob url instead of saving as NPZ. This may cause the page to load very slowly initially, but combines all data into a single html file.

  • serve_nonblocking – bool, if true, and display=True, open a server in another threading and continues execution

Returns:

dirname, if it was not provided, returns the generated folder name; url

display(*args, **kwargs) → Tuple[str, str]

Alias of Scene.export with display=True (serve using http.server)

Parameters:

  • dirname – output folder path, if not given then uses ./nerfvis_scenes/(0-9a-zA-Z_ from self.title)

  • world_up – (3,), optionally, world up unit vector for mouse orbiting (will try to infer from cameras from add_camera_frustum if not given)

  • cam_center – (3,), optionally, camera center point (will try to infer from cameras from add_camera_frustum if not given)

  • cam_forward – (3,), optionally, camera forward-pointing vector (will try to infer from cameras from add_camera_frustum if not given)

  • cam_origin – (3,), optionally, camera center of rotation point (will try to infer from cameras from add_camera_frustum if not given)

  • compress – whether to compress the output npz file (slower but smaller)

  • instructions – list of additional javascript instructions to execute (advanced)

  • url – str, URL for server (if display=True) default localhost

  • port – int, port for server (if display=True) default 8888 (if not available, tries next up to 32)

  • open_browser – bool, if true then opens the web browser, if possible (default False)

Returns:

dirname, if it was not provided, returns the generated folder name; url

embed(width: int = 1100, height: int = 600, *args, **kwargs)

Calls Scene.export but in addition embeds inside a notebook.

NOTE 1: if the notebook is opened from a different notebook root folder in the future it will not work.

NOTE 2: still writes files to ./nerfvis_scenes folder, and it may get large overtime. You may want to clean it up manually.

Parameters:

  • dirname – output folder path, if not given then uses ./nerfvis_scenes/(0-9a-zA-Z_ from self.title)

  • world_up – (3,), optionally, world up unit vector for mouse orbiting (will try to infer from cameras from add_camera_frustum if not given)

  • cam_center – (3,), optionally, camera center point (will try to infer from cameras from add_camera_frustum if not given)

  • cam_forward – (3,), optionally, camera forward-pointing vector (will try to infer from cameras from add_camera_frustum if not given)

  • cam_origin – (3,), optionally, camera center of rotation point (will try to infer from cameras from add_camera_frustum if not given)

  • compress – whether to compress the output npz file (slower but smaller)

  • instructions – list of additional javascript instructions to execute (advanced)

  • url – str, URL for server (if display=True) default localhost

  • port – int, port for server (if display=True) default 8888 (if not available, tries next up to 32)

  • open_browser – bool, if true then opens the web browser, if possible (default False)

Returns:

dirname, if it was not provided, returns the generated folder name; url

nerfvis.utils

split_mat4(mat4: ndarray) → Dict[str, ndarray]

Tiny utility to split a 4x4 or 3x4 affine matrix into translation, rotation outputs r, t