Skip to content

Video recorder

video_recorder

STREAMED_VIDEO_FORMAT module-attribute

Python
STREAMED_VIDEO_FORMAT = {'.mp4', '.webm'}

Formats encoded incrementally by piping frames to ffmpeg as they're captured, instead of buffering them in memory.

LabelConfig

Bases: TypedDict

Describes one frame label, returned fresh per-frame by VideoRecorder.frame_label.

text instance-attribute

Python
text: str

The label text to burn into the frame.

position instance-attribute

Python
position: NotRequired[tuple[int, int]]

Top-left pixel coordinate to draw at. Defaults to (10, 10).

color instance-attribute

Python
color: NotRequired[Vec3 | Vec4]

Text color, normalized [0, 1] RGB(A) (e.g. Color.WHITE.rgba). Defaults to opaque white.

background_color instance-attribute

Python
background_color: NotRequired[Vec3 | Vec4 | None]

Optional fill behind the text, normalized [0, 1] RGB(A). An RGBA alpha < 1 is true-blended with the frame beneath it. Defaults to no background.

border_color instance-attribute

Python
border_color: NotRequired[Vec3 | Vec4 | None]

Optional 1px outline around the padded label box, normalized [0, 1] RGB(A). Defaults to no border.

font_size instance-attribute

Python
font_size: NotRequired[int]

Font size in pixels. Defaults to 14.

font_path instance-attribute

Python
font_path: NotRequired[str | Path | None]

Path to a TrueType/OpenType font file, for custom styles/weights. Defaults to PIL's built-in font.

padding instance-attribute

Python
padding: NotRequired[int]

Padding in pixels around the text when drawing background_color. Defaults to 4.

VideoRecorder dataclass

Python
VideoRecorder(
    path: Path,
    camera_name: CameraName,
    show_loads: bool = False,
    show_net_force: bool = False,
    show_contacts: bool = False,
    show_proximities: bool = False,
    show_traces: bool = False,
    fps: float = 30,
    playback_speed: float = 1.0,
    width: int = 640,
    height: int = 480,
    recording_trigger: Callable[
        [MjState], bool
    ] = lambda state: True,
    frame_label: Callable[[MjState], LabelConfig]
    | None = None,
    max_frames: int | None = None,
    quality: int | None = None,
    encode_speed: int | None = None,
    _frames: list = list(),
)

Records a MuJoCo simulation to a video file.

Frames are captured at a fixed rate (fps) relative to simulation time, not wall-clock time, so the output plays back at the exact rate specified regardless of how fast the simulation runs. The recorder skips capture_frame calls that fall between the interval boundaries, which prevents duplicate frames when the physics step is finer than 1/fps.

Typical usage inside a runtime function, within a with runtime_manager as rm: block::

Text Only
1
2
3
4
5
6
7
recorder = VideoRecorder(
    path=model.trial_dir / "camera.mp4",
    camera_name="top_view",
    fps=30,
    width=1280,
    height=720,
).setup(mj_model).register_to_rm()

register_to_rm wires the recorder into the RuntimeManager so that capture_frame is called automatically on every physics step and save is called when the simulation finishes. If no runtime_manager is passed, it registers to the RuntimeManager of the active with block.

Supported output formats (determined by the path extension):

  • .mp4: H.264 via ffmpeg; widest browser and player compatibility.
  • .webm: VP9 via ffmpeg; smaller files, fully seekable in the Dojo viewer. Requires ffmpeg with libvpx-vp9 support.
  • .gif: via PIL; no audio, loops automatically; large file size, not seekable.

.mp4 and .webm frames are piped to ffmpeg as they're captured rather than buffered in memory, so recording length isn't limited by available RAM. .gif (and any other extension, which falls back to mediapy) still buffers every frame until save is called. quality and encode_speed tune the ffmpeg encoder for .mp4/.webm; both have no effect on .gif.

Visual overlays (contact forces, net forces, custom arrows/lines) are controlled by the show_* flags and the show_loads flag passed to capture_frame.

playback_speed scales the frame rate of the saved video relative to fps: 0.5 plays back in slow motion, 1 is real time, and 2 plays back at double speed. It does not change how many frames are captured per second of simulation time, only how quickly they are played back.

recording_trigger is a function of the current MjState that gates whether a due frame is actually captured, e.g. lambda state: 5.0 <= state.data.time <= 10.0 to only record a window of the simulation.

frame_label, if set, is called with the current MjState and the returned LabelConfig is burned into the frame - text, position, color, an optional (alpha-blended) background, and font are all configurable per-frame.

max_frames caps the number of frames held in memory; once reached, further capture_frame calls are silently ignored (with a one-time warning).

Call close once save has finished to release the renderer's GL context.

path instance-attribute

Python
path: Path

Output file path. The extension determines the container and codec.

camera_name instance-attribute

Python
camera_name: CameraName

Name of the MuJoCo camera to render from (must exist in the model).

show_loads class-attribute instance-attribute

Python
show_loads: bool = False

Whether to render custom arrow overlays (passed via custom_arrows in capture_frame).

show_net_force class-attribute instance-attribute

Python
show_net_force: bool = False

Whether to render net force visualizations (mjVIS_PERTFORCE).

show_contacts class-attribute instance-attribute

Python
show_contacts: bool = False

Whether to render contact force visualizations (mjVIS_CONTACTFORCE).

show_proximities class-attribute instance-attribute

Python
show_proximities: bool = False

Whether to render custom line overlays (passed via custom_lines in capture_frame).

show_traces class-attribute instance-attribute

Python
show_traces: bool = False

Whether to render Tracer trails (passed via custom_traces in capture_frame).

fps class-attribute instance-attribute

Python
fps: float = 30

Target frame rate of the output video. Frames are sampled every 1/fps seconds of simulation time.

playback_speed class-attribute instance-attribute

Python
playback_speed: float = 1.0

Target for playback speed. If using 0.5 the video will record in "slow motion", 2 will record as occurring twice as fast.

width class-attribute instance-attribute

Python
width: int = 640

Render width in pixels.

height class-attribute instance-attribute

Python
height: int = 480

Render height in pixels.

recording_trigger class-attribute instance-attribute

Python
recording_trigger: Callable[[MjState], bool] = (
    lambda state: True
)

Function evaluated against the current MjState on every step. Frames are only captured while it returns True.

frame_label class-attribute instance-attribute

Python
frame_label: Callable[[MjState], LabelConfig] | None = None

Optional function returning a LabelConfig to burn into each captured frame, e.g. lambda state: {"text": f"t={state.data.time:.2f}s"}.

max_frames class-attribute instance-attribute

Python
max_frames: int | None = None

Optional cap on the number of frames captured. Once reached, further capture_frame calls are ignored. For .gif and other buffered formats this also bounds memory use; .mp4/.webm are streamed to disk as they're captured, so it only bounds recording length for those.

quality class-attribute instance-attribute

Python
quality: int | None = None

Override for ffmpeg's -crf (constant rate factor) on .mp4/.webm: lower means higher quality and a larger file. Defaults to a codec-specific value (23 for .mp4/libx264, 33 for .webm/libvpx-vp9) when unset. Useful range is roughly 18-32; has no effect on .gif.

encode_speed class-attribute instance-attribute

Python
encode_speed: int | None = None

Override for the .mp4/.webm encoder's speed-vs-compression trade-off, on a 0-8 scale where 0 is slowest/best compression and 8 is fastest/worst (passed straight through as -cpu-used for .webm; mapped to an x264 -preset name for .mp4). Defaults to a codec-specific value (4 for .mp4, 2 for .webm) when unset. Has no effect on .gif.

setup

Python
setup(state: MjState) -> Self

Initializes the MuJoCo renderer for this model. Must be called before the simulation loop.

Source code in src/mujoco_mojo/runtime/video_recorder.py
Python
def setup(self, state: MjState) -> Self:
    """Initializes the MuJoCo renderer for this model. Must be called before the simulation loop."""
    self._validate(state)

    try:
        self._renderer = mujoco.Renderer(
            model=state.model, height=self.height, width=self.width
        )
    except Exception as e:
        msg = "Failed to initialize the MuJoCo Renderer. If on a server, try setting 'export MUJOCO_GL=egl' in your terminal."
        logger.error(msg)
        raise RuntimeError(msg) from e

    # initialize the vopt with defaults
    mujoco.mjv_defaultOption(self._vopt)

    # whether or not to show debug graphics
    self._vopt.flags[mujoco.mjtVisFlag.mjVIS_PERTFORCE] = int(self.show_net_force)
    self._vopt.flags[mujoco.mjtVisFlag.mjVIS_CONTACTFORCE] = int(self.show_contacts)
    return self

is_due

Python
is_due(state: MjState) -> bool

Returns whether capture_frame would actually capture a frame for state right now, without any side effects. Lets callers skip expensive work (e.g. building custom_traces) that would otherwise go unused on the steps between frames.

Source code in src/mujoco_mojo/runtime/video_recorder.py
Python
def is_due(self, state: MjState) -> bool:
    """Returns whether `capture_frame` would actually capture a frame for `state` right now, without any side effects. Lets callers skip expensive work (e.g. building `custom_traces`) that would otherwise go unused on the steps between frames."""
    if state.data.time < self._next_record_time:
        return False
    if not self.recording_trigger(state):
        return False
    if self.max_frames is not None and self._frame_count >= self.max_frames:
        return False
    return True

capture_frame

Python
capture_frame(
    state: MjState,
    custom_arrows: list[ArrowConfig],
    custom_lines: list[LineConfig],
    custom_traces: list[LineConfig],
)

Captures the current state as a video frame.

Source code in src/mujoco_mojo/runtime/video_recorder.py
Python
def capture_frame(
    self,
    state: MjState,
    custom_arrows: list[ArrowConfig],
    custom_lines: list[LineConfig],
    custom_traces: list[LineConfig],
):
    """Captures the current state as a video frame."""
    if state.data.time < self._next_record_time:
        return

    if not self.recording_trigger(state):
        return

    if self.max_frames is not None and self._frame_count >= self.max_frames:
        if not self._max_frames_warned:
            logger.warning(
                f"VideoRecorder for {self.path} reached max_frames={self.max_frames}; "
                "no further frames will be captured."
            )
            self._max_frames_warned = True
        return

    frame = self._render_frame(state, custom_arrows, custom_lines, custom_traces)

    if self.path.suffix.lower() in STREAMED_VIDEO_FORMAT:
        if self._encoder_proc is None:
            self._open_encoder()
        w, h = self._encode_size
        assert (
            self._encoder_proc is not None and self._encoder_proc.stdin is not None
        )
        self._encoder_proc.stdin.write(frame[:h, :w].tobytes())
    else:
        self._frames.append(frame)

    # increment the clock for the next frame
    self._frame_count += 1
    self._next_record_time += 1 / self.fps

snapshot

Python
snapshot(
    state: MjState,
    path: Path,
    custom_arrows: list[ArrowConfig] | None = None,
    custom_lines: list[LineConfig] | None = None,
    custom_traces: list[LineConfig] | None = None,
)

Renders the current state and saves it as a single image to path, regardless of recording_trigger or fps timing.

Source code in src/mujoco_mojo/runtime/video_recorder.py
Python
def snapshot(
    self,
    state: MjState,
    path: Path,
    custom_arrows: list[ArrowConfig] | None = None,
    custom_lines: list[LineConfig] | None = None,
    custom_traces: list[LineConfig] | None = None,
):
    """Renders the current state and saves it as a single image to `path`, regardless of `recording_trigger` or `fps` timing."""
    from PIL import Image

    frame = self._render_frame(
        state, custom_arrows or [], custom_lines or [], custom_traces or []
    )
    Image.fromarray(frame).save(path)
    logger.info(f"Snapshot saved to {path}")

save

Python
save()

Finishes writing the video file.

Supported formats: - .mp4: H.264 via ffmpeg; universally compatible. - .webm: VP9 via ffmpeg; smaller files and fully seekable. - .gif: via PIL; no audio, loops automatically, large file size, not seekable.

.mp4 and .webm are encoded incrementally: each frame is piped to ffmpeg as it's captured, so save only needs to close that pipe and wait for ffmpeg to finish. .gif (and any other format) buffers every frame in memory and is only encoded here.

The output format is determined by the extension of path.

Source code in src/mujoco_mojo/runtime/video_recorder.py
Python
def save(self):
    """
    Finishes writing the video file.

    Supported formats:
    - `.mp4`: H.264 via ffmpeg; universally compatible.
    - `.webm`: VP9 via ffmpeg; smaller files and fully seekable.
    - `.gif`: via PIL; no audio, loops automatically, large file size, not seekable.

    `.mp4` and `.webm` are encoded incrementally: each frame is piped to ffmpeg as it's captured, so `save` only needs to close that pipe and wait for ffmpeg to finish. `.gif` (and any other format) buffers every frame in memory and is only encoded here.

    The output format is determined by the extension of `path`.
    """
    if self._frame_count == 0:
        return

    if self._encoder_proc is not None:
        self._finish_encoding()
    elif self.path.suffix.lower() == ".gif":
        from PIL import Image

        # convert arrays to PIL images
        pil_images = [Image.fromarray(frame) for frame in self._frames]

        # save gif
        pil_images[0].save(
            self.path,
            save_all=True,
            append_images=pil_images[1:],
            duration=int(1000 / self._output_fps),  # ms per frame
            loop=0,  # loop forever
        )
    else:
        import mediapy as media

        media.write_video(path=self.path, images=self._frames, fps=self._output_fps)
    logger.info(f"Video saved to {self.path}")

close

Python
close() -> None

Releases the GL context held by the underlying MuJoCo renderer, and kills the ffmpeg encoder if save was never called. Call once recording is finished and save has been called.

Source code in src/mujoco_mojo/runtime/video_recorder.py
Python
def close(self) -> None:
    """Releases the GL context held by the underlying MuJoCo renderer, and kills the ffmpeg encoder if `save` was never called. Call once recording is finished and `save` has been called."""
    if self._encoder_proc is not None and self._encoder_proc.poll() is None:
        proc = self._encoder_proc
        try:
            if proc.stdin is not None:
                proc.stdin.close()
        except BrokenPipeError:
            pass
        proc.kill()
        proc.wait()
    self._renderer.close()