11.5.3.2. dVRK Data
Warning
The dVRK data collection suite is exclusively supported on ROS 2. ROS 1 is not supported.
The dvrk_data package has two related roles:
It provides GStreamer video utilities for acquisition, alignment, transport, and inspection. These utilities are designed around multiple cooperating processes that exchange frames through
unixfdsockets instead of routing high-bandwidth video through ROS topics.It provides data collection tools for synchronized experiments: video recording, ROS 2 bag recording, session metadata, video tagging, timestamp inspection, and extraction into files suitable for analysis or training.
The video path preserves as much timing information as possible. Hardware streams are captured directly with GStreamer, frames carry source timestamps, and recorded videos are paired with sidecar timestamp files. ROS 2 messages can be recorded alongside video, but the package does not force all data streams through a single synchronized transport while recording.
It is important to note that the data collection tools explicitly do not
synchronize the disparate video and topic streams together natively. Instead,
during extraction, individual per-source estimated_latency values are used
alongside accurate timestamps to align the sampled data together as closely as
chronologically possible.
Package Layout
Video transport and utility executables include stereo_source,
stereo_alignment, stereo_alignment_calibration,
video_configurator, gscam_socket, and gscam.launch.py. They can be
used to build local GStreamer pipelines, publish selected streams through
@dvrk_gst abstract Unix sockets, calibrate stereo alignment, or bridge a
stream back into ROS when needed. Use gscam_socket to list active sockets
and launch a gscam_node with a single command.
Data collection executables include record, video_tag,
video_latency, extract, and encord_to_tags. These tools share JSON
configuration files and session metadata so a recording can be reviewed,
annotated, and extracted without losing the original video timestamps.
Data Collection Workflow
The suite of tools provided by the dvrk_data package is designed to work together in a sequence. This section describes the data flow and how shared files connect the different programs.
Data Flow Overview
Configuration: A master JSON configuration file defines video sources, ROS 2 topics, optional stages, and recording parameters.
Recording: The Record application captures video, audio, and ROS 2 topics while preserving per-source timestamps.
Curation: The Video Tag tool loads the session to add temporal labels and frame-accurate tags.
Extraction: The Extract tool processes the recorded videos and ROS 2 bags to generate analysis-ready data such as images, videos, and CSVs.
Validation: The Tools for Latency tools verify timing consistency and measure system performance.
Session Directory Output
When record completes a session, it creates a directory (named by its timestamp, e.g., 2026/02/18_025457/) containing:
Video Files: Recorded
.mp4files for each enabled camera.Sidecar Timestamps: JSON files named
camera_name_YYMMDD_HHMMSS_timestamps.jsoncontaining nanosecond-accurate epoch timestamps for every frame.ROS bags: Recorded data topics stored in ROS 2 bag format.
index.json: A summary of durations, metadata, and data paths.
Downstream Consumption
The session directory and its contents are consumed by:
video_tag: Loads the video files and uses
index.jsonto identify available streams. It generates or updates atags.jsonfile in the session directory.extract: Uses the sidecar timestamps to extract frames from
.mp4at the exact original nanoseconds and converts ROS bags into easy-to-use CSVs.check_timestamps: Uses extracted frames to verify timestamp consistency and quantify timing offsets against burned-in overlays.