Boost AI Medical Device Streaming Workflows with the NVIDIA Clara Holoscan SDK

Advances in edge computing, video cameras, real-time processing, and AI have helped transform medical devices over the years. NVIDIA developed the NVIDIA Clara Holoscan platform to support the development of software-defined AI medical devices.

The latest release of the NVIDIA Clara Holoscan SDK 0.2 offers real-time AI inference capabilities and fast I/O for high-performance streaming applications in medical devices. This includes endoscopy, ultrasound, surgical robots, microscopy, and genomics sequencing instruments.

The release also consists of:

Core backend on NVIDIA Graphical eXecution Framework (GXF.)
A sample endoscopy AI application.
A customizable AI pipeline to add your own model.
Support for both the Clara AGX Developer Kit with the Jetson AGX Xavier and NVIDIA RTX 6000 and the Clara Holoscan Development Kit with the Jetson AGX Orin and NVIDIA RTX A6000.
Support on the NVIDIA JetPack 5.0 SDK, which includes Ubuntu 20.04.

GXF processes streaming data

The most significant change in the Clara Holoscan SDK v0.2 is the adoption of a new core backend called NVIDIA Graph eXecution Framework (GXF.) GXF is a framework supporting component-based programming for streaming data processing pipelines. It is built for efficient data ingestion, data transfer, and AI/ML workloads.

With GXF, you can create reusable components and combine them in graphs to build applications for different products quickly. GXF supports the processing of video and AU streams as well as user-defined streaming data types used in medical devices such as raw ultrasound, radiology imaging scanners, and microscopes.

A recent test using the NVIDIA Latency Display Analysis Tool on a 1080p video stream showed that GXF offers a significant speedup compared to previous solutions. In the test, GXF reduced the overhead in an AI Inferencing application by nearly 3x compared to a similar pipeline in the Clara Holoscan SDK 0.1.

Chart showing reduced latency with GXF in v0.2 at 51 ms compared to 146 ms in v0.1 — Figure 1: GXF in Clara Holoscan SDK v0.2 reduces streaming data pipeline overhead for improved latency of real-time applications compared to a similar pipeline running on the previous version of the SDK v.01

Additionally, GXF supports user-customizable components to support generic data processing pipelines. GXF handles the critical parts of building a high-performance application due to two important components.

First is a scheduler that determines when components execute. The scheduler supports single or multithreaded execution, with conditional execution, asynchronous scheduling, and other custom tools.

Second, GXF has a memory allocator that provides a system with an upfront allocation of a large contiguous memory pool and reuses regions as needed. To ensure zero-copy data exchange between components, memory can be pinned to the device.

AI-enabled medical instrument graph pipeline template — *Figure 2. An example of a generic pipeline for customizing the workflow, including data processing, model inference, storage, and visualization*

Endoscopy AI sample application on Clara Holoscan

Digital endoscopy has evolved as a key technology for medical screenings and minimally invasive surgeries. Using real-time AI platforms to process and analyze the video signal produced by the endoscopic camera has been growing.

This technology is helping with anomaly detection and measurements, image enhancements, alerts, and analytics. The Clara Holoscan SDK 0.2 includes a sample AI-enabled endoscopy application showcasing the end-to-end functionality of GXF and support for devices that interface with AJA with an HDMI input.

The endoscopy AI sample application has a deep learning model to perform object detection and tool tracking in real time on an endoscopy video stream.

The application uses several NVIDIA features to minimize the overall latency, including:

GPUDirect RDMA video data transfer to eliminate the overhead of copying to or from system memory.
NVIDIA Performance Primitive Library for CUDA-accelerated 2D image transformations before AI inference.
TensorRT runtime for optimized AI Inference and speed-up.
CUDA and OpenGL interoperability, which provides efficient resource sharing on the GPU for visualization.

To learn more about the endoscopy AI sample application, its hardware and software reference architecture on Clara Holoscan, as well as the path to production, download the Clara Holoscan Endoscopy Whitepaper.

Frame by frame identification and tracking in endoscopy — Figure 3. An endoscopy image from a gallbladder surgery showing AI-powered frame-by-frame tool identification and tracking. Image courtesy of Research Group Camma, IHU Strasbourg and the University of Strasbourg

Bring your own model AI application

Bring your own AI model into the Clara Holoscan reference pipeline to quickly create a streaming workflow. Swapping out of one model for another is accomplished by updating one configuration file and exporting data to the GXF-native data format. Models saved in portable ONNX, as well as the NVIDIA performance-optimized TRT format, can be run on GXF’s built-in inference engines.

Support for the Clara Developer Kit

The Clara Holoscan SDK 0.2 is supported on the Clara AGX and the new Clara Holoscan Developer Kit. The next-generation Clara Holoscan Development Kit is built with a high-performance NVIDIA Orin module, a powerful RTX A6000 GPU, and the connectivity performance of the ConnectX SmartNIC.

This kit is the ideal solution for developing the next generation of software-defined medical devices. Orin is geared for autonomous machines with high-speed interface support for multiple sensors and 8X the performance of the last generation for multiple concurrent AI inference pipelines.

Updated JetPack 5.0HP1 with Ubuntu 20.04

The NVIDIA JetPack SDK contains the base OS for the Clara Holoscan SDK. For version 0.2, the JetPack SDK is being upgraded from version 4.5 to version 5.0HP1. This upgrades the OS to L4T rel-34, to be on par with Ubuntu 20.04 with LTS Kernel 5.10.