Jetson Benchmarks

Jetson is used to deploy a wide range of popular DNN models and ML frameworks to the edge with high performance inferencing, for tasks like real-time classification and object detection, pose estimation, semantic segmentation, and natural language processing (NLP).

MLPerf Inference Benchmarks

The tables below show inferencing benchmarks from the NVIDIA Jetson submissions to the MLPerf Inference Edge category.

Jetson AGX Orin Jetson Orin NX MLPerf v3.0 Results

Model	NVIDIA Jetson AGX Orin (TensorRT)			NVIDIA Orin MaxQ (TensorRT)		NVIDIA Jetson Orin NX
Model		Single Stream (Samples/s)	Offline (Samples/s)	Multi Stream (Samples/s)	Offline (Samples/s)	System Power(W)	Offline (Samples/s)
Image Classification ResNet-50	1538	6438.10	3686	3525.91	23.06	2517.99
Object Detection Retinanet	51.57	92.40	60.00	34.6	22.4	36.14
Medical Imaging 3D-Unet	.26	.51	N/A	3.28	28.64	.19
Speech-to-text RNN-T	9.822	1170.23	N/A	14472	25.64	405.27
Natural Language Processing BERT	144.36	544.24	N/A	3685.36	25.91	163.57

Steps to reproduce these results can be found at v3.0 Results | MLCommons
These results were achieved with the NVIDIA Jetson AGX Orin Developer Kit running a preview of TensorRT 8.5.0, and CUDA 11.4
Note different configurations were used for single stream, offline and multistream. Reference the MLCommons page for more details

Jetson AGX Xavier and Jetson Xavier NX MLPerf v1.1 Results

Model	Jetson Xavier NX (TensorRT)	Jetson AGX Xavier 32GB (TensorRT)
Model	Image Classification ResNet-50	1245.10	2039.11
Object Detection SSD-small	1786.91	2833.59
Object Detection SSD-Large	36.97	55.16
Speech to Text RNN-T	259.67	416.13
Natural Language Processing BERT-Large	61.34	96.73

Full Results can be found at v1.1 Results | MLCommons
These results were achieved on Jetson AGX Xavier Developer Kit and Jetson Xavier NX Developer Kit running Jetpack 4.6, TensorRT 8.0.1, CUDA 10.2
ResNet-50, SSD-small, and SSD-Large were run on the GPU and both DLAs
These MLPerf Results can be reproduced with the code in the following link: https://github.com/mlcommons/inference_results_v2.0/tree/master/closed/NVIDIA

NVIDIA Pretrained Model Benchmarks

NVIDIA pretrained models from NGC start you off with highly accurate and optimized models and model architectures for various use cases. Pretrained models are production-ready. You can further customize these models by training with your own real or synthetic data, using the NVIDIA TAO (Train-Adapt-Optimize) workflow to quickly build an accurate and ready to deploy model.The table below shows inferencing benchmarks for some of our pretrained models running on Jetson modules.

Jetson Pretrained Model Benchmarks

Jetson Orin Results

Model	Jetson Orin Nano 4GB	Jetson Orin Nano 8GB	Jetson Orin NX 8GB	Jetson Orin NX 16GB	Jetson AGX Orin 32GB	Jetson AGX Orin 64GB
PeopleNet (v2.5 unpruned)	57	117	192	240	409	685
Action Recognition 2D	220	372	440	483	1158	1517
Action Recognition 3D	13	26	32	39	71	108
LPR Net	552	974	1314	1427	2800	4213
Dashcam Net	200	400	689	877	1482	2139
Bodypose Net	69	137	169	203	360	563

Model	Jetson Nano	Jetson TX2 NX	Jetson Xavier NX	Jetson AGX Xavier
PeopleNet (v2.5 unpruned)	2	5	120	195
Action Recognition 2D	32	88	245	472
Action Recognition 3D	1	3	21	32
LPR Net	47	86	714	1236
Dashcam Net	11	26	424	667
Bodypose Net	3	7	104	172

Jetson Orin & Jetson Xavier Benchmarks were run using Jetpack 5.1.1
Each Jetson module was run with maximum performance (MAXN for JAO64, JAO32, ONX16, ONX8; and 15W mode for JON8, and 10W mode for JON4)
For Jetson Nano and Jetson TX2 NX, these benchmarks were run using Jetpack 4.6.1
Each Jetson module was run with maximum performance (MAXN)
Reproduce these results by downloading these models from our NGC catalog

Jetson Family Benchmarks

Model	Jetson Orin Nano 4GB	Jetson Orin Nano 8GB	Jetson Orin NX 8GB	Jetson Orin NX 16GB	Jetson AGX Orin 32GB	Jetson AGX Orin 64GB
Inveption_V4	182	361	593	769	1337.8	1702.6
VGG19	174	361	442	532	937	1471
Super_resolution	102	203	280	386	610	882
UNET-sgmentation	76	148	183	217	387	584
Pose Estimation	280	546	665	800	1424	2048
Yolov3-tiny	371	731	1156	1440	2611	3179
Resnet50	621	1158	1725	2183	3717	4834
SSD-Mobilnet	1094	2156	2893	3457	6415	7671
SSD_Resnet34_1200x1200	18	34	52	72	120	163
Yolov5m	69	131	162	193	342	519
Yolov5s	158	301	379	449	785	1135

These Benchmarks were run using Jetpack 5.1.1
Each Jetson module was run with maximum performance (Max Frequencies in MAXN for JAO64, JAO32, ONX16, ONX8; and 15W mode for JON8, and 10W mode for JON4)
Steps to reproduce these results can be found here