778161 - REEF BOT-103 Robotic Perception using Deep Neural Networks with Jetson Orin & Depth Camera

Course Description

This course delivers an advanced understanding of deep learning for computer vision and robotics. This course examines advanced object detection frameworks, including YOLOv11 and the Ultralytics API, with emphasis on single-shot detection techniques. Through applied projects, participants gain experience in custom data labeling and model training using Roboflow and Google Colab, and deploy models of varying sizes to align with performance objectives and resource constraints.

A central component of the course focuses on edge implementation, including model optimization for the NVIDIA Jetson and the use of TensorRT to enable high-efficiency inference. The course also addresses advanced processing techniques, including small-object detection, multi-object tracking, and real-time streaming using GStreamer and DeepStream. It further incorporates orthorectification to enhance geospatial accuracy and multi-stream fusion to integrate and synchronize multiple video feeds.

Learner Outcomes

Course Goals and Objectives

By the end of this short course, students should have a solid understanding of the following:

Deep Learning Models for Object Detection

• Explore and compare deep learning architecture designed for single-shot detection.
• Develop a deep understanding of the YOLO 11 model and its capabilities.
• Leverage the Ultralytics API to fully exploit YOLO model features and workflows.

Custom Data Labeling and Training

• Apply best practices for data labeling using tools such as Roboflow.
• Train custom models using Google Colab.
• Deploy custom-trained models of varying sizes to balance accuracy and performance.

Hardware Acceleration on NVIDIA Jetson

• Implement and benchmark optimized models on the NVIDIA Jetson Orin to evaluate performance and resources efficiency.
• Understand and apply the TensorRT inference SDK through both C++ and Python APIs.
• Optimize deep learning models for efficient execution on Jetson hardware.

Advanced Deep Learning Processing

• Apply Sliced Aided Hyper Inference (SAHI) to detect minute objects.
• Integrate multi-object detection and tracking with deep neural networks.
• Develop hardware-accelerated video streaming using GStreamer and NVIDIA DeepStream.

Orthorectification and Geospatial Mapping

• Apply orthorectification to produce geometrically corrected, consistently scaled imagery for improved localization, navigation, mapping, and object recognition.
• Render detections and tracks in UTM coordinate space for real-world geospatial analysis.

Sensor Fusion

• Combine and correlate detections across 2D and 3D data streams for richer situational awareness.

Testimonials

This syllabus represents the instructor’s current plans and objectives. As we go through the semester, those plans may need to change to enhance the class learning opportunity. Such changes, communicated clearly, are not unusual and should be expected.