This book provides a thorough overview of the state-of-the-art FPGA-based robotic computing accelerator designs and summarizes their adopted optimized techniques. This book consists of ten chapters, delving into the details of how FPGAs have been utilized in robotic perception, localization, planning, and multi-robot collaboration tasks. In addition to individual robotic tasks, this book provides detailed descriptions of how FPGAs have been used in robotic products, including commercial autonomous vehicles and space exploration robots. Some key observations of this book has been published as a survey paper in IEEE Circuits and Systems Magazine, 2021.

This book provides a comprehensive overview of state-of-the-art embodied AI robotic systems, advancing toward artificial general intelligence. It comprises 14 chapters. Part 1 (chapters 1-2) introduces embodied AI robots background and recent developments. Part 2 (chapters 3-6) explores key systems for embodied robots, including perception, localization, planning, and control. Part 3 (chapters 7-9) deeps dive into LLMs for robotic frameworks. Part 4 (chapters 10-13) discusses efficiency, robustness, safety, and data challenges in embodied AI, along with solutions. Part 5 (chapter 14) demonstrates embodied robotic applications.

(Research Topics: Embodied AI Robotic System / Reliability of AMC / Hardware Architecture of AMC / Benchmarking and Design Flow of AMC)

Embodied AI System Design

We present a cognitive-inspired modular framework for cooperative embodied AI systems and identify the system inherent characteristics and optimization opportunities. Evaluated on long-horizon multi-objective tasks, our cross-layer optimization achieves an average 3.93x speedup in end-to-end task execution.

We analyze the neuro-symbolic workload chracteristics, and present a hardware acceleration case study for vector-symbolic architecture to improve the performance, efficiency, and scalability of neuro-symbolic computing.

Reliability of Autonomous Machine Computing

We characterize the inherent resilience of different compute kernels in autonomous vehicles and drones systems. We analyze the protection design landscape and propose the lightweight Vulnerable-Adaptive Protection (VAP) paradigm for resilient autonomous machines.

We propose MulBERRY, a multi-agent robust learning framework to enhance bit error robustness and energy efficiency for autonomous swarm systems.

We introduce ROSFI, the first Robot Operating System (ROS) resilience analysis methodology, to assess the effect of silent data corruption (SDC) on safety-critical applications.

We propose BEERY, a robust learning framework to improve bit error robustness and energy efficiency for RL autonomous systems. BEERY enables robust low-voltage operation on UAVs, leading to high energy savings in both compute-level operation and system-level quality-of-flight.

We build a ROS-based end-to-end fault analysis framework to understand the resilience of Micro Aerial Vehicles (MAVs) system, and propose two low overhead anomaly-based transient fault detection and recovery schemes.

We explore the various originations of fault sources across the computing stack of autonomous systems, and discuss the diverse fault impacts and fault mitigation techniques of different scales of autonomous systems.

We characterize the hardware transient fault impact on federated reinforcement learning system, a swarm intelligence paradigm in autonomous machines. We further propose application-aware cost-effective fault detection and mitigation scheme to enable autonomy reliability.

We evaluate the resilience of learning-based navigation systems to transient and permanent hardware faults. We further propose two efficient fault mitigation techniques for both RL training and inference.

Hardware Architecture of Autonomous Machine Computing

We propose ORIANNA, a framework leverageing a common abstraction factor graph to generate accelerators for diverse robotic applications (e.g., manipulators, vehicles, drones) containing multiple optimization-based algorithms (e.g., localization, planning).

We present a benchmarking framework and conduct a comprehensive measurement study of prediction-time DNN adaptation techniques, encompassing both supervised and unsupervised approaches, on CIM hardware substrates at the edge.

We present a heterogeneous programmable ARM Cortex-based SoC with power-efficient RRAM compute-in-memory for CNN and high-speed SRAM compute-near-memory for SNN for the modality-matched acceleration of the hybrid vision.

We propose a lightweight, fully unsupervised and real-time adaptation algorithm for safety-critical lane detection of autonomous driving, and demonstrate its state-of-the-art performance on Nvidia Jetson Orin.

We propose a fully-programmable heterogeneous ARM Cortex-based SoC with an in-memory low-power RRAM-based CNN and a near-memory high-speed SRAM-based SNN in a hybrid architecture, for high-speed target identification and tracking applications.

We present an energy-efficient and runtime-reconfigurable FPGA-based accelerator for robotic localization tasks. We exploit SLAM-specific data locality, sparsity, reuse, and parallelism, and achieve >5x performance improvement over state-of-the-art.

We present the cross-layer robotic computing stack, illustrate the current progress and key design techniques. We summarize and highlight the challenges, research opportunities, and roadmap for the next-generation FPGA-based robotic computing systems.

We present a series of ultra-low-power accelerator and system designs on enabling the intelligence in edge robotic platforms, with an emphasis on mixed-signal circuit, neuro-inspired computing, benchmarking, software infrastructure, and algorithm-hardware co-design.

We provide an overview of recent work on FPGA-based robotic accelerators. An analysis of software and hardware optimization techniques and main technical issues is presented, along with some commercial and space applications.

We present a real-time and energy-efficient ORB (Oriented-Fast and Rotated-BRIEF) based visual system on FPGAs. Compared to Nvidia TX1 and Intel i7 CPU, our FPGA-based implementation achieves 5.6x and 3.4x speedup, as well as 3.0x and 34.6x power reduction, respectively.

We present an energy-efficient hardware architecture for real-time ELAS (Efficient Large-scale Stereo) based stereo matching on FPGAs. Compared to SOTA SoC and Intel i7 CPU, our FPGA design achieves 3.32x and 38.4x speedup, and 1.13x and 27.1x power reduction, respectively.

Benchmarking and Design Flow of Autonomous Machine Computing

We introduce RobotPerf, a benchmarking suite to evaluate robotics computing performance across a diverse range of hardware platforms. As an open-source initiative, RobotPerf remains committed to evolving with community input to advance the future of hardware-accelerated robotics.

This report summarizes our recent efforts in facilitating the development of scalable, efficient, adaptive, and reliable autonomous machine computing, including automatic domain-specific SoC exploration, software-hardware co-design, and performance-efficiency-resilience co-optimization.

We propose a machine learning-based design space exploration framework, Autopilot, that can automate the full system cyber-physical co-design for aerial robots. AutoPilot consistently outperforms general-purpose processors and specialized accelerators built for drones.

We present a bottleneck analysis tool, Skyline, for designing compute systems for autonomous Unmanned Aerial Vehicles (UAV). The tool provides insights by exploiting the fundamental relationships between various components in the autonomous UAV such as sensor, compute, body dynamics.

We present AutoSoC, a algorithms-hardware co-design framework for end-to-end learning-based aerial autonomous machines. AutoSoC runs the ASIC flow of place and route and generates layouts of the floor-planed accelerators with varying performance, area, and power consumption.

We introduce a roofline-like model to understand the role of computing in aerial autonomous machines. The model provides insights by exploiting the fundamental relationships between various components in an aerial robot, such as sensor framerate, compute performance, and body dynamics.