Project Overview
Drone swarms require more than object detection: they must understand how aerial agents move, interact, and change their collective behavior. This project develops a pixels-to-semantics pipeline that converts visual observations into compact descriptions of formation, mission intent, cooperation, and potential risk directly on UAV-class edge hardware.
The project combines two complementary implementations. HARE uses panoramic multi-camera perception, multi-object tracking, spatiotemporal graph learning, compact tokenization, and a lightweight semantic decoder. The ISQED edge-semantics system uses a highly efficient FPGA tracker and a quantized language model running on the embedded ARM processor. Together, they explore how FPGA fabric and embedded processors can share perception and reasoning while reducing cloud dependence, bandwidth use, and communication delay.
System Approach
Panoramic perception: Three synchronized RGB cameras are mapped into a common panoramic view using ORB feature matching and RANSAC. Detection and ByteTrack-style association produce stable object identities and trajectories.
Interaction-aware reasoning: Tracked agents form a dynamic spatiotemporal graph. Attention-based message passing and temporal modeling capture relative motion, proximity, and evolving group behavior.
Compact semantics: Air-VQ converts graph embeddings into a bounded token sequence, while SlotLLM predicts formation, mission intent, and cooperation state.
On-device language understanding: The ISQED implementation converts FPGA-generated track and interaction records into schema-constrained summaries using an 8-bit quantized language model on the KV260 processing system.
Edge Deployment and Results
HARE: The KV260 implementation closes timing at 100 MHz and uses 61,857 LUTs, 58,797 registers, 142 BRAM tiles, and 524 DSPs. Post-implementation Vivado analysis estimates 3.513 W of on-chip power. Its bounded interface uses at most 145 tokens per window, reducing a 4,096-byte floating-point representation to 290 bytes. Functional behavior was validated on a synchronized three-camera laboratory sequence; complete on-board frame-rate and end-to-end latency measurements remain future work.
ISQED system: The 320 × 240 visual front end processes a frame in 66.7 ms, or approximately 15 FPS. Its tracker uses only 2,182 LUTs, 2,165 flip-flops, one BRAM, and no DSPs, while the reported tracker-plus-LLM configuration operates at approximately 2.9 W. In a representative 508-frame sequence containing five UAVs performing a rounding maneuver, the adapted on-device model correctly identified both the object count and behavior.
Impact
The combined work demonstrates a practical path from raw multi-camera video to high-level multi-agent understanding on resource-constrained platforms. It supports autonomous monitoring, search and rescue, infrastructure inspection, environmental observation, and operation in bandwidth-limited or contested environments.
Publications
Z. Wang, W. A. Fortney, P. O. Forcha, Y. Feng, G. Bendix, and C. Bobda, “Real-Time Edge Semantics for Drone Swarms via FPGA Perception and On-Device LLMs,” 2026 27th International Symposium on Quality Electronic Design (ISQED), 2026. DOI: 10.1109/ISQED69900.2026.11534722.