move_detect/docs/hardware-bringup.md

Hardware Bring-up & Benchmark Procedure (Raspberry Pi 5 + Hailo-8L)

This document describes the validation procedure for the edge detection core on actual hardware (Pi 5 + Hailo accelerator). Development PC testing is covered in the main test suite; this procedure applies to the Pi deployment phase.

1. PCIe Gen3 Activation

Enable PCIe Gen3 for Hailo-8L optimal throughput:

1. SSH to Pi 5 and edit /boot/firmware/config.txt:

   sudo nano /boot/firmware/config.txt
   

2. Add or update the line:

   dtparam=pciex1_gen=3
   

3. Reboot:

   sudo reboot
   

4. Verify PCIe Gen3 negotiation:

   lspci -vv | grep -i speed
   

   Expected output: LnkSta: Speed 8GT/s (Gen3) or higher, not 2.5GT/s (Gen1).

---

2. HEF Benchmark: Single, 2-Stream, 4-Stream

Benchmark yolov8s_pose.hef (fall detection backbone) under different load scenarios:

1. Single-stream baseline:

   • Load HEF via HailoRT VDevice, run 1 stream continuously (640×640 RGB input).
   • Measure: FPS, CPU usage, Hailo die temperature.
   • Target: ≥ 20 FPS @ <60°C (baseline headroom).

2. 2-stream round-robin:

   • Process 2 video streams (CSI + USB UVC) round-robin into Hailo.
   • Measure: Combined FPS per stream, CPU, temperature.
   • Target: ≥ 10 FPS per stream (full fall detection at 2 streams).

3. 4-stream round-robin (stress test):

   • Process 4 USB UVC streams into Hailo round-robin.
   • Measure: FPS per stream, CPU, temperature, memory.
   • Target: ≥ 7–8 FPS per stream (performance envelope; may require USB controller separation).

Benchmark tooling:

• Use hailo-benchmark or custom Python loop with HailoRT + OpenCV frame capture.
• Log FPS, CPU (top/mpstat), temperature (vcgencmd measure_temp).
• Store results in docs/benchmarks/YYYY-MM-DD-pi5-hef-results.txt.

---

3. FPS 3-Tier Validation

Measure frame processing under three operational modes:

Tier	Mode	Description	Target FPS	Notes
T1: Full	Detection on all frames	Hailo inference on every frame; maximum load.	≥ 12 fps (crucial: fall detection responsiveness)	For 1–2 primary cameras (fall risk zones). Limits concurrent streams.
T2: Detection	Inference every 2–3 frames	Trade-off: skip frames to reduce Hailo load.	≥ 8 fps	For additional context cameras; acceptable for non-critical zones.
T3: Motion	Hailo only on motion keyframes	Inference triggered by motion (v4l2 MJPEG keyframe or frame diff).	≥ 5 fps avg	For background monitoring; many cameras, low overhead.

1. Configure edge detection core to support tier selection (via thresholds file or env var).
2. Run each tier with realistic camera setup (e.g., T1 on main fall cam, T2/T3 on perimeter).
3. Log FPS, E2E latency (frame capture → alert), CPU, Hailo temp for 30 min per tier.
4. Confirm T1 ≥ 12 fps for the target scene (well-lit, medium activity).

---

4. Multi-Camera Stability (Long-run Test)

Validate sustained operation and resource cleanup:

1. Setup:

   • 1 CSI camera (picamera2 via PiCameraSource) + 2–4 USB UVC cameras (UvcSource).
   • Route USB cameras to 2 separate USB3 controllers (Pi 5 has 2 controllers; check with lsusb -t).

2. Test duration: 4–8 hours continuous frame capture and optional Hailo inference.

3. Monitor:

   • FPS stability (no drift or sudden drops).
   • Memory usage (RSS, no unbounded growth; confirm video buffer cleanup).
   • CPU distribution (no single core maxed; picamera2 process isolation working).
   • Hailo temperature (sustained < 75°C).
   • Error logs (no hang or exception cascades).

4. Pass criterion: No crashes, memory leaks, or FPS degradation after 4 hours.

---

5. Reflect Results in Thresholds & Camera Roles

After benchmark completion, update configuration:

1. edge/config.py thresholds:

   • Adjust motion.cooldown_ms, motion.min_duration_ms based on measured T2/T3 FPS.
   • Tune fall.confidence_threshold if fall recall vs. precision trade-off differs on real video.

2. edge/config.py camera roles:

   • Assign role="fall_primary" to high-fps T1 cam (≥ 12 fps).
   • Assign role="context" to T2 cameras (8–10 fps).
   • Assign role="perimeter" to T3 motion-only cameras.

3. Benchmark report:

   • Document actual Pi hardware, kernel version, HailoRT version, Hailo FW version.
   • Archive FPS/latency/temp results.
   • Flag any anomalies (thermal throttling, USB contention, CPU saturation).

---

Troubleshooting

PCIe speed not Gen3:

• Confirm cable seating and power to Hailo.
• Check firmware update: sudo rpi-update (experimental; use with caution).

HEF FPS < target:

• Verify HailoRT version matches Hailo FW.
• Check for CPU threads contention (picamera2 process + Python inference + OS).
• Monitor Hailo die temperature; throttle if > 85°C.

Memory leak in USB capture:

• Ensure OpenCV's VideoCapture.release() is called on cleanup.
• Check v4l2-ctl -D for buffer count and memory mapping.

USB controller contention:

• Separate high-bandwidth cameras (USB3 MJPEG) across 2 controllers.
• Use lsusb -t and /sys/bus/usb/devices/ to map.

---

References

• hailo-rpi5-examples — Hailo + Pi 5 reference (MIT).
• picamera2 docs.
• Raspberry Pi 5 datasheet.