move_detect/docs/hardware-bringup.md

Hardware Bring-up & Benchmark Procedure (Raspberry Pi 5 + Hailo-8L)

This document describes the validation procedure for the edge detection core on actual hardware (Pi 5 + Hailo accelerator). Development PC testing is covered in the main test suite; this procedure applies to the Pi deployment phase.

1. PCIe Gen3 Activation

Enable PCIe Gen3 for Hailo-8L optimal throughput:

1. SSH to Pi 5 and edit /boot/firmware/config.txt:

   sudo nano /boot/firmware/config.txt
   

2. Add or update the line:

   dtparam=pciex1_gen=3
   

3. Reboot:

   sudo reboot
   

4. Verify PCIe Gen3 negotiation:

   lspci -vv | grep -i speed
   

   Expected output: LnkSta: Speed 8GT/s (Gen3) or higher, not 2.5GT/s (Gen1).

---

2. HEF Benchmark: Single, 2-Stream, 4-Stream

Benchmark yolov8s_pose.hef (fall detection backbone) under different load scenarios:

1. Single-stream baseline:

   • Load HEF via HailoRT VDevice, run 1 stream continuously (640×640 RGB input).
   • Measure: FPS, CPU usage, Hailo die temperature.
   • Target: ≥ 20 FPS @ <60°C (baseline headroom).

2. 2-stream round-robin:

   • Process 2 video streams (CSI + USB UVC) round-robin into Hailo.
   • Measure: Combined FPS per stream, CPU, temperature.
   • Target: ≥ 10 FPS per stream (full fall detection at 2 streams).

3. 4-stream round-robin (stress test):

   • Process 4 USB UVC streams into Hailo round-robin.
   • Measure: FPS per stream, CPU, temperature, memory.
   • Target: ≥ 7–8 FPS per stream (performance envelope; may require USB controller separation).

Benchmark tooling:

• Use hailo-benchmark or custom Python loop with HailoRT + OpenCV frame capture.
• Log FPS, CPU (top/mpstat), temperature (vcgencmd measure_temp).
• Store results in docs/benchmarks/YYYY-MM-DD-pi5-hef-results.txt.

---

3. FPS 3-Tier Validation

Measure frame processing under three operational modes:

Tier	Mode	Description	Target FPS	Notes
T1: Full	Detection on all frames	Hailo inference on every frame; maximum load.	≥ 12 fps (crucial: fall detection responsiveness)	For 1–2 primary cameras (fall risk zones). Limits concurrent streams.
T2: Detection	Inference every 2–3 frames	Trade-off: skip frames to reduce Hailo load.	≥ 8 fps	For additional context cameras; acceptable for non-critical zones.
T3: Motion	Hailo only on motion keyframes	Inference triggered by motion (v4l2 MJPEG keyframe or frame diff).	≥ 5 fps avg	For background monitoring; many cameras, low overhead.

1. Tier selection is INFER_EVERY in /etc/move-detect/edge.env (per device): 1 = T1 (every tick, the default), 2–3 = T2. On a skipped tick the frame is still read and discarded — the camera buffer stays current — but no pose runs and nothing enters the evidence ring, so no unblurred frame can reach a snapshot. T3 (motion-gated inference) is deliberately NOT implemented. A motion gate that misjudges would stop inferring on a motionless fallen worker — the exact case this product exists to catch — and its threshold cannot be tuned without real footage. Decide it during bring-up with measured video, not in advance.
2. Run each tier with realistic camera setup (e.g., T1 on main fall cam, T2 on perimeter).
3. Log FPS, E2E latency (frame capture → alert), CPU, Hailo temp for 30 min per tier.
4. Confirm T1 ≥ 12 fps for the target scene (well-lit, medium activity).

---

4. Multi-Camera Stability (Long-run Test)

Validate sustained operation and resource cleanup:

1. Setup:

   • 1 CSI camera (picamera2 via PiCameraSource) + 2–4 USB UVC cameras (UvcSource).
   • Route USB cameras to 2 separate USB3 controllers (Pi 5 has 2 controllers; check with lsusb -t).

2. Test duration: 4–8 hours continuous frame capture and optional Hailo inference.

3. Monitor:

   • FPS stability (no drift or sudden drops).
   • Memory usage (RSS, no unbounded growth; confirm video buffer cleanup).
   • CPU distribution (no single core maxed; picamera2 process isolation working).
   • Hailo temperature (sustained < 75°C).
   • Error logs (no hang or exception cascades).

4. Pass criterion: No crashes, memory leaks, or FPS degradation after 4 hours.

---

5. Reflect Results in Thresholds & Camera Roles

After benchmark completion, update configuration:

Every knob below lives in /etc/move-detect/edge.env (read by edge/config.py::load_config), not in a code file — no redeploy is needed to retune a site. All are validated at startup: a non-finite or non-positive value refuses to boot rather than silently disabling a detector. The full list is in docs/pilot-runbook.md §3.

1. Detector thresholds:

   • MOTION_THR (default 0.25) — per-second stillness threshold shared by fall no-motion and immobility. This is the primary re-tuning knob on real video: it is frame-rate normalised, so a device running at a different measured fps needs it re-checked.
   • FALL_PRONE_S (3.0) / FALL_NO_MOTION_S (10.0) — how long prone before a fall confirms, and before it escalates to the 위급 no-motion event. Raise if real footage over-triggers.
   • IMMOBILE_S (120.0), GHOST_REAPPEAR_S (10.0), ZONE_DEBOUNCE_S (0.75), COOLDOWN_S (30.0).

2. Per-camera load:

   • INFER_EVERY (1) — 1 for the fall-critical camera, 2–3 for context cameras that missed the T1 target. Set per device; each camera on a device shares it today.
   • TARGET_FPS (12) — loop pacing. It only slows the loop down; it cannot buy throughput.

3. Benchmark report:

   • Document actual Pi hardware, kernel version, HailoRT version, Hailo FW version.
   • Archive FPS/latency/temp results.
   • Flag any anomalies (thermal throttling, USB contention, CPU saturation).

---

Troubleshooting

PCIe speed not Gen3:

• Confirm cable seating and power to Hailo.
• Check firmware update: sudo rpi-update (experimental; use with caution).

HEF FPS < target:

• Verify HailoRT version matches Hailo FW.
• Check for CPU threads contention (picamera2 process + Python inference + OS).
• Monitor Hailo die temperature; throttle if > 85°C.

Memory leak in USB capture:

• Ensure OpenCV's VideoCapture.release() is called on cleanup.
• Check v4l2-ctl -D for buffer count and memory mapping.

USB controller contention:

• Separate high-bandwidth cameras (USB3 MJPEG) across 2 controllers.
• Use lsusb -t and /sys/bus/usb/devices/ to map.

---

References

• hailo-rpi5-examples — Hailo + Pi 5 reference (MIT).
• picamera2 docs.
• Raspberry Pi 5 datasheet.