141 lines
5.4 KiB
Markdown
141 lines
5.4 KiB
Markdown
# Hardware Bring-up & Benchmark Procedure (Raspberry Pi 5 + Hailo-8L)
|
||
|
||
This document describes the validation procedure for the edge detection core on actual hardware (Pi 5 + Hailo accelerator). Development PC testing is covered in the main test suite; this procedure applies to the Pi deployment phase.
|
||
|
||
## 1. PCIe Gen3 Activation
|
||
|
||
Enable PCIe Gen3 for Hailo-8L optimal throughput:
|
||
|
||
1. SSH to Pi 5 and edit `/boot/firmware/config.txt`:
|
||
```bash
|
||
sudo nano /boot/firmware/config.txt
|
||
```
|
||
|
||
2. Add or update the line:
|
||
```
|
||
dtparam=pciex1_gen=3
|
||
```
|
||
|
||
3. Reboot:
|
||
```bash
|
||
sudo reboot
|
||
```
|
||
|
||
4. Verify PCIe Gen3 negotiation:
|
||
```bash
|
||
lspci -vv | grep -i speed
|
||
```
|
||
Expected output: `LnkSta: Speed 8GT/s` (Gen3) or higher, not 2.5GT/s (Gen1).
|
||
|
||
---
|
||
|
||
## 2. HEF Benchmark: Single, 2-Stream, 4-Stream
|
||
|
||
Benchmark `yolov8s_pose.hef` (fall detection backbone) under different load scenarios:
|
||
|
||
1. **Single-stream baseline:**
|
||
- Load HEF via HailoRT VDevice, run 1 stream continuously (640×640 RGB input).
|
||
- Measure: FPS, CPU usage, Hailo die temperature.
|
||
- Target: ≥ 20 FPS @ <60°C (baseline headroom).
|
||
|
||
2. **2-stream round-robin:**
|
||
- Process 2 video streams (CSI + USB UVC) round-robin into Hailo.
|
||
- Measure: Combined FPS per stream, CPU, temperature.
|
||
- Target: ≥ 10 FPS per stream (full fall detection at 2 streams).
|
||
|
||
3. **4-stream round-robin (stress test):**
|
||
- Process 4 USB UVC streams into Hailo round-robin.
|
||
- Measure: FPS per stream, CPU, temperature, memory.
|
||
- Target: ≥ 7–8 FPS per stream (performance envelope; may require USB controller separation).
|
||
|
||
**Benchmark tooling:**
|
||
- Use `hailo-benchmark` or custom Python loop with HailoRT + OpenCV frame capture.
|
||
- Log FPS, CPU (top/mpstat), temperature (vcgencmd measure_temp).
|
||
- Store results in `docs/benchmarks/YYYY-MM-DD-pi5-hef-results.txt`.
|
||
|
||
---
|
||
|
||
## 3. FPS 3-Tier Validation
|
||
|
||
Measure frame processing under three operational modes:
|
||
|
||
| Tier | Mode | Description | Target FPS | Notes |
|
||
|------|------|-------------|-----------|-------|
|
||
| **T1: Full** | Detection on all frames | Hailo inference on every frame; maximum load. | ≥ 12 fps (crucial: fall detection responsiveness) | For 1–2 primary cameras (fall risk zones). Limits concurrent streams. |
|
||
| **T2: Detection** | Inference every 2–3 frames | Trade-off: skip frames to reduce Hailo load. | ≥ 8 fps | For additional context cameras; acceptable for non-critical zones. |
|
||
| **T3: Motion** | Hailo only on motion keyframes | Inference triggered by motion (v4l2 MJPEG keyframe or frame diff). | ≥ 5 fps avg | For background monitoring; many cameras, low overhead. |
|
||
|
||
1. Configure edge detection core to support tier selection (via thresholds file or env var).
|
||
2. Run each tier with realistic camera setup (e.g., T1 on main fall cam, T2/T3 on perimeter).
|
||
3. Log FPS, E2E latency (frame capture → alert), CPU, Hailo temp for 30 min per tier.
|
||
4. Confirm T1 ≥ 12 fps for the target scene (well-lit, medium activity).
|
||
|
||
---
|
||
|
||
## 4. Multi-Camera Stability (Long-run Test)
|
||
|
||
Validate sustained operation and resource cleanup:
|
||
|
||
1. **Setup:**
|
||
- 1 CSI camera (picamera2 via PiCameraSource) + 2–4 USB UVC cameras (UvcSource).
|
||
- Route USB cameras to 2 separate USB3 controllers (Pi 5 has 2 controllers; check with `lsusb -t`).
|
||
|
||
2. **Test duration:** 4–8 hours continuous frame capture and optional Hailo inference.
|
||
|
||
3. **Monitor:**
|
||
- FPS stability (no drift or sudden drops).
|
||
- Memory usage (RSS, no unbounded growth; confirm video buffer cleanup).
|
||
- CPU distribution (no single core maxed; picamera2 process isolation working).
|
||
- Hailo temperature (sustained < 75°C).
|
||
- Error logs (no hang or exception cascades).
|
||
|
||
4. **Pass criterion:** No crashes, memory leaks, or FPS degradation after 4 hours.
|
||
|
||
---
|
||
|
||
## 5. Reflect Results in Thresholds & Camera Roles
|
||
|
||
After benchmark completion, update configuration:
|
||
|
||
1. **`edge/config.py` thresholds:**
|
||
- Adjust `motion.cooldown_ms`, `motion.min_duration_ms` based on measured T2/T3 FPS.
|
||
- Tune `fall.confidence_threshold` if fall recall vs. precision trade-off differs on real video.
|
||
|
||
2. **`edge/config.py` camera roles:**
|
||
- Assign `role="fall_primary"` to high-fps T1 cam (≥ 12 fps).
|
||
- Assign `role="context"` to T2 cameras (8–10 fps).
|
||
- Assign `role="perimeter"` to T3 motion-only cameras.
|
||
|
||
3. **Benchmark report:**
|
||
- Document actual Pi hardware, kernel version, HailoRT version, Hailo FW version.
|
||
- Archive FPS/latency/temp results.
|
||
- Flag any anomalies (thermal throttling, USB contention, CPU saturation).
|
||
|
||
---
|
||
|
||
## Troubleshooting
|
||
|
||
**PCIe speed not Gen3:**
|
||
- Confirm cable seating and power to Hailo.
|
||
- Check firmware update: `sudo rpi-update` (experimental; use with caution).
|
||
|
||
**HEF FPS < target:**
|
||
- Verify HailoRT version matches Hailo FW.
|
||
- Check for CPU threads contention (picamera2 process + Python inference + OS).
|
||
- Monitor Hailo die temperature; throttle if > 85°C.
|
||
|
||
**Memory leak in USB capture:**
|
||
- Ensure OpenCV's `VideoCapture.release()` is called on cleanup.
|
||
- Check `v4l2-ctl -D` for buffer count and memory mapping.
|
||
|
||
**USB controller contention:**
|
||
- Separate high-bandwidth cameras (USB3 MJPEG) across 2 controllers.
|
||
- Use `lsusb -t` and `/sys/bus/usb/devices/` to map.
|
||
|
||
---
|
||
|
||
## References
|
||
|
||
- [hailo-rpi5-examples](https://github.com/hailo-ai/hailo-rpi5-examples) — Hailo + Pi 5 reference (MIT).
|
||
- [picamera2 docs](https://datasheets.raspberrypi.com/camera/picamera2-manual.pdf).
|
||
- [Raspberry Pi 5 datasheet](https://datasheets.raspberrypi.com/rpi5/raspberry-pi-5-datasheet.pdf).
|