# Hardware Bring-up & Benchmark Procedure (Raspberry Pi 5 + Hailo-8L)

This document describes the validation procedure for the edge detection core on actual hardware (Pi 5 + Hailo accelerator). Development PC testing is covered in the main test suite; this procedure applies to the Pi deployment phase.

## 1. PCIe Gen3 Activation

Enable PCIe Gen3 for Hailo-8L optimal throughput:

1. SSH to Pi 5 and edit `/boot/firmware/config.txt`:
   ```bash
   sudo nano /boot/firmware/config.txt
   ```

2. Add or update the line:
   ```
   dtparam=pciex1_gen=3
   ```

3. Reboot:
   ```bash
   sudo reboot
   ```

4. Verify PCIe Gen3 negotiation:
   ```bash
   lspci -vv | grep -i speed
   ```
   Expected output: `LnkSta: Speed 8GT/s` (Gen3) or higher, not 2.5GT/s (Gen1).

---

## 2. HEF Benchmark: Single, 2-Stream, 4-Stream

Benchmark `yolov8s_pose.hef` (fall detection backbone) under different load scenarios:

1. **Single-stream baseline:**
   - Load HEF via HailoRT VDevice, run 1 stream continuously (640×640 RGB input).
   - Measure: FPS, CPU usage, Hailo die temperature.
   - Target: ≥ 20 FPS @ <60°C (baseline headroom).

2. **2-stream round-robin:**
   - Process 2 video streams (CSI + USB UVC) round-robin into Hailo.
   - Measure: Combined FPS per stream, CPU, temperature.
   - Target: ≥ 10 FPS per stream (full fall detection at 2 streams).

3. **4-stream round-robin (stress test):**
   - Process 4 USB UVC streams into Hailo round-robin.
   - Measure: FPS per stream, CPU, temperature, memory.
   - Target: ≥ 7–8 FPS per stream (performance envelope; may require USB controller separation).

**Benchmark tooling:**
- Use `hailo-benchmark` or custom Python loop with HailoRT + OpenCV frame capture.
- Log FPS, CPU (top/mpstat), temperature (vcgencmd measure_temp).
- Store results in `docs/benchmarks/YYYY-MM-DD-pi5-hef-results.txt`.

---

## 3. FPS 3-Tier Validation

Measure frame processing under three operational modes:

| Tier | Mode | Description | Target FPS | Notes |
|------|------|-------------|-----------|-------|
| **T1: Full** | Detection on all frames | Hailo inference on every frame; maximum load. | ≥ 12 fps (crucial: fall detection responsiveness) | For 1–2 primary cameras (fall risk zones). Limits concurrent streams. |
| **T2: Detection** | Inference every 2–3 frames | Trade-off: skip frames to reduce Hailo load. | ≥ 8 fps | For additional context cameras; acceptable for non-critical zones. |
| **T3: Motion** | Hailo only on motion keyframes | Inference triggered by motion (v4l2 MJPEG keyframe or frame diff). | ≥ 5 fps avg | For background monitoring; many cameras, low overhead. |

1. Tier selection is `INFER_EVERY` in `/etc/move-detect/edge.env` (per device):
   `1` = T1 (every tick, the default), `2`–`3` = T2. On a skipped tick the frame is still read
   and discarded — the camera buffer stays current — but no pose runs and nothing enters the
   evidence ring, so no unblurred frame can reach a snapshot.
   **T3 (motion-gated inference) is deliberately NOT implemented.** A motion gate that misjudges
   would stop inferring on a *motionless fallen worker* — the exact case this product exists to
   catch — and its threshold cannot be tuned without real footage. Decide it during bring-up
   with measured video, not in advance.
2. Run each tier with realistic camera setup (e.g., T1 on main fall cam, T2 on perimeter).
3. Log FPS, E2E latency (frame capture → alert), CPU, Hailo temp for 30 min per tier.
4. Confirm T1 ≥ 12 fps for the target scene (well-lit, medium activity).

---

## 4. Multi-Camera Stability (Long-run Test)

Validate sustained operation and resource cleanup:

1. **Setup:**
   - 1 CSI camera (picamera2 via PiCameraSource) + 2–4 USB UVC cameras (UvcSource).
   - Route USB cameras to 2 separate USB3 controllers (Pi 5 has 2 controllers; check with `lsusb -t`).

2. **Test duration:** 4–8 hours continuous frame capture and optional Hailo inference.

3. **Monitor:**
   - FPS stability (no drift or sudden drops).
   - Memory usage (RSS, no unbounded growth; confirm video buffer cleanup).
   - CPU distribution (no single core maxed; picamera2 process isolation working).
   - Hailo temperature (sustained < 75°C).
   - Error logs (no hang or exception cascades).

4. **Pass criterion:** No crashes, memory leaks, or FPS degradation after 4 hours.

---

## 5. Reflect Results in Thresholds & Camera Roles

After benchmark completion, update configuration:

Every knob below lives in `/etc/move-detect/edge.env` (read by `edge/config.py::load_config`),
**not** in a code file — no redeploy is needed to retune a site. All are validated at startup:
a non-finite or non-positive value refuses to boot rather than silently disabling a detector.
The full list is in `docs/pilot-runbook.md` §3.

1. **Detector thresholds:**
   - `MOTION_THR` (default 0.25) — per-second stillness threshold shared by fall no-motion and
     immobility. This is the primary re-tuning knob on real video: it is frame-rate normalised,
     so a device running at a different measured fps needs it re-checked.
   - `FALL_PRONE_S` (3.0) / `FALL_NO_MOTION_S` (10.0) — how long prone before a fall confirms,
     and before it escalates to the 위급 no-motion event. Raise if real footage over-triggers.
   - `IMMOBILE_S` (120.0), `GHOST_REAPPEAR_S` (10.0), `ZONE_DEBOUNCE_S` (0.75), `COOLDOWN_S` (30.0).

2. **Per-camera load:**
   - `INFER_EVERY` (1) — 1 for the fall-critical camera, 2–3 for context cameras that missed
     the T1 target. Set per device; each camera on a device shares it today.
   - `TARGET_FPS` (12) — loop pacing. It only slows the loop down; it cannot buy throughput.

3. **Benchmark report:**
   - Document actual Pi hardware, kernel version, HailoRT version, Hailo FW version.
   - Archive FPS/latency/temp results.
   - Flag any anomalies (thermal throttling, USB contention, CPU saturation).

---

## Troubleshooting

**PCIe speed not Gen3:**
- Confirm cable seating and power to Hailo.
- Check firmware update: `sudo rpi-update` (experimental; use with caution).

**HEF FPS < target:**
- Verify HailoRT version matches Hailo FW.
- Check for CPU threads contention (picamera2 process + Python inference + OS).
- Monitor Hailo die temperature; throttle if > 85°C.

**Memory leak in USB capture:**
- Ensure OpenCV's `VideoCapture.release()` is called on cleanup.
- Check `v4l2-ctl -D` for buffer count and memory mapping.

**USB controller contention:**
- Separate high-bandwidth cameras (USB3 MJPEG) across 2 controllers.
- Use `lsusb -t` and `/sys/bus/usb/devices/` to map.

---

## References

- [hailo-rpi5-examples](https://github.com/hailo-ai/hailo-rpi5-examples) — Hailo + Pi 5 reference (MIT).
- [picamera2 docs](https://datasheets.raspberrypi.com/camera/picamera2-manual.pdf).
- [Raspberry Pi 5 datasheet](https://datasheets.raspberrypi.com/rpi5/raspberry-pi-5-datasheet.pdf).