---
name: iot-airgap-delivery
description: Use when working on IoT sensor systems with air-gapped deploy bundles (Pi nodes + shared-hosting PHP server) — covers derived-key auth, bundle regeneration workflow, 403/WAF debugging, display render verification, and pre-delivery audit. Trigger on deploy bundle work, HMAC key mismatch/403 errors, Pi display rendering, or "납품" (delivery) prep.
---

# IoT Air-gap Delivery Workflow

Battle-tested workflow for IoT systems shaped like: sensor nodes (Raspberry Pi / STM32)
→ shared-hosting PHP server (cafe24-style) → optional mini display node, deployed as
self-contained offline bundles generated by a bundle-generator script.

## 1. Key architecture (get this right first — most repeated confusion)

- **API_KEY** = master secret. Lives ONLY in: server `config.local.php` + build machine
  `deploy/.secrets/api_key.txt`. Never on leaf devices.
- **Derived keys** = `HMAC-SHA256(message=<label>, key=API_KEY)`:
  - `STATUS_KEY = hash_hmac('sha256', 'status', API_KEY)` → display node only
  - `CRON_SECRET = hash_hmac('sha256', 'cron', API_KEY)` → cron HTTP access
- The server stores NO derived keys — it recomputes them per request and compares with
  `hash_equals`. So "server-side status key" does not exist as a stored value.
- To print the expected derived key on the server:
  `php -r 'require "config.php"; echo hash_hmac("sha256","status",API_KEY),"\n";'`
- NEVER put the same value in both API_KEY and a derived-key slot. They are different values.
- Regenerating `api_key.txt` breaks EVERY deployed node. The generator must reuse an
  existing secret file; verify the printed key prefix is unchanged after regeneration.

## 2. Bundle workflow (source is truth)

1. Edit source files only (never hand-edit files inside `deploy/`).
2. Re-run the generator (e.g. `python tools/make_deploy_bundles.py`).
3. Confirm output reuses the existing API_KEY (prefix printed unchanged).
4. Deploy: server bundle → hosting upload; node bundle → copy to Pi →
   `sudo systemctl restart <service>`.
5. `deploy/` must be gitignored (contains live secrets); remind to back up
   `deploy/.secrets/` offline — losing it orphans every deployed node.

## 3. 403 / auth-failure debugging ladder (in order)

1. Distinguish WHO rejected: `curl -sS -i -H "X-<Key-Header>: <key>" <url>`
   - JSON error body (`{"error":"forbidden"}`) → app rejected → key mismatch.
   - HTML 403 → Apache/WAF rejected → NOT a key problem.
2. Key mismatch: print server-side derived key (see §1), diff against device config byte-for-byte
   (watch for pasted whitespace/newlines).
3. curl works but device fails with SAME key →
   a. Service not restarted after config edit (config imported once at process start).
   b. Service loads a DIFFERENT config file (check systemd `WorkingDirectory`/`ExecStart`,
      grep the actual file the service imports).
   c. **WAF blocks default `python-requests/x.y` User-Agent** (very common on shared hosting;
      curl's UA passes). Fix: always send a custom UA (`"<AppName>/2.0"`) in every
      `requests` call. Reproduce with `curl -A "python-requests/2.31.0"`.
4. 503 `server_not_configured` → server API_KEY still default → upload real config.local.php.
5. Placeholder junk keys (e.g. `wnegfjk...`) in a config = never-configured file; find which
   file the process actually loads before editing anything.

## 4. Display node changes (small TFT, Pillow rendering)

- Any render-code change: verify OFF-DEVICE first. Write a scratchpad script that stubs
  `config` + driver modules via `sys.modules`, feeds synthetic data (including worst cases:
  all-sections node, missing history, single data point, extreme values), saves PNGs at the
  panel's exact resolution.
- Read the PNGs back (vision) to check: no overlapping text, no clipping at panel edges,
  no dead zones; then send previews to the user via SendUserFile before deploying.
- Layout: slot-based adaptive layout (only sections the node type needs, evenly divided)
  beats fixed y-coordinates. Guard: clamp page index when sensor count shrinks; fall back
  to a smaller font when a formatted value exceeds panel width.
- SPI display init fail `No such file or directory` → `/dev/spidev*` missing → enable SPI
  (`sudo raspi-config nonint do_spi 0`) + reboot; then check user in `spi,gpio` groups.

## 5. Server API endpoints (shared hosting)

- Poll-heavy endpoints (device polls every ~30s): every time-window query MUST ride an
  index. Composite `(sensor_id, measured_at)` needs a `sensor_id` predicate — loop
  per-sensor with a prepared statement instead of one global range scan.
- Cap unbounded result sets (`LIMIT`) — a flapping sensor must not inflate the JSON payload.
- Wrap the whole handler in try/catch → generic `{"error":"internal"}` + `error_log`;
  set `ini_set('display_errors','0')` centrally (shared hosting often ships On).
- Rate-limit per IP (file-based fixed window, LOCK_EX, same pattern as login throttle);
  fail-open on storage errors so monitoring devices never die from the throttle itself.
- Guard optional tables with a `table_exists` helper before querying.
- Additive JSON only: add keys, never rename/remove — old deployed clients keep working.

## 6. Pre-delivery audit (납품 전)

1. Launch TWO parallel review agents: security-reviewer + code-reviewer, scoped to the
   changed surfaces. Verify each finding against actual code before acting.
2. Always check FIRST: real secret files (`config_local.py`, `config.local.php`) covered by
   `.gitignore` AND never committed (`git log --all --full-history -- <path>`); rotate if leaked.
3. Fix by severity: CRITICAL (secrets/injection) → HIGH (perf/full scans) → MEDIUM
   (error disclosure, unbounded payloads) → LOW (render edge cases).
4. `php -l` every touched PHP file; re-run render verification; regenerate bundles; then commit.