# FlyLo Gesture FX — Outboard Pose-Triggered Visuals

A standalone outboard app for gesture-triggered live visuals. MediaPipe
Pose tracks the performer's skeleton from the camera feed and fires
particle bursts, eruptive beams, and full-frame slams on specific arm
gestures. NDI into Resolume via Screen Capture HX.

Installs side-by-side with the main `FlyLo CamFX.app` (own Bundle ID).
Built off the same engine — same audio coupling, same OUTPUT picker,
same render-resolution / framerate controls, same MIDI/keyboard mapping
system. Stripped FX bank to just gesture-focused work.

## FX bank (4 slots)

- **Slot 1 · CLEAN CAM** — raw camera passthrough with optional
  background removal (MediaPipe Selfie + DepthAnything depth-key,
  combined or independent). Use as the baseline NDI source for
  Resolume when you just want clean signal.
- **Slot 2 · ERUPT** — eruptive particle burst spawned from wrist
  positions when wrist velocity exceeds a threshold. State-free
  particle pool, additive blending, 4 palettes.
- **Slot 3 · BEAMS** — eruptive light beams shoot outward along the
  arm vectors on specific gesture triggers. Per-gesture beam pattern:
  arms-up = vertical pillars, sweep = horizontal pair, invoke =
  V-pattern + max brightness.
- **Slot 4 · INVOKE** — full-frame slam (center flash + expanding
  rings) when both wrists go above the head with upward velocity.
  The "drop the ceiling" trigger. Peak-reignites during the decay
  envelope so audio impacts re-light the moment.

## Gesture vocabulary (from MediaPipe Pose · lite model)

States (true while held):
- **ARMS UP** — both wrists above head height (nose Y).
- **ARMS OUT** — both wrists at shoulder height, extended outward
  (T-pose).
- **REACH FORWARD** — both wrists closer to camera than hips
  (z-depth check).
- **CROUCH** — hips significantly below their rolling rest position.

Triggers (single-shot):
- **FIST PUMP (left / right)** — sharp upward wrist velocity per side.
- **SWEEP (left / right)** — sharp horizontal wrist velocity in either
  direction.
- **ARMS UP RISING** — edge: arms-up state went false → true.
- **INVOKE** — both wrists fast-up AND above head simultaneously.

A top-left chip strip lights as each gesture fires, so you can verify
the detector is reading your motion correctly.

## Show-time flow (NDI → Resolume)

1. Launch `FlyLo Gesture FX.app`. Permit camera + mic.
2. Hit `Tab` to cycle to slot 2 / 3 / 4 (the gesture FX).
3. Click the `⤴ OUTPUT` button → pick your secondary display. Drop
   render resolution to `1920 × 1080` or `1280 × 720` if your Mac
   needs the headroom. Drop framerate cap to `30 fps` if pose
   detection is bogging.
4. On the Mac, open **NDI Tools Screen Capture HX** and point it at
   the output popup window. Resolume picks up the NDI source.
5. Stand in front of the camera. Move the performer. Watch the chip
   strip light up as your gestures fire.

The popup output mirror window includes the gesture FX visually but
NOT the chip overlay or HUD (those are operator-side only) — so
Resolume sees a clean frame.

## Build the .dmg

Same recipe as CamFX:

```bash
unzip flylo-gesture-fx-source.zip
cd flylo-gesture-fx-bundle

npm install                       # ~1-2 min, ~600 MB
./scripts/download-depth-model.sh # ~99 MB ONNX (for CLEAN CAM's depth key)
npm run dist:mac                  # ~3-5 min, produces release/FlyLo Gesture FX-1.0.0.dmg

open release/                     # drag "FlyLo Gesture FX" to /Applications
```

First launch: right-click → Open (Gatekeeper bypass, one-time). Permit
camera + microphone. App ID is `com.strangeloop.flylo-gesture-fx` so it
co-exists with `FlyLo CamFX.app` and `FlyLo CamFX Text.app`.

## Tuning gesture sensitivity

If gestures fire too easily / not easily enough, every threshold lives
in `src/pose/GestureDetector.ts` under the `DEFAULT_OPTS` constant:

- `fistPumpSpeed` — minimum wrist velocity (image-units/sec) for a
  fist-pump trigger. Lower = more sensitive.
- `sweepSpeed` — same for sweeps.
- `armsUpClearance` — how far above the nose Y the wrists must be.
- `triggerCooldown` — minimum seconds between same-side triggers (so
  one big motion doesn't fire 10 times).

Rebuild after editing — these aren't exposed as live params yet.

## Known caveats

- MediaPipe Pose **lite** model — accurate enough for arms / hips at
  stage distance. For finer joints (heel articulation, finger-level)
  switch the model URL in `PoseTracker.ts` to `_full` (~25 MB) or
  `_heavy` (~75 MB). Lite runs at 60fps on Apple Silicon; full ~30fps;
  heavy ~15fps.
- Camera should be reasonably steady — pose detection is image-space,
  not world-space, so dramatic camera moves will register as motion.
- Single performer only. The detector tracks `numPoses: 1`. Bump it in
  `PoseTracker.ts` if you want multi-performer.