Preprint · not yet submitted

Seeing Through the Blackout

Gesture-Based Theatrical Lighting Control Under Adversarial Illumination

Amogh Bajpai · Independent Researcher

The constraint

The system’s own output is an adversary to its own input. A blackout removes the illumination the camera depends on, and the system must stay controllable in the state it has just created. A saturated wash shifts skin tone outside the distribution the tracker was trained on. Haze scatters the image. In every case the disturbance is produced by the artistic intent of the production, so it is not something the system may ask to have reduced.

We call this adversarial illumination, and argue it is what separates stage lighting from every adjacent gesture-control domain. The proposed resolution is spectral: near-infrared sensing places the control channel outside the visible band entirely. That is argued in the paper and not yet demonstrated — the prototype uses a visible-light webcam.

Four findings from building it

The tracker goes blind as the gesture completes

A clap cannot be detected from contact, because contact is the moment tracking fails. When two palms meet the merged region stops resembling two hands and the landmark estimator drops the pair — the frame in which the gesture completes is the frame in which observation is lost. The detector reads the envelope instead: fast closing motion, then loss of the pair while separation was still small.

Any gesture whose defining moment is self-occluding — contact, grasp, fist closure — is invisible at its own climax to a landmark tracker.

Clap twice on the demo and watch the hand count drop to zero at the instant it fires.

Cross-cue interference cascade

A clap throws both hands laterally at speed, and at the moment the pair merges the tracker reports a single hand travelling fast — an exact match for a lateral swipe. Every clap fired a spurious colour cue. That cue then held the global cooldown open, and the cooldown suppressed the second clap that had caused it.

Neither detector was wrong alone. Presenting gesture i and recording whether cue i fires reports every detector as correct; the cascade only appears under continuous operation with a false-trigger log.

This is the independent argument for false triggers per hour as the headline metric.

Cues must be performable without looking

The master dim originally required a flat palm facing downward. It worked in isolation and failed in use, because an operator watches the stage, not their own hand. A gesture needing a specific orientation needs visual confirmation to perform reliably, and visual attention is the one resource an operator running a show does not have.

In an operator context, a cue that must be looked at in order to be performed is not a usable cue. That eliminates most precision poses and anything depending on finger-level configuration.

Orientation testing was removed entirely — any open hand now drives the master.

Latency is epistemic, not computational

Inference accounts for roughly 8% of the end-to-end budget. The dominant terms are temporal evidence accumulation — a classifier cannot commit until it has seen enough of the gesture — and fixture rise time, which is electromechanical.

Faster hardware does not meaningfully help. Reducing inference cost to zero would recover under a tenth of the observed latency; the available lever is early classification from partial sequences.

The live telemetry panel reports each stage separately, and says which terms a browser cannot observe.

Table II, live

These are not transcribed from the paper. They are read at build time from the same constants the running system uses, so the table here, the table in the PDF and the deployed detectors cannot drift apart. A parameter table copied out by hand is wrong by the second revision.

Cue	Parameter	Value
LX 1	Contact separation	1.3 spans
LX 1	Re-arm separation	2.2 spans
LX 1	Separation at track loss	2.4 spans
LX 1	Closing speed	4.5 spans/s
LX 1	Grace after pair lost	260 ms
LX 1	Re-arm timeout	240 ms
LX 1	Inter-clap gap	120--900 ms
LX 2	Hold	400 ms
LX 2	Wrist height	0.62
LX 3	Hold	500 ms
LX 3	Stillness per frame	0.035
LX 4	Minimum extended fingers	3
LX 4	Smoothing coefficient	0.4
LX 5	Window	350 ms
LX 5	Travel	0.3 frame widths
LX 5	Minimum samples	4 frames
LX 5	Cooldown	1500 ms
LX 5	Two-hand lockout	800 ms
---	Global cooldown	350 ms
---	Hand-count debounce	110 ms

The rig under test

29 fixtures across 119 channels of a 512-channel universe, hung as a mid-size proscenium house actually is. Channel counts follow real personalities, so the universe the browser generates is patchable without translation.

Front of house

4 × profile · 4 ch each · patch 1–16

Overhead

4 × fresnel · 4 ch each · patch 17–32

Boom stage left

3 × par · 4 ch each · patch 33–44

Boom stage right

3 × par · 4 ch each · patch 45–56

Backlight

4 × par · 4 ch each · patch 57–72

Cyclorama

4 × cyc · 4 ch each · patch 73–88

Breakup

2 × profile · 4 ch each · patch 89–96

Moving heads

2 × moving · 8 ch each · patch 97–112

Followspot

1 × followspot · 2 ch each · patch 113–114

Footlights

1 × batten · 4 ch each · patch 115–118

House lights

1 × house · 1 ch each · patch 119–119

6 profile · 4 fresnel · 10 par · 4 cyc · 2 moving · 1 followspot · 1 batten · 1 house

The five cues

Two static, two dynamic, one continuous; two two-handed against three one-handed. The axes are close to independent, which separates most pairs before any classifier is asked to. This is a design-time substitute for discriminative training, and it is what lets a purely rule-based implementation work at all.

LX 1

Blackout

Clap twice

dynamic · 2H · FTR risk low

LX 2

General cover

Both palms open, raised

static · 2H · FTR risk low

LX 3

Centre special

Point, hold

static · 1H · FTR risk medium

LX 4

Master dim

One open hand, move up or down

continuous · 1H · FTR risk medium

LX 5

Colour state

Swipe sideways

dynamic · 1H · FTR risk high

What this does not contain

— The system has been built, deployed and operated by a single user.
— It has not been evaluated with participants, and no FTR figure is reported.
— The adversarial illumination argument is motivated but not demonstrated.
— The thresholds above were set by iterative observation by one operator. They are a starting configuration, not a tuned one.
— The vocabulary is five cues; a production cue stack is an order of magnitude larger, and whether the block-diagonal property survives that scaling is untested.

The instrumentation exists to serve a protocol that has not yet been run: a two-hour armed idle recording for the FTR figure, a participant study for per-cue F1 and cross-user generalisation, a rules-versus-learned comparison on identical recorded sessions, and physical fixtures.

Cite the artefact

@misc{bajpai2026lights,
  author       = {Bajpai, Amogh},
  title        = {Lights: Gesture-Based Theatrical Lighting Control},
  year         = {2026},
  howpublished = {\url{https://lights.amoghbajpai.com}},
  note         = {Source: https://github.com/amoghgg/lights}
}