Engineering

The perception and decision loop.

How the system is meant to see a room, decide on light for each person, check the result and learn, and how much of that runs today. Every part on this page carries one of three marks, and nothing here is promised beyond its mark.

Status as of 2 November 2026

Running today

What exists · source on GitHub

Works today

The den's lights are four Hue bulbs run by den_lighting, a Home Assistant integration. They switch on when someone is in the room and off 120 seconds after it empties, dim to a warm, low scene while a Steam game runs, and hold any bulb someone changes by hand.

A camera and its “is anyone here?” check also run in the den. Everything else on this page is partly built or not built yet.

No engineered spotlight is installed. The moving spotlight and its fixture are still bench and board work: see Hardware.

Development of the den, the Qwen vision model and the backend has been paused since 25 October 2026.

Screenshot to comeHome app chat window
Home app · chat, tool calls, device actions
Screenshot to comeHome tool den view: lit scan and virtual fixtures
Home tool · den model and virtual fixtures
Screenshot to comeHome Assistant card, without camera frames
Lovelace card · Home Assistant dashboard

How it works

Simple view · eight steps in a loop

The loop watches the room, works out what light each person needs, gives each of them that light, checks the result and learns. Then the camera sees the room in its new light, and it goes round again. It is being built in a den in Victoria.

  1. 01

    See the room

    A camera watches the room. The rest of the system stays asleep until someone walks in.

    Works todayThe camera and its “is anyone here?” check run in the den.

  2. 02

    Spot people and things

    It finds each person’s body and hands, and things like a knife or a book.

    Partly builtThe software exists and has been tried out in the den, but is switched off for now.

  3. 03

    Follow each person

    It marks where each person is on a map of the room and keeps track of each one separately. No names needed: people are A and B.

    Partly builtIt can roughly place one person, in the same switched-off software. Keeping people apart is still to come.

  4. 04

    Understand what each needs

    It works out what light each person needs: bright and focused for chopping, soft and steady for reading.

    Partly builtAn AI describes the whole room every few seconds. Working out each person’s needs is still to come.

  5. 05

    Plan the light

    It turns those needs into one plan that also looks good, worked out around any light set by hand, and tries it on a 3D copy of the room before changing a real light.

    Partly builtSimple fixed rules, tried only on pretend lights on a screen.

  6. 06

    Change the lights

    It tells the lights what to do, changing them gently. A light someone changed is never changed back by an old plan, and the others carry on.

    Partly builtFour ordinary smart bulbs in the den switch on and off with the room, dim warm while a game runs, and hold a bulb someone changed. The planned spotlights exist only on screen.

  7. 07

    Check the result

    It looks again: did the light land on the surfaces where it should? If not, it adjusts. Glare in someone’s eyes is predicted, not measured.

    Not built yetNothing checks the result yet.

  8. 08

    Learn from you

    When someone turns a light up or down, it keeps planning around that setting and learns what they like for next time.

    Partly builtThe bulbs notice a change by hand and hold that bulb until two minutes after the room empties or it was last touched, whichever is later, or during a game until the game ends. Learning from it is still to come.

Back to step 1: the camera sees the room in its new light.

The Simple view as drawn: the eight steps in a loop, each with a camera picture of what it handles, faces blurred, and its status today. The pictures for steps 6 and 7 are tagged AI-edited.
Simple view · eight steps, as drawn · photos from the owner's apartment, not the den · faces blurred, AI-edited pictures tagged

The build ladder

Sheet A14 · the order to build

Each rung builds one part into today's loop, behind a gate written before the rung starts. The numbers in the gates are targets set before the work, not measured results.

  1. RRecording capabilityNot built yetKeep single-person den sessions with checked capture timing, cutting any stretch with a second person in view. Waits on the owner's decision about what may be kept.
  2. 0Record and replayPartly builtKeep what today's loop writes, and replay every logged decision to the same scene.
  3. N1A single-person referenceNot built yetA labelled reference for one person, labelled blind to a protocol fixed before labelling starts.
  4. N2Need on one person, offlineNot built yetAn offline verdict, feasible, not feasible or insufficient, on whether one person's need can be read. Changes nothing in the loop.
  5. 1Camera registrationPartly builtTie the camera to the room scan: three control pairs and two checks, all within 5 cm.
  6. 2Observations in the room framePartly builtEvery observation carries its time, room position, uncertainty and freshness.
  7. 3Tracks, from geometryNot built yetFollow each person with no silent identity swaps.
  8. 4A two-person gold setNot built yetLabelled two-person evenings to test the tracks against.
  9. 5Person bundlesNot built yetEach person's evidence gathered in one place, and marked unsafe whenever two people cross.
  10. 6Person needNot built yetEach person's working area and need: area within 15 cm median, onset within 3 s median.
  11. 7Measure lightNot built yetMeasure the room's light with a calibrated camera, checked against a light meter.
  12. 8Light modelPartly builtPredict the light a scene makes: median error 8%, 90th percentile 19%.
  13. 9Coordinator v0Partly builtOne plan for the room, replayed against real evenings, each decision in under 100 ms.
  14. 10VerifyNot built yetCheck the light landed: within ±15%, on at least 85% of attempts.
  15. 11Hand the bulbs to the coordinatorPartly builtOnly if it needs fewer corrections by hand per occupied hour than today's den_lighting.
  16. 12Correction attributionPartly builtWhen someone corrects a light, work out which part of the loop got it wrong.
The build ladder as drawn on sheet A14: the loop's thirteen parts in three rows, perception, light, and the record underneath, each with its state today and the rung that builds it.
Sheet A14 · the loop, and the rung that builds each part · as drawn
The architecture's title block and key: revision B, a draft of 27 October 2026 covering sheets A1 to A14, three notes on what the set is and how to read it, and the key to the people, line styles and status marks used on every sheet.
How to read the sheets · title block and key · as drawn

What's next

Written · not yet run

Every part

Detailed view · sheets A1–A14

The architecture is drawn on fourteen sheets. Sheet A1, the whole loop on one sheet, is below. The others are not published yet, apart from the build-ladder drawing from A14 above.

  1. A1One closed loop between people and light
  2. A2What runs each part, on which machine, how fast
  3. A3Seventeen records carry everything between stages
  4. A4Forty seconds in the den, through every component and into every record
  5. A5From pixels to one track per person, with the doubt kept
  6. A6Today one Qwen call describes the whole room; the proposal reads each person
  7. A7Each person is a handful of need variables
  8. A8Qwen teaches a small per-person student
  9. A9Measure the light, predict it, choose by objective, then check what landed
  10. A10Beauty is authored once, in perceptual terms, and solved for in every room
  11. A11A correction says the light was wrong; the logged decision says which link broke
  12. A12The owner sets the room's numbers by hand today
  13. A13Score the light people live with, then find the link that failed
  14. A14Eight problems, what each costs, and the order to build
Sheet A1 as drawn: the whole loop on one sheet. Perception runs along the top from the room's cameras to a separate person state for A and for B, the light side runs back along the bottom through the coordinator, actuation and verify, and slow loops for calibration, teaching, learning and evaluation sit underneath. Each box has a picture of what it handles; faces are blurred and three pictures are tagged AI-edited.
Sheet A1 · the whole loop on one sheet, as drawn · photos from the owner's apartment; the den appears only as a scan · faces blurred, AI-edited pictures tagged

Earlier plans in the notebook

Written for the spotlight · plans, not running code