Engineered Lighting vision
Lighting is the entry point into ambient computing. Contextual lighting comes first; projected information and interaction follow as the underlying capabilities mature. The current Home app is a development and operating tool, not the consumer product. The goal is minimal routine app interaction: physical controls, gestures, voice, and eventually projected surfaces.
This page is the vision. Where the build stands today lives in the hardware notebook and the software pages. The films are Blender concept renders of the intended experience, not recordings of working fixtures.
Lighting is underestimated in terms of its impact on our lives. It enhances our mood, well-being, and productivity, significantly elevating the aesthetic appeal and perceived value of our spaces. Simply put, good lighting creates a good vibe—and a good vibe is often underappreciated. Thoughtful lighting makes us happier, and our spaces more beautiful. Although we intrinsically value it, no company has effectively offered this considered approach to lighting as a complete product.
The challenge is that good lighting is highly contextual. For example, a gallery benefits from carefully placed track lighting that highlights artwork. A cozy movie night calls for soft, warm lighting gently diffused off the walls. Conversely, a workspace demands cool, direct lighting to maintain productivity. Thus, home lighting must be dynamic and adaptable to various contexts and moods.
Yet achieving truly dynamic lighting today is essentially impossible. The ideal lighting experience requires multiple specialized fixtures—lamps for warmth, recessed lights for brightness, track lights for directional focus. Even if one manages to overcome the complexity and expense of installing these fixtures, another significant challenge arises: the hardware and software are typically developed independently, leading to fragmented solutions.
There are companies that produce both fixtures and associated software—usually basic smartphone apps for controlling RGB colors and brightness. (I hope this doesn't need to be said—but we aren't here to talk about RGB LEDs; we are talking about creating beautiful lighting experiences that look like a director of photography composed them for an Architectural Digest shoot.)
These limited apps typically restrict users to the company's own ecosystem, immediately limiting their appeal and usability. Consequently, users are forced to integrate their systems with broader smart home platforms like Google Home, HomeKit, or, for the particularly ambitious, Home Assistant. Upon navigating to these platforms, users discover that they must also install an array of sensors to enable basic automations. Subsequently, users need almost college-level logic skills to program brittle, hard-coded if-then statements, which inevitably break down when anything unscripted occurs. In short, this is precisely why achieving genuinely good lighting today remains impossible.
Most lighting companies sell individual bulbs or fixtures, leaving users to navigate installation and configuration themselves. However, if Jobs and Ive were creating a lighting product, they wouldn't simply sell a bulb—they would craft a complete, intentional, and holistic lighting experience.
My research is focused precisely on developing this kind of holistic lighting solution. Imagine if lighting just worked—intuitively adjusting itself based on your activities, without manual input or complex setups. This solution involves dynamic and adaptable hardware combined with intelligent software, creating a seamless experience in which lighting perfectly aligns with the human context of the space at any given moment.
- 01Can a single, versatile fixture (or set of fixtures) effectively replicate dozens of unique lighting scenes, replacing multiple specialized fixtures?
- 02By being vertically integrated—controlling both the hardware and the software—and leveraging cutting-edge technology like AI, can we automatically create highly personalized lighting scenes and seamlessly manage transitions between them?
What is good lighting?
Good lighting is contextual: it depends on how you are using your space. The right lighting for working is not the same as the right lighting for relaxing, which is not the same as the right lighting for eating, which is not the same as the right lighting for entertaining, and so on. When working, you might prefer cool, direct light. When relaxing, you likely prefer warm, diffuse light. Put simply—good lighting should adapt to your needs.
Yet today, achieving this kind of dynamic lighting is annoying. Homes rely on a mix of fixtures—lamps for warmth, recessed lights for brightness, track lights for directional focus—each serving a specific function. If you want to make your home “smart,” you need to install sensors and manually configure rigid automations using platforms like HomeKit or Home Assistant. This process is unintuitive and time-consuming. As a result, very few people actually go through the trouble—despite nearly everyone wanting better lighting.
But what if lighting simply worked? What if your home could intuitively adjust its lighting based on how you use the space, without requiring manual input or complex setup?
The ideal home lighting experience
Imagine installing a lighting system that self-calibrates upon setup, requiring no further configuration. You simply live your life, and the lighting adapts seamlessly:
- You sit down for dinner, and the lights gradually adjust to provide natural illumination on the table while casting a warm glow throughout the room.
- When it’s time for a movie night, you settle onto the couch, and the lighting shifts—soft, warm light bouncing off the walls at 10% luminosity, creating the perfect ambiance.
- You hang a new painting, and without any manual intervention, the home recognizes it as artwork and casts a precisely framed beam of natural light, as if it were displayed in a museum.
- When you start working, the lighting transitions to provide cool, direct light that enhances focus.
- Playing a board game? The scene adjusts again, ensuring optimal visibility. If someone gets up to grab a drink from the bar, a delicate glow follows them, illuminating the area just enough for them to pour a glass of whiskey.
The lights feel alive, responding fluidly to human presence and activity.
Beyond lighting: a new computing paradigm
Why stop at just lighting? Once you have a system that dynamically manipulates light, where do the boundaries end?
- Want to check the time? A digital clock subtly appears on the wall.
- Curious about the weather? The forecast is projected onto a nearby surface.
- Heading out for work? Real-time traffic updates appear on the door.
- Cooking? A recipe is projected onto the countertop, hands-free and always in view.
Consumer computing has long been defined by screens—the way information is delivered to our eyes. But what if, even for just 1% of the time, you didn’t need to reach for your phone? What if a simple query like “When was Lincoln born?” displayed the answer on the nearest surface?
This could be a seismic shift in consumer computing—not as a replacement for screens, but as a new category in the way we interact with information. It’s about creating a new layer of contextual computing, where information is delivered in a way that feels organic, integrated into the environment rather than confined to a device.
This is the true vision of ambient computing—the world teased to us by sci-fi, from Minority Report to Star Wars.
A world where light isn’t just illumination—it’s information.
Defining ambient interfaces
What do these ambient interfaces look like?
The key to success in ambient computing is not competing directly with traditional consumer display technology. The more you try to replicate the capabilities of high-contrast, high-resolution consumer screens, the more you set yourself up for failure. Projected interfaces are not designed to match the vibrancy and contrast of OLED or LED displays, particularly in spaces with ambient light. Instead of trying to replicate these screens, we should embrace what projected interfaces do best—providing low friction, contextual information without the need for constant device interaction.
What does it mean to compete with consumer display technology? If you attempt to project full-color interfaces in a well-lit space, contrast diminishes, making the experience look cheap. This is why the opportunity lies in defining a new paradigm for ambient computing—one that leverages the unique strengths of projection rather than forcing it into an existing mold.
Our vision centers on projected white-light interfaces: a design language that embraces simplicity, elegance, and functionality. These interfaces are flat, cohesive, and distinct from conventional consumer computing. Rather than mimicking high-resolution screens, they take inspiration from the clarity of well-designed physical signage, the subtlety of shadow and light, and the timeless appeal of minimalism. The result is a system that feels intentional, sophisticated, and seamlessly integrated into everyday life—an ambient computing experience that is both beautiful and uniquely its own.
Building an OS from scratch: creating a fully functional system at launch
Consumer expectations have evolved—today, launching a new device with only a handful of core functionalities is no longer enough. Users will expect a fully realized operating system that supports a wide range of capabilities, much like a smartphone— likely expecting every ‘app’ aside from those that enable content consumption (As every user will already have a TV). When I refer to "apps," I use the term broadly to encompass features and functionalities that shape the user experience.
So how do we create an operating system that feels complete from day one? The key is dynamically generating an infinite number of "apps," ensuring that any expected functionality is available immediately. To achieve this, we can leverage a transformer-based AI model—specifically, a multimodal model capable of:
Two forms of input
1. Passive – Observing and understanding human context
2. Active – Responding to inquiries and requests
Two forms of output
1. Updating the state of the home – Adjusting light scenes and other connected devices
2. ‘Ambient Computing’ – Translating information-seeking requests into front-end interfaces that always maintain a cohesive design language, delivering bespoke, elegant, and functional UI
That may sound abstract, so let’s break down what this means in practical terms and how to build it.
Leveraging Home Assistant as the state model
At the core of this system is Home Assistant, an open-source smart home platform that will serve as the state model for all sensors and devices in the home. Home Assistant provides real-time data on everything from motion sensors to lighting states, allowing our system to both interpret and modify the environment dynamically. By integrating Home Assistant, the system gains a reliable interface for controlling and responding to smart home inputs.
Building a system of prompt chaining
Beyond simply integrating with the home, what we’re really building is a dynamic chain of prompts—a structured workflow where different AI-generated responses are strung together to create rich, interactive experiences. I’ve been looking into a platform named Fabric, which provides an intuitive tool for linking multiple prompts (which they call "fabrics") into complex, yet easily configurable workflows.
Effectively, the system functions as a series of prompts and confidence checks, ensuring that interactions remain smooth, relevant, and personalized. By structuring the OS around dynamic prompts rather than static pre-built apps, we create an infinitely extensible framework where new functionalities can emerge in real time, tailored to the user's needs and the context of the home environment.
This is the foundation of our OS—a system that doesn’t require a traditional "app store" but instead builds interfaces on demand, translating any request into an elegant, projected UI that seamlessly integrates into the ambient computing experience.
High-level architecture overview
-
01
Sensors & Environment Layer
- Collects data from cameras, motion/occupancy sensors, temperature sensors, contact switches (doors/windows), and any other smart devices.
- Publishes events to an Event Bus in real time.
-
02
Home Assistant (State Model & Communication Intermediary)
- Maintains a unified view of device states and sensor data.
- Receives command updates (e.g., “set brightness to 20% in LivingRoom”) and executes them.
- Exposes an API or event-based interface (e.g., MQTT or WebSocket) to higher-level modules.
-
03
Core AI ServicesVision Model (Local Inference)
- Runs on edge hardware (e.g., NVIDIA Jetson) to detect activities or context from video feeds.
- Sends structured events (e.g., “User is sitting on the couch,” “User is cooking,” “No occupants detected”) to the Event Bus.
- Respects privacy by avoiding cloud streaming of raw video.
Request Type Model (Natural Language Understanding)- Classifies user utterances into environment-related requests (“Dim the lights”), information-seeking (“How do I make a martini?”), or mixed intent.
- Potentially includes an intent extractor and entity recognizer for more fine-grained control (e.g., “LivingRoom,” “Movie mode,” “Alice’s preference”).
Knowledge Graph / Contextual Store (Optional)- Maintains structured knowledge about the home (rooms, devices, user preferences) and potentially external knowledge (recipes, time-of-day usage patterns, etc.).
- Provides quick lookups to minimize repeated LLM calls.
Intuition Model (LLM or Rules + LLM)- Suggests environment state changes based on the user’s activity or request.
- Could be partly rule-based (e.g., “If user is watching a movie, set brightness to 20%”) and partly LLM-driven for more nuanced recommendations.
- Considers context from the Knowledge Graph and Preferences/Profiles.
-
04
Preferences & Profile Manager
- Stores user- or household-level overrides (e.g., user likes brighter lights than normal, or user is extremely sensitive to overhead lighting).
- Exposes these preferences to the Intuition Model and the Validation module.
- Could be updated automatically when users frequently override recommended settings.
-
05
Validation & Contextual Overrides
- Replaces a simple confidence check with a multi-factor analysis.
- Input includes: proposed change from Intuition Model; current Home Assistant states (e.g., lights are already at 25% brightness); preferences & profiles (e.g., user consistently overrides bright settings after 10pm); time-of-day / location-based triggers (e.g., after sunset, or if occupant is in bed).
- Produces a final recommended action (or no action), plus a confidence score or reason codes for auditing.
- Allows for special overrides (party mode, guests present, child-friendly settings, etc.).
-
06
Orchestrator & Event Bus
- Coordinates the flow of data and commands among the Vision Model, NLU, Intuition Model, Preferences Manager, and Home Assistant.
- Manages concurrency (e.g., prevents conflicting lighting commands if two different triggers fire simultaneously).
- Could queue or batch updates if the system is under heavy load.
-
07
Feedback Loop & Learning
- Logs all final decisions and user responses (e.g., user accepted, user manually overrode the system).
- Periodically retrains or fine-tunes AI models with anonymized usage data, or updates preference rules automatically.
- Helps the system evolve to better match real user behavior.
-
08
Information Retrieval & Summarization
- When the Request Type Model identifies an information-seeking query (“How do I make a martini?”), the system calls an LLM or knowledge source.
- Summarizes the response in a concise format.
- Passes the final output to an Ambient UI Renderer for user consumption.
-
09
Ambient UI Renderer
- Generates cohesive projected interfaces or minimal UI elements.
- Ensures a consistent white-light, flat-design aesthetic across all projected data (light states, recipe instructions, short messages).
Example flows
APassive environment adjustment
- 1Vision Model detects “User is lying on the couch.”
- 2Event Bus publishes “PassiveTrigger.Activity:LyingOnCouch.”
- 3Orchestrator calls Intuition Model with context: “User lying on couch in LivingRoom. No other occupants.”
- 4Intuition Model outputs recommended YAML:
LivingRoom: brightness: 20% color: warm white
- 5Validation & Contextual Overrides checks user’s “Alice” preference for relaxing, current time (10pm), sees no conflict, sets confidence to 90%.
- 6If above threshold, Orchestrator sends command to Home Assistant → adjusts lights.
- 7Log outcome in Feedback Loop.
BActive user-initiated change
- 1User says, “I want movie mode in the living room.”
- 2Request Type Model: “Environment-related.”
- 3Intuition Model: “Movie mode → brightness: 15%, color: warm white.”
- 4Validation & Contextual Overrides: Looks at time (9pm), user’s preference (Bob likes 10% brightness), final sets brightness to 10%, confidence 95%.
- 5Orchestrator → Home Assistant updates lighting.
- 6Feedback Loop logs acceptance.
CMixed intent
- 1User says, “I’m having dinner with guests; dim the living room and show me a recipe for carbonara.”
- 2Request Type Model: “Mixed.” Splits into environment + information requests.
- 3Intuition Model handles environment: recommends 30% brightness, warm color.
- 4Validation & Contextual Overrides adjusts if preferences or context.
- 5Information Retrieval & Summarization retrieves a quick carbonara recipe.
- 6Ambient UI Renderer: Projects or displays the recipe + sets lights.
- 7Feedback Loop logs results.
DInformation-seeking only
- 1User asks: “What’s the weather forecast?”
- 2Request Type Model: “Information-seeking.”
- 3Information Retrieval & Summarization: “Sunny, 75°F, no rain.”
- 4Ambient UI Renderer: Projects small weather widget in the living room.
- 5Feedback Loop logs usage.
The hardware that enables this vision
To project interfaces, we need the ability to control and manipulate millions of tiny points of light with precision. At first glance, this might seem like an insurmountable challenge—but humanity has already solved this problem. The technology is called Digital Light Processing (DLP), and it has been widely used in everything from home theater projectors to the digital projectors found in movie theaters.
At the core of DLP technology is the Digital Micromirror Device (DMD) chip, a marvel of modern engineering fabricated using similar techniques to microprocessors. DMD chips contain millions of microscopic mirrors, each capable of tilting on or off to reflect light, effectively creating a high-resolution projection. This allows projections to function much like OLED displays, where individual pixels can be turned completely on or off to create crisp, high-contrast images.
The challenge with traditional projectors, however, is that they are optimized for full-color RGB output, designed for displaying images and video rather than producing high-quality, natural lighting for a home environment. This is where our innovation comes in: while the DMD chip controls the projection itself, the light engine that supplies the illumination is a separate component. By replacing the standard RGB light engine with a tunable white light source, we can create a projection system designed specifically for ambient computing.
Tunable white light refers to the ability to adjust both the warmth and coolness of white light, enabling a spectrum of color temperatures that can shift to match different environments and needs. By integrating this capability into our projection system, we can achieve a much brighter, cleaner, and more visually comfortable experience—one that is purpose-built for ambient computing rather than retrofitted from existing display technology. This approach allows us to leverage the best qualities of DLP projection while designing for an entirely new use case: a seamless, elegant, and dynamic ambient computing experience.
The next version of the hardware prototype
When exploring how to achieve 360-degree projection, I initially went deep into the rabbit hole—researching and even attempting to design a GRIN (Gradient-Index) lens. GRIN lenses manipulate light through a gradual change in the refractive index, enabling complex optical behavior without traditional curved surfaces. My goal was to create a distortion-free projection system capable of seamlessly covering an entire space.
After discussing this idea with an optomechanical expert friend of mine — someone who prototypes hardware for major tech companies—he grounded me in reality. He explained that manufacturing a custom GRIN lens would cost hundreds of thousands of dollars, and even then, the results would likely be flawed. Instead, he gave me invaluable advice:
"Come up with the dumbest, most dead-simple idea to achieve the same result."
An optomechanical expert friend
That’s when it hit me: instead of a single complex lens, why not just mount multiple projectors radially in a circle, each covering a different direction… Simple…Effective….
For the next iteration of the prototype, I will use eight Texas Instruments DLP2000EVM projectors, arranged in a radial configuration to provide full 360-degree coverage. These projectors will serve two purposes:
- 01Projecting interfaces – Delivering ambient computing visuals throughout the space.
- 02Acting as dynamic spotlights – Traditional gallery and museum lighting relies on barn doors to shape light precisely within the frames of a painting. With the DMD chip inside a DLP projector, we can achieve the same precision dynamically—whether it’s illuminating artwork, a cutting board, or any other focal point.
While this version of the prototype will still rely on standard projector light engines (meaning light quality will remain suboptimal), future iterations will replace these with tunable white light sources optimized for home environments.
Adding tunable white LED strips
Between each projector, tunable white LED strips will be mounted radially, allowing fine control over light temperature and intensity. Each strip can be dimmed or disabled independently to create specific lighting effects. For example, during a movie night, the system could turn off inward-facing LEDs while keeping outward-facing ones active, allowing soft, warm light to reflect off the walls without disrupting the immersive experience.
System architecture and communication
The projectors will communicate with an onboard computer via Ethernet using a small onboard gigabit switch. Given the number of projectors, leveraging IP-based communication will be the most scalable and reliable approach. The current computing platform I’m designing around is the NVIDIA Jetson, which offers the necessary processing power while supporting future AI-driven enhancements.
Powering the system
Powering this setup is no small feat. The prototype will include a 24V 340W AC/DC power adapter—a staggering amount of power, roughly equivalent to a small gaming PC. According to my calculations, this is the minimum required to run all projectors simultaneously. The system will also integrate the necessary power components to step down 24V to the various voltage levels needed by different hardware components.
Future additions
A critical next step will be incorporating an onboard camera, though initially, the camera will be an external module. In future iterations, it will be fully integrated to enable more advanced interaction and automation features.
This prototype represents the first major step toward a truly dynamic and intuitive ambient computing system—leveraging projected interfaces, intelligent lighting, and adaptable interaction in a way that has never been done before.