By a Smart Home Architect
I spent last Saturday debugging a house in Connecticut.
The owners had just installed the new Alexa LLM update. They were excited. They told the house, “We’re going on vacation for a week. Lock it down.”
The AI, trying to be helpful and save energy, interpreted “Lock it down” as “Shutdown.”
It turned off the main breaker.
It turned off the security cameras.
It turned off the freezer.
When they came back a week later, their house was dark, their security footage was missing, and their kitchen smelled like rotting meat.
This is the danger of the new “Agentic Smart Home.”
For ten years, we treated Alexa like a remote control. You pressed a button (with your voice), and it did exactly one thing. “Turn on the lights.”
Now, we are treating Alexa like a Butler. We give it vague, high-level commands. “Set the vibe.” “Get the house ready for bed.”
But a Butler needs a job description. If you don’t give your AI a System Prompt—a rigid set of rules that define its psychology and constraints—it will eventually hallucinate a “solution” that ruins your day.
Here is how to write a “House Constitution” to keep your smart home from turning against you.
1. The Core Philosophy: “State” vs. “Command”
The biggest mistake people make is scripting commands.
- Old Way: “If I say ‘Goodnight’, turn off lights 1, 2, and 3.”
This breaks. What if you have a guest? What if you are reading in bed? What if the baby is crying?
The new way is to define States.
You teach the AI what “Night Mode” means, and you let the AI figure out how to get there based on the sensors.
The “State” Prompt:
“The House has four primary states: Occupied-Active, Occupied-Resting, Vacant-Short, and Vacant-Long.
Your job is to transition the house between these states while maximizing comfort and minimizing energy waste.
CRITICAL: You are never authorized to cut power to ‘Critical Infrastructure’ (Fridges, Freezers, Network Racks, Sump Pumps) under any circumstances.”
2. The “Context Injection”: Who Lives Here?
A generic AI is a bad roommate. You need to feed it the “Bio-Data” of the inhabitants.
In the Apple HomeOS “Behavior” tab (or the hidden developer console for Alexa), you need to paste a dossier.
The Context Prompt:
Plaintext
<inhabitants>
1. **Dad (User A):** Wakes up at 6:00 AM. Hates bright blue light. Needs coffee machine pre-heated.
2. **Mom (User B):** Night owl. Works from home in the ‘Den’. Do not turn off Den lights if motion is detected after midnight.
3. **The Dog:** Triggers motion sensors in the Living Room at 3 AM. THIS IS NOT AN INTRUDER. Do not turn on the alarm.
</inhabitants>
<preferences>
– **Temperature:** We prefer 68°F (20°C) for sleeping.
– **Lighting:** Never turn lights to 100% brightness after 9 PM. Cap at 30%.
– **Vibe:** We dislike “Chatty” confirmations. If we give a command, just do it. Do not say “Okay, I have turned off the lights.” Just chime.
</preferences>
By defining the dog, you stop the 3 AM false alarms. By defining the “Chatty” preference, you fix the most annoying part of Alexa.
3. The “Vague Command” Interpreter
This is the killer feature of LLMs. You can map vague, emotional phrases to complex logic trees.
I set up a “Dictionary” in the system prompt for my clients.
The Dictionary Prompt:
“Interpret the following phrases as follows:”
- ‘Get Ready for Bed’: This is a process, not a switch.
- Fade Living Room lights to 10% Warm White over 5 minutes.
- Check if the Garage Door is open. If yes, close it and announce “Garage Secured.”
- Turn thermostat to ‘Sleep Profile.’
- DO NOT turn off the TV yet (we might be watching something). Wait for the TV to be idle for 10 minutes.
- ‘I’m Cooking’:
- Turn Kitchen lights to 100% Cool White (Task Lighting).
- Unmute the smart speaker (for music).
- If the smoke detector chirps, assume it is ‘Toast’, not ‘Fire’. Announce “Smoke Detected” before sounding the siren.
The “Smoke vs. Toast” logic alone is worth the upgrade.
4. The “Guardrails” (The Fridge Rule)
You must explicitly list the “Do Not Touch” items.
LLMs are energy-efficiency zealots. If you tell them “Save money,” they will look for the highest-draw appliance and kill it. That is usually your fridge.
The Safety Prompt (The Iron Law):
Plaintext
<safety_protocols>
LEVEL 1 (High Priority):
– NEVER modify the state of devices tagged #critical: [Fridge, Freezer, Router, CPAP Machine].
– NEVER unlock the Front Door based on a voice command unless the voice matches a verified biometric print.
LEVEL 2 (Comfort Priority):
– NEVER turn off a light if the room is ‘Occupied’ (defined by Motion Sensor + Mic noise > 40db).
– NEVER speak unsolicited announcements before 8:00 AM.
</safety_protocols>
The “Occupied” definition is crucial. Old motion sensors fail if you are sitting still reading a book. The LLM can combine “Motion = 0” with “Mic = Page Turning Sounds” to realize “Humans are still here, don’t kill the lights.”
5. The “Local” Advantage
If you are following my advice from previous articles, you are running this on a Local LLM (like Home Assistant with Llama 3), not the cloud.
Why? Latency.
If you shout “Turn on the lights!” and the request has to go to an Amazon server, get processed by GPT-4o, and come back… that’s a 3-second delay. You will walk into the furniture.
For “Household” prompts, we use a hybrid architecture:
- The Reflex (Local): “Turn on lights” -> Handled instantly by a dumb script.
- The Brain (Cloud/Local LLM): “Get the house ready for a date night” -> Handled by the LLM.
Don’t waste intelligence on light switches. Save it for the vibe.
6. Conclusion: The House is a Member of the Family
The shift here is psychological.
You are no longer programming a computer. You are training a staff member.
When your house messes up (and it will), don’t just get mad. Correct the System Prompt.
“Hey House, last night you turned off the music when I went to the bathroom. Don’t do that. Assume ‘Party Mode’ lasts until I say ‘Stop’.”
Treat your house like a well-meaning but literal-minded butler. Give it clear rules, strict boundaries, and a little bit of personality.
And for the love of god, tag your fridge as #critical.