By a Smart Home Architect
I spent last Saturday debugging a house in Connecticut.
The owners had just installed the new “AI Update” for their generic smart home hub. They were excited. They stood in the living room and told the house, “We’re going on vacation for a week. Lock it down.”
The AI, trying to be helpful and maximize energy efficiency, interpreted “Lock it down” as a total shutdown command.
It turned off the main breaker.
It turned off the security cameras.
It turned off the freezer.
When they came back a week later, their house was dark, their security footage was missing, and their kitchen smelled like rotting meat.
This is the danger of the “Agentic Smart Home.”
For ten years, we treated smart assistants like remote controls. You pressed a button (with your voice), and it did exactly one thing. “Turn on the Kitchen Lights.” It was dumb, but it was predictable.
Now, we are treating them like Butlers. We give them vague, high-level commands. “Set the vibe.” “Get the house ready for bed.” “I’m stressed.”
But a Butler needs a job description. If you don’t give your AI a System Prompt—a rigid set of rules that define its psychology and constraints—it will eventually hallucinate a “solution” that ruins your day.
Here is how to build a “House Constitution” using a local Raspberry Pi and Microsoft’s Phi-4 model to keep your smart home from turning against you.
1. The Hardware: Why Local is Non-Negotiable
First, a rule: Never let the cloud control your light switches.
If you shout “Turn on the lights!” and the request has to go to a server in Virginia, get processed by GPT-4o, and come back… that’s a 3-second delay. You will walk into the furniture before the lights turn on.
Furthermore, if your internet goes down, your house shouldn’t turn into a brick.
In 2026, we have a solution: Small Language Models (SLMs).
Specifically, Microsoft Phi-4.
This model is tiny (approx 3.8 billion parameters). It isn’t smart enough to write a novel, but it is a genius at following instructions and formatting JSON.
The Build:
- Computer: Raspberry Pi 5 (8GB RAM).
- Storage: NVMe SSD (SD cards are too slow for AI).
- Software: Home Assistant OS + Ollama.
- Model: phi4:q4_k_m (4-bit quantized to fit in RAM).
This setup costs about $100. It runs entirely on your local network. It responds in 400 milliseconds. And it keeps your data inside your walls.
2. The Core Philosophy: “State” vs. “Command”
The biggest mistake people make is scripting commands.
- Old Way: “If I say ‘Goodnight’, turn off lights 1, 2, and 3.”
This breaks easily. What if you have a guest? What if you are reading in bed? What if the baby is crying?
The new way is to define States.
You teach the AI what “Night Mode” means, and you let the AI figure out how to get there based on the active sensors.
The System Prompt:
“You are HomeOS. The House has four primary states: Occupied-Active, Occupied-Resting, Vacant-Short, and Vacant-Long.
Your job is to transition the house between these states while maximizing comfort and minimizing energy waste.
CRITICAL: You are never authorized to cut power to ‘Critical Infrastructure’ (Fridges, Freezers, Network Racks, Sump Pumps) under any circumstances.”
3. The “Context Injection”: Who Lives Here?
A generic AI is a bad roommate. It doesn’t know you. You need to feed it the “Bio-Data” of the inhabitants so it understands the intent behind the noise.
In Home Assistant, we create a text helper that is fed into the LLM’s context window every time it runs.
The Context Prompt:
Plaintext
<inhabitants>
1. **Dad (User A):** Wakes up at 6:00 AM. Hates bright blue light. Needs coffee machine pre-heated.
2. **Mom (User B):** Night owl. Works from home in the ‘Den’. Do not turn off Den lights if motion is detected after midnight.
3. **The Dog:** Triggers motion sensors in the Living Room at 3 AM. THIS IS NOT AN INTRUDER. Do not turn on the alarm.
</inhabitants>
<preferences>
– **Temperature:** We prefer 68°F (20°C) for sleeping.
– **Lighting:** Never turn lights to 100% brightness after 9 PM. Cap at 30%.
– **Vibe:** We dislike “Chatty” confirmations. If we give a command, just do it. Do not say “Okay, I have turned off the lights.” Just chime.
</preferences>
By defining the dog, you stop the 3 AM false alarms. By defining the “Chatty” preference, you fix the most annoying part of Alexa/Siri.
4. The “Vague Command” Interpreter
This is the killer feature of LLMs. You can map vague, emotional phrases to complex logic trees without writing code.
Phi-4 is excellent at Zero-Shot Classification. It can look at a list of available tools (lights, locks, thermostat) and map a sentence to the right tool.
The Dictionary Prompt:
“Interpret the following phrases as follows:”
- ‘I’m having a bad day’:
- Set Living Room lights to ‘Warm White’ at 20%.
- Play ‘Lo-Fi Beats’ playlist on the Sonos.
- Turn on the Kettle (comfort tea).
- ‘I’m Cooking’:
- Turn Kitchen lights to 100% Cool White (Task Lighting).
- Unmute the smart speaker (for music).
- If the smoke detector chirps, assume it is ‘Toast’, not ‘Fire’. Announce “Smoke Detected” before sounding the siren.
The “Smoke vs. Toast” logic alone is worth the upgrade.
5. The “Guardrails” (The Fridge Rule)
You must explicitly list the “Do Not Touch” items.
LLMs are energy-efficiency zealots. If you tell them “Save money,” they will look for the highest-draw appliance and kill it. That is usually your fridge or your server rack.
The Safety Prompt (The Iron Law):
Plaintext
<safety_protocols>
LEVEL 1 (High Priority):
– NEVER modify the state of devices tagged #critical: [Fridge, Freezer, Router, CPAP Machine].
– NEVER unlock the Front Door based on a voice command unless the voice matches a verified biometric print (FaceID check required).
LEVEL 2 (Comfort Priority):
– NEVER turn off a light if the room is ‘Occupied’ (defined by Motion Sensor + Mic noise > 40db).
– NEVER speak unsolicited announcements before 8:00 AM.
</safety_protocols>
The “Occupied” definition is crucial. Old motion sensors fail if you are sitting still reading a book. The LLM can combine “Motion = 0” with “Mic = Page Turning Sounds” to realize “Humans are still here, don’t kill the lights.”
6. Real-World Example: “Movie Night”
Here is how this plays out in real-time on the Raspberry Pi.
User: “Hey House, let’s do a movie.”
Old Automation: (Requires a hard-coded script called ‘Movie Night’)
- Checks if script exists. If yes, runs it. If no, says “I don’t understand.”
Phi-4 on Pi (The Inference):
- Analyze Request: “User wants to watch a movie.”
- Analyze Context: “Time is 8:30 PM. Living Room is Occupied. TV is currently Off.”
- Infer Needs:
- Movies require darkness -> Dim lights.
- Movies require silence -> Turn off HVAC fan if loud.
- Movies require the screen -> Turn on TV and Apple TV.
- Movies require focus -> Suppress phone notifications (via API).
- Execute: Calls Home Assistant services to perform these actions.
The beauty is that I never programmed a “Movie Night” scene. The AI deduced the requirements of a movie from its training data. If I say “Let’s play video games” instead, it might dim the lights but leave the volume louder. It adapts.
7. Conclusion: The House is a Member of the Family
The shift here is psychological.
You are no longer programming a computer with if/then statements. You are training a staff member.
When your house messes up (and it will), you don’t rewrite code. You Correct the Prompt.
“Hey House, last night you turned off the music when I went to the bathroom. Don’t do that. Assume ‘Party Mode’ lasts until I say ‘Stop’.”
Treat your house like a well-meaning but literal-minded intern. Give it clear rules, strict boundaries, and a little bit of personality.
And for the love of god, tag your fridge as #critical.