I Gave My Cameras a Brain

Security cameras generate footage. Footage is homework. I wanted answers, so I taught the house to ask its own cameras questions.

the pattern

Five small automations run in my house, and each one follows the same four steps. An event happens. The automation waits a beat. It grabs one camera frame and hands it to Gemini with a specific question. The answer lands on my phone as one line of text.

No streams, no scrubbing through clips, no “motion detected” spam. A question goes out, an answer comes back.

the five questions

Is the trash at the curb? The night before pickup, at 8pm, an automation snapshots the driveway camera and asks: “Is there a trash bin at the curb in this image? Answer YES or NO.” A NO becomes a notification with a shame emoji. This one automation has saved me two missed pickups, which in my house pays for the whole hobby.

Did the delivery person leave a package? My doorbell fires a person event. The automation waits fifteen seconds for the person to leave, snapshots the porch camera, and asks whether a package is visible on the mat. Only a YES notifies. The doorbell’s own “person detected” alert used to fire for joggers; this fires for boxes.

Did the garage door close, visually? Home Assistant knows my garage cover reports “closed.” Sensors drift, though, and a cover sensor reporting closed while the door sits open is the exact failure you care about. So after the cover reports closed at night, a camera looks at the door and Gemini confirms: does this garage door appear fully closed? Disagreement notifies me. The sensor provides state; the camera provides proof.

Anything left in the driveway tonight? One frame at 10pm: “List any objects that don’t belong overnight in this driveway: bikes, toys, tools, packages, open vehicle doors.” Kids leave bikes out. Now the house tattles.

Who’s at the door? A doorbell ring grabs a frame and asks for a one-line description: “delivery driver holding a box,” “two kids selling something,” “neighbor with a dog.” That line is the notification. I stopped opening the live view.

cooldowns, or how to keep this from becoming spam

An AI answer per event sounds great until a spider builds a web on the porch camera. Each automation carries a cooldown timer, fifteen minutes to an hour depending on the question. A timer entity per automation, checked as a condition, restarted after each send. Boring and load-bearing.

Two variants worth copying: vacation mode swaps the driveway question to a stricter “list ANY people or vehicles” version, and late-night doorbell rings skip the phone and go to both phones plus the bedroom speaker.

the config shape

One example, sanitized, using the trash-night check. The others differ in trigger, camera, and question. Camera names are generic; use your own. My frames come from an NVR, but any camera entity in HA works the same.

- id: trash_at_curb_check
  alias: "Camera Q&A — trash at curb (night before pickup)"
  trigger:
    - platform: time
      at: "20:00:00"
  condition:
    - condition: template
      value_template: "{{ now().isoweekday() == 3 }}"  # night before pickup
  action:
    - service: camera.snapshot
      target:
        entity_id: camera.driveway
      data:
        filename: /config/www/snapshots/driveway_check.jpg
    - service: ai_task.generate_data
      data:
        task_name: trash_check
        instructions: >
          Look at this image of a residential driveway and curb.
          Is there a wheeled trash bin at or near the curb?
          Answer with exactly YES or NO.
        attachments:
          media_content_id: media-source://media_source/local/snapshots/driveway_check.jpg
          media_content_type: image/jpeg
      response_variable: verdict
    - condition: template
      value_template: "{{ 'NO' in verdict.data | upper }}"
    - service: notify.my_phone
      data:
        title: "🗑️ Trash is NOT out"
        message: "Pickup is tomorrow morning. The curb is empty."
  mode: single

The question design carries this whole system. “Describe this image” produces essays. “Answer with exactly YES or NO” produces automations. Ask for the shape you can act on, and give the model an out (“if the image is too dark to tell, answer UNKNOWN”) so bad frames fail safe instead of failing confident.

costs and privacy

Each check is one frame and a short prompt, so cost per question is fractions of a cent, and the automations fire a handful of times a day. On privacy: frames go to a cloud model, which I accept for exterior cameras only. Interior cameras stay out of this system. Decide where your line is before you wire it up, because moving the line later means auditing prompts you wrote months ago.

a note on screenshots

The notification examples in this post are re-created with test payloads. Publishing real exterior camera frames means publishing your house number, your cars, and your street to whoever’s reading. Sanitize yours the same way if you write about this.

One frame plus one specific question beats an hour of live streaming. My cameras answer more questions in a day now than I asked them in the previous year.