LOCI Companion is a personal AI assistant with a memory for what you have seen. It is a small camera worn on the chest that remembers what it saw, where and when, so you can ask about it later: where did I leave my keys, what was on that whiteboard, what was the name of the wine at dinner. I have a working demo on my desk. This series is me building the rest of it in public.
This is Part 1 of Building LOCI Companion. It covers the idea, the hardware I am using today and where the project stands. It also opens up the architecture and LOCI-DB, the memory underneath, with diagrams and code you can run. Later parts go deeper into the firmware, battery life and the first people to wear it.
The idea: an assistant that was there
Today’s AI assistants know a lot about the world and almost nothing about your day. They can explain quantum field theory but cannot tell you where you put your passport, which slide your colleague showed on Tuesday, or whether the shelf in the garage was already empty last week. Everything they know about your life is what you remember to type in.
LOCI Companion is my attempt to close that gap. It does three things:
- Sees. A small camera at chest height captures what is in front of you, when you want it to.
- Remembers. What it saw is stored with where and when it saw it, so the assistant builds up a memory of your physical world over time instead of starting from zero every conversation.
- Answers. You ask in plain language, by voice, and it answers from that memory. When something has changed in a place you care about, it can tell you.
The everyday version is mundane on purpose: finding things, recalling things, noticing things. Where I parked. What the label on the medicine box said. Who was at the table. The same memory is also powerful for people who need it most: someone who is blind or has low vision, or someone living with memory loss, gets far more from an assistant that remembers than from one that only describes the current moment. I want both to work, and I will test with both.
Why the memory matters more than the camera
The camera is the easy part. The hard part, and the part I care about most, is the memory underneath it. That is LOCI-DB, an open-source spatial memory I have been building for robots, drones and world models. It stores each observation with its position in space and its moment in time, indexes them together on a four-dimensional Hilbert curve, and returns a novelty score that says how different the current view is from anything seen before at that place.
Those two properties are what make it an assistant rather than a camera. Indexing by place and time means a question like “where did I last see my keys” is a lookup, not a search through hours of footage. Novelty means the assistant can stay quiet most of the time and speak up only when something actually changed. An assistant that talks less and says more is the whole product.
A robot and a person wearing a pendant have the same underlying problem: they move through a world, see it from one point of view, and need to remember what they saw. LOCI Companion is the first place I am putting that memory on a human.
How the pieces fit
The pendant is a thin client. It does the things that have to happen on your body: look, listen, speak, and keep working when the network does not. Everything heavy runs on a host I own, which today is a laptop and could as easily be a Raspberry Pi in a cupboard. The two talk HTTPS over one keep-alive connection, either on the home network or through a Cloudflare Tunnel when I am out.
A frame goes up roughly twice a second. The host runs object detection on it, and every detection becomes a memory in LOCI-DB: a label, a position, a time and a confidence. When I hold the button and ask a question, the audio goes up the same pipe, is transcribed locally, answered from the memory, turned into speech and played from the board’s speaker.
The part I am proudest of is the offline path. If the host disappears, the pendant keeps capturing to its microSD card and uploads the backlog later with the original timestamps, so the memory has no holes. It also caches a spoken answer per remembered object, and an on-device speech recogniser (Espressif’s MultiNet7, no wake word, no cloud) can match “where is my cup” and play the cached answer with no host at all.
Inside LOCI-DB
Most AI memory today is a vector database: you turn things into embeddings and find the ones most similar to a question. That covers what. A wearable also needs where and when, and it needs all three in a single query. “Where did I leave my cup” is a similarity question with a place answer. “What was on the table this morning” is a place question with a time window. Bolting coordinates onto a vector store as ordinary filters works for a demo and falls over as the memory grows. LOCI-DB makes space and time part of the index.
A memory is what, where and when
The unit of memory is a WorldState: an embedding vector for what was seen, an x, y, z position, a timestamp in milliseconds, a scene_id that groups observations of the same thing, a confidence and free-form metadata. States in the same scene are linked to the one before them, so the database can replay how something moved over time.
The Hilbert curve: one number for space and time
The core trick is a space-filling curve. A Hilbert curve visits every cell of a grid in a single continuous line, and it has a useful property: cells that are close together in space are mostly close together along the line. LOCI-DB encodes each memory’s (x, y, z, t) onto a four-dimensional Hilbert curve, so every memory gets an integer address. It does this at three resolutions (16, 256 and 4,096 bins per axis) so both coarse and fine questions stay cheap.
A question about a place then becomes a short list of curve addresses to match, an integer filter the storage engine is very good at, instead of a geometric scan over every memory. The bucket list only narrows the search. Exact coordinate, time and confidence ranges are applied in the same filter before the nearest neighbours are picked, and an exact geometric check at the end means nothing just inside the edge of the box is missed.
Memory that ages like memory
A pendant that remembers everything forever is a storage problem and a privacy problem. LOCI-DB keeps two collections: raw memories, and summaries. Time is cut into logical epochs, and as epochs get old their raw states are folded into per-scene summaries and deleted. Recent memory stays sharp, old memory becomes gist, and storage stays bounded however long the device runs. Results are also re-ranked by recency with an exponential decay that halves every hour by default, so the cup you put down five minutes ago beats the one from last week.
Novelty: knowing when to speak
The call I care most about for an assistant is predict_and_retrieve. It takes what you are looking at now, optionally runs it through a world model to predict what comes next, and searches memory for the closest match. Alongside the results it returns a novelty score between 0 and 1: one minus the best similarity, optionally calibrated against the history of scores. Near 0 means “I have seen this here before”. Near 1 means “this is new”. That single number is what lets the assistant stay quiet when nothing has changed and speak up when something has.
Show me the code
LOCI-DB is a Python library, Apache 2.0, and runs in-process with no server, Docker or GPU. This is the pendant’s memory in miniature, and it runs as written:
# pip install loci-stdb
import time, zlib
import numpy as np
from loci import LocalLociClient, WorldState
memory = LocalLociClient(vector_size=128)
def embed(label: str) -> list[float]:
# Stand-in for a real image or label embedding (CLIP, DINOv2, ...)
rng = np.random.default_rng(zlib.crc32(label.encode()))
v = rng.normal(size=128)
return (v / np.linalg.norm(v)).tolist()
# Every detection from the pendant's camera becomes one memory:
# WHAT it saw (a vector), WHERE (x, y, z in [0, 1]) and WHEN (ms).
now = int(time.time() * 1000)
sightings = [ # label, x, y, z, seconds ago
("cup", 0.48, 0.52, 0.40, 3 * 3600),
("keys", 0.12, 0.80, 0.55, 1800),
("scissors", 0.55, 0.50, 0.40, 600),
]
for label, x, y, z, ago in sightings:
memory.insert(WorldState(
x=x, y=y, z=z,
timestamp_ms=now - ago * 1000,
vector=embed(label),
scene_id=f"{label}@kitchen",
confidence=0.9,
metadata={"label": label, "room": "kitchen"},
))
# "Where did I leave my cup?" Similarity in, place and time out.
hit = memory.query(vector=embed("cup"), limit=1)[0]
mins = (now - hit.timestamp_ms) // 60_000
where = f"({hit.x:.2f}, {hit.y:.2f}, {hit.z:.2f})"
print(f"{hit.metadata['label']} at {where}, {mins} min ago")
# cup at (0.48, 0.52, 0.40), 180 min ago
# "What was on the table in the last hour?"
# A box in space plus a window in time.
table = {"x_min": 0.4, "x_max": 0.6,
"y_min": 0.4, "y_max": 0.6,
"z_min": 0.3, "z_max": 0.5}
recent = memory.query(vector=embed("scissors"),
spatial_bounds=table,
time_window_ms=(now - 3_600_000, now),
limit=5)
print([r.metadata["label"] for r in recent])
# ['scissors'] (the cup is on the table too, but three hours ago)
# "Is this new?" 0.0 = seen it here before, near 1.0 = new.
for label in ["cup", "umbrella"]:
r = memory.predict_and_retrieve(
context_vector=embed(label),
predictor_fn=lambda v: v, # plug a world model in here
current_position=(0.48, 0.52, 0.40),
)
print(label, round(r.prediction_novelty, 2))
# cup 0.0
# umbrella 0.96
Swap LocalLociClient for LociClient("http://localhost:6333", ...) and the same code runs on a persistent Qdrant backend. In the real system the vector comes from the detector’s label rather than a stand-in, and positions come from where the object sits in the frame or, on the depth rig, from real 3D coordinates.
On the board, the firmware is ESP-IDF C. Stripped of logging, its main loop is short:
// The pendant's main loop, simplified from firmware/esp32-p4-client/main/app_main.c
for (;;) {
if (button_down()) { // push to talk
record_and_ask(); // host answers; if it is unreachable,
continue; // MultiNet7 on the board answers from the card
}
bool online = host_reachable();
if (online && local_backlog_count() > 0)
upload_one_backlog_frame(); // frames captured offline, original timestamps
if (time_for_next_frame()) {
if (online) live_frame(); // JPEG to the host, labels (and any audio) back
else capture_to_card(); // no host: keep remembering to microSD
}
vTaskDelay(pdMS_TO_TICKS(5));
}And because an assistant is increasingly an agent, LOCI-DB also ships an MCP server, so Claude or any MCP client can use the same memory:
pip install "loci-stdb[mcp]"
claude mcp add loci-memory --env LOCI_MCP_MODE=local -- loci-mcp
# tools: remember, recall, novelty, trajectory, memory_statsWhy a pendant
I tried to answer the form-factor question before writing any code. Glasses are the obvious choice and the wrong one for a first version: expensive, hard to fit, and a big ask for someone who does not already wear them. A phone has to be taken out and pointed, which means you only capture what you already knew was important. A pendant on a lanyard sits at chest height, sees roughly what your body is facing, leaves both hands free, and comes off in a second. It is also the cheapest thing to build and iterate on, which matters when you are one person with a soldering iron.
What is on my desk today
The current build is deliberately off the shelf, so anyone can reproduce it:
| Part | What it does |
|---|---|
| Waveshare ESP32-P4-WIFI6 Kit A | The brain. An ESP32-P4 handles the camera and on-device processing; a companion ESP32-C6 provides Wi-Fi 6 and Bluetooth. The kit includes the camera module and a speaker. |
| NULLLAB PM11 battery module, 1200 mAh | Power, so the device can be worn untethered. |
| Lanyard | The mount. Low-tech on purpose. |
| LOCI-DB | The memory: what was seen, where and when, plus novelty detection. |
[Ziggy: add a photo of the current build here, worn and on the desk.]
Where I am right now
There is a working demo, end to end. The board streams frames to the host, I hold the button and ask “where is my cup?”, and the answer plays from the speaker about 1.3 seconds after I let go. On the home network a frame makes the round trip in 70 to 240 ms; through the Cloudflare Tunnel, 200 to 400 ms. With the host switched off, the board recognised “where is my cup” on its own in 730 ms and played the cached answer from the card. [Ziggy: one moment where it surprised you.]
In parallel I am building a second rig to stress the memory layer: a Raspberry Pi with an OAK-D Lite depth camera that I carry around while LOCI-DB builds a map. The depth camera gives real 3D positions, which is what the “where” in spatial memory needs to be accurate. Lessons from that rig feed back into the pendant.
What I do not have yet is a prototype I would hand to someone else. That is the next milestone.
The hard problems
Four problems decide whether this becomes something people actually wear:
- Privacy. This is the first problem, not the last. A camera on your chest sees other people, and a memory of your day is the most personal dataset you will ever own. My default is that images stay on the device or on hardware you own, that what gets stored is memory rather than video, and that the assistant can be told to forget. This is the same principle behind Enclave, my other project on keeping sensitive data out of reach of the models that use it.
- Power. Continuous vision on a 1200 mAh battery is a budget problem. The device has to choose when to look, not stare all day.
- Latency. An answer that arrives three seconds late is an answer about where you used to be. Deciding what runs on the ESP32-P4 and what runs elsewhere is the central architecture question.
- Trust. If it is confidently wrong once in a way that matters, people stop wearing it. It has to know when it does not know, and say so.
What is next in the series
- Part 2: The hardware. The full parts list, wiring, the enclosure and what I would change.
- Part 3: Giving an assistant a memory. LOCI-DB in depth: storing observations from a moving wearer, the benchmarks, where it is slow, and how novelty decides when to speak.
- Part 4: Privacy and power. What runs on the device, what never leaves it, how forgetting works, and the battery numbers.
- Part 5: First wearers. Putting the prototype on people who are not me, and what they tell me.
If you build wearables, work on personal AI or accessibility, or simply have a strong opinion about whether you would wear one of these, I would like to hear from you. Subscribe below to get each part when it is published.