AIOT Smart Office — Multi-Camera Person Tracking with a Thai Voice Assistant

The office wanted to know who was in which room and to ask about it out loud in Thai. Three constraints fought each other. First, every capability had to be demonstrable on a development machine with only 8 GB of VRAM. Second, camera tracking is realtime and cannot be paused, deprioritised, or moved off the GPU to make room for another model. Third, the voice assistant needs Whisper, an LLM, pyannote, and TTS — all competing for that same VRAM.

More importantly, "who is in this room" is a question you cannot afford to answer wrongly. A system that confidently says the wrong name is worse than one that says it doesn't know.

I split the system into two projects that share no code and no dependencies. They communicate over exactly one Redis Stream, `location-events`, with no HTTP anywhere in the location data path.

That decision let both sides be developed, tested, and fail independently. A camera outage does not take the assistant down, and the contract between the two systems is a single schema pinned in a file.

For correctness, I set thresholds so the tracker returns `Unknown` rather than guessing when face and ReID evidence falls short, and I made the assistant answer location questions with exact SQL lookups against PostgreSQL — no vector search on location data. Approximate retrieval on a question that demands an exact answer is an invitation to hallucinate.

RTSP cameras → YOLO person detection → per-camera tracker (BoT-SORT/ByteTrack) → face identification with InsightFace (SCRFD + ArcFace) → OSNet ReID to link people across cameras → room enter/exit events → `XADD` to the Redis Stream.

On the AICORE side, a location worker consumes with `XREADGROUP`, writes to PostgreSQL, and auto-enrolls unseen people and rooms. The Thai assistant (Whisper STT → Ollama → RAG → Thai TTS) then answers by querying the location tables directly.

The seam is pinned by `location_event_contract.json`, a golden fixture both projects' contract tests bind to. A one-sided schema change turns CI red immediately instead of silently dropping events in production. GitLab CI filters jobs by path, but editing the contract file deliberately triggers both.

I started the project and wrote nearly all of it — 73 of 95 commits on the main repository. Others joined later, after the structure had settled.

The decisions that were mine: splitting the two systems behind a Redis Stream instead of building a monolith; the two-mode GPU coordination design; pinning the seam with a golden fixture; choosing exact SQL over vector search for location data; and writing the architecture documentation in three layers (repo root → integration → each subproject) so newcomers and AI tools could navigate without asking.

Sharing 8 GB of VRAM without ever stopping the cameras. I deliberately did not build a per-request VRAM accountant — too complex, too easy to break. Instead there are two GPU coordination modes: on the 8 GB demo machine, exclusive keeps YOLO resident while Ollama and Whisper/pyannote take turns; on the 16 GB machine, shared lets them coexist when VRAM allows. I benchmarked the 4B and 9B models under identical context sizes before choosing, rather than inferring from parameter count.

A silent ONNX Runtime clash. InsightFace pulls in the CPU ONNX Runtime and overwrites the GPU package's files — the system still runs, just far slower, with no error. I fixed the install ordering and added a healthcheck that only passes when the container actually sees CUDAExecutionProvider, reporting DEGRADED · CPU instead of staying quiet.

Auditing my own code and writing down the debt. I reviewed both systems in two full passes (49 findings, 47 fixed, then 45 more) and kept the four I could not fix in a document, each with the reason it remains and the condition that must be met first — for example, Redis still has no auth and Postgres still uses default credentials, a risk that is only acceptable inside a controlled network and must be fixed before the system leaves it. Debt written down with an explicit release condition is worth more than debt nobody knows about.

The system was designed for an office of 6 rooms and 40 or more cameras. Those numbers were set at the start of the project, and they are the reason the components were separated from the beginning rather than written together and pulled apart later.

What has actually been validated so far: the full loop runs on the 8 GB development machine (RTX 2070), which this build targets at 2 cameras — tracking runs continuously while the assistant answers Thai-language questions, and the cameras never have to yield to another model. Real-model tests pass for Ollama 4B, Whisper, and pyannote on CUDA, and CI validates the cross-system contract on every change.

Testing at 30–40 cameras on an RTX 4080 is the next step. The components are already separated for it, but I have not measured it yet — being clear about what is measured and what is still a target matters more than quoting a large number I cannot back.

It remains a prototype by design — not cleared for real door control until it passes licensing, privacy, presentation-attack detection, and capacity acceptance as written in the production requirements document.