Compute-Efficient Video AI
How we engineered a multi-scenario AI surveillance platform that runs 7 concurrent AI capabilities across live CCTV feeds — ensuring AI models are invoked only when genuinely warranted, keeping real-time inference viable on shared on-premises hardware.
AI at the Edge — Without the Compute Bill
Running deep learning inference on live video is expensive. A typical deployment of 16 cameras at 15 fps generates 240 frames per second. Passing every frame through a YOLO object detection model would demand continuous GPU utilisation across all cameras simultaneously — making the system unviable for any deployment that doesn't have data-centre infrastructure on site.
The platform needed to support multiple AI scenarios concurrently — face recognition, intruder detection, litter detection, parking occupancy, licence plate recognition, doorstep visitor detection, and multi-camera person tracking — all running on commodity edge hardware, without sending video to the cloud.
The solution required a disciplined engineering principle: AI models are the last resort, not the first step. Every workflow must exhaust lightweight mathematical techniques before invoking inference. The AI model sees only the small fraction of frames where something meaningful may have actually happened.
"If we run AI on every frame of every camera, the compute cost makes the system unviable for any but the largest deployments. We need to be smarter about when we invoke the model — the AI runs at the very end of the workflow, not the beginning."
— Engineering PrincipleCompute-Efficient AI Capabilities
Each scenario is designed so that AI models are engaged only when the platform determines something of significance may have occurred — keeping 7 concurrent AI capabilities viable on shared on-premises hardware in production.
On-Premises Inference
All AI processing happens locally — no video leaves the site. The platform is designed to run on commodity edge hardware without cloud compute dependencies.
7 Concurrent Scenarios
Litter detection, parking occupancy, intruder detection, ANPR, doorstep visitor detection, face identity verification, and multi-camera person tracking run simultaneously from shared infrastructure.
Scenario-Specific Design
Each capability is independently engineered for its domain — the approach that works for parking doesn't apply to face verification. No generic one-size-fits-all inference loop.
Production-Ready Scale
The system has been deployed across multi-building residential sites with dozens of cameras and thousands of daily events, with inference loads that remain manageable on shared hardware.
Temporal Sampling
Each scenario operates at a sampling rate calibrated to its event frequency — well below the native camera frame rate. Scenario-appropriate sampling is enforced at the source, so the AI pipeline never receives more input than is meaningful for that scenario.
Scene Change Detection
Computer vision techniques evaluate whether a candidate frame contains a meaningful scene change before any deep learning is invoked. Frames without relevant change are discarded at this stage without reaching the inference stack.
Scene Context Awareness
Several scenarios maintain a representation of the scene's expected state. Incoming frames are evaluated against this reference — only deviations beyond a significance threshold proceed. This encodes domain knowledge about what 'normal' looks like per camera, reducing scenario-specific false positives.
Spatial Region Filtering
Only the spatial region relevant to each scenario is processed. Analysing a targeted crop of each frame is orders of magnitude cheaper than full-frame inference, and also reduces false detections from irrelevant areas of the scene.
Frame Deduplication
Visual deduplication identifies consecutive frames with negligible perceptual differences and discards redundant copies before inference. This is particularly effective in slow-moving scenes where many consecutive frames contain the same visual information.
Context-Aware Scene Differencing
In residential buildings, residents routinely leave tied garbage bags outside their flats for scheduled collection. A naive litter detection model would flag these bags as litter in every processed frame — generating continuous false alerts and destroying trust in the system.
The solution encodes scene context directly into the detection workflow. Rather than triggering on any object present in the frame, the system evaluates each frame against an expected scene state — alerting only on deviations from that expected state. This dramatically reduces false alerts in environments where some objects are always present and expected.
The expected scene state is maintained per-camera and can be refreshed by the platform backend when the scene legitimately changes. Only frames that show a meaningful deviation from this expected state proceed to AI-based object detection.
Pixel Differencing (Consecutive Frames)
Too sensitive to lighting shifts, shadows, and reflections. No contextual understanding — cannot distinguish intentional placements from litter.
Full-Frame YOLO on Every Frame
Extremely high computational cost and maximum false positives — garbage bags flagged as litter in every single frame without context awareness.
Context-Aware Scene Baseline + AI Detection
The expected scene state is encoded per-camera. Evaluation against this baseline ensures AI only sees frames with genuinely new content — not expected objects. Dramatically reduces false positives in cluttered environments.
Real-Time Occupancy Monitoring
Parking management requires knowing whether each slot is occupied or vacant in real time. Running YOLO on every frame to detect vehicles is prohibitively expensive, particularly across multi-deck or multi-bay parking zones with many slots in a single camera's field of view.
The system accurately tracks occupancy state in real time across all configured slots. It is designed to scale to any number of slots without proportional increases in processing load — making it viable for large multi-bay or multi-deck facilities monitored by a single camera.
Slot positions are configured per-camera via annotation. The system reliably detects vehicle entry and exit events and reports occupancy state to the platform in real time.
Continuous YOLO on Every Frame
High compute cost with no accuracy benefit for fixed cameras. Slot positions are static — re-detecting vehicles in the same coordinates adds no new information.
Object Tracking (DeepSORT / ByteTrack)
Tracking is designed for moving objects. Parked vehicles are stationary — tracking adds complexity and overhead for a problem that doesn't involve motion.
Scalable Real-Time Slot State Tracking
Occupancy state is tracked accurately in real time. The approach scales to any number of slots without proportional compute increases, maintaining accuracy across diverse lighting conditions and vehicle types.
Event Formation + Selective AI Validation
Detecting unauthorised human presence in a monitored zone during off-hours requires distinguishing between meaningful intrusion events and incidental motion — wind through foliage, lighting flickers, or an animal crossing a doorway. Running YOLO on every frame wastes compute on events that will never classify as intrusion; using only motion detection produces too many false triggers.
The system uses a staged approach. Motion detection identifies candidate frames before involving deep learning. Rather than acting on individual motion frames, the system evaluates motion over a time window — this temporal validation step eliminates transient noise while ensuring genuine movement patterns are detected. Only confirmed motion events proceed to AI-based person classification.
If the AI model confirms human presence, an intrusion alert is generated. Non-human motion events — falling objects, animals, lighting changes — are handled without generating false alerts. The system adapts over time to its environment, reducing repeated false triggers from recurring non-intrusion patterns.
YOLO on Every Frame
Very high compute cost. Not scalable across multiple cameras. Detects every person in the frame, not just intruders — no context about whether detection is anomalous.
Frame Differencing (Consecutive Frames Only)
Sensitive to all movement including noise. Cannot distinguish meaningful events. No AI validation means human vs. non-human motion cannot be separated.
Autoencoder-Based Anomaly Detection
Requires extensive training data per scene. Hard to generalise across diverse camera placements. Complex to deploy at scale per-camera without retraining.
Motion Pre-Screening + Temporal Validation + AI Person Detection
Motion filtering eliminates static frames. Temporal validation eliminates transient noise. AI runs only on confirmed events. The system adapts to its environment over time, reducing repeated false triggers.
The system distinguishes human intrusion from incidental environmental motion and adapts over time to its deployment environment — reducing nuisance alerts without compromising detection accuracy. Alerts are generated with timestamps and camera identifiers for downstream action.
Multi-Stage Pipeline with Fuzzy Validation
Licence plate reading from a live video stream is inherently multi-step: the plate must be localised in the frame before text can be extracted, and OCR output requires cleaning and validation before matching against a registered vehicle database.
The optimisation strategy applies spatial and temporal filters before any deep learning runs. Frames are sampled at intervals rather than continuously. Within each frame, only the lower portion (the region where vehicle plates appear at entry gates) is cropped and processed — reducing the area of interest from a full 1080p frame to a small crop. YOLO then detects licence plate bounding boxes within this ROI only; full-frame inference is never performed.
Extracted plates go through text normalisation, fuzzy similarity matching against the vehicle register, and a stability filter — the plate must be detected consistently across multiple observations before a confirmed match is logged, suppressing false triggers from partially visible or blurred plates.
Traditional Image Processing (Canny / Contours)
Highly sensitive to lighting, blur, and shadows. Fails in real-world conditions without extensive manual tuning per installation. Poor generalisation.
OCR Directly on Full Frame
No plate localisation. Background text, signs, and objects produce high false-positive rates. Accuracy degrades significantly without plate bounding.
YOLO on Full Frame Continuously
High continuous compute cost. YOLO on the full frame includes sky, walls, and irrelevant content. ROI cropping achieves equivalent accuracy at a fraction of the cost.
Multi-Stage Plate Recognition with Validation
Each stage reduces false candidates before the next. Fuzzy matching handles real-world OCR noise. A stability filter ensures only confirmed, consistent reads generate access events.
The system achieves high read accuracy in real-world conditions — handling low-resolution plates, varying lighting, partial occlusion, and motion blur. Only confirmed, stable reads trigger access decisions, preventing spurious events from momentary or ambiguous detections.
Zone-Based Doorstep Detection
In shared residential foyers, a single CCTV camera monitors multiple flat entrances. Detecting when someone is specifically at a flat's doorstep — rather than merely walking through the shared corridor — requires spatial awareness, not just object detection.
The system defines a detection zone for each flat entrance. YOLOv8 detects all persons in the frame, but only persons whose position falls within the relevant zone trigger a doorstep event. A person walking through the corridor generates no event; a person stopping at the entrance reliably triggers one.
One camera can simultaneously serve multiple flat zones without separate models or camera instances per flat. Zone coordinates are configured per-camera via annotation.
Full-Frame Detection Without ROI Filtering
Fires on every person in the foyer regardless of their location. Produces continuous false alerts for residents passing through or visiting other flats.
Motion Detection Only (MOG2 / Frame Differencing)
Cannot distinguish humans from other motion sources. No spatial specificity — any movement near the doorstep area triggers an alert including shadows, pets, or lighting changes.
YOLOv8 Person Detection + Zone-Based Spatial Filtering
Person detection ensures only humans trigger events. Zone filtering adds spatial precision — the person must be at the doorstep, not merely visible. One model serves multiple flat zones per frame.
Doorstep Visitor Detection — Spatial Validation Pipeline
pHash Deduplication Before Deep Face Embeddings
Identifying whether a specific individual appears in a video — by comparing against a reference image — requires running face detection and embedding generation on candidate frames. At 15–25 fps, most consecutive frames are near-identical; running InsightFace's face detection and embedding pipeline on every frame would repeat the same computation dozens of times per second for what is functionally the same visual input.
The pre-filter uses perceptual hashing. Each incoming frame generates a compact visual fingerprint; frames that are perceptually near-identical to the previous frame are discarded. Only visually distinct frames advance to InsightFace for face detection, alignment, and embedding extraction.
Face embeddings from the video are compared against the reference image's embedding using cosine similarity. Frames where similarity exceeds the match threshold generate a timestamp record. The result is a list of timestamps at which the identified individual appears in the footage, produced with a fraction of the inference calls that a naive approach would require.
InsightFace on Every Frame
Near-identical consecutive frames incur full embedding computation for zero additional information. Scales linearly with fps — prohibitively expensive for long videos.
SSIM Frame Similarity
Structurally more sensitive than needed for this use case. Computationally heavier than pHash and more susceptible to minor lighting changes that don't affect face identity.
Pixel Difference (Raw)
Too sensitive to micro-movements and noise. Treats every frame as unique even when the scene is unchanged. Does not eliminate redundant inference.
pHash Deduplication + InsightFace + Cosine Similarity
pHash provides fast, lightweight uniqueness detection. InsightFace runs only on genuinely distinct frames. Cosine similarity is robust to embedding-space scale variation.
Face Identity Verification — pHash Pre-Filter Pipeline
Three-Stage Pipeline: Detection · Tracking · Re-ID
Route compliance monitoring — verifying that a visitor reaches only their approved destination — requires linking a person's identity across multiple non-overlapping camera views. This is a problem where AI at all three stages is unavoidable, but where using the right architecture at each stage is the optimisation.
YOLO — Per-Frame Person Detection
YOLOv8 detects persons in each frame of each camera, outputting bounding boxes, class labels, and confidence scores. Detection is frame-local — no memory, no identity. Frame sampling reduces the input rate to the detection stage. YOLO is the only option here; lightweight motion detection cannot distinguish humans from other objects.
ByteTrack — Intra-Camera Tracking
ByteTrack assigns stable local identifiers to detected persons within a single camera stream. It uses Kalman filter prediction combined with IoU-based Hungarian algorithm matching to maintain IDs across frames — including through brief occlusions or low-confidence detections. ByteTrack is far more computationally efficient than Re-ID, making it ideal for the per-camera, per-frame tracking problem.
OSNet ReID — Cross-Camera Re-Identification
When a person transitions from one camera to another, their ByteTrack ID resets. OSNet generates a compact appearance embedding (Re-ID descriptor) for tracked individuals. These embeddings are matched across camera boundaries to link the same person's trajectory across the full camera network — enabling end-to-end route verification from entry gate to assigned destination.
Why All Three Stages Are Necessary
Detection alone cannot track; tracking alone cannot cross cameras; Re-ID alone without detection and tracking produces no stable subjects to re-identify. Each stage handles a distinct sub-problem. Skipping any stage degrades the system to something that cannot solve cross-camera route verification at all.
Multi-Camera Tracking — Three-Stage AI Pipeline
The AI & Computer Vision Toolkit
Building AI that needs to run at the edge?
Whether it's optimising inference pipelines, designing multi-scenario video AI, or deploying on-premises at scale — these problems are solved in production here.