Intelligent resident surveillance is one of the most essential smart community services. The increasing demand for security needs surveillance systems to be able to detect anomalies in surveillance scenes. Employing high-capacity computational devices for intelligent surveillance in residential societies is costly and not feasible. Therefore, we propose anomaly detection for intelligent surveillance using CPU-only edge devices. A modular framework to capture object-level inferences and tracking is developed. To cope with partial occlusions, posture deformations, and complex scenes, we employed feature encoding and trajectory association governed by two metrices complementing to each other. The elements of an anomaly detection framework are optimized to run on CPU-only edge devices with sufficient frames per second (FPS). The experimental results indicate the proposed method is feasible and achieves satisfactory results in real-life scenarios.