High Speed low latency real-time data processing and analytics for IOT data for a Large US Retailer

Problem Statement

Real-time streaming data from IOT sensors monitoring Cooking equipment like fryers and ovens, Hot and Cold Storage containers, Transport vehicles that are continuously sending data on Power, Temperature, HVAC, etc
High Speed low latency real-time data processing and analytics for IOT data
Order of 100K messages per second from IOT Devices for monitoring food storage and food processing in hot and cold environments, 24 x 365
Need for real-time parsing, enrichment, predictive monitoring and deliver data in near-real time to Advanced analytics and Data Science users.

Real-time Data processing layer using Kafka, Spark, Cassandra, Redis
Data enrichment to add master data attributes, tagging for asset and process, Business rules-based Data Quality checks.
Real time Data Aggregation to build calculated aggregate metrics, Time window functions, moving averages and checks with Thresholds and allow Search and real time dashboard functionality
Analytics Layer for near-real time data analysis, AI/ML modeling
Persistent Data Storage on Cassandra for historical Data Analysis.

Seamless ingestion, processing of real time data with high volume/second
Technology agnostic robust architecture for high performance, scalability
Near real-time Data enrichment, in-process analytics, monitoring dashboards and enabling AI/ML based advanced analytics.