Cloud DataOps and Data migration for a Large European Bank

Problem Statement

Most of the data resided on-prem and there was a need to move them to cloud along with centralization of all information. Along with migration of huge amounts of data, there was also a need to replicate data between on-prem and cloud and maintain both instances. Complete ramp-down of on-prem data was envisioned in 2 years.

Solution Overview

• Ensured current on-prem deployment is working effectively by increasing efficiencies in data synchronization and transfer.
• Effective reporting engine to clearly identify problem areas.
• Automation of many tasks in operations and testing that reduced human dependency.
• Introduction of innovations that have transformed the BAU operations. Details in the innovation section below. Data architecture simplification and design programs
• We worked to create a roadmap that enables a simplified architecture that all divisions of the bank can benefit. Deployed migration teams to replicate dataset on cloud with minimal latency.
• Created new data sets with a cloud first/ cloud only approach.
• Data connections via Ab Initio and Kafka with minimal latency to ensure near real-time connections of info.
• Suitable big-data architecture to create single view of data at a timeframe.
• AWS Cloud migration with S3 enabled clients to store much higher volumes of data at lower costs, while also being faster to operate via Snowflake deployment.

Outcomes

• Automated provision of dashboards at the time of project creation in PaaS using DaaS along with near real time unified views. Watcher is a plugin for Elastic Search, which was implemented to automatically detect unaccounted data changes.
• Integration with Dashboards for near real time alerts. 3 unique automated attempts set up for self-heal.
• Machine Learning models set up for anomaly detection. False alerts have been minimized to <1%.

Combining Watchers, Anomaly detection and Self-Healing to create cognitive events and processes to bring in intelligent operations