Establish a Data Management Platform with Azure Synapse Analytics
About the Client:
Our customer is a wholly owned subsidiary of a large German group. With a holistic, customer-specific offering, it is successfully driving forward the digitization of all the group’s companies with around 5,100 employees in an integrative and value-creating manner.
In order to find and implement the right solutions for the companies, the customer relies on a comprehensive, market-oriented product portfolio in areas such as cloud, big data, Internet of Things and artificial intelligence, as well as on a high level of consulting and methodological expertise.
In particular, with its own ventures, the customer is in the dynamic development of open data platforms. These enable the comprehensive transformation from classic work and organizational structures to self-organization and company-wide, agile networks.
The Challenge:
Since mid-2020, the customer has been faced with the task of developing a uniform, central and open data platform in accordance with the group’s operating specifications.
A large number of group-wide projects with requirements for high data throughput should be able to benefit from the advantages of a modern data platform through automated provision within a very short time. The in-house specified configurations meet all corporate requirements in terms of security and compatibility in a future-proof manner and offer cost savings, reliable performance and high reliability in addition to quick and easy integration into existing systems.
The Implementation:
The aim of the project is to offer a Group-wide and uniform data management platform for IT projects within the Group. In the process, the entire infrastructure will be set up according to Group-wide operating specifications. The connections run mainly via the Microsoft backbone and not publicly via the Internet. The Data Management Platform can be ordered via a Group portal for all IT teams and is then provided automatically.
An ordering process triggers a Gitlab pipeline, which uses Terraform to set up the infrastructure within 30 minutes, set necessary permissions, and perform configuration.
Microsoft Azure Cloud was chosen as the provider to build a data management platform. This option offers intelligent solutions for storing, managing and analyzing growing volumes of complex customer data.
For data storage, a Data Lake Gen2 architecture is deployed which, in combination with Synapse Analytics, provides a simple interface for data scientists. Processing is performed on the underlying Apache Spark pools and SQL pools.
In the future, integration with the data catalogue will be ensured with Synapse Analytics.
The Result:
The introduction of the Data Management Platform enables fast, barrier-free and cost-efficient access to the collected data of the client. Thus, not only the access and management of the existing data catalog will be improved, but also the profitable use of the data will be facilitated.
Thanks to the professional support of PROTOS Technologie GmbH, it is possible to receive the platform completely set up and pre-configured within approximately 30 minutes after ordering. The user-friendly operation lowers the initial inhibition to work with data.
For data scientists, the use of Synapse Analytics brings elementary operational and organizational advantages, as it supports the integration of different systems. While internal data integration handles the management and administration of data, data connectivity prevents data from disappearing into the data swamp. Synapse Analytics also supports the innovative data lake house paradigm. Big Data Warehouse (Big Data SQL Pools) queries and Spark jobs (Apache Spark) can also be executed in Synapse Analytics.
PROTOS Technologie GmbH, as a certified HashiCorp & Microsoft Partner (Silver – Data Analytics), has been supporting the client since the beginning of 2020 in building the Azure reference architecture for the data management platform, which can be self-provisioned via a service portal. Thanks to the collaboration, the customer was able to develop a unified, centralized and open data platform, according to the group’s operator specifications. In addition to the consulting services, PROTOS is implementing the highly automated provisioning of the platform based on HashiCorp Terraform.
The Benefits:
Provision and Operation
The DMP provides fast, accessible and cost-effective access to the group’s collected data.
Compliance compliant Set Up
The in-house specified configurations meet all group specifications regarding security and compatibility.
Reliability
The modern data platform offers reliable performance and high reliability.
Grafiken: Prosymbols, Freepik, Becris, Ralf Schmitzer from Flaticon.com