{"id":5706,"date":"2022-09-12T12:39:31","date_gmt":"2022-09-12T10:39:31","guid":{"rendered":"https:\/\/www.metakraftwerk.de\/patterns\/"},"modified":"2024-11-21T11:27:45","modified_gmt":"2024-11-21T10:27:45","slug":"patterns","status":"publish","type":"page","link":"https:\/\/www.metakraftwerk.de\/en\/patterns\/","title":{"rendered":"Patterns"},"content":{"rendered":"\t\t
\n\t\t\t\t\t\t\t
\n\t\t\t\t\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t
\n\t\t\t
\n\t\t\t\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t
<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t
\n\t\t\t
<\/div>\n\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t

Design Pattern for Data Integration<\/h1>\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\tIn the context of data warehousing, data engineering or related disciplines, each data integration process initially looks unique. Specific data structures in sources and targets are connected via a sequence of transformations, everything looks individual at first.\r\n\r\nIf you look at a large number of such data integration processes from a distance, a few patterns often become visible. Many projects have certainly already made this observation. But how can we benefit from this?\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t
\n\t\t\t
\n\t\t\t\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t
\n\t\t\t
<\/div>\n\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t
\n\t\t\t\t
\n\t\t\tIn the context of data warehousing, data engineering or related disciplines, each data integration process initially looks unique. Specific data structures in sources and targets are connected via a sequence of transformations, everything looks individual at first.\r\n\r\nIf you look at a large number of such data integration processes from a distance, a few patterns often become visible. Many projects have certainly already made this observation. But how can we benefit from this?\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t
\n\t\t\t
\n\t\t\t\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t
\n\t\t\t
<\/div>\n\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t
\n\t\t\t
\n\t\t\t\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t
\n\t\t\t
<\/div>\n\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t
\n\t\t\t
\n\t\t\t\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t
<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t
\n\t\t\t
<\/div>\n\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t
\n\t\t\t
\n\t\t\t\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t
<\/linearGradient><\/defs><\/g><\/g><\/g><\/g><\/g><\/g><\/g><\/g><\/g><\/g><\/g><\/g><\/g><\/g><\/g><\/g><\/g><\/g><\/g><\/g><\/g><\/g><\/g><\/g><\/g><\/g><\/g><\/g><\/g><\/g><\/g><\/g><\/g><\/g><\/g><\/g><\/g><\/g><\/g><\/g><\/g><\/g><\/g><\/g><\/g><\/g><\/g><\/g><\/g><\/g><\/g><\/g><\/g><\/g><\/g><\/g><\/g><\/g><\/g><\/g><\/g><\/g><\/g><\/g><\/g><\/g><\/g><\/svg><\/div><\/div><\/div><\/div>\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t
\n\t\t\t
\n\t\t\t\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t

Metadata Normalization<\/h2>\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\tIf we take a closer look at the data integration processes, we notice that they have some things in common. Basic logics and transformations are the same. So is the data modeling used (e.g., Data Vault, Dimensional Modeling). On the other hand, some parts are very specific, especially the processed data structures differ between each process.\r\n\r\nIn a metadata normalization, one separates between the common and the specific parts of the processes. The common parts are implemented in a design pattern. The specific parts are extracted from each process as so-called instance metadata.\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t
\n\t\t\t
\n\t\t\t\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t
\n\t\t\t
<\/div>\n\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t
\n\t\t\t
\n\t\t\t\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t
<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t
\n\t\t\t
<\/div>\n\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t
\n\t\t\t
\n\t\t\t\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t

Design Pattern<\/h3>\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\tA pattern is first designed, implemented and tested as an abstract prototype. This is done either in a data integration tool or as an SQL\/Spark template. The prototype is then imported into MetaKraftwerk and supplemented with dynamic rules that define where instance metadata should later be inserted and used. The pattern also defines naming rules and other dynamic components, such as DDL templates. When developing a pattern, there are no limits to creativity, and the functionality is completely based on the needs of the customer project.\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t
\n\t\t\t
\n\t\t\t\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t
\n\t\t\t\t\t