The concept of an enterprise data catalog is quite simple: it is an inventory of your data assets. It enables users to understand and find the data they need to generate insights and make business decisions. However, building and managing such a catalog is not at all simple. Without ML/AI powered automation, your implementation of a data catalog could be unproductive and inefficient, to say the least.
Connecting different technologies to make the data catalog impactful
To be meaningful and impactful, a data catalog needs to include a broad set of IT assets. In other words, a data catalog is as good as the metadata it can ingest. The challenge is that enterprise’s IT landscape is becoming increasingly complex with the adoption of all kinds of technologies. It is impossible to manually define, discover and upload metadata from all these complex systems. Therefore, the capability of automatically scanning metadata from all key technologies, ranging from mainframe, DBMS, data warehouses, big data sources, ETL and BI reporting tools, to programming languages, is the prerequisite to a successful data catalog. Connecting different technologies in a knowledge graph builds the foundation of a data catalog.
Identifying data assets to enable fact-based data governance and management
With automation, organizations can identify strategic data assets faster and more accurately. They can discover golden/certified data sources to help drive and control new application development. Also, they are able to categorize data assets that have most value and risk. With these findings at their fingertips, data governance and compliance practitioners can exert more efficient and targeted control.
This fact-based visibility and traceability enable data stewards to work on practical data management improvement. The automated traceability can help reduce data ambiguity by identifying the “true” one among data assets with the same look-and-feel. Data stewards also find the most optimal places (“hot spots”) in the overall data architecture to address data quality with least effort and most impact.
Automated mapping of terms with assets to facilitate the reuse of data assets
Too often data and analytical services are built with strong technical focus. In addition, different teams manage different projects and create silos. As a result, redundant and duplicate sources and logic are commonplace; data assets are hard to find, understand and trust. All these contribute to the difficulty of reusing data assets and waste of resources. One of the ways to solve this problem is to leverage automated term2asset mapping. With this automation, organizations can ensure creation, validation, and maintenance of business views over data assets. They can also establish processes and governance regimes to control the quality of business glossary content. In this scenario, reuse of data assets and business self-service become the norm. Data users can now gain insights to impact their decisions and thus bring value to the business more rapidly.
Activating the catalog to enable timely decision making
Many data catalogs in the marketplace are static. A static catalog is very much like a stack of printed documents, providing the user with stale information. Stale information is not very useful in helping decision making. An enterprise data catalog should be part of a living and breathing knowledge graph. Automation helps activate such a catalog with features like dynamic data lineage, impact analysis, and near real time monitoring of data quality. Furthermore, automation makes a data catalog more powerful and active with prescriptive analytics such as recommendation engines for data shopping.
In short, automation is essential to an effective enterprise data catalog. Orion’s Enterprise Information Intelligence Graph (EIIG) offers the most automated data catalog in the industry. EIIG has the ability to ingest 60+ technologies. It uses machine learning/artificial intelligence to unify data, connect business terminology and create a data catalog of all assets. By connecting the dots, Orion’s EIIG allows one to make more informed decisions in near real-time.
About the author: Niu Bai, Ph.D. is the Head of Global Business Development and Partnerships at Orion Governance, Inc.