Thoughts from Databricks Data & AI Summit

Despite the mostly dreary weather in San Francisco, Data and AI were hot inside the Moscone Center, where Databricks’ annual user conference was being held. The theme of this year’s Data & AI Summit was “Generation AI.” Under this thematic umbrella, besides LLMs and lakehouse architectures, an important topic is data governance around Databricks’ offering Unity Catalog. As a sponsor of this event, Orion Governance, had a chance to interact with Databricks customers and partners and found great interest in our Enterprise Information Intelligence Graph (EIIG), a self-defined data fabric which offers tremendous synergy with Databricks Lakehouse and Unity Catalog.

This synergy is multifaceted with two main use cases.

Accelerate Lakehouse Migration

Migrating to a Databricks Lakehouse, which combines data lakes and data warehouses into a unified platform, is a complicated process and presents challenges. A prerequisite to such migration is the understanding of the legacy environment. EIIG automatically ingests metadata from 70+ technology sources, ranging from the mainframe (PL1, COBOL, JCL, IMS), AS400 (RPG), SAP, ETL and BI Reporting tools, to programming languages including Python, Java, Scala, Javascript and Hive, and intelligently stitches it into an industry-leading knowledge graph. With this comprehensive picture of their information landscape, enterprises can assess their migration readiness and make migration plans much more effectively.

Along with end-to-end granular data lineage and dynamic data catalog, EIIG provides near real-time impact analysis to visualize what effect one change may have on downstream systems and identify dependencies. In addition, EIIG’s data quality augmentation and trust propagation capabilities enable organizations to have confidence in the data they are migrating.

Enterprises can also leverage EIIG to manage costs during migration and lakehouse implementation. One such example is the use of metadata analytics such as similarity analysis. EIIG can help automatically and visually identify duplicate reports, ETL (Extract Transform and Load) jobs, and tables so that organizations can delete them and prioritize the right workloads to migrate. As a result, they can reduce cost and minimize risks.

In short, as the next-generation metadata management and data governance platform, EIIG can help speed up the migration process and enable enterprises to implement their Databricks Lakehouse projects on time and within budget.

Complement Unity Catalog and Enable Enterprise-wide Data Governance

Unity Catalog is Databricks’ unified governance solution for data and AI on the lakehouse. EIIG complements Unity Catalog to create significant synergy.

The reality of enterprise IT landscape is a hybrid cloud of which lakehouse is a part. EIIG connects disparate systems in this hybrid cloud and creates one federated view of enterprise information flow. With bidirectional API, EIIG integrates with Unity Catalog. This integration enables organizations to send metadata feeds from other systems to Unity Catalog to enrich its coverage. And at the same time, enterprises can use EIIG to get information from Unity Catalog to be included in an enterprise-wide data catalog.

Since EIIG is one product that natively integrates all the data governance capabilities including lineage, catalog, impact analysis, active metadata, and metadata analytics, organizations can implement a global data governance regime to cover the entire enterprise with the lowest total cost of ownership.

To sum up, Unity Catalog and EIIG can work together to help enterprises achieve unified governance and comply with regulations so as to harness the power of data and AI more effectively, quickly, and economically.

For more information, please schedule a demo.

 

About the Author: Niu Bai, Ph.D. is the Head of Global Business Development at Orion Governance, Inc. Connect with Niu on LinkedIn.

Sign up for emails you actually want to read

Join the Orion Governance email list

Ready To Check Out Orion Governance?

Schedule a demo to quickly discover how Orion works for you

Recent Articles