Thoughts from Databricks Data & AI Summit
Despite the mostly dreary weather in San Francisco, Data and AI were hot inside the Moscone Center, where Databricks’ annual user conference was being held. The theme of this year’s Data & AI Summit was “Generation AI.” Under this thematic umbrella, besides LLMs and lakehouse architectures, an important topic is data governance around Databricks’ offering Unity Catalog. As a sponsor of this event, Orion Governance, had a chance to interact with Databricks customers and partners and found great interest in our Enterprise Information Intelligence Graph (EIIG), a self-defined data fabric which offers tremendous synergy with Databricks Lakehouse and Unity Catalog.
This synergy is multifaceted with two main use cases.
Accelerate Lakehouse Migration
Migrating to a Databricks Lakehouse, which combines data lakes and data warehouses into a unified platform, is a complicated process and presents challenges. A prerequisite to such migration is the understanding of the legacy environment. EIIG automatically ingests metadata from 70+ technology sources, ranging from the mainframe (PL1, COBOL, JCL, IMS), AS400 (RPG), SAP, ETL and BI Reporting tools, to programming languages including Python, Java, Scala, Javascript and Hive, and intelligently stitches it into an industry-leading knowledge graph. With this comprehensive picture of their information landscape, enterprises can assess their migration readiness and make migration plans much more effectively.
Along with end-to-end granular data lineage and dynamic data catalog, EIIG provides near real-time impact analysis to visualize what effect one change may have on downstream systems and identify dependencies. In addition, EIIG’s data quality augmentation and trust propagation capabilities enable organizations to have confidence in the data they are migrating.
Enterprises can also leverage EIIG to manage costs during migration and lakehouse implementation. One such example is the use of metadata analytics such as similarity analysis. EIIG can help automatically and visually identify duplicate reports, ETL (Extract Transform and Load) jobs, and tables so that organizations can delete them and prioritize the right workloads to migrate. As a result, they can reduce cost and minimize risks.
In short, as the next-generation metadata management and data governance platform, EIIG can help speed up the migration process and enable enterprises to implement their Databricks Lakehouse projects on time and within budget.
Complement Unity Catalog and Enable Enterprise-wide Data Governance
Unity Catalog is Databricks’ unified governance solution for data and AI on the lakehouse. EIIG complements Unity Catalog to create significant synergy.
The reality of enterprise IT landscape is a hybrid cloud of which lakehouse is a part. EIIG connects disparate systems in this hybrid cloud and creates one federated view of enterprise information flow. With bidirectional API, EIIG integrates with Unity Catalog. This integration enables organizations to send metadata feeds from other systems to Unity Catalog to enrich its coverage. And at the same time, enterprises can use EIIG to get information from Unity Catalog to be included in an enterprise-wide data catalog.
Since EIIG is one product that natively integrates all the data governance capabilities including lineage, catalog, impact analysis, active metadata, and metadata analytics, organizations can implement a global data governance regime to cover the entire enterprise with the lowest total cost of ownership.
To sum up, Unity Catalog and EIIG can work together to help enterprises achieve unified governance and comply with regulations so as to harness the power of data and AI more effectively, quickly, and economically.
For more information, please schedule a demo.
About the Author: Niu Bai, Ph.D. is the Head of Global Business Development at Orion Governance, Inc. Connect with Niu on LinkedIn.
Sign up for emails you actually want to read
Join the Orion Governance email list
Recent Articles
A Solution to Solving the Stubborn Data Silo Problem | Evanta NYC 2024 Review
Thank you to Evanta for organizing this CDAO Executive Summit in New York, where leaders in digital transformation shared insights and best practices. Several key takeaways emerged, including the leverage of Generative AI and [...]
EIIG Identifies and Enables Efficient Usage of Golden Data Sources
Unlock the full potential of your data assets with Orion Governance's EIIG (Enterprise Information Intelligence Graph). This in-depth explainer video showcases how EIIG identifies and enables more efficient usage of “golden” data sources, [...]
Video Explainer: How does EIIG identify data assets that have the most value and risk to ensure governance and compliance?
In this video, we explore how Orion Governance's Enterprise Intelligence Information Graph (EIIG) helps organizations identify data assets that hold the most value and risk, ensuring effective governance and compliance. Through EIIG’s powerful [...]