2023 Data Catalogs in Review: Key Trends Shaping Data Management in 2024

As the digital landscape continues to evolve, the importance of effective data management and governance has never been more critical. This article delves into the key trends that shaped data management in 2023, offering insights into the future of data catalogs and how they are poised to transform data management practices in 2024 and beyond. With a focus on emerging needs such as data mesh, governance, impact analysis, and the quest for more personalized and efficient data cataloging solutions, we explore the evolving landscape of data management and the critical role of technology in enabling organizations to harness the true power of their data.

The year 2023 marked a pivotal moment in the journey of data catalogs, with the market seeing an unprecedented influx of over fifty vendors, each promising to offer the ultimate solution for enterprise data management challenges. Amidst this crowded marketplace, only a select few managed to stand out, highlighting the industry’s urgent need for innovation and specificity in data management solutions. 

Summarized version 

– In 2023, the Data Catalog market was flooded with over 50 vendors, but only four managed to distinguish themselves.

– Three major players targeted specific user personas, while another vendor focused on active metadata.

– Many organizations were dissatisfied with their chosen catalogs, leading them to seek alternatives.

– There are currently no catalog solutions available that fully satisfy the needs of enterprise customers.

– The concept of data marketplaces and data product libraries is evolving, but there is a lack of a lightweight solution for data consumers.

– Organizations are moving towards Attribute-Based Access Control (ABAC) for more intricate access control policies.

– Data enablement is being embraced as a positive approach to data governance, empowering businesses through tools and collaborative processes.

– Data literacy is gaining recognition, leading to a demand for subject matter experts and initiatives to educate employees.

– Shifting left in data management platforms involves integrating metadata within the platform itself, but challenges exist in terminology disparity.

– The rise and fall of Data Mesh has been a topic of discussion, with organizations facing challenges in implementation and change management.

– Despite challenges, Data Mesh has the potential to revolutionize data management with the right implementation and focus on change management.

– AI governance is a buzzworthy topic, but many vendors’ claims lack tangible progress toward implementation.

Full Article

The year 2023 proved to be a disappointing one for Data Catalogs as more than 50 vendors flooded the market, vying for attention from organizations. However, only a mere four of them managed to distinguish themselves. 

Data Catalog Frustration

Among these, three major players with significant market shares targeted specific user personas such as data stewards, data engineers, and data analysts, while another vendor emphasized the concept of active metadata. In my interactions with various organizations, I found that most of them had already conducted thorough evaluations and either selected the market leader or were compelled by their IT leadership to fulfill their yearly commitment goals. Nevertheless, none of these organizations expressed genuine satisfaction with their chosen catalogs, as they failed to meet the requirements of many enterprise stakeholders. 

In fact, many organizations were frustrated with the catalogs’ functionality, implementation, overall journey, and user experience, prompting them to seek alternatives. Unfortunately, there are currently no catalog solutions available that truly satisfy the needs and demands of enterprise customers. It remains to be seen how this market will evolve, and it is eagerly anticipated that disruptive players will emerge to shake things up.

Data Marketplace Continuously Evolves

The concept of data marketplaces and data product libraries is continuously evolving. Organizations have recognized the importance of providing a user-friendly interface that allows any employee to easily discover data products. These data products can encompass various types of data or analytical assets that do not require extensive productization processes. 

However, the market currently lacks a lightweight solution as a data consumer portal. To address this gap, many organizations have relied on major consulting partners who may not fully grasp the concept or possess relevant experience, except for their access to UI experts. There is a strong desire to witness the emergence of a solution that can eliminate the need for custom-built solutions and provide a more efficient and effective platform for data discovery and utilization. 

Organizations have initiated the process of automating data provisioning and are now delving into the space of more intricate access control policies using Attribute-Based Access Control (ABAC) instead of the conventional Role-Based Access Control (RBAC). As a result, there has been a notable enhancement in the domain of policy-based access controls.

Data Enablement,  Data Literacy and Data Mesh

Data enablement is a term that brings a positive connotation to the concept of data governance. To foster a more optimistic atmosphere, organizations have begun embracing data enablement as a means to empower businesses through the implementation of appropriate tools and collaborative processes. I have personally advocated for the rebranding and realignment of data governance to data enablement, recognizing the potential benefits it can bring.

The importance of data literacy has gained significant recognition within organizations, leading to a growing emphasis on educating employees about the data they possess, its utilization, and its role in decision-making processes. Consequently, there has been a surge in the demand for subject matter experts (SMEs) in data literacy, prompting the development of various resources and experimental initiatives aimed at transforming this concept into a tangible reality.

The concept of shifting left in data management platforms involves the integration of document/enriching metadata within the platform itself. This approach holds great potential in aiding the data engineering team to gain a deeper understanding of their projects. However, I still have some reservations regarding the tangible benefits that the business will derive from this process, especially considering the potential challenges arising from the disparity in terminologies between the business area and the data engineering teams. To truly maximize the advantages, vendors must collaborate and establish standardized processes for sharing metadata across platforms. By achieving this level of coordination and consistency, we can anticipate significant improvements in results. I am eagerly looking forward to witnessing the evolution of these developments by the year 2026.

The rise and fall of Data Mesh has been a topic of discussion in the tech world. By early 2023, there was a tremendous amount of excitement surrounding this new approach to data management. Everyone wanted to jump on the bandwagon and explore the possibilities it offered. However, as organizations began to implement Data Mesh, they quickly realized that it was not as simple as it seemed. The complexity of this journey became apparent, and the lack of proper change management became a significant hurdle. Many organizations failed to see the desired results, and analysts started questioning the effectiveness of Data Mesh, pushing it toward obscurity. Interestingly, the concept of Data Mesh had been around for years, even before the term was coined. 

In 2009, I had the opportunity to build a data strategy around this innovative approach for a major financial organization. It was ahead of its time, and I firmly believe that Data Mesh is here to stay. Despite the challenges organizations face in recent years, the potential of Data Mesh to revolutionize data management is undeniable. With the right implementation and a focus on change management, I am confident that we will witness the true fruits of our labor soon. 

The buzz surrounding AI governance has reached a fever pitch. Suddenly, everyone is chiming in on the topic without truly understanding its intricacies or how it sets itself apart from traditional governance. To add to the confusion, numerous vendors have flooded the market with flashy marketing materials, each claiming to possess the ultimate framework for success. However, upon closer inspection, it becomes evident that these claims are nothing more than empty promises, with no tangible progress made toward actual implementation.

Finding Solutions in 2024

In conclusion, 2023 was a year of introspection and realization in the data catalog market, underscoring the imperative for solutions that not only meet but exceed the multifaceted needs of enterprises. The challenges highlighted throughout the year—from the dissatisfaction with existing catalogs to the demand for more sophisticated data governance and management frameworks—underscore a clear market gap. It is within this context that Orion Governance emerges as a beacon of hope, offering a comprehensive platform that addresses all the highlighted issues. 

With Orion’s Enterprise Information Intelligence Graph’s (EIIG) ability to cater to a wide array of needs including data mesh, data governance, impact analysis, and more, Orion Governance stands out as a singular solution that effectively solves the myriad data catalog issues faced by enterprises today. As we move into 2024, it is evident that the future of data management and governance lies in the hands of innovative platforms like Orion Governance, which are capable of adapting to the ever-evolving data landscape and fulfilling the complex requirements of modern enterprises.

Ready to learn more? Schedule a demo to learn more about how EIIG’s data catalog can make a difference for your enterprise.

About the Author: Ram Pratti is the Chief Evangelist at Orion Governance, Inc. Connect with Ram on LinkedIn.