The great promise of AI is the ability to create powerful new applications to enhance productivity. We now have chatbots that can speak confidently about the contents of large databases, like CRMs.
But what if you wanted an AI application that could look across several databases? What if some of that data resided in different applications, where it was formatted differently? This type of analysis is still a challenge for AI. The data needs to be integrated while the AI is “thinking” and before it responds to a user’s prompt. Data cannot be AI ready if it’s stored across different data sources.
What Makes Data AI Ready?
- Governed, so that it is aligned with the policies of the developing organization.
- Trusted, or vetted for accuracy, so that users have confidence using the application.
- Delivered immediately, even if it is drawn from multiple disparate data sources, so it can be incorporated into an AI’s response.
Not Ready for AI
The problem isn’t just data integration. At least, the problem isn’t just physical data integration. You can store zettabytes of data in the same data lake, but if it’s actually stored across different applications, it won’t be AI ready. Each application will use slightly different semantics, and an AI can’t understand that two data sets with similar names, such as “CUST-ID” and “CUSTOMER,” might both refer to the same entity.
AI readiness gets even more challenging, though. AI applications are “intelligent,” but they’re also made of code, so they do exactly what you tell them to do — no more and no less. This is why prompting is so important when using AI. You often need to provide an AI with context so that it knows exactly what to do and which data source to use.
Much of this type of contextual information can also be provided on the back end to simplify the prompting process. You can accomplish this by adding descriptive metadata to each applicable data source via a metadata management application. If an AI uses a certain data source, it would then know the context that is important to apply, such as the conditions under which it can use the data set.
Finally, AI-ready data also has to be accurate and trustworthy, because ultimately, the AI itself will only be as accurate and trustworthy as the data it is trained on. This is why delivering AI-ready data is such a tall order.
Enter Logical Data Management
Most data management platforms rely on extract, transform and load (ETL) processes that move data into a single system where it can be integrated into a format that an AI application can readily consume. This is why many companies struggle to get their data AI ready; ETL processes simply take time to run. Some are faster than others, but they deliver data in scheduled batches. Thus, they can never deliver data in real time. For example, they would be unable to furnish an AI application with data to support an in-progress “conversation” with a user.
Logical data management platforms, in contrast, create logical views, or representations, of the different data sources. It brings these views into a logical layer that is independent from the physical layer composed of the systems that house the data. That is, elements in the logical layer can be reorganized or renamed without affecting the organization or naming structure of the data sources in the physical layer.
This separation between the logical and physical layers provides an abstraction layer between the two in which the logical layer contains abstract representations of the data sources that nonetheless enable actions on the underlying data sources. For example, users can access data directly from the logical layer, using a data visualization tool like Tableau, a data catalog or data marketplace, without even knowing where the data is actually stored.
The most powerful benefit of this approach is that users can access data in the logical layer in real time. This is because the logical layer is always kept up-to-date with the physical layer. Even when data changes in an underlying source system, the data is updated at subsecond speeds in the logical layer. In a traditional data management approach, if data is not available in the central repository, which might be a cloud data warehouse or a data lakehouse, users simply need to wait for the data to be physically replicated before it can be accessed.
A Flexible, Powerful Approach
One of the most promising aspects of logical data management is that it is implemented as a layer rather than a point solution. It’s one that you can implement over just about any data infrastructure, no matter how complex. Logical data layers are “light” in that they don’t store any data. They only store the metadata required to access the underlying data sources.
Because it establishes a real-time data access layer above existing sources, it enables organizations to implement semantic transformations within the logical layer, which automatically “translate” the data into different terms, depending on the needs of the data consumers, effectively unifying it, and also provide a ready interface for augmenting data sources with descriptive metadata, to further aid AI applications. This real-time access across disparate data sources also enables organizations to establish end-to-end data governance and security policies across the entire data estate. This is managed from one interface connected to the logical layer, rather than multiple interfaces for each of the individual data sources.
Get Your Data Ready for AI
Logical data management offers a flexible, straightforward way to make organizational data – no matter how diverse or dispersed – ready for AI
