Data democratisation, a push to enable everyone in the organisation to access, manipulate and analyse the organisation’s rich data stores, has been a dream for some years. In theory, data democratisation would eliminate data gatekeepers and bottlenecks, and make data available and usable across the business. This would help unlock more value from data and drive smarter, data-driven businesses.
No sooner had the concept emerged than organisations realised there would be challenges in achieving data democratisation. There were concerns about data governance, data quality and the risks of inexperienced staff not adhering to data management best practices.
The myth of the one-stop shop
Data democratisation raises questions such as what data should be accessed, and where this should be enabled. Simply making all data available to all users could prove chaotic and risky. The notion that data can be provisioned easily through a “one-stop shop” for the digital marketplace and individual consumers is misleading.
To democratise and share data with the right people or processes, a huge amount of preparation is required in the background – from creation to consumption to destruction. The data must be qualified (legally accessed, profiled, verified, cleansed, validated) and integrated (matched, related, de-duplicated, enriched) before it can be provisioned as trustworthy information.
Moreover, democratising data requires underlying governance, that is, the application of controls in the preparation of the data to ensure that the right data accesses and processing is done at the right times and by the right people for the right purpose.
All this work comes at a cost.
The challenge of multiple sources
Even if all this work is done at the data’s source and served directly to the consumer, it should be noted that the provisioning will inevitably take from many points, and will still need to come together.
Take the example of a taxi driver looking for the shortest route to a destination. This involves getting possible routes from one information provider (eg Google Maps), which in turn has to access and bring many datasets together from other disparate sources, for maps, route parameters, world dates/times, road maintenance and control status, nearby events from the newscasts and internet resourcing, and then channel the collated and integrated data product to the taxi driver.
Optimising the ‘how’
The potential and opportunities for channelling all data to users is exciting. But the “how” of this is still being developed. Factors such as data sourcing, preparation and provisioning; the processes, techniques, methods and patterns (functions) used to access and process the data are yet to be optimised. In future, it is likely that the data emerging from addressing the “how” will also offer up new and useful insights for the source providers themselves.
They may be in a position in future to analyse function performance statistics, compute and throughput resource consumption, affinities, preferences and the least resource consuming patterns through the systems and applications, for example.
Data democratisation
To serve up data while addressing data management and governance concerns, organisations need to define key architectural and governance principles behind the sharing and ownership of data centrally and use enabling technology to execute these principles. These concepts are co-dependent, as just the definition of the principles will make for slow to no progress, while the use of technology without the defined principles will lead to poor adoption of the technology and failure of the project.
A potential solution is a data fabric. The core principle behind the data fabric architecture is to provide a curation and abstraction layer for data across multiple disparate sources, with this layer having the ability to deliver governance and data management uniformly.
Depending on the requirement and size of an organisation, a data mesh may enable the concept of data democratisation, by assigning responsibility for data management across disparate teams in silos, but still maintaining some central governance principles while executing day to day operations.
About the KID Group
The KID Group is a private group of companies providing expert consulting, professional services and technologies in the Southern African region. With the head office in Johannesburg and satellite offices in Cape Town, Durban and Pretoria, the KID Group employs experts across a range of business intelligence and mobile data services. Our combined 300+ strong team is diverse, not only in terms of the racial and gender mix, but also in terms of the skills and experiences it brings to the projects undertaken.
- The author, Veemal Kalanjee, is MD of InfoFlow
- Read more articles from the KID Group on TechCentral
- This promoted content was paid for by the party concerned