A recent roundtable event brought together business and technical leaders from a diverse range of industries to discuss the paradigm of event streaming.

This event, sponsored by Confluent and LSD Open and hosted by TechCentral, provided space for thought and feedback regarding the challenges and opportunities for streaming data in real time.

Themes and key discussions

1. The data journey: where are we now and where are we going?

Industry cohesion: Experience among the group attending the roundtable was spread across various industries, including fintech, retail, medical and the public sector. As the conversation flowed, it became clear that although the product and go-to-market varied drastically, many of the underlying services below the surface struggled with identical challenges of modernisation. Value was noted within streaming’s ability to replicate success across mission-critical use cases that many shared in common (transactions, customer portal, microservice acceleration, and so on).
Stages of maturity: Different stages of maturity regarding data utilisation were obvious from the discussion. From monolithic isolation to the advanced concept of a data mesh, company representatives discuss how to get from A to B to eliminate data siloes and accelerate innovation. Obstacles commonly included manpower and funding as well as the sunk cost fallacy of which outsourcing management was considered as a potential wholesome solution.

2. Reliability vs speed

Technological advancements: The prominence of AI and machine learning has made companies review at least one key aspect of their data: is it reliable? Participants noted the dichotomy between requiring a vast amount of insight in real time and its association with bad data and the importance of robust tools to maintain data integrity. Those further along in the maturity curve explained how they have enabled both speed and structure for these modern use cases by shifting their thinking.

3. Scalability and speed to market

Scaling for new customers: The need for scalable solutions to ensure timely and relevant data usage was outlined and discussed as a priority within new customer acquisition. Participants shared their difficulties in expanding systems that were built ad hoc on legacy middleware. The focus developed towards their vision of configuring data systems for the growth of the business to achieve swift data processing and actionability. They drilled into what they would need it to do, when, and how quickly to generate the most revenue.
Data integration for agility: Breaking down data siloes was identified as a critical step towards effective data integration across organisations. This requires forward-thinking investment in technology essential for making data accessible and actionable for decision makers. Some great insight flowed from the LSD Open and Confluent teams on accelerators to break these siloes and what can be gained in terms of business agility and the ability to pivot.

4. Company culture and data utilisation

Cultural shift: Emphasising the triad of people, process and culture, the conversation revolved around fostering a data-centric culture within organisations. Raising awareness about the business potential of streaming data and the importance of educating teams across different lines of business were seen as foundational steps.
Internal customers: Recognising that customers can be internal teams, as well as external clients, participants discussed strategies to make data more accessible and useful internally making sharing insight more frictionless.
Expertise: Leaders laughed together at how often the topic of data becomes an administrative nightmare. This sparked a discussion around prioritising “the business needs” and in many cases choosing to identify low-hanging fruit to tackle the opportunity piece by piece.

5. Sharing success stories, overcoming challenges

Case studies: Attendees shared success stories and common pitfalls from their journey leveraging data as a whole. These narratives provided valuable lessons on navigating the complexities of making it effective.
Common pitfalls: Issues such as siloed data and lack of clear data ownership were recurrent themes. Participants discussed strategies to overcome these challenges and move towards higher data maturity.

6. Practical solutions and steps forward

Real-time data value: Delegates highlighted the importance of demonstrating the value of real-time data to drive business growth and direction. Concrete use cases were discussed to illustrate the benefits. This would be monumental in getting the required buy-in of the business to start using this for decision making.
Actionable steps: The group outlined practical steps to achieving higher data maturity, including implementing real-time data streaming solutions and integrating them into business processes.

Conclusion

The roundtable was a dynamic and insightful exchange of ideas, reflecting the diverse experiences and stages of data maturity across different industries. Emphasising the need for reliable, scalable and timely data processing, the discussions underscored the importance of a strong data culture and the sharing of knowledge and strategies. The issues around whether this would be a right fit per industry and per organisation gave a unique feel to what could be the next steps for their decision-making processes for their business growth and agility.