Data is the lifeblood of the digital age, and the ability to analyse ever-increasing volumes of data quickly and accurately is often what sets organisations apart from their competition.
The sheer volume of data, coming from various modern and legacy systems, can make the analytics process a complex, time-consuming and expensive task – even with the assistance of sophisticated cloud services.
Capitec Bank found itself in a position where it has to process thousands of tables from its data lake every day. The content ranges from card processing data to financial reporting, and originated in a variety of systems, both legacy platforms and modern database management systems like PostgreSQL and Microsoft SQL.
The bank is migrating to the cloud, but it still runs some of its systems on-premises and on a broad range of technologies.
“There are a lot of interactions and dependencies between the various systems,” explains Hamish Whittal, founder of Cloud Fundis.
Due of the volume, scale and complexity of its data analytics requirements, Capitec – like many other customers in the same position – found that data processing using standard cloud services was very costly, often contributing the lion’s share of the cloud costs for the business. Migration from on-premises to the cloud can be both challenging and expensive.
“It’s a reality that every customer is cost aware and that the cost of processing large volumes of data can quickly become painful,” Whittal says.
Cloud savings
Cloud Fundis is an IT consultancy in the analytics and data environment, offering AWS solutions. It developed a solution for Capitec that not only makes their data processing task quicker and easier, but has also reduced the costs by a minimum of 50% (in some cases, 80% or more in savings).
“For any modern business, processing analytics data using standard AWS services like AWS Glue, AWS EMR and AWS EMR on EKS is expensive,” Whittal explains.
Cloud Fundis solved the problem by containerising data processing for Capitec.
“We created a new service, based off the successes we had for Capitec Bank – dubbed BrightSpark – built on native services such as AWS ECS, AWS EKS and AWS Batch. Our big change was creating a simple-to-use, custom-written API (application programming interface) with which customers can interact to manage and run their jobs. The core components of BrightSpark can be deployed into a customer’s existing AWS estate, allowing them full control over important aspects like data security and governance,” explains Alex Morton, DevOps lead at Cloud Fundis.
“From the customer’s side, they simply send a Spark or Python script through our engine, which modifies and optimises the code. This transformed and optimised code can then be executed using BrightSpark.”
The difference with BrightSpark is that the customer jobs are completed at substantially less cost, so significantly less is spent on cloud compute.
“Optimised code allows customers to process data faster, enabling quicker decision-making and enhancing competitiveness,” Whittal adds.
The system was first conceptualised about eight months ago and went into “proof of concept” two months later. After various tweaks and modifications, it went live about six months after the first concept.
The results have exceeded expectations, with Capitec’s cost to extract and transform thousands of data sets each day dropping by more than 50%. By automating and simplifying the process, customers save time and are also able to free up valuable human resources.
“We have taken away the complexity,” Morton says. “The customer doesn’t need to have the containerisation skills; they simply make an API call with the code and we take care of the technicalities and orchestration of services like Kubernetes, EKS and ECS.”
Importantly, BrightSpark also includes a full reporting capability, so customers can see exactly what each job costs in time, compute and money.
With such promising early results, Cloud Fundis is now working on expanding the scope of BrightSpark. “Today it is generalisable to Apache Spark and standard Python jobs,” Morton says. “But it is a fairly universal platform and in future will be able to handle additional data science processes, streaming jobs and Jupyter notebooks. And this will pave the way for more cost savings across many workload applications.”
Exclusively with AWS
Whittal emphasises that the re-platformed solution is still built on standard AWS services such as AWS ECS, AWS EKS and AWS Batch. Cloud Fundis only works with AWS as its cloud provider. “It is a great platform, easy to use, and with great support,” he says.
While developing BrightSpark, the Cloud Fundis team found AWS to be massively supportive and responsive. “We had to lean on AWS support a bit to get the project across the line,” Morton says. “Whenever we ran into issues during our build, they were fast and efficient in fixing them. Some of the functionality that we needed isn’t a common use case for these services, but AWS was proactive in helping us to do it.”
The company believes its relationship with First Distribution will help it develop and further market its BrightSpark service.
About Cloud Fundis
Cloud Fundis is an AWS Select Tier partner and provider of cloud-based analytics, big data, machine learning and DevOps solutions, helping its customers discover the incredible insights they can gain from their data and the efficiencies of automation. Visit Cloud Fundis at www.cloud-fundis.co.za.
About First Distribution
First Distribution is the leading value-added distributor in Africa. Our offering includes a product portfolio of global leading data centre, cloud, networking, and security products and services. This embraces hybrid solutions with access to any combination of on-premises, private cloud and public cloud end-to-end solutions from multiple vendors. The portfolio, together with highly skilled staff to support and manage vendors with an unrivalled level of expertise, ensures that our partners have the key elements required for success. First Distribution additionally hosts one of the largest technology partner-to-partner networks in Africa allowing partners to extend the reach of their solutions across the continent. Start building on AWS today with First Distribution: contact us on +27115402640 or e-mail us at [email protected].
- Read more articles by First Distribution on TechCentral
- This promoted content was paid for by the party or parties concerned