Why the SKA is the ultimate in big data - TechCentral

Why the SKA is the ultimate in big data

Construction on the SKA Karoo site (image: SKA South Africa)

Construction on the SKA Karoo site (image: SKA South Africa)

Telescopes have come a long way since the days when they were all about lone astronomers watching the night sky through their upstairs windows. Today, teams of astrophysicists build and use much more modern instruments, not only to observe light visible to our eyes, but also radio emissions from the universe.

Radio telescopes used to use large single dishes to pick up these emissions, but have since graduated to arrays of antennae. These act as one dish over much a wider area and make it possible to receive signals from further away. We refer to them as radio-interferometric arrays because they study the interference of radio waves between pairs of antennae.

Radio interferometry was developed by the English scientists Antony Hewish and Martin Ryle, who received the Nobel prize in 1974 for using such telescopes to discover pulsars — highly magnetised rotating neutron stars. Pulsars rotate with extreme stability, making them the most precise clocks in the universe.

In 1993, the American astrophysicists Russell Hulse and Joseph Taylor demonstrated the importance of pulsars in understanding our universe by discovering them in a gravitationally coupled pair known as a binary system. This provided the first indirect evidence for gravitational waves, which are an essential unproven part of Einstein’s theory of general relativity.

There have been numerous arrays of radio telescopes developed since the 1970s. The state of the art includes the Low-Frequency Array (Lofar) in western Europe, the Murchison Widefield Array (MWA) in western Australia and the Atacama Large Millimetre Array (Alma) in Chile. These are all just coming online and are regarded as a major step forward in our ability to understand the universe.

Artist's impression of SKA dishes (image: SKA Organisation/Swinburne Astronomy Productions)

Artist’s impression of SKA dishes (image: SKA Organisation/Swinburne Astronomy Productions)

Yet they will be soon followed by the Square Kilometre Array (SKA), which will be made up of an unprecedented number of antennae spread across two continents for the first time (Australia and Africa). The SKA will see the radio sky with unprecedented sensitivity and resolution, enabling us to pick up extremely small and faint objects and probe the more distant universe. It is due to come online around 2023.

It is already the case that interferometric data holds only partial information and does not provide immediately recognisable visual images. This is because the antenna array cannot be dense enough to cover each and every position on the ground. To convert the information into images, complex mathematical algorithms are already used.

To add to this existing complexity, the SKA will record unprecedented volumes of data. To put it in perspective, astronomers anticipate that it will produce as much as 10 times the data as global Internet traffic. Radio astronomical imaging therefore urgently needs to be reinvented in this ultra-precision and big data context.

This is where we come in. We have recently received funding to develop new ways to acquire data with radio telescopes using a recent theory called compressed sensing.

This theory enables us to build up whole pictures with far more fragmented data than before, ensuring that each data point contains the maximum amount of information. We will also design image-recovery algorithms that are scalable to the SKA’s big data regime and can reconstruct ultra-high resolution images of the distant universe.

ET, Einstein and the dark universe
This forthcoming radio telescope throws up numerous exciting possibilities. It will be able to observe the formation of Earth-like planets in other galaxies, and detect possible signals sent by extraterrestrial intelligence. It will also be able to detect amino acids and carbon biomolecules, which are key building blocks for organic life.

The SKA has the potential to produce new pulsar discoveries. It should be sensitive enough to allow tens of thousands of pulsars to be detected, with very good chances of finding one orbiting a black hole for the first time. This would allow us to test Einstein’s theories about what happens in the unexplored strong gravity regime around black holes. We might also be able to directly observe gravitational waves for the first time by monitoring the change of distance between pulsars as waves pass by.

The coming leap forward in radio telescopes will also help us to understand the origins of the universe. Although we can infer the existence of the epoch when the first stars formed from our general understanding of the universe and other observations, we have never observed it directly.


Direct observations of hydrogen emission from this period would tell us a lot about how the first stars formed, and also about cosmology in general, including our understanding of the very first moments following the Big Bang. The Lofar and MWA telescopes will look for this very weak signal. But if they are able to detect it, it would only be in a statistical sense. The SKA will be the first telescope capable of producing images of this epoch when the first stars formed.

Another science goal of the SKA is to develop a deeper understanding of dark energy and dark matter. One way of doing this is to study the distortion of the light from background galaxies by intervening matter as it travels towards us.

In modern cosmology this effect, called weak lensing, has typically been studied with optical telescopes. Yet there is a lot to be gained from similar studies using radio wavelengths. Very high resolution is required, however, so it can’t be done with existing radio telescopes. The SKA will open up this new area, potentially uncovering a deeper understanding of the nature of dark energy and dark matter.

In short, we could be on the verge of a great leap forward in our understanding of the universe. If so, the new theory of compressed sensing could play an essential role in recovering images from tiny fragments of information far in the distant universe.

Beyond this, compressed sensing has a wide range of other applications. For instance we will transfer our techniques from astronomical to biomedical imaging, using compressed sensing to tackle the challenges of fast high-resolution magnetic resonance imaging. This is likely to give researchers a much more detailed understanding of neuronal pathways in the human brain.

  • The ConversationJason McEwen is assistant professor at University College London
  • This article was originally published on The Conversation


  1. Tuesday Is Soylent Green Day on

    SKA will be an important testing ground for SA’s IT industry. Innovation will play a key role.

© 2009 – 2019 NewsCentral Media