The volume of data available to research and industry is ever-increasing.
This increase in data collection is not always matched by comparable increases in data storage, utilization and analysis capabilities. If the space to store the data is limited, then data has to be discarded or not recorded at all. Even when that is not the case, most of the data produced is recorded and stored without ever being analysed.
High Energy Physics (HEP) experiments at the Large Hadron Collider (LHC) at CERN have the ability to produce hundreds of gigabytes of data per second. It is untenable to retain all this data with the current resources; novel techniques are needed in order to extract crucial physics information that would otherwise have to be discarded. Commercial applications face a similar challenge: the amount and complexity of the data grows while the resources and time to take a decision using that information do not scale accordingly. This is especially relevant in transport (self-driving cars, in-vehicle apps for fleet control, traffic prediction apps, and public transport networks) and finance (transaction strategy and fraud detection).
In order to make the most of the available information in a cost-effective way, and to reduce the time to insight, data-taking and data-analysis need to become fast and efficient. Meeting this challenge requires the training of a new generation of researchers with cross-cutting expertise both in problems related to fundamental research and in disruptive algorithms used in industry. They will break from the traditional paradigm of "first collect data, then analyze it" and move towards Real-Time Analysis (RTA) where data collection and analysis become synonymous, so that unprocessed information that would be expensive to store can be discarded. SMARTHEP (Synergies between MAchine learning, Real Time analysis and Hybrid architectures for efficient Event Processing and decision making) is an network that aims to deliver this paradigm shift.