Powerful Data Processing Pipelines with C++20 Coroutines

16:00 - 17:30 Friday 8th April 2022 BST

Intermediate

Data processing is at the core of many applications. A useful abstraction for building such systems is the notion of a data pipeline, composed of a number of distinct processing steps. Designing a general solution for building such pipelines turns out to be surprisingly challenging though, in particular if we want to execute the different processing steps concurrently.

We will start by exploring various examples of processing steps and will see how those introduce different requirements that complicate the design of a general solution. We will explore what a universal composable interface for a generic data processing step could look like. We will then motivate how the resulting system can be simplified further by the introduction of coroutines. We will investigate the different ways of how data can be passed between processing steps with C++20 coroutines and arrive at a system that is both easy to use for developers and efficient at runtime.

This talk will be a hands-on exploration with lots of code examples. No prior knowledge of coroutines is required.