Streams ‘R’​ Us: Distributed Data Processing in the 21st Century

Plenty is being said about the amount of data that exists and the amount that will be created in our world. More is implied than said about how the processing of data has fundamentally changed over the years and how application architectures and hardware infrastructure have evolved.

The combination of fast and affordable computing and pervasive connectivity has turned data processing on its head. The approach to processing has gone from store-and-process to process-and-store. Couplethis with scale-out architectures for processing and we’re cooking…cooking data like we’ve never done before.

Pervasive Connectivity and Computing

Looking back, the blending of connectivity and computing that underlies both client-server and cloud-mobile opened up a plethora of options to combine compute and network to distribute intelligent processing. We are now able to optimize for costs and responsiveness and realize low-latency processing at the best cost.

We put these choices to work when we choose to develop a native mobile app vs. web/HTML5 mobile app, or when we decide how much processing Amazon Echo does at home vs. in the cloud.

Scaled Out Stream Processing

Taking advantage of this compute-network merger meant we could scale-out the processing required, not just across distributed data centers and cloud, but also to the edges where end-points like wearables, phones, and Alexa exist. Further, we are not processing data in batch, but streaming data (“data-in-motion”) before it is stored.

Our infrastructure now processes streaming data like humans do, on-the-fly. Not after the data is parked in some flash storage equivalent to our sub-conscious.

The net result of stream processing on a scaled-out infrastructure looks like this. We are used to calling it IoT.

If distributed artificial and human intelligence across a labyrinth of connected devices is IoT, so be it.

However, when at my command, Alexa streams Bach’s Cello Concerto performed by Yo Yo Ma, I won’t be thinking IoT. I’ll be thinking Streams ‘R’ Us. That is, if a comparison between biological information processing and computers enters my mind at all!