The combination of fast and affordable computing and pervasive connectivity has turned data processing on its head. The approach to processing has gone from store-and-process to process-and-store. Couplethis with scale-out architectures for processing and we’re cooking…cooking data like we’ve never done before.

Pervasive Connectivity and Computing

Looking back, the blending of connectivity and computing that underlies both client-server and cloud-mobile opened up a plethora of options to combine compute and network to distribute intelligent processing. We are now able to optimize for costs and responsiveness and realize low-latency processing at the best cost.

We put these choices to work when we choose to develop a native mobile app vs. web/HTML5 mobile app, or when we decide how much processing Amazon Echo does at home vs. in the cloud.

Scaled Out Stream Processing

Taking advantage of this compute-network merger meant we could scale-out the processing required, not just across distributed data centers and cloud, but also to the edges where end-points like wearables, phones, and Alexa exist. Further, we are not processing data in batch, but streaming data (“data-in-motion”) before it is stored.

Our infrastructure now processes streaming data like humans do, on-the-fly. Not after the data is parked in some flash storage equivalent to our sub-conscious.

The net result of stream processing on a scaled-out infrastructure looks like this. We are used to calling it IoT.

If distributed artificial and human intelligence across a labyrinth of connected devices is IoT, so be it.

However, when at my command, Alexa streams Bach’s Cello Concerto performed by Yo Yo Ma, I won’t be thinking IoT. I’ll be thinking Streams ‘R’ Us. That is, if a comparison between biological information processing and computers enters my mind at all!

The post Streams ‘R’​ Us: Distributed Data Processing in the 21st Century appeared first on The Cerebrus Group.