Sign up to save your podcasts
Or
Share Data Streams: The Next Fundamental IT Revolution
Share to email
Share to Facebook
Share to X
Share to email
Share to Facebook
Share to X
Or
Data Streams: The Next Fundamental IT Revolution
Our guest on the podcast this week is James Urquhart, cloud computing thought leader and creator of Digital Anatomy.
We discuss how data streams, or real-time stream processing, is the next fundamental architectural and organizational revolution in IT. Today, the capability is available though it is complex to implement at scale. James predicts that within ten years using data streams will become mainstream. Traditionally, an application owns a set of data which comes to rest and is taken somewhere else to correlate with another data set through batch processing. This is an expensive process and can miss key events, which is crucial in industries like banking or finance. What’s available now for data streams takes directly from open-source software and becomes a utility.
How data streams work is that you can start with one stream and decide which functions you want to use on it. You can pass one stream through a different stream, then combine and recombine new streams in a relatively agile way so you can evolve them as you go. You could be adding new data compliance requirements or regulations as they emerge. The application is no longer the focus of how data gets managed, the data becomes the center of attention. There is still a lot about data streams that needs to be built from how to operate them, how to optimally monetize the cost, and what baseline services are required for it to be most useful.
The use of data streams does not eliminate the ability to analyze historical data stores, but allows you to bring AI, machine learning, and pattern recognition into that process. For instance, you can compare two online historical campaigns and do a reasonable prediction of what you’ll expect from a third campaign. This allows for event-processing and anomaly detection with machine learning.
Data stream technology is available now, but you would need a team of very strong developers to get it working at scale. The question is whether the average enterprise can adopt this quickly enough. If adopted, data streams will drive how organizations work and make real-time decisions in the future. It is an exciting trend and we look forward to seeing how real-time stream processing evolves.
View all episodes
By Cloud Technology PartnersOur guest on the podcast this week is James Urquhart, cloud computing thought leader and creator of Digital Anatomy.
We discuss how data streams, or real-time stream processing, is the next fundamental architectural and organizational revolution in IT. Today, the capability is available though it is complex to implement at scale. James predicts that within ten years using data streams will become mainstream. Traditionally, an application owns a set of data which comes to rest and is taken somewhere else to correlate with another data set through batch processing. This is an expensive process and can miss key events, which is crucial in industries like banking or finance. What’s available now for data streams takes directly from open-source software and becomes a utility.
How data streams work is that you can start with one stream and decide which functions you want to use on it. You can pass one stream through a different stream, then combine and recombine new streams in a relatively agile way so you can evolve them as you go. You could be adding new data compliance requirements or regulations as they emerge. The application is no longer the focus of how data gets managed, the data becomes the center of attention. There is still a lot about data streams that needs to be built from how to operate them, how to optimally monetize the cost, and what baseline services are required for it to be most useful.
The use of data streams does not eliminate the ability to analyze historical data stores, but allows you to bring AI, machine learning, and pattern recognition into that process. For instance, you can compare two online historical campaigns and do a reasonable prediction of what you’ll expect from a third campaign. This allows for event-processing and anomaly detection with machine learning.
Data stream technology is available now, but you would need a team of very strong developers to get it working at scale. The question is whether the average enterprise can adopt this quickly enough. If adopted, data streams will drive how organizations work and make real-time decisions in the future. It is an exciting trend and we look forward to seeing how real-time stream processing evolves.