Data Processing

Pedro Henriques

In this post, we'll dive into the world of data processing, exploring concepts such as streaming processing, batch processing, real-time processing, and micro-batching.

Data Ingestion: Where It Al Begins 🥣

The journey of data processing starts with data ingestion, the phase where we gain control over the data flow. During this critical stage, we establish connections with data sources and channel data into our processing pipelines. Here, we shape the way data will be processed, gaining insights into its scale and complexity. The data we have to deal with is inherently streaming, data is nearly always produced and updated continuously at its source. The difference is how we process the data, and provide it for downstream systems.

Batch Processing 🧱

The most common approaches to data processing is batch processing. In this method, data is collected over time and processed in large chunks at scheduled intervals. While batch processing might introduce some latency, it is a robust technique for handling significant volumes of data. The process has a distinct start and end point, and it doesn't process data in a continuous flow.

Streaming Processing 🍃

Refers to the real-time or near-real-time processing of data. Streaming processing emerges as a powerful technique for handling data that is generated and updated continuously at its source. In contrast to batch processing, which accumulates data over time for scheduled processing, streaming processing treats data as a constant flow of events or records. As new data arrives, it is processed incrementally in small units, ensuring timely insights.

Micro Batching (Near real-time) ⚖️

A compelling compromise between traditional batch processing and real-time streaming is micro-batching. In this approach, data is collected and processed in small, finite-sized batches at regular intervals or upon reaching a predefined data volume. While not true real-time streaming, micro-batching significantly reduces processing latency compared to traditional batch processing. Despite working with discrete batches, micro-batching bridges the gap between real-time streaming and batch processing.

Real-time Processing'⏰

Real-time processing represents a subset of streaming processing, prioritizing minimal latency and instantaneous response times. In a true real-time processing system, each event is swiftly processed without delay, catering to dynamic conditions and enabling immediate actions.

Dependencies and Constraints

Both batch and streaming processing approaches come with dependencies and constraints that warrant careful consideration.

Upstream

  • • Data Availability
  • • Extraction Method
  • • Data Size per Extraction
  • • Data Structure Changes

Downstream

  • • Align processing with downstream systems' transaction timing.

Adherence to Data Contract Requirements

  • • Fulfill data contract specifications, covering aspects like latency, structure, and availability.

Decoding Real-Time Data Processing: What to Consider and Why

The technology landscape is undergoing a transformation towards democratizing real-time streaming processes. Organizations aspire to access data rapidly, circumventing the complexities and costs associated. However, we haven't reached that tipping point yet. Implementing a streaming solution requires a thorough evaluation of trade-offs. Here are some factors that need to be considered:

  • • Does the use case in question require the processing of streaming data in real time?
  • • Is this processing required at the millisecond, second level?
  • • Are the downstream systems prepared to receive this data at the same speed as it is created and sent?
  • • What is the speed of data generation and the size of data I can pull from the data sources?
  • • Will this streaming approach cost more in terms of time, money, maintenance, and downtime, than a batch alternative?
  • • Do we have the knowledge and expertise within the organization to do this?

Despite these issues, you should think about whether real-time streaming is a current need that will bring tangible benefits to the organization, or whether you want it because it's trendy, regardless of all the advantages it could bring.

Considerations and conclusion

Most common use cases don't require real-time streaming and work perfectly well with batch processing, or even using a micro batching approach. Other use cases will require a straightforward stream approach. Embracing the right technology is advantageous when it aligns with objectives and use cases. However, when it doesn't, simplicity remains a valuable guiding principle.

In a world where data insights drive innovation, understanding the nuances of data processing approaches is crucial. Whether you're riding the wave of streaming processing, harnessing real-time capabilities, sticking with the reliability of batch processing, or choosing the right compromise, you should always take informed decisions.

I hope you enjoyed it, see you soon. 👋