Apache Beam provides a portable API layer for building sophisticated data-parallel processing pipelines that may be executed across a diversity of execution engines, or runners. The core concepts of this layer are based upon the Beam Model (formerly referred to as the Dataflow Model), and implemented to varying degrees in each Beam runner. To help clarify the capabilities of individual runners, we’ve created the capability matrix below.
Individual capabilities have been grouped by their corresponding What / Where / When / How question:
What results are being calculated?
Where in event time?
When in processing time?
How do refinements of results relate?
For more details on the What / Where / When / How breakdown of concepts, we recommend reading through the Streaming 102 post on O’Reilly Radar.
Note that in the future, we intend to add additional tables beyond the current set, for things like runtime characteristics (e.g. at-least-once vs exactly-once), performance, etc.
How to read the tables
Tools we are comparing
Properties
Does this tool have this property?Yes/Partially/No/Unverified