Stream Applications

In this repository, you will find a collection of components that can meet various data integration use cases and requirements.

The repository’s primary focus is to provide a set of standalone Java functions that can be useful in the end-user applications as-is.

Besides, this repository builds on the Java functions to generate standalone Spring Cloud Stream applications that can run against Spring Cloud Stream’s RabbitMQ or Apache Kafka binder implementations. It is also possible to extend the generator to bundle the Java functions with the other supported binder implementations.

These applications can run standalone or as part of a data flow, such as the one orchestrated using Spring Cloud Data Flow.

Project Structure

The repository includes two major sections - Functions and Applications. The former hosts the various Java functions, and the latter is for generating the standalone Spring Cloud Stream applications and hosting their related components.

The following are the various components of this repository.

• Standalone Java Functions

• Common Core Components for Applications

• Spring Cloud Stream Applications

• Build Parent

• Release Train

Reusable Functions

java.util.Supplier	java.util.Function	java.util.Consumer
Debezium supplier	Aggregator	Analytics
File	Filter	Cassandra
FTP	Header-Enricher	Elasticsearch
	Header-Filter	File
HTTP	HTTP Request	FTP
JDBC	Image Recognition(Tensorflow)
JMS	Object Detection(Tensorflow)	JDBC
Mail	Semantic Segmentation(Tensorflow)	Log
MongoDB	SpEL	MongoDB
MQTT	Splitter	MQTT
RabbitMQ	Task Launch Request	RabbitMQ
AWS S3	Task Launcher	Redis
SFTP		RSocket
Syslog		AWS S3
TCP		SFTP
Time		TCP
Twitter	Twitter	Twitter
Websocket		Websocket
		Wavefront
XMPP		XMPP

Reusable Spring Cloud Stream Applications

Source	Processor	Sink
Debezium supplier	Aggregator	Analytics
File	Bridge	Cassandra
FTP	Filter	Elasticsearch
Groovy	Fiile	HTTP
Header-Enricher	FTP	JDBC
HTTP Request	JMS	Image Recognition(Tensorflow)
JDBC	Load-Generator	Object Detection(Tensorflow)
Log	Mail
MongoDB	MongoDB	Semantic Segmentation(Tensorflow)
MQTT	MQTT	Script
Pgcopy	RabbitMQ	Splitter
RabbitMQ	AWS S3	Transform
Redis	SFTP	Twitter Trend
Router	Syslog
RSocket	TCP
SFTP	Time
Task Launcher	Twitter Message
TCP	Twitter Search
Throughput	Twitter Stream
Twitter Message	Websocket
Twitter Update	XMPP
Wavefront
Websocket

Build

This build requires GraalVM JDK 17 distribution in order to build the script function and processor.

If you are building locally and don’t want to publish to a repository set: export LOCAL=true

Quick Compile

This will compile all jars and publish to local repo.

./quick-compile.sh

Boil the Ocean

This will build and test / integration test all the modules.

./full-build-test.sh

Core

This will build the core functionality and all functions.

./build-core.sh

You can then build the desired apps.

Application

./build-app.sh . applications/sink/log-sink

Note	In order to disable metrics by default there needs to be application properties configured like in default-application.properties. The build-app.sh script will copy default-application.properties into src/main/resources if no application.properties,yml,yaml or json is present.

Additional Resources

Here is a list of resources where you can find out more about using and developing functions and stream applications:

• Function Composition

• Contributing a New Function or Application to this Repository

Code of Conduct

Please see our Code of Conduct