TCEX order replay server

The purpose of this server is to replay historical trade messages as if it were simulating production data flow at requested replay rate (1x, 5x, 60x, etc). This server currently loads a static set of trades from a file, and stream them on-demand via WebSocket.

The server hosts the subscription and unsubscription API for clients, and the demo webpage displays the simple data visualization as the client receives the replayed order messages over subscribed webSocket.

The replay server runs on http://localhost:8080/.

Installation

git clone git@github.com:denniswon/tcex.git

cd tcex

cp .env.example .env

Commands

# runs the initial setup script
make setup

# generate protobufs
make proto_gen

# installs backend go mod dependencies
go mod tidy
# builds backend
make build
# run backend
./tcex

# or to build & run together
make run

# run frontend
cd demo; yarn dev;

# builds and runs both back and frontend locally on separate processes
make demo

High-Level Architecture

The webserver has been designed with concurrency and scalability in mind. That is, it tries to minimize the number and durations of files being opened on the server for processing. Also, it utilizes processing queue kappa architecture for processing order replays in a streaming manner. In particular, the system uses Redis cache and pubsub based processing queues, one for processing trades input file and another for handling actual order replays.

This way, the server is able to handle multiple requests concurrently without blocking any of them while minimizing the resources used and modularity among the subcomponents for easier incremental optimization in the future.

The server is built in Golang, using Gin http server. This project uses goroutines, go channels, and Threadpool for concurrency and pipelining.

Frontend

• Next.js
• React
• Typescript
• Styled Components
• UiReact
• Klinechart

Demo

Kline mode with replay rate of x60

tcex-demo-encoded.mp4

Replay Order at x60 rate (demo video below trimmed in the middle due to github readme upload size limit)

tcex-demo-x60-skipped.mov

Replay Order at x600 rate

tcex-demo-x600.mp4

Replay Order at x0.1 rate

tcex-demo-encoded.mp4

Subscribing with Order Replay Requests

For requesting and listening to orders being replayed, connect to /v1/ws endpoint using websocket client library & once connected, send subscription request with payload ( JSON encoded )

{
  "type": "subscribe",
  "name": "order", // "order" or "kline"
  "filename": "trades.txt", // optional, defaults to 'trades.txt' if not supplied.
  "replay_rate": 60, // optional, defaults to 60 for x60 replay rate.
  "granularity": 60 // optional, defaults to 60. used only for "kline" requests. in seconds.
}

Subscription confirmeation response ( JSON encoded )

{
  "code": 1,
  "message": "Subscribed to <subscription_id>"
}

Real-time notification about orders being replayed at requested rate:

{
  "price": "1347.41",
  "quantity": 200,
  "aggressor": "ask",
  "timestamp": 1722527801638
}

Real-time Kline OHLC data of order replay:

{
  "timestamp": 1722527800000, // bucket start time in unix timestamp
  "low": 1340.0, // lowest price during the bucket interval
  "high": 1348.41, // highest price during the bucket interval
  "open": 1340.0, // opening price (first trade) in the bucket interval
  "close": 1347.41, // closing price (last trade) in the bucket interval
  "volume": 5, // net quantity volume of trading activity during the bucket interval
  "turnover": 587647.4, // total usd volume of trading activity during the bucket interval
  "granularity": 60 // granularity field is in "seconds"
}

Cancel subscription:

{
  "type": "unsubscribe",
  "id": "<subscription_id>" // subscription id returned from subscription request above
}

Unsubscription confirmation response:

{
  "code": 1,
  "message": "Unsubscribed from <subscription_id>"
}