Data Gateway

“Data Gateway” ~ reads data from an Aerosense receiver and shoves it into the cloud.

Get Started Quick

Data Flow

The data flow from the aerosense sensor modules looks like this:

Node (edge processor on-blade)
->  Base Station (bluetooth equipment on-tower)
--->  Gateway (data manager and uploader on-tower)
----->  Ingress (Cloud Function to receive data on-cloud)
------->  Google Cloud BigQuery + Google Cloud Store (database / object storage system)
    |---->  Digital Twin (data analysis system)
    |---->  Jupyter Notebooks (data analysis/introspection for researchers)
    |---->  Dashboard (data visualisation for researchers and system installers)

A Node streams data to the Base Station via bluetooth. The Base Station writes the bytestream directly to a serial port. The Gateway (this library) reads the bytestream from the serial port, decodes it and buffers it in local storage. The Gateway then is responsible for:

establishing a connection (websocket) to Ingress and writing the buffered data, or
packaging the data into events and files which are posted to Ingress.

The Gateway is also responsible for managing the buffer and local storage to minimise data loss in the event of internet outages.

The code for the Cloud Function Ingress is also included in this repository.