No off-the-shelf ETL processor for metering and billing

Problem

We describe an ETL process in data model and data flow for billing. To implement this process we can use an off-the-shelf ETL processing framework (ETL processor) or create custom tools.

Relevant requirements

  • The billing process reads data from the Prometheus API

  • The billing process has an intermediate storage (see Relevant decisions)

Relevant decisions

  • In Access metering data we decided that we don’t directly read metering data from the Prometheus API.

Proposals

There’s two approaches we could take to implement the billing process:

  1. Off-the-shelf ETL processor

    We setup an off-the-shelf ETL processor (for example Singer, Airbyte or Apache NiFi) to implement our billing process. We implement the billing process in the framework provided by the ETL processor.

  2. Custom implementation of our ETL process

    We implement the billing process as a collection of custom tools which operate on a shared relational database which acts as a data warehouse.

Decision

We create a custom implementation of our ETL process for billing.

Rationale

Most off-the-shelf ETL processors don’t natively support ingesting data from a Prometheus API.

Some ETL processors are aimed at non-programmers and their primary way to implement processes is a graphical programming interface. Other processors mainly provide a job scheduling, execution and monitoring platform and provide the user with toolkits to develop custom tools which model their process.

In conclusion, it’s unlikely that we’d significantly reduce the complexity of implementing the business logic of the billing process by using an off-the-shelf ETL processor.

There will be a future decision regarding the scheduling, execution and monitoring platform for the billing process.