No off-the-shelf ETL processor for metering and billing
We describe an ETL process in data model and data flow for billing. To implement this process we can use an off-the-shelf ETL processing framework (ETL processor) or create custom tools.
The billing process reads data from the Prometheus API
The billing process has an intermediate storage (see Relevant decisions)
In Access metering data we decided that we don’t directly read metering data from the Prometheus API.
There’s two approaches we could take to implement the billing process:
Off-the-shelf ETL processor
We setup an off-the-shelf ETL processor (for example Singer, Airbyte or Apache NiFi) to implement our billing process. We implement the billing process in the framework provided by the ETL processor.
Custom implementation of our ETL process
We implement the billing process as a collection of custom tools which operate on a shared relational database which acts as a data warehouse.
Most off-the-shelf ETL processors don’t natively support ingesting data from a Prometheus API.
Some ETL processors are aimed at non-programmers and their primary way to implement processes is a graphical programming interface. Other processors mainly provide a job scheduling, execution and monitoring platform and provide the user with toolkits to develop custom tools which model their process.
In conclusion, it’s unlikely that we’d significantly reduce the complexity of implementing the business logic of the billing process by using an off-the-shelf ETL processor.
|There will be a future decision regarding the scheduling, execution and monitoring platform for the billing process.|