APPUiO Control API Billing Entity as Virtual Resource
Problem
We want to allow users to create and manage billing relevant data (billing entity) on their own using the APPUiO Control API (self-service). We use an ERP which is not Kubernetes aware for all of our billing and accounting needs. Some customers might have other products than APPUiO Cloud and our accounting team needs to edit billing entities in the ERP.
Since billing entities will be edited through both the APPUiO Portal and the ERP, we need to make sure that the billing entities are always in sync.
Proposals
Virtual Resource Stored in ERP
The billing entity is not stored in the APPUiO Control API but fetched on demand from the ERP. This is implemented as an aggregated API server which translates the APPUiO Control API API to the ERP API. The adapter respects the RBAC rules of the APPUiO Control API and only allows users to access their own billing entities.
This solves any synchronization issues as the billing entity is always up-to-date.
The APPUiO Control API might become partly unresponsive if the ERP is down or slow.
Aggregated apiserver empowers you to customize your apiserver to do following things cannot achieved by CR[D]:
Makes your apiserver adopt different storage APIs rather than ETCDv3
Extends long-running subresources/endpoints like websocket for your own resources
Integrates your apiserver with whatever other external systems
apiserver-builder-alpha Readme
-
The API transparently loads the billing entity from the ERP.
-
A user edits the billing entity in the APPUiO Portal.
-
The changes are saved in the ERP through the API.
Custom Resource Stored in Control API
The billing entity is stored in the APPUiO Control API as a custom resource. The billing entity is synchronized with the ERP periodically and on demand.
The controller synchronizing the billing entity with the ERP is implemented as a Kubernetes controller. An implementation based on a controller would create a system with multiple sources of truth. In the general case, there’s no right order for independent updates in such a system.
The controller must have a way to track changes from both the ERP and the APPUiO Control API. The sync state tracking might be inspired by Kubernetes three-way merge patching and resource versioning.
It could be possible to use Webhooks in the ERP to notify the APPUiO Control API about changes. There still needs to be backup mechanism if changes made in the ERP happen during downtime of the APPUiO Control API or vice-versa.
Since the custom resources are stored locally performance is not affected by the ERP.
We still need to implement an aggregated API server since list operations are not supported through Kubernetes RBAC.
-
The controller loads the billing entity from the ERP and stores it in the APPUiO Control API.
-
A user edits the billing entity in the APPUiO Portal.
-
The changes are saved in the control API.
-
The controller compares versions and synchronizes the changes with the ERP.
Rationale
The virtual resource stored in the ERP approach has been chosen because it does not have any synchronization issues. Debugging synchronization issues can be tricky.
If we go for the locally stored CRD approach we still need to implement an aggregate API server for list operations.
The following reasons favor the virtual resource stored in the ERP over the CRD approach:
- Debuggability
-
The virtual resource stored in the ERP approach is easier to debug. The CRD approach requires a lot of synchronization logic and debugging might be tricky. The virtual resource stored in the ERP approach is easier to understand and debug.
- Implementation effort
-
The syncing approach has more open variables and might change substantially if switching to a different ERP. We need to implement an aggregate API server for list operations anyway.