Production Operations

This page describes our rules and principles when working with production systems. As always, it’s important to apply common sense to these rules. It might and will be the case that not everything applies and makes sense all the time. The important thing is that you’re aware of what you’re doing.

This complements and extends the document "Betriebsverfahren für Informations- und Kommunikationstechnik" from our ISMS.

Be aware — Ask

Always be aware of what you’re doing and about the consequences of your actions. If in doubt, ask! It’s better to ask too much than to make mistakes, especially when it comes to working on production environments.

It’s also not always crystal clear what a production system actually is. A service having "test" in its name could still be an important system for the customer doing software tests with it, and therefore from our perspective it’s still a production service. Because of that: If in doubt, treat any service as if it’s a production system unless proven otherwise.

Also make sure you’re aware of the impact your actions can have on other systems, which might rely on the behavior of the system you’re working on.

Shutdown before deletion

First shut down a service before actually deleting it. This allows a user of it to realize that this service is gone.

4-Eye data deletion

When deleting any server or storage systems (virtual and physical) a second person (four-eyes principle) has to verify first that the data on the affected system is really intended to be deleted and then supervise the deletion, so only the intended data gets deleted and nothing else.

The scope includes, but is not limited to:

  • Server deletion on any cloud or physical infrastructure

  • Virtual and physical disks

  • Backups

  • Any other form of storage where there is the possibility of having customer data

Mention the person who verified the deletion in the ticket.

Keep backups after deletion

Keep backups of the deleted data for at least 30 days before also deleting the backups. Unless agreed otherwise with the customer.

Before production — Test!

Always test and validate changes on test environments. Let the customer test changes as well to be sure we’ve done it the right way.

Plan and agree on production changes

Coordinate with the customer for any planned changes to production systems, and keep them in the loop when rolling out changes. Also agree on the procedure for any change potentially directly affecting the service we provide to the customer.

The best way is to inform a few days in advance about planned changes and get the agreement from the customer for the planned changes. This incorporates Before production — Test!.

Be on the safe side — it is better to communicate more rather than too little. Put yourself in the perspective of your communication partner and ask yourself if your explanation can be understood properly — your communication partner might not have the same technical understanding you have.

Mistakes as Learning Moments

We strive to prevent mistakes, but acknowledge unexpected situations.

When mistakes happen, we learn from them in order not to repeat the same mistakes again. For that, we use a blameless post-mortem process to help us reflect and what happened and how we’ll prevent it from happening again. The process starts immediately when mistakes and incidents are discovered.

As we value transparency very highly, we share our findings with the affected parties and, if possible, make them available to other interested parties.