Security and Vulnerability Handling Process

Critical security vulnerabilities are announced on mailing lists from suppliers, on security newsletters, by MELANI, and by many other sources. This security advisories are analysed and if a vulnerability affects the system directly, the update is planned as soon as possible.

Urgent upgrades are assessed and planned immediately and less urgent vulnerabilities are patched according to our maintenance process.

General

Our maintenance process ensures regular system updates (there are some exceptions on customer’s systems which are maintained according to special agreements). We manage server systems based on Ubuntu and RedHat. These two big players usually provide updates rapidly. With this regular maintenance the known vulnerabilities are eliminated within one week.

Internal security incidents are reported according our ISO27001 certified incident management process.

Our Responsible Ops engineers are supervising our monitoring dashboard during office hours according to our Monitoring Concept. The Responsible Ops engineers are able to react quickly during office hours.

Customers can report vulnerabilities to VSHN according their Service Level Agreement.

We regularly discuss security issues in our vshn.chat #security channel.

Contact VSHN Regarding Security

For all security-related communication please send an email to support@vshn.ch which will create a ticket in our ticket system.

As a customer you could directly create a ticket in our VSHN Portal.

In urgent cases call us additionally at +41 44 545 53 00 (Mon-Fri 0900-1800 CET/CEST) and ask to speak to the security team.

Please get in touch with any incidents about VSHN internal systems, shared services, products like appuio.cloud, our open source code, customer systems or anything else you think we can help - we’re happy to forward requests to a more specific party.

In sensitive cases you can also reach out to some people from the ISM Domain:

We are committed to fixing problems as fast as possible, depending on the severity within 24h, 7 days or 28 days.

We will give credit to reporters if requested and send out some swag (t-shirts, coffee mug, stickers, etc) at our discretion.

Outside Office Hours

Customers with a 24/7 contract can contact us as follows outside office hours:

  1. Create a ticket in control.vshn.net with high priority selected

  2. AND call our 24/7 on-call number.

Vulnerability Process for VSHNeers

Office Hours

Our standard procedure to handle security issues or vulnerabilities.

  1. Someone (customer, employee, Responsible Ops, etc.) becomes aware of a vulnerability and a ticket is created.

  2. Responsible Ops ensures VSHN is working on the ticket.

  3. Responsible Ops checks for a published CVSS score in the NIST Vulnerability search with the CVE number. If a CVSS score is published and has a value of 9 or higher, the following steps have to be taken:

    • Responsible Ops ensures to find someone to be the "Vulnerability Manager", who should have the big picture of the vulnerability and the possibly affected systems.

    • Vulnerability Manager ensures that each affected Team sends a delegate who can work on a solution for their team to the task force.

    • Vulnerability Manager creates a channel in vshn.chat with #CVE-YYYY-XXXXXX and invites all taskforce members.

    • Vulnerability Manager is responsible to provide updates regarding our handling of the vulnerability on our status pages status.vshn.net/ and status.appuio.ch/.

    • All VSHNeers ensure that tickets related to the incident are labeled with the CVE number.

    • Vulnerability Manager creates wiki page below of CVE Responses, and they:

      • Create a filter with all tickets with the CVE as label.

      • Use this page to document vulnerable systems, known mitigations and workarounds, and all relevant information.

    • Vulnerability Manager defines the rhythm of short update stand-ups with all taskforce members, where the status of the mitigation is assessed and next steps are discussed.

    • Vulnerability Manager ensures the communication to customers via tickets.

    • Taskforce members ensure that the technical tasks are handled.

    • The Taskforce assesses if an immediate action is required and schedules changes accordingly. Consider VSHN Canada, someone who is flexible, or as last resort an on-call engineer, to do the actual change if its outside office hours.

  4. If a CVSS score is lower than 9:

Outside Office Hours

VSHN doesn’t actively supervise vulnerability announcements outside office hours.

If we get aware of a critical vulnerability outside office hours, the on-call engineer will assess the issue and has the possibility to call 2nd level on-call engineers, if there is a bigger problem to tackle down.

During on-call we implement workarounds to ensure the vulnerability is mitigated. If no workarounds are possible 1st and 2nd level on-call assess, together with customer where possible, other mitigations such as shutting down the service until more analysis can be done on next business day.

Further analysis of the vulnerability will be done on the next business day.

VSHNeers' Checklist for High Critical Vulnerabilities

If there are new vulnerabilities revealed, such as Remote Code Executions (RCE) which need immediate action, this checklist helps to ensure mitigation is done.

This checklist should be used usually for very urgent unauthenticated remote code execution vulnerabilities with a CVSS score 9 and higher.

Technical Checklist

  • Create a ticket per server for customer communication and steps done for mitigation

  • Put the service behind a VPN if possible

  • Ensure it is not available directly via the Internet

  • If available via Internet and no patch available yet

    • Shut the service down

    • Ensure service does not restart automatically after reboot

    • Ensure service is not restarted via configuration management (e.g. Puppet) or other means like GitOps (e.g. ArgoCD)

    • Ensure someone review your change also with this checklist

  • (?) it is in the nature of such an incident, that we cannot predict everything; discuss with your peers what should be done additionally.

  • (?) …​

Process Checklist

  • Create internal ticket to handle all separate customer / server tickets

  • Ensure an internal communication channel (e.g additional chat channel '#CVE-XXXX-YYY') is created to discuss with all involved VSHNeers

  • Ensure that VSHNeers are informed at #security and/or #technical

  • Make sure all Responsible Ops could support in handling the vulnerability if needed.

Lists

VSHN is subscribed to important security announcement lists, such as: