Just saw this thread, allow me to retort.
Firstly, we make no excuses for this omission. Security measures that should have been in place were not. Going forward, the following is true:
1. All keys required for server function are no longer stored permanently on any of our servers and exist solely in memory after they are put into operation
2. All servers have unique short lived certificates and keys generated from our new CA which are rotated
3. Each server certificate has uniquely identifying Common Name + SANs
4. New OpenVPN client configurations enforce server certificate X509 name verification using the common name which is unique.
This beings me to my next point: transparency. Had we followed "standard" industry practices, you would have never known this happened. We could have just sunset the CA with no explanation, under the guise of "new better network",
kinda like PIA did over 6 months after
their servers were seized in Russia, which required people to download new OpenVPN configs. There is no reason to do that, unless your keys were compromised.
Secondly, providers like Nord and Torguard, who had their servers hacked back in 2019, only came clean when their keys hit the web. Otherwise you can bet you would never hear about it.
Thirdly, the above 2 providers, and pretty much every VPN we tested (except IVPN which is doing things correctly, same as we do
now), are vulnerable to the exact same issue, right now. Which you can empirically test yourself, by
fetching their OpenVPN certificates over the wire. Once you do, you will discover the following:
1. Pretty much everyone uses long lived certificates 1-50 years until expiry
2. Pretty much everyone is either deploying the exact same certificate + key to every server OR have unique certificates + keys, but without anything that's uniquely identifying in them. Which makes it useless.
3. Those who DO have unique common name or SANs on their certificates (nord and express for example), fail to actually make use of this data and perform X509 verification at the client (you should see verify-x509-name in the config). Express actually tried to do this, and simply copy/pasted the example from the OpenVPN manual, which is completely useless and has zero security benefits.
Effectively, a compromise of those certs and keys will yield the exact same situation, and considering how opaque the consumer VPN industry is, it likely happened and you just don't know about it.
We made a mistake, we researched the best possible way to mitigate it, and did so in under 3 weeks. It's in production, and you can verify that this is the case by looking at the OpenVPN certs + our configs. This guarantees that the same issue cannot happen, even if the server keys were to be compromised again for some reason. You can also verify that those providers that were previously affected did no research, and implemented an equally insecure "solution". Same goes for most other ones. Sure, some claim that their servers are all "RAM based", but can you truly prove that? The only thing you can prove is how their public facing OpenVPN infrastructure is setup, and that leaves no room for interpretation. If that's implemented incorrectly, what makes you think their RAM solution holds water?