After our database rebuild everything has returned to being stable and operating at full capacity
Posted Mar 03, 2019 - 02:36 UTC
Monitoring
We've entirely rebuilt our database from a backup and everything appears to be stable now. We're continuing to monitor the situation but things appear stable now.
Posted Mar 02, 2019 - 22:28 UTC
Update
We're continuing to see database sync errors between our Read and Write servers. We are recreating our entire database stack now.
Posted Mar 02, 2019 - 22:06 UTC
Identified
We've noticed a few more connection issues and are currently rebuilding some of our database read replicas. There may be degraded performance for a few minutes while the scaling occurs.
Posted Mar 02, 2019 - 21:37 UTC
Monitoring
Write access is restored and we are monitoring the situation.
Posted Mar 02, 2019 - 20:40 UTC
Identified
We are experiencing issues with write access to our databases. This is causing things to appear in the Monitor etc. properly but is preventing you from creating or editing reservations.
Posted Mar 02, 2019 - 20:38 UTC
This incident affected: Services (Operator Panel, VR Launcher and Services).