All Systems Operational

About This Site

Service status for Presslabs, Smart Managed WordPress Hosting

Public Websites ? Operational
WordPress admin ? Operational
WordPress admin on Kubernetes Operational
Europe ? Operational
US ? Operational
CDN ? Operational
Standard Support Availability ? Operational
Cache Manager ? Operational
Presslabs Dashboard Operational
Git service Operational
Email delivery Operational
SendGrid SMTP Operational
90 days ago
100.0 % uptime
Today
AWS route53 Operational
90 days ago
100.0 % uptime
Today
Log Search ? Operational
90 days ago
99.96 % uptime
Today
Log Collection ? Operational
90 days ago
99.96 % uptime
Today
Google Cloud Platform Google Compute Engine Operational
Google Cloud Platform Google Kubernetes Engine Operational
Google Cloud Platform Google Cloud Networking Operational
Operational
Degraded Performance
Partial Outage
Major Outage
Maintenance
Major outage
Partial outage
No downtime recorded on this day.
had a major outage
had a partial outage
Scheduled Maintenance
On 12-04-2020 starting with 02:00 EDT for 2 hours, we'll be performing an upgrade to Kubernetes Clusters in US region. During this time, the WordPress Admin can become inaccessible for brief periods of time while we apply patches and work on your backend. The public sites will not be affected and they will work normally for not logged-in users. For any inquiries during this time don't hesitate to contact us at support@presslabs.com.
Posted on Apr 9, 19:58 EEST
On 13-04-2020 starting with 04:00 EEST for 2 hours, we'll be performing an upgrade to Kubernetes Clusters in Europe region. During this time, the WordPress Admin can become inaccessible for brief periods of time while we apply patches and work on your backend. The public sites will not be affected and they will work normally for not logged-in users. For any inquiries during this time don't hesitate to contact us at support@presslabs.com.
Posted on Apr 9, 19:51 EEST
Past Incidents
Apr 10, 2020

No incidents reported today.

Apr 9, 2020

No incidents reported.

Apr 8, 2020

No incidents reported.

Apr 7, 2020
Resolved - Today's incident was caused by a combination of factors, as follows:

1. An upgrade to our API backend service has failed due to a large DB migration. That caused instances to no longer be recognized via the API. This happened despite our automatic Unit Tests and local testing procedures that are supposed to prevent this from happening.
2. While this deployment error happened, the old DB tables for the API were becoming unavailable. Because of that, the API was returning the default data for the docker image that was supposed to run on the GKE cluster for all sites.
3. The API returning an empty docker image caused a full network deployment for all sites on all clusters over GKE, causing wp-admin areas to restart with the "new" image.
4. The default image from wordpress-operator (https://github.com/presslabs/wordpress-operator) was used instead because none was specified by the API, which in turn was not working properly on the latest version of our infrastructure, throwing a Not Found error for wp-admin areas.
5. Since our Backend API service controls the entire GKE deployment process, being down didn't allow us to quickly revert the changes. We had to manually disable the deployment service and manually update the images for each site. Hence the delays in the recovery process.
6. At this point, the automatic deployment workflow is turned off for GKE clusters and we're still working on fully repairing the API backend servers. Most of them are now partially working in normal conditions.

Going forward we'll be taking extra steps in our development/deployment process to prevent such issues from occurring again.
Apr 7, 20:07 EEST
Monitoring - The sites are now fully back online. We'll continue to monitor the status and we're gradually re-enabling cache flushing.
Apr 7, 15:09 EEST
Identified - We're working on getting the issue fixed ASAP.
Apr 7, 14:03 EEST
Apr 6, 2020

No incidents reported.

Apr 5, 2020

No incidents reported.

Apr 4, 2020

No incidents reported.

Apr 3, 2020

No incidents reported.

Apr 2, 2020
Completed - The scheduled maintenance has been completed.
Apr 2, 10:01 EEST
Verifying - Verification is currently underway for the maintenance items.
Apr 2, 09:44 EEST
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Apr 2, 08:00 EEST
Scheduled - We are going to deploy changes in order to improvie the reliability of MySQL connections on the US Kubernetes cluster. During this period, the site wp-admins are expected to be down for brief moments. As always there is no expected impact for non-logged-in users.
Apr 1, 18:17 EEST
Apr 1, 2020

No incidents reported.

Mar 31, 2020

No incidents reported.

Mar 30, 2020

No incidents reported.

Mar 29, 2020
Completed - The scheduled maintenance has been completed.
Mar 29, 02:00 EET
Verifying - Verification is currently underway for the maintenance items.
Mar 29, 01:08 EET
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Mar 29, 00:02 EET
Scheduled - Following the yesterday's incidents we are going to deploy a fix in order to reduce the likelihood of it happening again. During this period, the sites are expected to show brief database conectivity issues. As allways there is no expected impact for non-logged-in users.
Mar 28, 20:41 EET
Mar 28, 2020

No incidents reported.

Mar 27, 2020

No incidents reported.