Degraded Performance of Project Management, Editors, API and File Processing Services
Incident Report for Memsource
Postmortem

Introduction

We would like to share more details about the events that occurred with Memsource between 12:15 CEST and 16:06 CEST on July 2, 2021 which led to a partial performance degradation of the Project Management, Editors, API and File Processing services and what Memsource engineers are doing to prevent these sorts of issues from happening again.

Timeline

12:15 PM CEST: Initial long running requests alerts arrive, but are automatically resolved.

13:27 PM CEST: We start working on optimizing the application code.

13:30 PM CEST: More response time alerts arrive, indicating larger parts of the system are being affected by a performance degradation.

14:26 PM CEST: We deploy new servers, including first code optimizations.

16:06 PM CEST: New servers with fixes are fully deployed resulting in a significant decrease in overall system load. Incident is considered to be resolved.

Root Cause

Newly released functionality caused an overall increase of load on our servers.

Actions to Prevent Recurrence

  • We optimized the code that impacted the performance.
  • We also deployed additional servers to bring the overall server load down.

Conclusion

Finally, we want to apologize. We know how critical our services are to your business. Memsource as a whole will do everything to learn from this incident and use it to drive improvements across our services. As with any significant operational issue, Memsource engineers will be working tirelessly over the next coming days and weeks on improving their understanding of the incident and determine how to make changes that improve our services and processes.

Posted Jul 08, 2021 - 14:50 CEST

Resolved
The incident has been resolved.
Posted Jul 02, 2021 - 16:16 CEST
Update
Our engineers found that the API and File Processing Services may also be affected by the issue.
Posted Jul 02, 2021 - 15:50 CEST
Update
Our engineers are continuing to work on a fix for this issue.
Posted Jul 02, 2021 - 15:30 CEST
Identified
Our engineers have identified the issue and are implementing a fix.
Posted Jul 02, 2021 - 15:01 CEST
Investigating
Our engineers are currently investigating why some Memsource services are not available for our users.
Posted Jul 02, 2021 - 14:30 CEST
This incident affected: Memsource (SLA) (API, Editor for Web, File Processing, Project Management).