We would like to share more details about the events that occurred with Memsource between 10:42 AM and 12:07 PM CET on the 28th of January, 2020 which led to the disappearance of jobs and projects for linguist users and what Memsource engineers are doing to prevent these sorts of issues from happening again.
10:42 AM CET: Deployment of a new version of Project Management started.
11:13 AM CET: While the deployment is still in progress, the new version is deployed on several servers. The engineering team notices that projects and jobs are not being displayed for some linguists.
11:32 AM CET: The engineering team identified that the new version is causing the problem and decided to revert to the previous version.
12:07 PM CET: The previous version is deployed and linguists are able to access the jobs.
The new version of Project Management requires an update of existing project and job data to make it compatible with newly introduced database changes. The update process took some time and DB queries in the new version were not robust enough to support the partially updated database. The combination of these two issues lead to the data not being fully visible in the new version of the application.
As a reaction to the problems:
Finally, we want to apologize. We know how critical our services are to your business. Memsource as a whole will do everything to learn from this incident and use it to drive improvements across our services. As with any significant operational issue, Memsource engineers will be working tirelessly over the next coming days and weeks on improving their understanding of the incident and determine how to make changes that improve our services and processes.