Content processing issues

Incident Report for Scanii.com

Postmortem

Hello everyone, late yesterday 12/17/19 around 22:30 EST a change made as part of the rollout of our new regions caused servers running in our EU1 region (Ireland) to believe they were running in AP1 (Australia). The issue was addressed today 12/18/19 at 15:30:00 EST with a change that corrected the location identification for those EU1 based servers.

Here is what happened during the incident:

Content processed in Ireland was being incorrectly tagged as being processed in Australia
Processing results (which includes findings and metadata) from Ireland were stored in Australia

Needless to say we’re not happy that this incident happened and that it took so long to identify the problem hence we are taking actions to prevent this from reoccurring in the future.

Posted Dec 18, 2019 - 21:57 EST

Resolved

Issue resolved, and we'll keep monitoring. What we know thus far is that , due to a user error, servers in EU1 were mistakenly claiming to be in AP1 and thus marking files as being processed in AP1 instead of EU1. Files were being processed in the correct regions but results were being stamped as coming from AP1.

At this point we also believe that results (which only include some metadata and the processing result) were also stored in the incorrect region. We'll perform a postmortem to assess how to prevent this problem from happening again in the future.

Posted Dec 18, 2019 - 15:46 EST

Update

We have identified the source of the problem and we're deploying a fix

Posted Dec 18, 2019 - 15:20 EST

Identified

We are investigating issues processing files from this region

Posted Dec 17, 2019 - 22:00 EST

This incident affected: API Endpoints (api-eu1.scanii.com).