CloudSight API Outage

Incident Report for CloudSight API

Postmortem

Approximately 17:30 UTC on 28 February, 2017, CloudSight detected issues with receiving blank images to our system and not being able to provide tags to images already in our system.

It was determined that issues upstream with Amazon Web Services (AWS) Simple Storage Service (S3) had experienced a significant outage, which was preventing images submitted to the CloudSight API and by CloudSight apps (CamFind and TapTapSee) from being received and stored on AWS S3, and returned to our main application for tagging said images.

We worked closely with AWS' support to monitor the situation, and created additional instances of our application so that when services became available, approximately at 22:00 UTC on 28 February, 2017, CloudSight was able to handle the queued volume of images to be tagged.

CloudSight experienced a high volume of images once AWS service was restored, but was back to normal operational capacity at approximately 23:00 UTC on 28 February, 2017.

We have continued to monitor this situation and monitor AWS services used by CloudSight, and will do so through 2 March, 2017.

Posted Mar 01, 2017 - 18:21 PST

Resolved

All components for the CloudSight API and cloudsight.ai portal have returned to an operational status based on average throughput and response times. There may still be some outlying performance that are beyond our control, and that we will continue to monitor. However, as of now, the CloudSight API is considered to be fully operational.

Posted Feb 28, 2017 - 15:45 PST

Monitoring

The CloudSight API is returning to normal operation, but still experiencing slightly higher than normal latency. We are continuing to monitor.

Posted Feb 28, 2017 - 14:08 PST

Investigating

Additional third party services have been disrupted again. We are working again with these services to restore operations as quickly as possible.

Posted Feb 28, 2017 - 13:21 PST

Monitoring

Our third party services appear to be mostly restored, and CloudSight API operations are returning to normal, but at a slower than average pace. We will continue to monitor and update as our API recovers and returns closer to average speed.

Posted Feb 28, 2017 - 13:13 PST

Update

Our third party vendors have identified additional issues causing outages, affecting several parts of our applications, one of which continues to be our CloudSight API. We are continuing to monitor and work with our vendors to resolve these issues.

Posted Feb 28, 2017 - 11:48 PST

Update

Additional third party services are having issues, causing additional outages for CloudSight. We are continuing to work with our third party to resolve these issues as quickly as possible.

Posted Feb 28, 2017 - 11:08 PST

Identified

We have identified the issue upstream and our third party service has acknowledged they have discovered the incident and are investigating. We are working closely with them to stay up to date and get this issue resolved as quickly as possible.

Posted Feb 28, 2017 - 10:24 PST

Update

Issues impacting the CloudSight API appear to be upstream from our application. We are continuing to investigate the outage and working jointly with our third party providers to determine a root cause.

Posted Feb 28, 2017 - 10:05 PST

Investigating

At 9:31 a.m. PST, services for CloudSight's API were interrupted. We are currently investigating.

Posted Feb 28, 2017 - 10:04 PST