Temporarily Slow Queries

Incident Report for Heap

Resolved

All clear!
Posted Feb 07, 2015 - 00:15 PST

Update

Our data processing infrastructure is still backfilling incoming data, and events may be backlogged by up to 12 hours for a subset of customers.

No data has been lost, and we expect data to be current by EOD Friday.

We're working on a number of features to help our query and ingestion infrastructure scale more gracefully, and also investing in new hardware.
Posted Feb 06, 2015 - 13:36 PST

Update

Our data processing infrastructure is still backfilling incoming events, so events might be backlogged for up to 12 hours for a subset of users.

No data has been lost, and we expect to clear the backlog by 9am PST.
Posted Feb 06, 2015 - 05:41 PST

Identified

Query speeds should be back to normal, but recent data may still be backlogged. No data has been lost.

We're working on backfilling data now and expect to have everything up-to-date by 10 PM PST.
Posted Feb 05, 2015 - 17:52 PST

Investigating

You may notice that reports have been loading more slowly today than normal. This is caused by downtime on some of our data-processing servers, which is increasing load to the rest of our cluster.

We're actively digging into the issue now, and we hope to bring query speed back to normal by the end of the day.

Data collection is unaffected.
Posted Feb 05, 2015 - 14:22 PST