The backfill of missing data is complete, and data should be up-to-date in your dashboard.
Posted Feb 24, 2016 - 07:33 PST
Update
We are still working to fill the backlog, and expect it to be clear by the end of the day today. New data may be delayed on your dashboard by up to 6 hours.
Posted Feb 23, 2016 - 10:54 PST
Monitoring
We have backfilled approximately half of the missing data from this weekend, and expect to clear the remaining backlog over the next 24 hours.
Posted Feb 22, 2016 - 08:57 PST
Update
New data is now being processed for all accounts. We are beginning to process the backlog, however, which may cause further slowdowns, so new data may be delayed on your dashboard by up to 4 hours.
Posted Feb 21, 2016 - 18:57 PST
Update
We have begun to re-enable our ingress pipeline for a small subset of users, and are working on restarting it for everyone.
Posted Feb 21, 2016 - 18:11 PST
Update
Our ingress pipeline is still disabled for most customers while we recover from the failure of multiple hardware nodes in our cluster.
We plan to re-enable ingress for new data within the next 6 hours, after which we will begin backfilling missing historical data from the past two days. We believe the backfill may take up to three additional days.
We will post another status update when we are able to re-enable ingress for new data.
Posted Feb 21, 2016 - 13:05 PST
Identified
Our systems team has identified the root cause as degraded hardware. We are replacing the affected hardware, and in the meantime rerouting queries to unaffected hardware. Data collection is unaffected, and no data has been lost.
Due to these issues, our ingestion pipeline has become backlogged, so you may notice missing data in your dashboard from earlier today. We expect to have this resolved by end of day on Sunday. We will provide updates here.
Posted Feb 19, 2016 - 12:36 PST
Investigating
Some dashboard queries are failing for a subset of our users. Our infrastructure team is investigating. Data collection is unaffected.