On Tuesday July 25, 2017 we experienced elevated data collection error rates and data ingestion latency for about an hour between ~2:45pm and ~3:45pm PDT. During this period, 4.06% of all requests to our web servers were served with an error and this data was never received by Heap. As a result, you may notice approximately 4% of your events missing for this hour in Heap. Data ingestion was also delayed by up to an hour during this window.

We understand that customers rely on Heap to capture all of their customer data and we sincerely apologize for any issues this may cause you. As discussed below, we’re making a variety of improvements to ensure a similar issue does not repeat itself in the future. Please don’t hesitate to reach out to support@heapanalytics.com if you have any questions about how this incident may have affected your dataset or if there’s anything else we can help out with!

What happened?

As part of our ingestion infrastructure, we leverage Apache Kafka to temporarily store data between when it arrives at our collection servers and when we load it into our primary customer-facing databases. At 2:52 pm, automated logging surfaced that Kafka was not properly receiving messages from our collectors but was to be properly handled by our spillover/retry processes. At 3:07pm, we noticed that latency for the ingestion pipeline increased and Kafka errors were surfaced in our logs at 3:12pm. At roughly the same time, automated logging pointed to increasing error rates for our collection service that we notified our customers of via our status page. At 3:29pm, we determined that the issue stemmed from one Kafka broker in particular, and by 3:40pm we had killed all processes and restarted this machine. By 3:45pm, all issues were resolved.

What was the root cause and how will we prevent this from happening again?

The root cause of the issue was a bug in Apache Kafka that can cause individual brokers to become out of sync with the rest of the Kafka cluster. This is known as “split brain”, and can cause lots of problems. In our case, it caused in-sync replica sets to shrink. Since this incident, we have upgraded to a new version of Kafka that includes a fix for the underlying issue that caused this outage. Going forward, we will be updating our version of Kafka (and other services) on a more regular basis to prevent any issues surfacing from underlying bugs with open source software we use in our infrastructure.

Our ingestion infrastructure assumes that Kafka will rarely be down, but we also have some fail-safes for when this does happen. Our first line of defense is buffering logic in our collection servers, which can store up to two weeks of data locally and ingest it later. Unfortunately, this buffering logic exacerbated this outage, because these attempts to replay buffered data synchronized and caused spikes in network utilization, up to 13x our normal traffic. This further destabilized Kafka and caused additional 500s to be returned from our collection servers. (This is known as the “thundering herds” problem.) We are making multiple changes to prevent this issue in the future:

1. Adding a safety check to ensure that Kafka machines are healthy before retrying buffered data
2. More aggressively throttling the number of requests sent during a given interval when retrying buffered data
3. Adding logic to ensure that each collection server retries data in succession rather than all at once

We’re also adding additional alerting to notify us of Kafka issues earlier before they begin to affect other collection services so that we can address potential issues earlier if they do occur.

As always, please don’t hesitate to reach out if you have any questions about this incident or how we’re going to mitigate future incidents. We take data collection uptime extremely seriously and do everything we can to make sure you can trust your Heap dataset. Thanks again for your understanding and we apologize for any complications this incident may have caused!