Summary

Last week, between 10/22 and 10/28, Heap accrued significant data latency within our ingestion pipeline resulting in data being delayed in the Heap dashboard and delayed Heap SQL syncs. This ingestion latency increased over the course of last week (see below graph) and was delayed by a maximum of ~16 hours.

On the night of 10/26 (PT), we identified a previously unknown bottleneck in our ingestion pipeline that had been gradually slowing down over the past year. We subsequently deployed a fix addressing this bottleneck which resulted in an immediate decrease in latency. By 2pm on 10/27 (PT), all data was being ingested for all customers normally and latency stabilized at under 5 minutes.

We recognize that our customers rely on Heap to ingest data in near-real-time and we treat latency accrual as a mission critical issue that we strive to prevent. Most importantly, if data latency exceeds multiple hours, Heap SQL syncs may also become delayed (as there is no new data to sync) and this results in other business systems that rely on this data presenting incorrect information. We apologize sincerely for any confusion in reporting this may have caused your organization.

Given the importance of this issue, we’ve included below technical details concerning the underlying issue as well as steps we’re taking to prevent this problem from surfacing again in the future. Please don’t hesitate to reach out to support@heapanalytics.com if you have any questions about how this incident may have affected your reporting or if there’s anything else we can help out with!

Technical Details

As part of our ingestion pipeline, Heap needs to determine which database location the event received should be written to in our primary database. This operation is quite CPU-intensive as it requires searching a master database that contains millions of locations across our entire customer base. Over the past few months, this lookup operation become slower and computationally more expensive due to the following factors:

1. Overall data footprint & resulting database location growth
2. Growth in data being sent to Heap & increased size of ingestion pipeline
3. Addition of a separate database cluster used for testing infrastructural improvements, which resulted in an increased number of locations to consider within this operation

The combination of these factors ultimately manifested themselves last week in Heap’s inability to ingest data fast enough to keep up with inbound data being sent to our systems.

This computationally expensive operation was previously done on a single thread. Once the bottleneck was identified, we changed our logic to parallelize the operation across several threads and cores. This change resulted in increased throughput within our ingestion pipeline to quickly ingest backlogged data and handle data being sent to our systems in near-real-time going foward.

We are instituting the following changes to prevent latency of this type from accruing again the in the future:

1. Adding additional monitoring of bottlenecks in our ingestion pipeline to better detect changes in behavior and begin alerting on leading indicators of throughput issues. Specifically, when operations slow overtime, we’ll have monitoring in place to address these types of problems before they result in customer-facing issues.
2. Increasing the size and performance of our machines that write data to our primary data store to remove any computation bottlenecks within these systems.

As mentioned above, we recognize that our customers rely on Heap to be ingesting data in near-real-time and we treat latency accrual as a mission critical issue that we strive to prevent. Please don’t hesitate to reach out to us at support@heapanalytics.com if you have any questions about this incident, how we plan to prevent similar incidents in the future, or any other questions about Heap!