Outage: On Friday, August 21st Heap experienced an outage during which app functionality and data collection were unavailable. This outage lasted from 10:53am to 12:15pm PDT.
Root cause: We deployed a bad configuration change to our request proxying layer. Our deployment process is built to abort deploys which contain faulty configuration changes, however due to a bug in our deployment tool, the bad configuration was allowed to roll out to our full fleet of web-tier machines.
We have modified our deployment process so that we are no longer impacted by this issue and are investigating further preventative action.
Remediation:
We are able to reprocess the following requests made to Heap’s pixel endpoint in order to recover data:
Unfortunately we are unable to recover the following events:
We are in the process of launching this data recovery effort, and expect it to be finished no later than Friday, September 4th. We will be sure to update you in the event of any changes to this timeline! Sincere apologies from the entire Heap team for any inconvenience this has caused