Ugh this just reminded me that I ran into this exact issue a couple years ago. We were running jobs every hour to ingest data from an API into our data warehouse. Eventually we got reports from users about having gaps in our data. We dug into it for days trying to find a pattern, but couldn’t pinpoint anything. We were just missing random pieces of data, but our jobs never reported any failures.
Eventually we were able to determine the issue. HTTP 200 with “error: true” in the response. Fml
Ugh this just reminded me that I ran into this exact issue a couple years ago. We were running jobs every hour to ingest data from an API into our data warehouse. Eventually we got reports from users about having gaps in our data. We dug into it for days trying to find a pattern, but couldn’t pinpoint anything. We were just missing random pieces of data, but our jobs never reported any failures.
Eventually we were able to determine the issue. HTTP 200 with “error: true” in the response. Fml