Data Gaps Incident [Resolved]

Updated by Elizabeth Smith

Incident Report

Final Report: 05/07/2025

Starting Monday, 03/31/2025, clients in ZENTRA Cloud saw data gaps in visual and API graphed data. Data was still present and could be displayed on individual ZC organizations when the server cache was cleared and data recalculated. This Issue affected ZENTRA Cloud 1.0, ZENTRA Cloud 2.0, and the API.

Root cause: An automated data process from a previous code release caused a conflict with the most recent code release. This was a deviation from the SOP.

Fix: On 04/04/2025, the system was restarted, and a new data automation process was implemented to replace the previous one. This was monitored for a few days to verify it was working correctly. By 04/06/2025, we were confident the issue was resolved.

Continuous Improvement:

  • Expanded Deployment Coverage: We have added additional data automation processes to the standard deployment.
  • Process Naming Standardization: All data automation processes are now explicitly named, enabling quick identification and removal of old automation processes before new code is released.

Resolved: April 6th - 17:00 UTC

We have resolved the issue with data gaps in the visual and API calls.

Monitoring: April 6th - 18:00 UTC

We are monitoring the fix applied on April 4th to verify data is behaving the way it should.

Monitoring: April 5th - 18:00 UTC

We are monitoring the fix applied on April 4th to verify data is behaving the way it should.

Identified: April 4th - 18:00 UTC

This issues has been identified and a push to fix has been implemted.

ZENTRA Cloud is seeing data gaps in visual and API graphed data. We have confirmed data is present once the server cache is cleared and data is re-calculated. we are determining why this incident is occurring.

Investigating: April 4th - 02:00 UTC

ZENTRA Cloud is seeing data gaps in visual and API graphed data. We have confirmed data is present once the server cache is cleared and data is re-calculated. we are determining why this incident is occurring.

Investigating: April 3rd - 16:00 UTC

ZENTRA Cloud is seeing data gaps in visual and API graphed data. We have confirmed data is present once the server cache is cleared and data is re-calculated. we are determining why this incident is occurring.

[Resolved] Tilt angle on ATMOS 41 and ATMOS 41W is now available via API.

Investigating: April 3rd - 02:00 UTC

ZENTRA Cloud is seeing data gaps in visual and API graphed data. We have confirmed data is present once the server cache is cleared and data is re-calculated. we are determining why this incident is occurring.

Tilt angle on ATMOS 41 and ATMOS 41W is no longer available via API. we are working to resolve this issue in addition to the cache issue.

Investigating: April 2nd - 018:00 UTC

ZENTRA Cloud is seeing data gaps in visual and API graphed data. We have confirmed data is present once the server cache is cleared and data is re-calculated. we are determining why this incident is occurring.

Investigating: April 1st - 01:00 UTC

ZENTRA Cloud is seeing data gaps in visual and API graphed data. We have confirmed data is present once the server cache is cleared and data is re-calculated. we are determining why this incident is occurring.

Investigating: March 31st - 018:30 UTC

ZENTRA Cloud is seeing data gaps in visual and API graphed data. We have confirmed data is present once the server cache is cleared and data is re-calculated. we are determining why this incident is occurring.


How did we do?