Data Gaps Incident [Resolved]
Incident Report
Final Report: 05/07/2025
Starting Monday, 03/31/2025, clients in ZENTRA Cloud saw data gaps in visual and API graphed data. Data was still present and could be displayed on individual ZC organizations when the server cache was cleared and data recalculated. This Issue affected ZENTRA Cloud 1.0, ZENTRA Cloud 2.0, and the API.
Root cause: An automated data process from a previous code release caused a conflict with the most recent code release. This was a deviation from the SOP.
Fix: On 04/04/2025, the system was restarted, and a new data automation process was implemented to replace the previous one. This was monitored for a few days to verify it was working correctly. By 04/06/2025, we were confident the issue was resolved.
Continuous Improvement:
- Expanded Deployment Coverage: We have added additional data automation processes to the standard deployment.
- Process Naming Standardization: All data automation processes are now explicitly named, enabling quick identification and removal of old automation processes before new code is released.
Resolved: April 6th - 17:00 UTC
We have resolved the issue with data gaps in the visual and API calls.
Monitoring: April 6th - 18:00 UTC
We are monitoring the fix applied on April 4th to verify data is behaving the way it should.
Monitoring: April 5th - 18:00 UTC
We are monitoring the fix applied on April 4th to verify data is behaving the way it should.
Identified: April 4th - 18:00 UTC
This issues has been identified and a push to fix has been implemted.
ZENTRA Cloud is seeing data gaps in visual and API graphed data. We have confirmed data is present once the server cache is cleared and data is re-calculated. we are determining why this incident is occurring.
Investigating: April 4th - 02:00 UTC
ZENTRA Cloud is seeing data gaps in visual and API graphed data. We have confirmed data is present once the server cache is cleared and data is re-calculated. we are determining why this incident is occurring.
Investigating: April 3rd - 16:00 UTC
ZENTRA Cloud is seeing data gaps in visual and API graphed data. We have confirmed data is present once the server cache is cleared and data is re-calculated. we are determining why this incident is occurring.
[Resolved] Tilt angle on ATMOS 41 and ATMOS 41W is now available via API.
Investigating: April 3rd - 02:00 UTC
ZENTRA Cloud is seeing data gaps in visual and API graphed data. We have confirmed data is present once the server cache is cleared and data is re-calculated. we are determining why this incident is occurring.
Tilt angle on ATMOS 41 and ATMOS 41W is no longer available via API. we are working to resolve this issue in addition to the cache issue.
Investigating: April 2nd - 018:00 UTC
ZENTRA Cloud is seeing data gaps in visual and API graphed data. We have confirmed data is present once the server cache is cleared and data is re-calculated. we are determining why this incident is occurring.
Investigating: April 1st - 01:00 UTC
ZENTRA Cloud is seeing data gaps in visual and API graphed data. We have confirmed data is present once the server cache is cleared and data is re-calculated. we are determining why this incident is occurring.
Investigating: March 31st - 018:30 UTC
ZENTRA Cloud is seeing data gaps in visual and API graphed data. We have confirmed data is present once the server cache is cleared and data is re-calculated. we are determining why this incident is occurring.
