IOT Service Interruption
Incident Report for Trimble Cloud
Postmortem

Acknowledgment of interruptions in the Trimble IoT Services on October 5, 2019. For more information see the Incident Analysis:

https://docs.google.com/document/d/1M_Ob2tiMGkjzUO-OEsrM-5QW9NjUlZF1jD1B_jMeDxI/edit#

Posted Oct 07, 2019 - 11:45 UTC

Resolved
we are closing the incident since the actual IoT Message pipeline is working as expected and there is no impact
for customers
Posted Oct 05, 2019 - 03:15 UTC
Identified
Update from AWS : We have identified the root cause of the errors in scheduled event creation. We can confirm that AWS Lambda functions that are invoked in response to scheduled CloudWatch events are not getting invoked successfully. We are actively working towards recovery.

IoT Status : IoT message pipeline operation is normal and only probes are impacted. We are closely following with AWS.
Posted Oct 05, 2019 - 02:52 UTC
Update
AWS team has confirmed that there are issues in Cloudwatch rules. Our probes trigger based on Cloudwatch rules and that's why we are facing this issue. we are closely following up with AWS. We will notify once we hear back from them.
Posted Oct 05, 2019 - 01:41 UTC
Update
We confirm the issue in the probe, No impact to IoT message pipeline.
Posted Oct 05, 2019 - 01:16 UTC
Update
IoT Message pipeline is normal. The issue seems to be in the probe. We are investigating this and update you soon about the progress of a resolution to this issue.
Posted Oct 05, 2019 - 00:58 UTC
Investigating
Our team is responding to alerts and investigating a disruption in IOT APIs. We sincerely apologize for the service interruption, and will provide an update soon about the progress of a resolution to this issue.
Posted Oct 05, 2019 - 00:26 UTC
This incident affected: IoTData (PTO, ESP, LMU-EU).