UCaaS Status Page - Notice history

Major Incident Report: 2025-07-08 Inbound and Outbound call failures on GRR

At 2:26 PM ET, we received a notification from our NOC monitoring that the NMS service, which handles registration and call processing on GRR crashed. Active calls on this server did drop, and devices on GRR failed over to ATL successfully.

What was Affected:

• Device Registration on GRR

• Active Calls on GRR

When It Began: 06/05/2025 2:26 PM ET

Current Status: This incident is now resolved.

Next Steps:

• Major Incident Report is now available

• MIR report: https://voipdocs.io/announcements/-2025-06-05-registration-failure-on-grr-

MIR now available for Loss of Communication to GRR Server

At 10:30am ET, our monitoring system alerted us to the GRR server being unreachable for two minutes, resulting in device registrations and inbound calls to failover to other servers. Shortly after, the same datacenter lost communication again.

What Was Affected: GRR Server

When It Began: 6/04/2025 10:30 AM ET

Resolution: 

• The degraded circuit was immediately removed from the routing profile to prevent further instability

• Enhanced monitoring was implemented during the recovery period to ensure sustained stability

• The Major Incident Report is now available: https://voipdocs.io/announcements/-2025-06-04-loss-of-communication-to-grr-server

At 10:30am ET, our monitoring system alerted us to the GRR server being unreachable for two minutes, resulting in device registrations and inbound calls to failover to other servers. Shortly after, the same datacenter lost communication again.

What was Affected: GRR Server

When It Began:  10:30 AM ET

Current Status:

• We continue to see stability on GRR's connections after 24 hours of monitoring.

• This incident is now considered Resolved.

Next Steps:

• Major Incident Report will be available within 48 hours.

Current Status: At this time we continue to see stability on GRR's connections. The datacenter will be performing maintenance and repair to the MPLS configuration tonight at 2AM and is expected to be completed before 8AM. No downtime or service interruptions are expected. Our NOC will be monitoring throughout the monitoring phase.

Next Steps:

• Continue to monitor

• Maintenance: Tonight, 2:00 - 8:00 AM ET

Next Update: 1:15 PM ET 6/25/2025

Degraded Services: None

Operational Services: All

What Is Affected: GRR Server

When It Began: 10:30 AM ET

Current Status: We have determined that the loss of traffic was due to a degraded circuit. The offending circuit was removed from the routing profile. Traffic has remained stable since the change. Moving to the monitoring stage.

Next Steps: Continue to monitor

Next Update: 6/5/2025 1:15 PM ET

Degraded Services: None

Operational Services: All

At 11:06AM ET we were alerted by our monitoring tools that WSS connections on GRR, LAS, IAD, and PHX were failing. SNAPmobile Web uses WSS for the SIP connection.

What Is Affected: All SNAPmobile Web Connections on GRR, LAS, IAD, & PHX

When It Began: 11:06AM ET

Current Status:

• This incident is considered Resolved.

• Our vendor resolved an SSL issue with the affected FQDNs

Next Steps:

• Major Incident Report will be available within 48 hours.

Next Update: N/A

Offline Services: N/A

Degraded Services: N/A

What Is Affected: All SNAPmobile Web Connections on GRR, LAS, IAD, & PHX

When It Began: 11:06AM ET

Current Status:

• After working with vendor support, we have successfully applied a fix to resolve WSS connections failing on LAS, GRR, IAD, & PHX.

• Per our internal policy, we will keep all WSS traffic redirected to ATL while we continue to monitor for 24 hours. This will prevent further downtime if the problem reoccurs.

Next Steps:

• Users will need to completely close SNAPmobile Web and reopen it to ensure a new connection. If the problem persists, please clear your browser cache and test again. If after clearing your cache, you still cannot connect, please submit a ticket to support@oit.co

• We understand that some users may experience poor call quality or dropped calls on ATL. If you experience any of these, please email support@oit.co with call examples or call traces so that we can investigate and potentially move affected clients back to their respective cores in advance.

Next Update: 5/31/25 2:00 PM ET

What Is Affected: All SNAPmobile Web Connections on GRR, LAS, IAD, & PHX

When It Began: 11:06 AM ET

Current Status:

• We have temporarily redirected all WSS traffic to ATL, which is confirmed to be working. In the meantime, our engineers will continue to work on a permanent resolution.

Next Steps:

• Users will need to completely close SNAPmobile Web and reopen it to ensure a new connection. If the problem persists, please clear your browser cache and test again. If, after clearing your cache, you still cannot connect, please submit a ticket to support@oit.co

Next Update: 2:06 PM ET