UCaaS Status Page - Notice history

Major Incident Report: 2025-07-08 Inbound and Outbound call failures on GRR

At 2:26 PM ET, we received a notification from our NOC monitoring that the NMS service, which handles registration and call processing on GRR crashed. Active calls on this server did drop, and devices on GRR failed over to ATL successfully.

What was Affected:

• Device Registration on GRR

• Active Calls on GRR

When It Began: 06/05/2025 2:26 PM ET

Current Status: This incident is now resolved.

Next Steps:

• Major Incident Report is now available

• MIR report: https://voipdocs.io/announcements/-2025-06-05-registration-failure-on-grr-

MIR now available for Loss of Communication to GRR Server

At 10:30am ET, our monitoring system alerted us to the GRR server being unreachable for two minutes, resulting in device registrations and inbound calls to failover to other servers. Shortly after, the same datacenter lost communication again.

What Was Affected: GRR Server

When It Began: 6/04/2025 10:30 AM ET

Resolution: 

• The degraded circuit was immediately removed from the routing profile to prevent further instability

• Enhanced monitoring was implemented during the recovery period to ensure sustained stability

• The Major Incident Report is now available: https://voipdocs.io/announcements/-2025-06-04-loss-of-communication-to-grr-server

At 10:30am ET, our monitoring system alerted us to the GRR server being unreachable for two minutes, resulting in device registrations and inbound calls to failover to other servers. Shortly after, the same datacenter lost communication again.

What was Affected: GRR Server

When It Began:  10:30 AM ET

Current Status:

• We continue to see stability on GRR's connections after 24 hours of monitoring.

• This incident is now considered Resolved.

Next Steps:

• Major Incident Report will be available within 48 hours.

Current Status: At this time we continue to see stability on GRR's connections. The datacenter will be performing maintenance and repair to the MPLS configuration tonight at 2AM and is expected to be completed before 8AM. No downtime or service interruptions are expected. Our NOC will be monitoring throughout the monitoring phase.

Next Steps:

• Continue to monitor

• Maintenance: Tonight, 2:00 - 8:00 AM ET

Next Update: 1:15 PM ET 6/25/2025

Degraded Services: None

Operational Services: All

What Is Affected: GRR Server

When It Began: 10:30 AM ET

Current Status: We have determined that the loss of traffic was due to a degraded circuit. The offending circuit was removed from the routing profile. Traffic has remained stable since the change. Moving to the monitoring stage.

Next Steps: Continue to monitor

Next Update: 6/5/2025 1:15 PM ET

Degraded Services: None

Operational Services: All