UCaaS Status Page - 4/14/2025 Call Recording Failure – Incident details

All systems operational

4/14/2025 Call Recording Failure

Resolved
Operational
Started 7 months agoLasted about 22 hours

Affected

Portals and Services

Major outage from 3:31 PM to 6:29 PM, Partial outage from 6:29 PM to 7:25 AM, Operational from 7:25 AM to 1:15 PM

Call Recording (IAD)

Major outage from 3:31 PM to 6:29 PM, Partial outage from 6:29 PM to 7:25 AM, Operational from 7:25 AM to 1:15 PM

Call Recording (PHX)

Major outage from 3:31 PM to 6:29 PM, Partial outage from 6:29 PM to 7:25 AM, Operational from 7:25 AM to 1:15 PM

Updates
  • Resolved
    Resolved
    **Current Status:** * **This incident is considered Resolved** * After testing and confirmation, all call recordings on GRR & LAS are now available, and call recordings on ATL are available **except between 4/14 2 AM and 4/15 3:30 AM ET.** * If a specific call is needed from this time period, you can submit a ticket to [support@oit.co](mailto:support@oit.co) within the **next 6 calendar days** and we can pull that call recording from voipmonitor for you. **Next Steps:** * Major Incident Report will be available within 48 hours. **Next Update:** N/A
  • Monitoring
    Monitoring
    **Update: Monitoring: 4/14/25 Call Recording Failure** **What Is Affected**: Call Recording availability across affected tenants **When It Began**: 2:00 AM, April 14, 2025, ET **Current Status**: * As of 2:45 AM ET on April 15th, call recordings for calls on ATL placed after tonight's maintenance window are showing in the Manager Portal. **Next Steps**: * Call recordings for calls on ATL placed throughout the duration of the incident (2 AM ET on April 14th through 1 AM ET on April 15th) can be retrieved by submitting a support ticket to [support@oit.co](mailto:support@oit.co). * We will continue to monitor for the next 48 hours to ensure call recording functionality remains stable. **Next Update**: 9 AM ET, April 16th, 2025 **Offline Services**: * None **Degraded Services**: * None **Operational Services**: * Inbound Calling – ATL * Outbound Calling - ATL * Inbound Calling – GRR * Outbound Calling - GRR * Inbound Calling – LAS * Outbound Calling - LAS * API * Fax * Messaging * Teammate Connector * Mobile X * Mobile Apps * Integrations We appreciate your patience throughout this process. If you need support, please contact [support@oit.co](mailto:support@oit.co). For more real time status updates as well as discussion please join our Discord (Not official support)
  • Update
    Update

    We will be performing emergency maintenance on our ATL server on April 15th, at 1:00 AM ET, to resolve the call recording failures on ATL.

    During this time, there will be a brief window where calls and device registration will be rerouted to alternate servers.

    Maintenance Start: 1:00 AM ET, April 15th, 2025

    Estimated Time to Complete: 4:00 AM ET, April 15th, 2025 (3 Hours)

    Offline Services

    Inbound Calling – ATL

    Outbound Calling -ATL

    Degraded Services

    None

    Operational Services

    Inbound Calling – GRR

    Outbound Calling -GRR

    Inbound Calling – LAS

    Outbound Calling - LAS

    API

    Fax

    Messaging

    Teammate Connector

    Mobile X

    Mobile Apps

    Integrations

    We appreciate your patience throughout this process. For additional support, please contact support@oit.co.

  • Update
    Update

    Current Status:

    -Call recordings are available for LAS and GRR.

    -This includes past, current, and future call recordings, including recordings that took place during the incident timeline.

    Next Steps:

    -We continue to troubleshoot ATL with vendor support. Call recordings are still unavailable for past, active, and future recordings on ATL at this time.

    -Partners do not need to take action

    Next Update: 5:30 PM, April 14, 2025, ET

  • Update
    Update

    Current Status:

    -Call recordings are available for LAS and GRR.

    -This includes past, current, and future call recordings, including recordings that took place during the incident timeline.

    Next Steps:

    -We continue to troubleshoot ATL with the help of our vendors. Call recordings are still unavailable for past, active, and future recordings on ATL at this time.

    -Partners do not need to take action

    Next Update: 4:30 PM, April 14, 2025, ET

  • Update
    Update

    Current Status:

    • Call recordings are available for LAS and GRR.

    • This includes past, current and future call recordings, and recordings that took place during the incident timeline.

    Next Steps:

    • We are still investigating ATL call recordings. ATL recordings are still unavailable at this time, including active/future call recordings.

    • Partners do not need to take action at this time

    Next Update: 3:30 PM, April 14, 2025, ET

  • Identified
    Identified

    Current Status:

    • Call recordings are now available for LAS and GRR.

    • This includes past, current and future call recordings, and recordings that took place during the incident timeline.

    Next Steps:

    • We are still investigating, as ATL call recordings are still unavailable at this time. Call recordings for ATL are not accessible, this includes active/future call recordings.

    • Partners do not need to take action at this time

    Next Update: 2:30 PM, April 14, 2025, ET

  • Update
    Update

    Current Status: We've engaged vendor support to develop a short-term solution while we determine the root cause.

    Next Steps:

    • Partners do not need to take action at this time

    • We will provide an update for the next steps once the root cause is identified or a workaround is available

    Next Update: 1:30 PM, April 14, 2025, ET

  • Investigating
    Investigating
    At approximately 2:00 AM ET on April 14th, 2025, we completed maintenance on our call recording servers to migrate them to the new data centers in IAD and PHX. At 10:20 AM ET, we identified an issue where call history is not displaying call recordings. Remote call recording storage may also be unavailable, but no other functionality appears to be impacted at this time. --- **What Is Affected:** * **Call Recording** availability across affected tenants --- **When It Began:** 2:00 AM, April 14, 2025, ET --- **Current Status:** * **\[10:45 AM ET, April 14\]** Issue was confirmed by support team; it is currently under investigation by engineering * No known workarounds at this time * Engineering has been engaged to review recording service logs and assess potential causes --- **Next Steps:** * **\[10:45 AM ET, April 14\]** The Engineering team is reviewing logs and service dependencies * Partners do not need to take action at this time * We will provide an update once the root cause is identified, or a workaround is available --- **Next Update:** 12:30 PM, April 14, 2025, ET --- **Offline Services:** * Call Recording --- **Degraded Services:** * None