UCaaS Status Page - 4/14/2025 Call Recording Failure – Incident details

All systems operational

4/14/2025 Call Recording Failure

Resolved
Operational
Started 30 days agoLasted about 22 hours

Affected

Portals and Services

Major outage from 3:31 PM to 6:29 PM, Partial outage from 6:29 PM to 12:00 AM

Call Recording (IAD)

Major outage from 3:31 PM to 6:29 PM, Partial outage from 6:29 PM to 12:00 AM

Call Recording (PHX)

Major outage from 3:31 PM to 6:29 PM, Partial outage from 6:29 PM to 12:00 AM

Updates
  • Resolved
    Resolved

    Current Status:

    • This incident is considered Resolved

    • After testing and confirmation, all call recordings on GRR & LAS are now available, and call recordings on ATL are available except between 4/14 2 AM and 4/15 3:30 AM ET.

    • If a specific call is needed from this time period, you can submit a ticket to support@oit.co within the next 6 calendar days and we can pull that call recording from voipmonitor for you.

    Next Steps:

    • Major Incident Report will be available within 48 hours.

    Next Update: N/A

  • Monitoring
    Monitoring

    Update: Monitoring: 4/14/25 Call Recording Failure

    What Is Affected: Call Recording availability across affected tenants

    When It Began: 2:00 AM, April 14, 2025, ET

    Current Status:

    • As of 2:45 AM ET on April 15th, call recordings for calls on ATL placed after tonight's maintenance window are showing in the Manager Portal.  

    Next Steps:

    • Call recordings for calls on ATL placed throughout the duration of the incident (2 AM ET on April 14th through 1 AM ET on April 15th) can be retrieved by submitting a support ticket to support@oit.co.

    • We will continue to monitor for the next 48 hours to ensure call recording functionality remains stable.

    Next Update: 9 AM ET, April 16th, 2025

    Offline Services:

    • None

    Degraded Services:

    • None

    Operational Services:

    • Inbound Calling – ATL 

    • Outbound Calling - ATL 

    • Inbound Calling – GRR 

    • Outbound Calling - GRR  

    • Inbound Calling – LAS 

    • Outbound Calling - LAS 

    • API 

    • Fax 

    • Messaging 

    • Teammate Connector

    • Mobile X

    • Mobile Apps 

    • Integrations 

    We appreciate your patience throughout this process. If you need support, please contact support@oit.co. For more real time status updates as well as discussion please join our Discord (Not official support)

  • Update
    Update

    We will be performing emergency maintenance on our ATL server on April 15th, at 1:00 AM ET, to resolve the call recording failures on ATL.

    During this time, there will be a brief window where calls and device registration will be rerouted to alternate servers.

    Maintenance Start: 1:00 AM ET, April 15th, 2025

    Estimated Time to Complete: 4:00 AM ET, April 15th, 2025 (3 Hours)

    Offline Services

    Inbound Calling – ATL

    Outbound Calling -ATL

    Degraded Services

    None

    Operational Services

    Inbound Calling – GRR

    Outbound Calling -GRR

    Inbound Calling – LAS

    Outbound Calling - LAS

    API

    Fax

    Messaging

    Teammate Connector

    Mobile X

    Mobile Apps

    Integrations

    We appreciate your patience throughout this process. For additional support, please contact support@oit.co.

  • Update
    Update

    Current Status:

    -Call recordings are available for LAS and GRR.

    -This includes past, current, and future call recordings, including recordings that took place during the incident timeline.

    Next Steps:

    -We continue to troubleshoot ATL with vendor support. Call recordings are still unavailable for past, active, and future recordings on ATL at this time.

    -Partners do not need to take action

    Next Update: 5:30 PM, April 14, 2025, ET

  • Update
    Update

    Current Status:

    -Call recordings are available for LAS and GRR.

    -This includes past, current, and future call recordings, including recordings that took place during the incident timeline.

    Next Steps:

    -We continue to troubleshoot ATL with the help of our vendors. Call recordings are still unavailable for past, active, and future recordings on ATL at this time.

    -Partners do not need to take action

    Next Update: 4:30 PM, April 14, 2025, ET

  • Update
    Update

    Current Status:

    • Call recordings are available for LAS and GRR.

    • This includes past, current and future call recordings, and recordings that took place during the incident timeline.


    Next Steps:

    • We are still investigating ATL call recordings. ATL recordings are still unavailable at this time, including active/future call recordings.

    • Partners do not need to take action at this time


    Next Update: 3:30 PM, April 14, 2025, ET


  • Identified
    Identified

    Current Status:

    • Call recordings are now available for LAS and GRR.

    • This includes past, current and future call recordings, and recordings that took place during the incident timeline.


    Next Steps:

    • We are still investigating, as ATL call recordings are still unavailable at this time. Call recordings for ATL are not accessible, this includes active/future call recordings.

    • Partners do not need to take action at this time


    Next Update: 2:30 PM, April 14, 2025, ET

  • Update
    Update

    Current Status: We've engaged vendor support to develop a short-term solution while we determine the root cause.


    Next Steps:

    • Partners do not need to take action at this time

    • We will provide an update for the next steps once the root cause is identified or a workaround is available


    Next Update: 1:30 PM, April 14, 2025, ET

  • Investigating
    Investigating

    At approximately 2:00 AM ET on April 14th, 2025, we completed maintenance on our call recording servers to migrate them to the new data centers in IAD and PHX. At 10:20 AM ET, we identified an issue where call history is not displaying call recordings. Remote call recording storage may also be unavailable, but no other functionality appears to be impacted at this time.


    What Is Affected:

    • Call Recording availability across affected tenants


    When It Began:

    2:00 AM, April 14, 2025, ET


    Current Status:

    • [10:45 AM ET, April 14] Issue was confirmed by support team; it is currently under investigation by engineering

    • No known workarounds at this time

    • Engineering has been engaged to review recording service logs and assess potential causes


    Next Steps:

    • [10:45 AM ET, April 14] The Engineering team is reviewing logs and service dependencies

    • Partners do not need to take action at this time

    • We will provide an update once the root cause is identified, or a workaround is available


    Next Update:

    12:30 PM, April 14, 2025, ET


    Offline Services:

    • Call Recording


    Degraded Services:

    • None