Update 2023-01-25 12:26 CET
Summary of Impact: Between 07:05 UTC and 09:45 UTC on 25 January 2023, customers experienced issues with networking connectivity, manifesting as network latency and/or timeouts when attempting to connect to Azure resources in Public Azure regions, as well as other Microsoft services including M365 and PowerBI.
Preliminary Root Cause: We determined that a change made to the Microsoft Wide Area Network (WAN) impacted connectivity between clients on the internet to Azure, connectivity between services within regions, as well as ExpressRoute connections.
Mitigation: We identified a recent change to WAN as the underlying cause and have rolled back this change. Networking telemetry shows recovery from 09:00 UTC onwards across all regions and services, with the final networking equipment recovering at 09:35 UTC. Most impacted Microsoft services automatically recovered once network connectivity was restored, and we worked to recover the remaining impacted services.
Next Steps: We will follow up in 3 days with a preliminary Post Incident Report (PIR), which will cover the initial root cause and repair items. We’ll follow that up 14 days later with a final PIR where we will share a deep dive into the incident.
You can stay informed about Azure service issues, maintenance events, or advisories by creating custom service health alerts (https://aka.ms/ash-videos for video tutorials and https://aka.ms/ash-alerts for how-to documentation) and you will be notified via your preferred communication channel(s).
***************************************************
Update 2023-01-25 11:48 CET , see also https://azure.status.microsoft/de-de/status
Between 07:05 UTC and 09:45 UTC on 25 January 2023, customers may have experienced issues with networking connectivity, manifesting as network latency and/or timeouts when attempting to connect to Azure resources in Public Azure regions, as well as other Microsoft services including M365, PowerBI.
We’ve determined the network connectivity issue was occurring with devices across the Microsoft Wide Area Network (WAN). This impacted connectivity between clients on the internet to Azure, as well as connectivity between services in datacenters, as well as ExpressRoute connections.
Current Status:
We have identified a recent change to WAN as the underlying cause, and have taken steps to roll back this change. Our telemetry shows consistent signs of recovery from 09:45 UTC onwards across multiple regions and services. Most customers should now see full recovery as WAN networking has recovered fully.
We are working to monitor and ensure full recovery for services that were impacted.
The next update will be in 30 minutes or as soon as we have further information.
This message was last updated at 10:28 UTC on 25 January 2023
***************************************************
Update 2023-01-25 10:37 CET
Starting at 07:05 UTC on 25 January 2023, customers may experience issues with networking connectivity, manifesting as network latency and/or timeouts when attempting to connect to Azure resources in Public Azure regions, as well as other Microsoft services including M365, PowerBI.
We’ve determined the network connectivity issue is occurring with devices across the Microsoft Wide Area Network (WAN). This impacts connectivity between clients on the internet to Azure, as well as connectivity between services in datacenters, as well as ExpressRoute connections. The issue is causing impact in waves, peaking approximately every 30 minutes.
We have identified a recent WAN update as the likely underlying cause, and have taken steps to roll back this update. Our latest telemetry shows signs of recovery across multiple regions and services, and we are continuing to actively monitor the situation.
This message was last updated at 09:36 UTC on 25 January 2023
***************************************************
Dear Customer,
please be informed about the following current announcement on azure.status.microsoft:
“Starting at 07:05 UTC on 25 January 2023, customers may experience issues with networking connectivity, manifesting as network latency and/or timeouts when attempting to connect to Azure resources in multiple regions, as well as other Microsoft services.
We are actively investigating and will share updates as soon as more is known.
This message was last updated at 08:53 UTC on 25 January 2023 ”
Best regards
Your MCON Managed Service Team
