Author: All MCON

Resolved: Connectivity issues in AWS AZ in Region Frankfurt

https://status.aws.amazon.com/

7:24 AM PDT Starting at 5:07 AM PDT we experienced increase connectivity issues for some instances, degraded performance for some EBS volumes and increased error rates and latencies for the EC2 APIs in a single Availability Zone (euc1-az3) in the EU-CENTRAL-1 Region. By 6:03 AM PDT, API error rates had returned to normal levels, but some Auto Scaling workflows continued to see delays until 6:35 AM PDT. By 6:10 AM PDT, the vast majority of EBS volumes with degraded performance had been resolved as well, and by 7:05 AM PDT, the vast majority of affected instances had been recovered, some of which may have experienced a power cycle. A small number of remaining instances and hosted on hardware which was adversely affected by this event and require additional attention. We continue to work to recover all affected instances and have opened notifications for the remaining impacted customers via the Personal Health Dashboard. For immediate recovery, we recommend replacing any remaining affected instances if possible.

6:29 AM PDT We continue to make progress in resolving the connectivity issues affecting some instances in a single Availability Zone (euc1-az3) in the EU-CENTRAL-1 Region. The increased error rates and latencies for the RunInstance and CreateSnapshot APIs have been resolved, as well as the degraded performance for some EC2 volumes within the affected Availability Zone. We continue to work on the remaining EC2 instances that are still impaired as a result of this event, some of which may have experienced a power cycle. While we do not expect any further impact at this stage, we would recommend continuing to utilize other Availability Zones in the EU-CENTRAL-1 region until this issue has been resolved.
6:05 AM PDT We are seeing increased error rates and latencies for the RunInstances and CreateSnapshot APIs, and increased connectivity issues for some instances in a single Availability Zone (euc1-az3) in the EU-CENTRAL-1 Region. We have resolved the networking issues that affected the majority of instances within the affected Availability Zone, but continue to work on some instances that are experiencing degraded performance for some EBS volumes. Other Availability Zones are not affected by this issue. We would recommend failing away from the affected Availability Zone until this issue has been resolved.
5:29 AM PDT We are investigating increased error rates and latencies for the EC2 APIs and connectivity issues for some instances in a single Availability Zone in the EU-CENTRAL-1 Region

Patch Day July 2021 – System Upgrades MCON Managed Services

Dear customer,

In order to keep our server systems updated and secure, we need to apply patches on a regular basis.

The next patch days are scheduled :

Region Europe:

  • Thursday, July 15th 2021 : development starting at 08 am and QA starting at 12 noon*
  • Friday, July 16th 2021 between 0:00 am and 5:00 am: Production systems*

Region America:

  • Tuesday, July 20th 2021, development starting at 08 am and QA starting at 12 noon*
  • Wednesday, July 21st 2021 between 0:00 am and 5:00 am: Production systems*
  •  Region APAC:
  • Thursday, July 22nd 2021, development starting at 08 am and QA starting at 12 noon*
  • Friday, June 23rd 2021 between 0:00 am and 5:00 am: Production systems*

There may be short downtimes during service, or when the operating system restarts.

Best regards

Your MCON Managed Services Team

* The times are valid for the respective timezone

Status Patch Day June 2021 – System Upgrades MCON Managed Services

Dear Sir or Madam,

according to our announcement below the patches for Region Europe and Region America and Region APAC were successfully done.

All systems are up and running. However if you encounter any problem please contact service@mcon-group.support.

Best regards
Your MCON Managed Services Team

Announcement from 2nd June 2021:

Dear customer,

In order to keep our server systems updated and secure, we need to apply patches on a regular basis.

The next patch days are scheduled :

Region Europe:

  • Thursday, June 10th 2021 : development starting at 08 am and QA starting at 12 noon*
  • Friday, June 11th 2021 between 0:00 am and 5:00 am: Production systems*

Region America:

  • Tuesday, June 15th 2021, development starting at 08 am and QA starting at 12 noon*
  • Wednesday, June 16th 2021 between 0:00 am and 5:00 am: Production systems*
  •  Region APAC:
  • Thursday, June 17th 2021, development starting at 08 am and QA starting at 12 noon*
  • Friday, June 18th 2021 between 0:00 am and 5:00 am: Production systems*

There may be short downtimes during service, or when the operating system restarts.

Best regards

Your MCON Managed Services Team

* The times are valid for the respective timezone

while performing regular updates there is an issue on AWS (see below)
we are publishing this because it might have impact on further proceedings during our regular patching

https://status.aws.amazon.com/
1:24 PM PDT We are investigating connectivity issues for some EC2 instances in a single Availability Zone (euc1-az1) in the EU-CENTRAL-1 Region.

https://status.aws.amazon.com/
6:54 PM PDT RESOLVED: Connectivity Issues & API Errors connectivity issues for some EC2 instances in a single Availability Zone (euc1-az1) in the EU-CENTRAL-1 Region has been resolved and the service is operating normally.

 

DC3.HAM incident

we are encountering a problem in our colocation DC3.HAM since around 11:45, some workloads are not reachable. we are analyzing the problem and will update you soonest

update: recovery started at 11:53 and was finished latest 12:00, all systems are back. several VMware hosts lost storage connection, virtual machines have been migrated to other hosts automatically.

update: the interruption in the storage connection could have been caused by a bug in the switch firmware, a related patch will be implemented during the regular patching thursday to friday night

Patch Day June 2021 – System Upgrades MCON Managed Services

Dear customer,

In order to keep our server systems updated and secure, we need to apply patches on a regular basis.

The next patch days are scheduled :

Region Europe:

  • Thursday, June 10th 2021 : development starting at 08 am and QA starting at 12 noon*
  • Friday, June 11th 2021 between 0:00 am and 5:00 am: Production systems*

Region America:

  • Tuesday, June 15th 2021, development starting at 08 am and QA starting at 12 noon*
  • Wednesday, June 16th 2021 between 0:00 am and 5:00 am: Production systems*
  •  Region APAC:
  • Thursday, June 17th 2021, development starting at 08 am and QA starting at 12 noon*
  • Friday, June 18th 2021 between 0:00 am and 5:00 am: Production systems*

There may be short downtimes during service, or when the operating system restarts.

Best regards

Your MCON Managed Services Team

* The times are valid for the respective timezone

Status Patch Day May 2021 – System Upgrades MCON Managed Services

Dear Sir or Madam,

according to our announcement below the patches for Region America, Region Europe and Region APAC were successfully done.

All systems are up and running. However if you encounter any problem please contact service@mcon-group.support.

Best regards
Your MCON Managed Services Team

Announcement from 12th May 2021:

Dear customer,

In order to keep our server systems updated and secure, we need to apply patches on a regular basis.

The next patch days are scheduled :

Region America:

  • Tuesday, May 18th 2021, development starting at 08 am and QA starting at 12 noon*
  • Wednesday, May 19st 2021 between 0:00 am and 5:00 am: Production systems*

Region Europe:

  • Thursday, May 20th 2021 : development starting at 08 am and QA starting at 12 noon*
  • Friday, May 21st 2021 between 0:00 am and 5:00 am: Production systems*
  •  Region APAC:
  • Thursday, May 20th 2021, development starting at 08 am and QA starting at 12 noon*
  • Friday, May 21st 2021 between 0:00 am and 5:00 am: Production systems*

There may be short downtimes during service, or when the operating system restarts.

Best regards

Your MCON Managed Services Team

* The times are valid for the respective timezone

Patch Day May 2021 – System Upgrades MCON Managed Services

Dear customer,

In order to keep our server systems updated and secure, we need to apply patches on a regular basis.

The next patch days are scheduled :

Region America:

  • Tuesday, May 18th 2021, development starting at 08 am and QA starting at 12 noon*
  • Wednesday, May 19st 2021 between 0:00 am and 5:00 am: Production systems*

Region Europe:

  • Thursday, May 20th 2021 : development starting at 08 am and QA starting at 12 noon*
  • Friday, May 21st 2021 between 0:00 am and 5:00 am: Production systems*
  •  Region APAC:
  • Thursday, May 20th 2021, development starting at 08 am and QA starting at 12 noon*
  • Friday, May 21st 2021 between 0:00 am and 5:00 am: Production systems*

There may be short downtimes during service, or when the operating system restarts.

Best regards

Your MCON Managed Services Team

* The times are valid for the respective timezone

Finished: DC3HAM Maintenance at 11th May 2021

Dear customer,

according to our announcement below, the scheduled firewall maintenance was finished. All systems are up and running.

However if you encounter any problem please contact:
service@mcon-group.support.

Best regards
Your MCON Managed Services Team

Announcement from 5th May 2021:

Dear customer,

we have to announce a firewall maintenance in our DC3HAM location in the night of Tuesday May 11th to Wednesday May 12th.

This is an urgent hardware maintenance and can not be postponed.
The maintenance will be carried out between 11pm and 1am CEST.

We do not expect any outages for services, in the worst case redundancy mechanisms take a few minutes to fully failover, but there is equipment running at any given time.

The maintenance is scheduled for :
2021-05-11 from 23:00 till 2021-05-12 01:00 CEST (Central European Summer Time)

We will test the systems after the maintenance window and inform you about the status.

Best regards
Your MCON Managed Service Team

Finished: DC3HAM Maintenance at 7st Mai 2021 by Lumen

Dear customer,

according to our announcement below, the scheduled network maintenance by our carrier Lumen was finished. All systems are up and running.

However if you encounter any problem please contact:
service@mcon-group.support.

Best regards
Your MCON Managed Services Team

Announcement from 30th April 2021:

Dear customer,

please be informed about the following  scheduled network maintenance by our carrier Lumen (former CenturyLink) in the Datacenter DC3HAM.

Lumen intends to carry out internal maintenance within its network. This has been designated as ESSENTIAL. The nature of this work is to carry out fiber cable work. This GCR is to create a joint between former PEC network Frankfurt – Dernbach to Segment F Frankfurt to Katzenelnbogen. During the maintenance window you may experience one period of outage to the maximum specified below.

Lumen would like to apologise for any inconvenience caused by this maintenance.

The maintenance is scheduled for :
2021-05-7 from 00:00 till 07:00 CEST (Central European Time)

We will test the systems after the maintenance window and inform you about the status.

Best regards
Your MCON Managed Service Team