Status Patch Day November 2021 – System Upgrades MCON Managed Services

Dear Sir or Madam,

according to our announcement below the patches for Europe Region and America Region and APAC Region were successfully done.

All systems are up and running. However if you encounter any problem please contact service@mcon-group.support.

Best regards
Your MCON Managed Services Team

Announcement from 4th November 2021:

Dear customer,

In order to keep our server systems updated and secure, we need to apply patches on a regular basis.

The next patch days are scheduled :

Region Europe:

  • Thursday, November 11th 2021 : development starting at 08 am and QA starting at 12 noon*
  • Friday, November 12th 2021 between 0:00 am and 5:00 am: Production systems*

Region America:

  • Tuesday, November 16th 2021, development starting at 08 am and QA starting at 12 noon*
  • Wednesday, November 17th 2021 between 0:00 am and 5:00 am: Production systems*
  •  Region APAC:
  • Thursday, November 18th 2021, development starting at 08 am and QA starting at 12 noon*
  • Friday, November 19th 2021 between 0:00 am and 5:00 am: Production systems*

There may be short downtimes during service or when the operating system restarts.

Best regards

Your MCON Managed Services Team

* The times are valid for the respective timezone

Finished: DC3HAM Maintenance on 4th November 2021 by Lumen

Dear customer,

according to our announcement below, the scheduled network maintenance by our carrier Lumen was finished. All systems are up and running.

However if you encounter any problem please contact:
service@mcon-group.support.

Best regards
Your MCON Managed Services Team

Announcement from 26th October 2021:

Dear Customer,

please be informed about the following  scheduled network maintenance by our carrier Lumen in the Datacenter DC3HAM.

Lumen intends to carry out internal maintenance within its network. This has been designated as ESSENTIAL. The nature of this work is to perform a Network Element software upgrade. During the maintenance window you may experience one period of outage to the maximum specified below.

Lumen would like to apologize for any inconvenience caused by this maintenance.

The maintenance is scheduled for :
from 2021-11-4 23:00 till 2021-11-5 01:00 CET (Central European Time)

We will test the systems after the maintenance window and inform you about the status.

Best regards
Your MCON Managed Service Team

Patch Day November 2021 – System Upgrades MCON Managed Services

Dear customer,

In order to keep our server systems updated and secure, we need to apply patches on a regular basis.

The next patch days are scheduled :

Region Europe:

  • Thursday, November 11th 2021 : development starting at 08 am and QA starting at 12 noon*
  • Friday, November 12th 2021 between 0:00 am and 5:00 am: Production systems*

Region America:

  • Tuesday, November 16th 2021, development starting at 08 am and QA starting at 12 noon*
  • Wednesday, November 17th 2021 between 0:00 am and 5:00 am: Production systems*
  •  Region APAC:
  • Thursday, November 18th 2021, development starting at 08 am and QA starting at 12 noon*
  • Friday, November 19th 2021 between 0:00 am and 5:00 am: Production systems*

There may be short downtimes during service or when the operating system restarts.

Best regards

Your MCON Managed Services Team

* The times are valid for the respective timezone

DC3HAM Maintenance on 4th November 2021 by Lumen

Dear Customer,

please be informed about the following  scheduled network maintenance by our carrier Lumen in the Datacenter DC3HAM.

Lumen intends to carry out internal maintenance within its network. This has been designated as ESSENTIAL. The nature of this work is to perform a Network Element software upgrade. During the maintenance window you may experience one period of outage to the maximum specified below.

Lumen would like to apologize for any inconvenience caused by this maintenance.

The maintenance is scheduled for :
from 2021-11-4 23:00 till 2021-11-5 01:00 CET (Central European Time)

We will test the systems after the maintenance window and inform you about the status.

Best regards
Your MCON Managed Service Team

Status Patch Day October 2021 – System Upgrades MCON Managed Services

Dear Sir or Madam,

according to our announcement below the patches for Region Europe and Region America and Region APAC were successfully done.

All systems are up and running. However if you encounter any problem please contact service@mcon-group.support.

Best regards
Your MCON Managed Services Team

Announcement from 7th October 2021:

Dear customer,

In order to keep our server systems updated and secure, we need to apply patches on a regular basis.

The next patch days are scheduled :

Region Europe:

  • Thursday, October 14th 2021 : development starting at 08 am and QA starting at 12 noon*
  • Friday, October 15th 2021 between 0:00 am and 5:00 am: Production systems*

Region America:

  • Tuesday, October 19th 2021, development starting at 08 am and QA starting at 12 noon*
  • Wednesday, October 20th 2021 between 0:00 am and 5:00 am: Production systems*
  •  Region APAC:
  • Thursday, October 21st 2021, development starting at 08 am and QA starting at 12 noon*
  • Friday, October 22nd 2021 between 0:00 am and 5:00 am: Production systems*

There may be short downtimes during service or when the operating system restarts.

Best regards

Your MCON Managed Services Team

* The times are valid for the respective timezone

Patch Day October 2021 – System Upgrades MCON Managed Services

Dear customer,

In order to keep our server systems updated and secure, we need to apply patches on a regular basis.

The next patch days are scheduled :

Region Europe:

  • Thursday, October 14th 2021 : development starting at 08 am and QA starting at 12 noon*
  • Friday, October 15th 2021 between 0:00 am and 5:00 am: Production systems*

Region America:

  • Tuesday, October 19th 2021, development starting at 08 am and QA starting at 12 noon*
  • Wednesday, October 20th 2021 between 0:00 am and 5:00 am: Production systems*
  •  Region APAC:
  • Thursday, October 21st 2021, development starting at 08 am and QA starting at 12 noon*
  • Friday, October 22nd 2021 between 0:00 am and 5:00 am: Production systems*

There may be short downtimes during service or when the operating system restarts.

Best regards

Your MCON Managed Services Team

* The times are valid for the respective timezone

Resolved: Outage DC3HAM

Dear Customer,

a trivial change on our external firewall with a (normal) subsequent sync to the secondary device caused both firewalls to go into “disabled”
state and not forward any packets anymore.

An deactivation and activation of the High Availability solved the problem.

The connection interruption lasted from 11:08 – 11:41 CEST, we apologize for any inconvenience.

We are in contact with the developers to find the rootcause of this issue.

We apologize for any inconvenience.

Patch Day September 2021 – System Upgrades MCON Managed Services

Dear customer,

In order to keep our server systems updated and secure, we need to apply patches on a regular basis.

The next patch days are scheduled :

Region Europe:

  • Thursday, September 16th 2021 : development starting at 08 am and QA starting at 12 noon*
  • Friday, September 17th 2021 between 0:00 am and 5:00 am: Production systems*

Region America:

  • Tuesday, September 21st 2021, development starting at 08 am and QA starting at 12 noon*
  • Wednesday, September 22nd 2021 between 0:00 am and 5:00 am: Production systems*
  •  Region APAC:
  • Thursday, September 23rd 2021, development starting at 08 am and QA starting at 12 noon*
  • Friday, September 24th 2021 between 0:00 am and 5:00 am: Production systems*

There may be short downtimes during service or when the operating system restarts.

Best regards

Your MCON Managed Services Team

* The times are valid for the respective timezone

Resolved: DC3HAM Storage Outage

RESOLVED

UPDATE 2021-08-28 02:59PM CEST
On Friday 27th August 19:30 a redundant storage cluster in our colocation DC3.HAM was failing during normal operations.
After onsite ananlyis we found that the storage stopped all services due to a suspected split brain error.
As a result of the storage cluster virtual servers running on VMware could not run properly.

The repair of the storage cluster was started immediately after analysis and finished around 28th August 5a.m.
After storage recovery all running virtual servers have been restarted and checked, all production systems have been up and running after 28th August 09:45 a.m.

We are in further analysis of the root cause.

UPDATE 2021-08-28 10:25AM CEST
most systems are back, we are working to fix remaining problems mainly on QA system

UPDATE 2021-08-27 07:30PM CEST
VM storage cluster is currently not working as expected
thus sites and services are not available right now
we are working with high pressure to resolve this issue as fast as possible