Downtime 19/05/2018 – 20/05/2018

Deep Analysis
Why the mod went offline for several hours

Podcrash+ is an ever-growing mod (we just passed 50,000 downloads!) but unlike most mods, when they grow its just success and bug fixing the mod, for us however we have to consider the effect that has on our server that provides many of the anti-cheat checks and also provides many of the features such as IRC in the mod.

So what exactly happened you may be wondering? Lets explain

The Incident
What happened?

At 11pm GMT on 19/05/2018, 10pm UTC and 6pm EST the server became unresponsive to all attempts to access it, showing a non-responsive error message. Checking the statistics showed that there appeared to be issues with bot the CPU and the outbound network traffic.

First, the CPU usage, as you can see from the right image, the server showed to be using a maximum of 400% of its allotted allowance, now whilst it can use extra without performance effects or limitations, it does indicate the likelihood of an issue with the server side code.

A brief explanation of how much 40 petabytes of data is.

1petabyte
20 million
four-drawer filing cabinets filled with text

The next major issue was when we looked at the inbound and outbound connection traffic, we noticed something unusual, the server was sending out nearly 40 petabytes worth of data in one hour right before the crash, now whilst our mod handles a lot of information both inbound and outbound to make sure that you are playing fair and that your competitors are doing the same, 40 petabytes is a LOT of data! So we locked down the inbound and outbound firewall to kill this traffic at the source and stop it from sending out.

The Solution
what we are doing to fix the problem

As soon as it became apparent that there was some issue with the code that powers the server, we reached out to several experienced developers that specialise in data packet handling in Java and are closely working with them to improve the state of the server code, this will hopefully stop anything like this from occurring again in the future, but our team remains alert and watching the server performance during key hours to make sure that we stay online even throughout the implementation of a fix.

As of next patch we will be adding in a new method of handling offline servers which will include a system to move you to a backup server that will handle the traffic whilst we restore service to the old server, this will allow players who are online to remain online and players who are wanting to load up the mod to also continue to do so with as little interruption as possible.

We are truly sorry for the issues that were caused last night and we are constantly striving to make it the best anti-cheat and Minecraft enhancement mod available to the community.

x

We use cookies to give you the best online experience. By agreeing you accept the use of cookies in accordance with our cookie policy.

I accept I decline Privacy Center Privacy Settings