ImmortalSeed - Slaying server instabilty

Site stability

First, let's start with the good news. Ultimately we survived what could have been a catastrophic failure. Bad hardware has been replaced and we are running at full strength again!

Starting last Sunday the primary server began experiencing stability issues. We quickly began working on the issue to determine its cause. It took several days to nail down the issue, which was a RAID card that was given intermittent faults. The fact that the failures were intermittent increased the difficulty in tracking down the issue as it would pass test after test.

To add insult to injury, the faulty controller had also dished up several corrupt backups. Fortunately, our backup scheme protects against this. Unfortunately, it meant we had to fall back to data that was 2 days old.

Moving forward

iS Staff has been hard at work restoring data that has been lost. In fact they have been going full tilt since the crashes began on Sunday to keep things stitched together while we worked on fixing the issue. Thanks everyone for their hard work and dedication! We would not have survived without you.

There is some data that cannot possibly be restored, including any kind of ratio changes that occurred over those missing days. To compensate, we are going to give every user on the site 5GBs upload credit.

Keeping Everyone Informed

During the height of the instability and crashes, we deployed a fail-over page so that everyone could see that we were still here and had not gone anywhere. Additionally, we wanted this page to give updates about what was going on.

We would love to get feed back on this. Were you able to see the fail-over page? If you did was it helpful? What else could we add to the page or what could we do differently? Any suggestions or input would be greatly appreciated.

You can post comments here: https://immortalseed.me/tsf_forums/s...d.php?tid=3716

Donations

A huge extra loud shout out and thanks goes to all those that have donated. Without your help and support, we would not be able to keep the lights on around here and iS would not exist. From the bottom of our hearts we thank you!

Donations were a bit behind last month and again behind the curve this month. Having awesome hardware (when it is not eating RAID controllers) provides us all with a great user experience, but they are also very very money hungry.

If you are able, please consider donating! To sweeten the deal we are doubling the benefits you receive from the non-freeleech options (double the upload, download removal, etc). As always, the doubled rates will be shown in the donation options.

Additionally, we are going to bring back an old tradition: If we reach 100% site-wide free leech will be turned on either for the last week of the month, or the first week of October!

Thank you everyone for making iS the best place to be on the interwebs. You guys rock!