The Hack - Aftermath

Supergrobi

Technical Supervisor
Finally - cleaning up after the hack is done, time for a more detailed report to the interested user base. If you're not into this nerd sh*t you can stop reading here.

w.jpg


Prequel
Someone started a thread, asking for a dark theme for Drummerworld forums. Since I'm mostly working at night and because of how my screen looks like most of the time, I absolutely supported this request.

aes70browser.png Texteditor.png

So I talked to Bernhard, offering my help and after having a very fruitful call with him, he gave me administrative credentials for the forums, the webhoster panel and the server. He also asked me if I maybe could take care of some updates pending so I started digging into the setup at that time. I was confronted with quite outdated software - which in fact no one could be blamed for: Bernhard as a drummer is maintaining and providing a stunningly informative and comprehensive website, offering an unparalleled amount of information around maybe all the worlds drummers, but he is no IT guy - how should one know about modsec, EOL, Heartbleed or SSL certificates then. And the webhosters support - even though those guys are highly skilled people - doesn't do anything if not being directly asked for. So it came that the forums software, the webserver, the database and even the OS the server was running on was a single, gigantic security issue.

Some of you might have encountered several shut-downs of the forum before the hack - this was me, slowly digging through all those software updates, one-by-one, always followed by intense testing that everything is still up and running fine. This is an unsurprisingly boring process which takes its time.

giphy.gif


The Hack
At the time of the hack the forums software was updated only partially (since one cannot jump over multiple versions offhandedly but has to follow a time consuming update path), the server was running an outdated OS which ran out of support November 2020, the database was on a version from end of 2008, only the PHP compiler was on the latest stable version in order to allow for updating the forums software. So attacking this forum was kind of a comparably low hanging fruit.

This is my report I gave to the administration staff after we had the server and the forums back on the network that day:

We got hacked today.

Maybe you have seen an announcement set up by one of the administrators, asking if someone could help him out by sending 300 bucks via PayPal. Same was posted on a new thread under the same account. The thread had only one view at that time so seems I spotted it just in time. I started a conversation with the administrator offering my help by sending money but asking for a video chat to confirm it's really the person owning the account and not just a random hacker. He said that he can't chat right now because he's on mobile and that the money is needed for the forums. I got curious, asking for what exactly he needs the money. He answered "for the software" and that he doesn't want me to ask any questions but to send the money, he is already very angry and it's very urgent. At that point it was 100% clear that it's a hack so I immediately picked up the phone and gave Bernhard a call, telling him that we're getting hacked right now and that he should inform the administrator about his account being compromised. At the same time I pulled up the hosters admin panel, adding a high priority ticket asking for immediate shut-down of the server and starting a live chat with the support, while I opened the forums admin panel trying do lock out the compromised account. At the same time the hacker started deleting all the forums, seconds later everything was completely wiped out. At the same time the hoster shut down the server. This was around 14:20 UTC. Bernhard then received a mail telling him that he has to pay 1500 bucks for the forum to come back online within 10 minutes. I mean - LOL?

Since then I had a support session with three guys all in all (they changed shifts along the way), recovering from a back-up from two days ago, cloning the compromised harddrive image for further investigations, updating various software, fiddling with exclusive access from my IP to the server to do various tasks on the SSH console and in the admin panel and finally bring the forum back to the network. Now for me it's a quarter to three in the morning (1:45 UTC) and I just sent the final report to Bernhard.

Let's see how things develop, maybe the attack isn't over yet as the hacker might have got highly pissed. Please keep heads up, the server is under constant monitoring by the hoster, too.

Aftermath
Since then I'm constantly working together with the noticeably awesome support to set up a completely new server with the latest OS running the latest server software, migrating databases, converting content to modern charsets, hardening the server by installing and configuring various security layers and testing everything over and over again - everything with the least amount of outage possible. And today we finally managed to bring the new machine to life, hook it up to the internet and switch over the IP from the old server - all while still being under constant fire from someone trying to get access to the admin panel of the forum. This means that currently we are as secure as can be, although everything is hack-able to some extend.

Issues
Although I already spent hours on testing through all functionality the forum offers it's most likely that there's still some issues left. They normally show up as a pop-up saying something like "Oops we ran into some errors". Please don't hesitate to start a conversation with me in order to report those issues, at best with a timestamp when it happened and some information about what you were doing or trying to achieve.

One already known issue is that we weren't able to convert parts of the database to UTF-8 in time which (saved two full days of work, but also) comes with some minor issues. The most obvious is that signatures containing special chars within the UTF-8 range (which is e.g. emoticons or chars like 𝄆 𝄞 ) are represented as two question marks like so: ?? If your signature contains some of these chars please just edit it to your liking and it will work again afterwards.
 
Last edited:

rhumbagirl

Senior Member
"cloning the compromised harddrive image for further investigations"

You ARE a hack! Seriously, good work. I'm following you for some future discussions :)
 

rhumbagirl

Senior Member
One change I noticed was the Code Insert tool (</>) is located in a different spot than it used to be. I think it used to be an option under an "Insert" menu item.

Also, I'm curious how many years of PC admin experience it takes to do what you do. Thanks
 

Supergrobi

Technical Supervisor
One change I noticed was the Code Insert tool (</>) is located in a different spot than it used to be.
Yes, the latest forum comes with some changes to the text editor we all have to get used to.

Also, I'm curious how many years of PC admin experience it takes to do what you do.
My first machine was a C=64 on which I hacked a drum tracker, able to play three tracks with four bars of 16th, going like "pchhh - kshhh - pchpch - kshhh"

Your location in the signature is ??
Bwaha good one! 😄 Too busy testing..

Thanks guys for all your appreciation!
 

doggyd69b

Well-known member
Finally - cleaning up after the hack is done, time for a more detailed report to the interested user base. If you're not into this nerd sh*t you can stop reading here.

w.jpg


Prequel
Someone started a thread, asking for a dark theme for Drummerworld forums. Since I'm mostly working at night and because of how my screen looks like most of the time, I absolutely supported this request.

View attachment 103519 View attachment 103520

So I talked to Bernhard, offering my help and after having a very fruitful call with him, he gave me administrative credentials for the forums, the webhoster panel and the server. He also asked me if I maybe could take care of some updates pending so I started digging into the setup at that time. I was confronted with quite outdated software - which in fact no one could be blamed for: Bernhard as a drummer is maintaining and providing a stunningly informative and comprehensive website, offering an unparalleled amount of information around maybe all the worlds drummers, but he is no IT guy - how should one know about modsec, EOL, Heartbleed or SSL certificates then. And the webhosters support - even though those guys are highly skilled people - doesn't do anything if not being directly asked for. So it came that the forums software, the webserver, the database and even the OS the server was running on was a single, gigantic security issue.

Some of you might have encountered several shut-downs of the forum before the hack - this was me, slowly digging through all those software updates, one-by-one, always followed by intense testing that everything is still up and running fine. This is an unsurprisingly boring process which takes its time.

giphy.gif


The Hack
At the time of the hack the forums software was updated only partially (since one cannot jump over multiple versions offhandedly but has to follow a time consuming update path), the server was running an outdated OS which ran out of support November 2020, the database was on a version from end of 2008, only the PHP compiler was on the latest stable version in order to allow for updating the forums software. So attacking this forum was kind of a comparably low hanging fruit.

This is my report I gave to the administration staff after we had the server and the forums back on the network that day:



Aftermath
Since then I'm constantly working together with the noticeably awesome support to set up a completely new server with the latest OS running the latest server software, migrating databases, converting content to modern charsets, hardening the server by installing and configuring various security layers and testing everything over and over again - everything with the least amount of outage possible. And today we finally managed to bring the new machine to life, hook it up to the internet and switch over the IP from the old server - all while still being under constant fire from someone trying to get access to the admin panel of the forum. This means that currently we are as secure as can be, although everything is hack-able to some extend.

Issues
Although I already spent hours on testing through all functionality the forum offers it's most likely that there's still some issues left. They normally show up as a pop-up saying something like "Oops we ran into some errors". Please don't hesitate to start a conversation with me in order to report those issues, at best with a timestamp when it happened and some information about what you were doing or trying to achieve.

One already known issue is that we weren't able to convert parts of the database to UTF-8 in time which (saved two full days of work, but also) comes with some minor issues. The most obvious is that signatures containing special chars within the UTF-8 range (which is e.g. emoticons or chars like 𝄆 𝄞 ) are represented as two question marks like so: ?? If your signature contains some of these chars please just edit it to your liking and it will work again afterwards.
About an Hr ago 10:00 I was trying to reply to a poster's comment and got the OOOPs error message and then the site was not available. but I figure it would be back shortly..
 

Supergrobi

Technical Supervisor
About an Hr ago 10:00 I was trying to reply to a poster's comment and got the OOOPs error message and then the site was not available. but I figure it would be back shortly..
Yes, we're still working on some issues popping up from now and then which often means to restart the webserver.
 

felonious69

Well-known member
I just thought you folks were screening your calls. Was starting to get a complex.
Thank you Supergrobi!!!!
AND Drummerworld in general!
 

Bozozoid

Well-known member
You are the man supergrobi.....and a bit under appreciated for what you do!. I really had no idea to what extent your involvement was until now...whhhhoa!. My hats off...thanks for all your intense work!.
 

Bernhard

Founder Drummerworld
Staff member
Markus (Supergrobi) is a cool drummer and also a highly skilled computer specialist within his own company, writing software for some of the biggest industry names in the world.
Unbelievable that he finds time and just for fun did all this work...i'm so grateful for his help..... can't say more because he accepts no flowers....

Bernhard
 
Top