Website Issues

[JonT]

We are having issues with availability of the website again. You may see "503 Service Unavailable". This is an issue with our hosting service. I have alerted them and asked them to look into it.

[JonT]

Normal service seems to have been resumed, but we are still scratching our heads over exactly what caused the issues in the first place. Good news - it wasn't yet another Denial of Service attack.

Looking more closely at some of our website logs shows some really quite odd things. By a very large margin the majority of traffic to our website is from Singapore, Brazil, Iraq and Russia. These account for more than half of the pages accessed on the website last month. Last time I looked the majority of our members were from UK, Europe, North America and Australasia.

It's a statement of the obvious (I hope) but given the majority of our website and forum is in the public domain, and there is clearly a lot of access from outside our community, please be very careful what you post and also what you include in your profile. You should assume that anything you post here is absorbed by multiple organisations as they sweep through the internet. Information such as your email address is protected, so you don't need to worry about that. But I always think twice before posting any kind of personal information here (other than what Belgian beer I have been drinking!)

At one point I had considered closing all forums to non-members which would mean you would have to be registered and logged in to read or write anything from and to the forum. But this feels extreme and would limit access to others who are just looking for some useful information on rowing (or Belgian beer).

Are we having fun yet?

[JonT]

Famous last words. We still seem to have the problem.

[Mike Channin]

Might be time to try to shut out the bots.
I had to take a similar approach about 16 years ago when I was running a reasonably big e-commerce site - vast majority of the traffic and load was coming from crawlers and bots with no business value. Huge improvement once they were sent packing...

There's definitely a significant issue here with the conflict of interests between wanting to promote open discussion of training, and the dark-web trying to use anything and everything for nefarious purposes...

[JonT]

Might be time to try to shut out the bots.
I had to take a similar approach about 16 years ago when I was running a reasonably big e-commerce site - vast majority of the traffic and load was coming from crawlers and bots with no business value. Huge improvement once they were sent packing...

There's definitely a significant issue here with the conflict of interests between wanting to promote open discussion of training, and the dark-web trying to use anything and everything for nefarious purposes...

You could be right Mike. I'm going to start by tightening up the registration process so we don't get as many bogus accounts being registered and used to generate garbage posts. I am always looking at the logs to see if a lot of traffic is coming from certain ip addresses that I can then block. I'll also have a look at stopping bots/crawlers trawling through the site. One other thing I am thinking about is purging the huge number of registered users that have never posted anything to the forum. These are probably bogus - why register and then do nothing when most of the forum can be accessed anyway?

[JonT]

Apologies folks. You are probably still seeing issues. I'm battling with a number of challenges in terms of unwanted visitors trying to do a number of things to the website including doing huge numbers of searches. Our forum is very large so a search takes up a lot of resources on our server. Eventually it gets flooded and you will see an error message if you try to access the site. I'm shutting off various ways of getting to us. Hopefully I will be able to shut enough of them down for us to get back to some sort of stability.

Jon

[JonT]

Well that was a fairly frantic day and evening yesterday. I have had to be pretty draconian in terms of restricting access to our site for non-registered guests. Things seem to have calmed down this morning with 22,000 visitors to the site yesterday and just 400 so far today. I'll keep an eye on things now and there are a number of other things that I want to fine tune to try to avoid this tricky situation again.

[Wolfmiester]

thanks as always Jon. I hope you still find you "new hobby" interesting
I just approved a post for this topic from "test user" saying "does this work"

[Mike Channin]

Apologies folks. You are probably still seeing issues. I'm battling with a number of challenges in terms of unwanted visitors trying to do a number of things to the website including doing huge numbers of searches. Our forum is very large so a search takes up a lot of resources on our server. Eventually it gets flooded and you will see an error message if you try to access the site. I'm shutting off various ways of getting to us. Hopefully I will be able to shut enough of them down for us to get back to some sort of stability.

Jon

I've seen this sort of issue before.
The requirement is to restrict high-cost operations (such as open searching) so that they can't just be used by bogus register-and-trawl accounts.
Maybe limit full search to only people who have posted a bit. I rarely use it, and I suspect it is infrequently needed under 'normal'/genuine usage. (That said, I could be an outlier, and others may depend on it more; even then, having a limit on only contributors is a good defensive mechanism).

Also, just wanted to say a massive thank-you, Jon. I appreciate how hard this is to do, and how well you have managed to keep things working for us all. We are in your debt!

[Iain]

Many thanks Jon, you are doing sterling work as always!

[fkoene]

Great work again, Jon. Thanks.

[JonT]

Thanks for the thanks folks. Yesterday was a bit grim and I was up to my neck and approaching out of my depth. But we seem to have come through the other side.

Mike - I agree. Search is now restricted and only available to established users of the forum. We are also now closed down completely to large parts of the world. I won’t write too much here about what defences I have put in place. This is all a bit ridiculous given we’re just a bunch of people talking about punishing ourselves with a bit of exercise machinery!

[Draggon]

We re also now closed down completely to large parts of the world. I won’t write too much here about what defences I have put in place. This is all a bit ridiculous given we’re just a bunch of people talking about punishing ourselves with a bit of exercise machinery!

So true, JonT! Thanks for your efforts. The older I get the more I realize why old people from my past were so grumpy sometimes...

[plummy]

Nice work Jon - much appreciated.

[JonT]

I noticed the website was sluggish this evening. We are being bombarded by accesses to the website from Vietnam. I have blocked the specific website being used to hit us and also limited access from Vietnam. Happy days.

[fkoene]

Thanks again for keeping an eye out, Jon