New Server Now Online

[A.J]

Not sure if this is a related fault but apparently i'm not allowed to view member profiles.. including my own..

yep, same here.....net-weather 'lite' tonight?

[Paul]

Back up with all the bits and pieces switched on again now - sorry about the extended problems, we've now resized to a much more powerful server which should stand us in good stead.

[echobase]

yep, same here.....net-weather 'lite' tonight?

I think it has something to do with giving the server a break !!

[❄️❄️❄️❄️❄️❄️❄️❄️❄️]

[DAVID SNOW]

Back up with all the bits and pieces switched on again now - sorry about the extended problems, we've now resized to a much more powerful server which should stand us in good stead.

With respect paul, you must have known it would bloody busy in here, so why did you not switch to ( as you say a more powerful server) earlier?

[Paul]

DS, without stress testing a server it's impossible to judge how hard it can be pushed before falling over. We upgraded to this new system last month and it had coped to this point and showed no real signs of slow down or of an upcoming problem prior to it going. We've had an hour or so's worth of downtime tonight to resize upwards and whilst it's unfortunate and frustrating, it's something that no amount of forward planning would have changed.

I'd also add that the problem may not have been the server anyway, there appears to be a database issue somewhere which is why we're still having one or two issues.

[flagpole]

Hi Paul

I can't see any locations in peoples profiles which is a pain with possible snow reports coming in. Is this just temporary because of the downtime or a problem my end?

[❄️❄️❄️❄️❄️❄️❄️❄️❄️]

It must be annoying having to upgrade servers now, but in 6 months time hardly anyone is on(!), apart from during extreme weather events, and even then it's never going to be busier than a cold spell in winter.

[DAVID SNOW]

DS, without stress testing a server it's impossible to judge how hard it can be pushed before falling over. We upgraded to this new system last month and it had coped to this point and showed no real signs of slow down or of an upcoming problem prior to it going. We've had an hour or so's worth of downtime tonight to resize upwards and whilst it's unfortunate and frustrating, it's something that no amount of forward planning would have changed.

I'd also add that the problem may not have been the server anyway, there appears to be a database issue somewhere which is why we're still having one or two issues.

ok,and good luck with the bug solving probs

[meerkat63]

DS, without stress testing a server it's impossible to judge how hard it can be pushed before falling over. We upgraded to this new system last month and it had coped to this point and showed no real signs of slow down or of an upcoming problem prior to it going. We've had an hour or so's worth of downtime tonight to resize upwards and whilst it's unfortunate and frustrating, it's something that no amount of forward planning would have changed.

I'd also add that the problem may not have been the server anyway, there appears to be a database issue somewhere which is why we're still having one or two issues.

Error messages received earlier did indicate a database issue - no additional info. aside from that however.

[Crepuscular Ray]

Apparently, I'm forbidden from viewing member profiles (even my own) and can't send pms, etc. Did I offend someone?

If this gets through, it will be a wonder: the last few attempts fell into a black hole.

Edited January 4, 2010 by crepuscular ray

[Paul]

All these non working features are related to the server issue I'm afraid - there'll be back on when we've identified the problem we're having and sorted it.

[flagpole]

Locations are back now

I just can't view new content. Nevermind, I think I'll turn in anyway. Night all.

[snow raven]

Hi Paul

I can't see any locations in peoples profiles which is a pain with possible snow reports coming in. Is this just temporary because of the downtime or a problem my end?

Hi Flaggy,

Its due to the server problems - this happened a few weeks ago too. Just put a request in the thread to ask people to put their locations in their sig - solved the problem in the main in the SE thread anyway!

[Crepuscular Ray]

All these non working features are related to the server issue I'm afraid - there'll be back on when we've identified the problem we're having and sorted it.

TKS Paul. Why do you put yourself through this stuff? Stupid question probably.

Evening Flaggy (if you are still awake) and SR.

[peterf]

well done Paul,at the end of the day it's no real problem mate.keep up the good work.

[Paul]

Thanks Peter, touch wood everything seems ok on here now, but we'll be double checking everything in the morning to make sure..

Good news of course is that by having the forum on it's own server, the problems tonight didn't affect the rest of the site.

[Paul]

We're powering up an even bigger server currently, this will mean as we switch to it there will be 5 mins or so of downtime within the next hour (ish).

[Paul]

That's the upgrade complete - it should (touch wood) be able to handle just about anything thrown at it.

[Neilsouth]

That's the upgrade complete - it should (touch wood) be able to handle just about anything thrown at it.

Awesome!

Going to need all the power it can get through this week

[FetchCB]

Would that include the snowballs Paul ?

[Paul]

Lol snowballs would be bad!!

Quick reply to adamjones who posted elsewhere (but that's been removed)

Adam, in the last month there was 2 days with around 1-2 hours of downtime in mid december which is because traffic to the forum jumped hugely, so one day we upgraded the server (which at that time was shared with some other parts of the site), then the following day we put it onto a brand new server solely for the forum.

Last night another massive jump in traffic lead to approx 1 hr 30 of downtime during which time we upgrade the server again, this morning we could see that the traffic was again going to increase so instigated another server resize which took 10 minutes of downtime just now.

With the greatest of respect to you, I'd suggest that considering the challenges we're facing in terms of traffic that's not actually so bad at all - no one wants downtime but it is a fact of life particularly when you consider the huge spikes in traffic which can occur when there is severe weather around and the massive growth netweather has seen in recent months overall. We've invested in a cloud based server which means we can now resize to cope with spikes and yes, the pay off will be the odd 5-10 mins downtime while that happens (and it won't be that often at all - certainly not twice in 24 hours as we've done recently) but that's a much better solution than either having a monster server which sits there for 9/10 of the year being expensive but not being needed, as these things aren't cheap, or having a server which isn't as expensive but which may not cope in the busiest times.

I'd also add that during the forum downtime the rest of the site has been fully online..

Paul

PS - A final thought, I've worked something like 19 hours out of the last 24, going to bed at 2am last night having worked all evening on the server and was up at 7am this morning to make sure all was ok again. Karl was still working at 11pm last night too, even though he was sceduled to leave work at 5pm - so it's not like we don't care!!

[NorthNorfolkWeather]

Lol snowballs would be bad!!

Quick reply to adamjones who posted elsewhere (but that's been removed)

Adam, in the last month there was 2 days with around 1-2 hours of downtime in mid december which is because traffic to the forum jumped hugely, so one day we upgraded the server (which at that time was shared with some other parts of the site), then the following day we put it onto a brand new server solely for the forum.

PS - A final thought, I've worked something like 19 hours out of the last 24, going to bed at 2am last night having worked all evening on the server and was up at 7am this morning to make sure all was ok again. Karl was still working at 11pm last night too, even though he was sceduled to leave work at 5pm - so it's not like we don't care!!

Paul,

Thanks for the time and effort you and the rest of the team have put into this work. Anyone who works with computers knows they can be tempermental and, when they can't cope everyone suffers. As you said above, no one wants downtime, but it happens, not even the very big boys of the IT world will guarantee 100%, and whereas some of the members may think Netweather's forum is 'mission critical', I think you deserve a big pat on the back for keeping everything else going.

Thanks a lot and hope the whole team manage to get back to a normal routine asap

[in the vale]

Well said Paul - you are doing a cracking job keeping the site up against such demand. Lots of new users (like myself) and very excited existing ones will inevitably create capacity issues, and you seem to be dealing with them quickly and with little fuss. I manage lots of IT stuff so have some idea what is going on here...

[The PIT]

Oops I see it got over loaded again. Another server required.