| IRC status [message #55269] |
Mon, 17 November 2003 15:28  |
 |
Crimson
Messages: 7429
Registered: February 2003
Location: Phoenix, AZ
Karma:
|
General (5 Stars)
ADMINISTRATOR |
|
|
Hi guys,
I just wanted to let you know all that happened this weekend to make IRC hell. It was a conglomeration of coincidences of timing.
There were multiple outages and netsplits caused by:
1) Global Crossing runs part of the backbone of routers leading to the hub. They had some hardware failure this weekend that prevents any of the linked servers from reaching the hub.
2) While the above outage was going on, I linked home.n00bstories.com to irc.forcesofdiversion.com which brought 95% of the users together without services (which were on the unreachable hub).
3) When Global Crossing fixed their problem, I relinked both boxes back to the hub which caused another couple netsplits because I had to disconnect them from each other to link them to the hub.
4) I believe Global Crossing had a couple other minor outages after the big one. Probably due to re-routing traffic or rebooting hardware. Each of these caused short splits.
5) The box hosting the hub suffered from a hard drive problem which caused 3 or 4 lockups over the weekend from Friday night onto Monday morning.
6) The host for the n00bstories box, about 5 weeks ago, moved my box from one UPS to another so that they could do some sort of maintenance on the first UPS. They finished fixing it last week and scheduled a move back on Sunday afternoon. This dropped n00bstories off the internet. Last time they did this, everything restarted automatically as I have it configured to do. For some unknown reason, this didn't happen. I wasn't at home, but I knew only the IRC server would have to be started manually once the box was back up. I asked Scorpio9a to do this, which he did. However, my DNS didn't start, web server didn't start... nothing started correctly, so when I got home about an hour after the reboot, I started these processes back up again.
7) This morning, my shell account for the hub was moved to another box on another network. This caused another split. I also got a new IP, so I had to reconfigure the hub and all the links for the new IP. I also had to recompile the IRCd because the new box is running a higher version of FreeBSD.
 The ForcesOfDiversion box occassionally gets crazy in the head for reasons I have yet to uncover. I think it's when the rest of the network is unstable... the normally-stable FoD ircd decides if everyone else is disconnecting and stuff, he will too. The process just terminates on the Windows box, then is immediately restarted as it's configured to do. Basically as long as the rest of the network is happy, FoD is happy.
9) I made a horrible error choosing Sharpnet to host the "edexcel" shell. I have emailed them to close my account, but who knows if they'll even answer because it takes them days to respond to anything. I can log into the box but it won't connect to the hub... it's like that IP is null-routed. This is the second time it's done this in the last week, so I'm closing the account and leasing another shell somewhere else.
---
Phew... as of this moment, everything is stable, except the 'edexcel' shell which is still lost in cyberspace. I am looking into other places to host that link, because Sharpnet is the worst.
I don't expect you all to read, much less understand all that, but I hope you understand that this wasn't an attack, and wasn't anything I could control. It just happened, and I really appreciate your patience.
I'm the bawss.
|
|
|
|
Search
Help
Members
Register
Login
Home
