New Message Board Software

Josh Cryer · 2005-07-28 17:18:39

Many many apologies, I hope that wasn't going on all day.

Palomar · 2005-07-28 18:19:27

Someone, or some thing, is doing massive searches on the forum. This has been causing the SQL timeouts today. I am turning off search capablity for Guest users. Sorry Guest users, but you will now have to register to do that.

*Is that what was causing New Mars to not open? At first I thought it was my ISP (dial-up). Rebooted the computer. About an hour later while trying to access the board...it stalled. Then a 3rd time, maybe an hour later. Finally a 4th time, just before going offline for a couple of hours. Window would say it's opening New Mars but it'd just drag very slowly and never open. My ISP was fine and running briskly otherwise; different web pages opening quickly, etc. It was just New Mars.

Glad you took that precaution, Josh.

--Cindy

Josh Cryer · 2005-07-28 19:52:37

I went ahead and shortened the session lengths to (back to an hour), but the day is over with and I'll be applying the unread topics mod tonight, so no one should notice.

John Creighton · 2005-07-28 20:57:34

Someone, or some thing, is doing massive searches on the forum. This has been causing the SQL timeouts today. I am turning off search capablity for Guest users. Sorry Guest users, but you will now have to register to do that.

It could be a search engine. There probably is away to control what pages bots and spiders can see.

Josh Cryer · 2005-07-28 21:01:59

It's very possible, I'll put a robots.txt file up in the /forums directory.

Palomar · 2005-07-29 05:01:45

*Thought I'd mention: "View posts since last visit" feature isn't working.

--Cindy

Rxke · 2005-07-29 06:16:06

ditto; redirects to index page...

Palomar · 2005-07-29 10:00:53

*Hey guys. The board continues to experience major problems. Was inaccessible for nearly 2 hours. This is the most recent error message:

phpBB : Critical Error
Error updating last visit time
DEBUG MODE
SQL Error : 1053 Server shutdown in progress
UPDATE phpbb_users SET user_session_time = 1122651999, user_session_page = 0, user_lastvisit = 1122648656 WHERE user_id = 197
Line : 187
File : sessions.php

Just thought I'd mention it. :? Now I'd better hurry and post those two articles; honestly, the boards are getting very unpredictable and unreliable.

--Cindy

Cobra Commander · 2005-07-29 10:08:27

There be demons here. Exorcism pending.

Josh Cryer · 2005-07-29 13:18:19

Very bad news. NewMars has basically been crashing the SQL server due to, well, the traffic and posts that we have here. Now, it's almost a situation where we can laugh (it's a pretty impressive feat), but it's also a situation where we have to start questioning whether or not we stay with phpBB.

I might have to prune some forum posts. Probably will. Certainly full searching will have to be turned off ("view posts since last visit" and whatnot can stay on).

I'm in contact with out host and we'll work this out. But right now searching is going to have to go.

In the end it's probably my fault.

Palomar · 2005-07-29 13:28:03

Very bad news. NewMars has basically been crashing the SQL server due to, well, the traffic and posts that we have here. Now, it's almost a situation where we can laugh (it's a pretty impressive feat), but it's also a situation where we have to start questioning whether or not we stay with phpBB.
I might have to prune some forum posts. Probably will. Certainly full searching will have to be turned off ("view posts since last visit" and whatnot can stay on).
I'm in contact with out host and we'll work this out. But right now searching is going to have to go.
In the end it's probably my fault.

*Josh: There are a lot of old threads in Free Chat especially that you might want to consider "pruning." Of course I know nothing about running a board like this, so it's just a suggestion. There are old threads all around which maybe could use "pruning" as well.

You're suggesting we'd go back to the Ikonboard? If so, it'll be back to the old name.

BTW, it was 2-1/2 hours since my last post until I was able to access the boards again. Not sure what others are experiencing. Total of at least 4 hours today of inaccessibility.

--Cindy

P.S.: I'm surprised you mention traffic being an issue in this. We've had -less- registered users consistently online than from the Ikonboard, which seemed a lot busier. Also, we've had less UNregistered "Guests" here on average as well. Just thought I'd mention it.

Josh Cryer · 2005-07-29 13:38:10

Cindy, I've been struggling with the support people for about an hour and a half now, I pretty much got lucky here (catching it so soon and not working today). They shut down our database because of way too many queries. I take full blame for this, because I rebuilt the search tables and it doubled the size of our database in just a few minutes. No problems were occuring before I did that. It's totally my fault. And Im so sorry.

Going back to IkonBoard is not really something I want to do, despite phpBB being free, it is still pretty enterprise level. We just have a relatively cheapish host that needs to share its SQL databases with thousands of users.

If I have to I'll pay for a better host out of my own pocket.

clark · 2005-07-29 13:38:41

Random idea...

Could the new forum be part of the problem? It is a new variable, right?

I imagine a search function on the Martian Chronicles forum returns a null value of some sort on searches conducted by non-members. Could that cause a loop of some sort by the bots?

edit- sounds like this isn't the issue. Keep us updated on the options.

Josh Cryer · 2005-07-29 13:44:35

The issue is that we had an 80MB search table that basically was queried, in full, any time someone did a search. For one host, this would be no problem, but we share our SQL database with many many users. Our database is probably one of the most significant ones out there.

This is all because I had to be smart and rebuild the search tables all the way back to 2001. Heh.

clark · 2005-07-29 13:47:09

can you hack it to limit searches by year?

Josh Cryer · 2005-07-29 13:58:46

It's a possiblity, but I'm going to do some pruning anyway. The database problems should be fixed for good. I can't apologize enough for them.

clark · 2005-07-29 14:01:46

What do you mean by 'prune'?

Josh Cryer · 2005-07-29 14:05:51

I mean that I'm going to delete posts, say, to this time in 2003. Maybe this time in 2004. I'll still have them, and we'll bring them back one day when we get a host that can handle them. But I'm waiting for a support response from our host before I do that. Things are looking okay from my end, but I want them to give me the "Everything looks okay."

edit: and if anyone by chance got a "password required" thingy just a minute ago, I was doing something stupid (but I realized my error within half a heartbeat, so the odds of someone seeing it are slim), I'm sorry guys, I'm having a bad time right now.

clark · 2005-07-29 14:14:55

Not that many will care, but there goes some of the exsisting creative content on this board...

Josh Cryer · 2005-07-29 14:19:51

Nah, I'm going to delete threads, not posts. Special threads will remain. And as I said I'll still have all the old posts. Right now we just have to deal with SQL management stuff, which is something I never considered we'd have to do when we moved the forums. I mean, after all, SQL is better than any other database, right?

Palomar · 2005-07-29 14:21:23

I'm sorry guys, I'm having a bad time right now.

*Josh, don't stress. I hope I haven't sounded impatient or anything. It's been a busy day with stat items which need immediate attention in my work, slow news and etc. Sooooooo: I wouldn't have been at the boards much today anyway. My previous posts were simply keeping Mods/Admin appraised of the situation.

That last error message (the long one) did concern me, though: I thought, oh no...don't tell me the entire database has crashed. It didn't.

You're doing a great job, giving it your best, and I'm sure I speak for everyone in thanking you...

--Cindy

John Creighton · 2005-07-29 17:27:06

I don’t know if anyone noticed but when the SQL went down the wiki was still up. I was thinking it is not so bad being down for a bit if people used that time to work on the wiki. However, if we do too many SQL queries we can’t take every other site down with us. So maybe we need to do some trimming as was suggested.

Josh Cryer · 2005-07-29 20:21:05

The forums and wiki have two independent SQL servers. Our host randomly assigns us an SQL host when you add a new database. I've never have it give me the same host twice. So that's why the wiki was fine but the forums were not.

Here's the response from support when I asked why the SQL db was locked:

Hello,
The database was locked to maintain the stability of the server. It was driving up the load and causing it to crash. It's approximately 192MB in size and running 362,000 rows at a time. This had to be stopped. Several of your tables need to be pruned, such as phpbb_search_wordmatch, and phpbb_posts_text.
If you have any more questions or need any further assistance, please feel free to contact us.

Josh Cryer · 2005-07-30 08:54:30

More info:

Hello Mr. Cryer,
Right now, the database looks good so I don't think any more pruning is
necessary. We will let you know if any further problems arise. Thank you
for tending to this issue and getting back to us.

Guest, I'll see if I can't get google to index us. At least once a day google bots are here (to a RDNS lookup on a given Guest IP and it's probably google). I don't have a robots.txt file up, I don't think google will search a forum unless explicitly asked to do so.

I'm looking at simply having the search tables only contain data from the last, say, 3-4 months. 4 years of posts is a lot for even MySQL to search.

New Mars Forums

Announcement

#26 2005-07-28 17:18:39

Re: New Message Board Software

#27 2005-07-28 18:19:27

Re: New Message Board Software

#28 2005-07-28 19:52:37

Re: New Message Board Software

#29 2005-07-28 20:57:34

Re: New Message Board Software

#30 2005-07-28 21:01:59

Re: New Message Board Software

#31 2005-07-29 05:01:45

Re: New Message Board Software

#32 2005-07-29 06:16:06

Re: New Message Board Software

#33 2005-07-29 10:00:53

Re: New Message Board Software

#34 2005-07-29 10:08:27

Re: New Message Board Software

#35 2005-07-29 13:18:19

Re: New Message Board Software

#36 2005-07-29 13:28:03

Re: New Message Board Software

#37 2005-07-29 13:38:10

Re: New Message Board Software

#38 2005-07-29 13:38:41

Re: New Message Board Software

#39 2005-07-29 13:44:35

Re: New Message Board Software

#40 2005-07-29 13:47:09

Re: New Message Board Software

#41 2005-07-29 13:58:46

Re: New Message Board Software

#42 2005-07-29 14:01:46

Re: New Message Board Software

#43 2005-07-29 14:05:51

Re: New Message Board Software

#44 2005-07-29 14:14:55

Re: New Message Board Software

#45 2005-07-29 14:19:51

Re: New Message Board Software

#46 2005-07-29 14:21:23

Re: New Message Board Software

#47 2005-07-29 17:27:06

Re: New Message Board Software

#48 2005-07-29 20:21:05

Re: New Message Board Software

#49 2005-07-30 08:54:30

Re: New Message Board Software

Board footer