Jump to content

Board Performance

SDS

By SDS
in The Stadium Wall Archives

 Share

Recommended Posts

  • Replies 82
  • Created
  • Last Reply

Top Posters In This Topic

Popular Days

Top Posters In This Topic

Popular Days

DC Tom Hall of Famer

DC Tom

Posted
Posted (edited)

Money isn't the issue - I appreciate the offers, but that's not what I'm asking for. It's really just a matter of the past content being accessible or if I need to throw more hardware at the solution.

 

Given that the most practical (by far) access to past content in the time frame you're talking about is the search function, the real question becomes: how much less useful does the search function become without that past content?

 

First step towards answering that is answering the question: how often does the search function get used? Any metrics on that?

 

Also, you might start by deleting some of the older, larger, and/or more useless threads. Like Last Post Wins (which is still in the database). Or GBID's "Hot for..." threads. No realistic reason anyone's referring back to those.

 

Edit: also, search for anything with the author = 'DC Tom' and content like '%You're an idiot%'. That'll recover some space right quick...do us both some good, actually.

 

I heard it's Beerball's fault for meddling in football (message board) related decisions.

 

Beerball was heard to say "Just go ahead and fire me..."

Edited by DC Tom
Link to comment
Share on other sites
BringBackFergy Hall of Famer

BringBackFergy

Posted
Posted

 

 

Given that the most practical (by far) access to past content in the time frame you're talking about is the search function, the real question becomes: how much less useful does the search function become without that past content?

 

First step towards answering that is answering the question: how often does the search function get used? Any metrics on that?

 

Also, you might start by deleting some of the older, larger, and/or more useless threads. Like Last Post Wins (which is still in the database). Or GBID's "Hot for..." threads. No realistic reason anyone's referring back to those.

 

Edit: also, search for anything with the author = 'DC Tom' and content like '%You're an idiot%'. That'll recover some space right quick...do us both some good,

Whatever the price is to save the "Mario Williams" thread, I'm in. That was epic and a true rite of passage for many of us.
Link to comment
Share on other sites
Clippers of Nfl Veteran

Clippers of Nfl

Posted
Posted

Money isn't the issue - I appreciate the offers, but that's not what I'm asking for. It's really just a matter of the past content being accessible or if I need to throw more hardware at the solution.

Usually when someone says its not a money issue, it is a money issue! How much?

Jk.

I know of archived posts, but never go to review them. Like we really "Cutler" about an opinion from 7 years ago?

 

Instead of post count, just keep the year the poster joined. "Joined 20.."

 

 

Link to comment
Share on other sites
OCinBuffalo All Pro

OCinBuffalo

Posted
Posted (edited)

If you suspect it is database issues, many times it is disk I/O. If you host the database on the same server as the web server, chances are the drives are configured for serving web pages, and not the transactional nature of a database.

 

If you have an easy way to archive, that's probably your best bet. Also make sure you have proper indexing and standard database maintenance (nightly reindex, transaction log maintenance, updating statistics on a regular basis, etc.)

 

Or you could try the brute force method and either throw more hardware at it or move it to Amazon (or other cloud service of choice). Just my $.02.

This is right. Especially the part about nightly reindex(provided your transactions are hitting the same DB that is being searched). Much depends on the database design.

 

If you are doing your "searching" the same place(database) you are doing your "thread/post entering"? That can cause significant peformance problems, especially in row level locking DBs, like MySQL/SQL Server(and yes, Oracle too). Also, much depends on how the search SQL statements are constructed, which also depends on your data model design.

 

Dynamic SQL(SQL statements that are built a run time), rather than stored procedures, is much slower. That's cause one is interpreted at run time, and one is pre-compiled.

Given that the most practical (by far) access to past content in the time frame you're talking about is the search function, the real question becomes: how much less useful does the search function become without that past content?

 

First step towards answering that is answering the question: how often does the search function get used? Any metrics on that?

 

Also, you might start by deleting some of the older, larger, and/or more useless threads. Like Last Post Wins (which is still in the database). Or GBID's "Hot for..." threads. No realistic reason anyone's referring back to those.

This is also right. Especially getting the answers to those questions.

 

 

 

Also: it may be necessary to set up 2 DBs. One designed for transactions(new threads, new posts, the "right now" stuff"), and one designed for searching(the "older stuff"). Each DB would be designed differently, and perhaps even use different DBMS.

 

A good rule: Design your database...primarily... for writes(inserting lots of data) or reads(searching/retrieving data), not both. Remember Mr. Myagi: "if you try to walk down the middle of the road? Sqush, just like grape!"

 

 

That's if we want this to work like a "big time" system. :D Use your MySQL for new threads/posting(inserts), use something like MongoDB for the archive/threads older than 1 week.

 

(.....and here come the Redis people, and their usual horsecrap about "flexibility". Answer to Redis people: its a football board that does the same thing every year, with the same data/objects, not a 50-division multinational, or a massive e-commerce site. Thus, Redis's main advantage? Useless to this case. So, no, Redis is not better here. MongoDB is the right call here, especially if the people who do this come from SQL backgrounds.)

 

This way, you're keeping the data you are most likely to be working with right now, "writes" in an SQL-based system(EDIT, and no, we don't just use Mongo for everything, because the board software probably only supports SQL-based systems): today's threads, posts, etc.in one DB. Even if there are a lot of searches to the "write" DB, the amount of data doesn't cause you trouble. Set a rule for thread "expiration", and archive older threads to the "read" DB(which is probably similar to what you are doing now)

 

You're keeping the older stuff in a DB that is designed for "reads"/storage, and literally blows SQL systems away when it comes to data retrieval.

 

If somebody hauls out an old thread and writes a new post to it? No big deal, just use this http://www.mulesoft.com/ to "ressurect" that old data back into the "write" DB. You use the same thing to do your weekly archiving going the other way. You just have to build a single field that holds the "old ID", and few other things.(I can explain this in detail).

 

The "read" DB is what we would call the "system of record". This way, threads that get closed/deleted/whatever, don't make it here, and they just get deleted when they "expire".

 

Finally, you add a feature to the search functionality, that basically says: you want to search "this week" stuff, or the "old" stuff? Essentially the searches would be directed to one DB or the other. Or you could do some fancy Mule stuff to do searches from both DBs at the same time....but, if that's what you want to do, please, please: just PM me, and I'll do it.

 

EDIT: Also, if you want to split HUGE threads between the 2 DBs, using this same method, it's doable, but again, let me/somebody who knows what they are doing do this.

 

If we really want to get nuts, we can try using Riak(which is a whole other "new" thing, which seems promising)....but I wouldn't yet.

Edited by OCinBuffalo
Link to comment
Share on other sites
l< j Veteran

l< j

Posted
Posted (edited)

I only bring this up because it may be helpful info to you: on Sunday and yesterday, the board 'mis-remembered' which threads I had read and would show all threads as read when I returned to TSW. It stopped. (I am not complaining; maybe this is helpful for diagnosing any problems.)

 

BTW: Your DB is about to get a good workout. Fingers crossed.

 

kj

Edited by l< j
Link to comment
Share on other sites
RyanC883 All Pro

RyanC883

Posted
Posted

Scott - I just started a website and we are going to use an Amazon cloud server ($150/month and can add space immediately for additional charges when needed). I don't know how yours is set up, but if it is a matter of some of us kicking in some cash, I'm sure many members would be willing to contribute (including myself). Great board so keep me and the others up to date if you need anything ($) so we can keep it going. Thanks again.

 

I'd contribute.

Link to comment
Share on other sites
KnightRider RFA

KnightRider

Posted
Posted

With over 3 million posts in the database, things are starting to creak, if not actually break down. Although I haven't officially diagnosed database size as being the reason it is slower these days, it seems to have gotten progressively worse as the seasons pass.

 

So, that's an issue. I'm open to suggestions on how to handle this.

 

Given that this is database size driven, I could possibly buy another license of IPB and offload a decade worth of posts, make it archive only. This will affect post counts, etc... on active board.

 

I could just also just delete those posts. Not a big fan of that option.

 

Or, perhaps there is a server tweak I have missed or there is something happening in the background that is causing the long page load times. Regardless, I'll need a plan of action for this season...

 

Thoughts?

 

Oh, boy, the timing of this is ironic...

Link to comment
Share on other sites
thewildrabbit All Pro

thewildrabbit

Posted
Posted

To be honest, I have not looked into these new server implementations. I assume Amazon knows how to do these things. I pay $310/month... The hardware is great, but I think we are just hitting the limit when too many requests come in at the same time. I assume RAM is the culprit here, but that's just my gut...

Jeez, SDS just ask for donations and get another server. I'm sure so many love this site that they will gladly chip in to help this work better. Send the old stuff up to the cloud.

 

BTW, thanks for enabling a such a great site for us Bills fans. Also, thanks goes out to the mods who help keep things temperate. :worthy:

Link to comment
Share on other sites
 Share
Go to topic listing
×
×
  • Create New...