Captain Caveman Posted September 9, 2014 Posted September 9, 2014 Scott is cheap. I heard it's Beerball's fault for meddling in football (message board) related decisions.
DC Tom Posted September 9, 2014 Posted September 9, 2014 (edited) Money isn't the issue - I appreciate the offers, but that's not what I'm asking for. It's really just a matter of the past content being accessible or if I need to throw more hardware at the solution. Given that the most practical (by far) access to past content in the time frame you're talking about is the search function, the real question becomes: how much less useful does the search function become without that past content? First step towards answering that is answering the question: how often does the search function get used? Any metrics on that? Also, you might start by deleting some of the older, larger, and/or more useless threads. Like Last Post Wins (which is still in the database). Or GBID's "Hot for..." threads. No realistic reason anyone's referring back to those. Edit: also, search for anything with the author = 'DC Tom' and content like '%You're an idiot%'. That'll recover some space right quick...do us both some good, actually. I heard it's Beerball's fault for meddling in football (message board) related decisions. Beerball was heard to say "Just go ahead and fire me..." Edited September 9, 2014 by DC Tom
BringBackFergy Posted September 9, 2014 Posted September 9, 2014 Given that the most practical (by far) access to past content in the time frame you're talking about is the search function, the real question becomes: how much less useful does the search function become without that past content? First step towards answering that is answering the question: how often does the search function get used? Any metrics on that? Also, you might start by deleting some of the older, larger, and/or more useless threads. Like Last Post Wins (which is still in the database). Or GBID's "Hot for..." threads. No realistic reason anyone's referring back to those. Edit: also, search for anything with the author = 'DC Tom' and content like '%You're an idiot%'. That'll recover some space right quick...do us both some good, Whatever the price is to save the "Mario Williams" thread, I'm in. That was epic and a true rite of passage for many of us.
BUNCH OF MULARKEY Posted September 9, 2014 Posted September 9, 2014 (edited) My hard drive just went floppy. But yea as far as keeping the archives available, one day I was bored and went back to the Donohoe is the devil posts, I cracked up. Edited September 9, 2014 by BUNCH OF MULARKEY
17 Josh Allen Posted September 9, 2014 Posted September 9, 2014 I would hire the IT guys who built the Obama care web site. I'm sure it would be fixed in a minute. LOL. I sure hope you get it figured out. thanks for all your hard work.
Clippers of Nfl Posted September 9, 2014 Posted September 9, 2014 Money isn't the issue - I appreciate the offers, but that's not what I'm asking for. It's really just a matter of the past content being accessible or if I need to throw more hardware at the solution. Usually when someone says its not a money issue, it is a money issue! How much? Jk. I know of archived posts, but never go to review them. Like we really "Cutler" about an opinion from 7 years ago? Instead of post count, just keep the year the poster joined. "Joined 20.."
Fan in San Diego Posted September 9, 2014 Posted September 9, 2014 If dumping posts would help that seems like a good way. Who really looks at posts more than a month old anyways. Maybe that ocean floor thread is the odd exception.
Jim in Anchorage Posted September 9, 2014 Posted September 9, 2014 I have noticed zero performance issues here. Seems as fast, or faster then ever.
Captain Caveman Posted September 9, 2014 Posted September 9, 2014 Generally when I've been on at non-peak times the performance seems to be a lot better. Seems like it might be related to large numbers of calls / writes to the database over a short time rather than overall DB size.
OCinBuffalo Posted September 9, 2014 Posted September 9, 2014 (edited) If you suspect it is database issues, many times it is disk I/O. If you host the database on the same server as the web server, chances are the drives are configured for serving web pages, and not the transactional nature of a database. If you have an easy way to archive, that's probably your best bet. Also make sure you have proper indexing and standard database maintenance (nightly reindex, transaction log maintenance, updating statistics on a regular basis, etc.) Or you could try the brute force method and either throw more hardware at it or move it to Amazon (or other cloud service of choice). Just my $.02. This is right. Especially the part about nightly reindex(provided your transactions are hitting the same DB that is being searched). Much depends on the database design. If you are doing your "searching" the same place(database) you are doing your "thread/post entering"? That can cause significant peformance problems, especially in row level locking DBs, like MySQL/SQL Server(and yes, Oracle too). Also, much depends on how the search SQL statements are constructed, which also depends on your data model design. Dynamic SQL(SQL statements that are built a run time), rather than stored procedures, is much slower. That's cause one is interpreted at run time, and one is pre-compiled. Given that the most practical (by far) access to past content in the time frame you're talking about is the search function, the real question becomes: how much less useful does the search function become without that past content? First step towards answering that is answering the question: how often does the search function get used? Any metrics on that? Also, you might start by deleting some of the older, larger, and/or more useless threads. Like Last Post Wins (which is still in the database). Or GBID's "Hot for..." threads. No realistic reason anyone's referring back to those. This is also right. Especially getting the answers to those questions. Also: it may be necessary to set up 2 DBs. One designed for transactions(new threads, new posts, the "right now" stuff"), and one designed for searching(the "older stuff"). Each DB would be designed differently, and perhaps even use different DBMS. A good rule: Design your database...primarily... for writes(inserting lots of data) or reads(searching/retrieving data), not both. Remember Mr. Myagi: "if you try to walk down the middle of the road? Sqush, just like grape!" That's if we want this to work like a "big time" system. Use your MySQL for new threads/posting(inserts), use something like MongoDB for the archive/threads older than 1 week. (.....and here come the Redis people, and their usual horsecrap about "flexibility". Answer to Redis people: its a football board that does the same thing every year, with the same data/objects, not a 50-division multinational, or a massive e-commerce site. Thus, Redis's main advantage? Useless to this case. So, no, Redis is not better here. MongoDB is the right call here, especially if the people who do this come from SQL backgrounds.) This way, you're keeping the data you are most likely to be working with right now, "writes" in an SQL-based system(EDIT, and no, we don't just use Mongo for everything, because the board software probably only supports SQL-based systems): today's threads, posts, etc.in one DB. Even if there are a lot of searches to the "write" DB, the amount of data doesn't cause you trouble. Set a rule for thread "expiration", and archive older threads to the "read" DB(which is probably similar to what you are doing now) You're keeping the older stuff in a DB that is designed for "reads"/storage, and literally blows SQL systems away when it comes to data retrieval. If somebody hauls out an old thread and writes a new post to it? No big deal, just use this http://www.mulesoft.com/ to "ressurect" that old data back into the "write" DB. You use the same thing to do your weekly archiving going the other way. You just have to build a single field that holds the "old ID", and few other things.(I can explain this in detail). The "read" DB is what we would call the "system of record". This way, threads that get closed/deleted/whatever, don't make it here, and they just get deleted when they "expire". Finally, you add a feature to the search functionality, that basically says: you want to search "this week" stuff, or the "old" stuff? Essentially the searches would be directed to one DB or the other. Or you could do some fancy Mule stuff to do searches from both DBs at the same time....but, if that's what you want to do, please, please: just PM me, and I'll do it. EDIT: Also, if you want to split HUGE threads between the 2 DBs, using this same method, it's doable, but again, let me/somebody who knows what they are doing do this. If we really want to get nuts, we can try using Riak(which is a whole other "new" thing, which seems promising)....but I wouldn't yet. Edited September 9, 2014 by OCinBuffalo
todd Posted September 9, 2014 Posted September 9, 2014 Just give me the keys - I'll fix it Is fix a synonym for "Rosen"?
Captain Caveman Posted September 9, 2014 Posted September 9, 2014 It also seems to be performing much better for me today, for whatever that's worth. Did you make any changes?
justnzane Posted September 9, 2014 Posted September 9, 2014 Is fix a synonym for "Rosen"? lol, Sorry Jay
l< j Posted September 9, 2014 Posted September 9, 2014 (edited) I only bring this up because it may be helpful info to you: on Sunday and yesterday, the board 'mis-remembered' which threads I had read and would show all threads as read when I returned to TSW. It stopped. (I am not complaining; maybe this is helpful for diagnosing any problems.) BTW: Your DB is about to get a good workout. Fingers crossed. kj Edited September 9, 2014 by l< j
RyanC883 Posted September 9, 2014 Posted September 9, 2014 Scott - I just started a website and we are going to use an Amazon cloud server ($150/month and can add space immediately for additional charges when needed). I don't know how yours is set up, but if it is a matter of some of us kicking in some cash, I'm sure many members would be willing to contribute (including myself). Great board so keep me and the others up to date if you need anything ($) so we can keep it going. Thanks again. I'd contribute.
KnightRider Posted September 9, 2014 Posted September 9, 2014 With over 3 million posts in the database, things are starting to creak, if not actually break down. Although I haven't officially diagnosed database size as being the reason it is slower these days, it seems to have gotten progressively worse as the seasons pass. So, that's an issue. I'm open to suggestions on how to handle this. Given that this is database size driven, I could possibly buy another license of IPB and offload a decade worth of posts, make it archive only. This will affect post counts, etc... on active board. I could just also just delete those posts. Not a big fan of that option. Or, perhaps there is a server tweak I have missed or there is something happening in the background that is causing the long page load times. Regardless, I'll need a plan of action for this season... Thoughts? Oh, boy, the timing of this is ironic...
thewildrabbit Posted September 9, 2014 Posted September 9, 2014 To be honest, I have not looked into these new server implementations. I assume Amazon knows how to do these things. I pay $310/month... The hardware is great, but I think we are just hitting the limit when too many requests come in at the same time. I assume RAM is the culprit here, but that's just my gut... Jeez, SDS just ask for donations and get another server. I'm sure so many love this site that they will gladly chip in to help this work better. Send the old stuff up to the cloud. BTW, thanks for enabling a such a great site for us Bills fans. Also, thanks goes out to the mods who help keep things temperate.
Recommended Posts