SDS Posted September 8, 2014 Posted September 8, 2014 With over 3 million posts in the database, things are starting to creak, if not actually break down. Although I haven't officially diagnosed database size as being the reason it is slower these days, it seems to have gotten progressively worse as the seasons pass. So, that's an issue. I'm open to suggestions on how to handle this. Given that this is database size driven, I could possibly buy another license of IPB and offload a decade worth of posts, make it archive only. This will affect post counts, etc... on active board. I could just also just delete those posts. Not a big fan of that option. Or, perhaps there is a server tweak I have missed or there is something happening in the background that is causing the long page load times. Regardless, I'll need a plan of action for this season... Thoughts?
infernus Posted September 8, 2014 Posted September 8, 2014 Do you have access to the physical server? Could be worth it to backup the db, temporarily remove data and diagnose if the obscene size is indeed causing the performance hit (then restore data back afterwards). That could take awhile though. Or if you have access to the db, run some queries to see how long a simple select query from the posts table takes to run
Fan in Chicago Posted September 8, 2014 Posted September 8, 2014 Given that this is database size driven, I could possibly buy another license of IPB and offload a decade worth of posts, make it archive only. This will affect post counts, etc... on active board. Thoughts? Is there an option to archive posts from say 5 years back ? I am not sure we need to refer to those older posts. I am not an IT guy, so probably don't understand why post count is related to activeness of the posts. But, I don't care about my post count but if most people like it, there should be a field under that person's profile where you could freeze the post counts till the date of the archive. So, if say, you archived all posts before Sept 5, 2009 then freeze the post count till that date with the new counter starting on 9/6/2009. Just a thought but yea I am seeing performance issues myself. (thanx as always for this board, Scott)
Dorkington Posted September 8, 2014 Posted September 8, 2014 With over 3 million posts in the database, things are starting to creak, if not actually break down. Although I haven't officially diagnosed database size as being the reason it is slower these days, it seems to have gotten progressively worse as the seasons pass. So, that's an issue. I'm open to suggestions on how to handle this. Given that this is database size driven, I could possibly buy another license of IPB and offload a decade worth of posts, make it archive only. This will affect post counts, etc... on active board. I could just also just delete those posts. Not a big fan of that option. Or, perhaps there is a server tweak I have missed or there is something happening in the background that is causing the long page load times. Regardless, I'll need a plan of action for this season... Thoughts? Post counts are kinda silly, so I vote archive/offload. That way, things could still be referenced, some how. I'd be willing to donate a bit, if there's big costs.
Captain Caveman Posted September 8, 2014 Posted September 8, 2014 Scott, Thanks a lot, I have been noticing some degradation in the last few weeks but didn't want to complain. I had assumed it was maybe due to increase in traffic / posting due with the start of the season.
justnzane Posted September 8, 2014 Posted September 8, 2014 (edited) I thought this thread was going about the quality of posting going down . Nonetheless, it would be nice to the post archived, but if it is too cost prohibitive, I'm sure most would understand. Edit: 6000th post...again woo hoo. Post count really is not a big deal for most except may be Tom, lol Edited September 8, 2014 by justnzane
BringBackFergy Posted September 8, 2014 Posted September 8, 2014 With over 3 million posts in the database, things are starting to creak, if not actually break down. Although I haven't officially diagnosed database size as being the reason it is slower these days, it seems to have gotten progressively worse as the seasons pass. So, that's an issue. I'm open to suggestions on how to handle this. Given that this is database size driven, I could possibly buy another license of IPB and offload a decade worth of posts, make it archive only. This will affect post counts, etc... on active board. I could just also just delete those posts. Not a big fan of that option. Or, perhaps there is a server tweak I have missed or there is something happening in the background that is causing the long page load times. Regardless, I'll need a plan of action for this season... Thoughts? Scott - I just started a website and we are going to use an Amazon cloud server ($150/month and can add space immediately for additional charges when needed). I don't know how yours is set up, but if it is a matter of some of us kicking in some cash, I'm sure many members would be willing to contribute (including myself). Great board so keep me and the others up to date if you need anything ($) so we can keep it going. Thanks again.
4BillsintheBurgh Posted September 8, 2014 Posted September 8, 2014 This runs in a sql server database I assume? I can ask some of our folks for tuning recommendations who work with that database, I generally stay with oracle. In my experience and what I've noticed hitting the website you seem to be hitting i/o back to disk more often. If you can check i/o statistics over time that may be shown there. Since by the nature of this website you have a lot of inserts but not much other dml I don't think object rebuilds would help unless as you said there is data removed. If possible, can you add memory to the database cache? I think this could help if there has been an increase in i/o's due to the size of the underlying data. Let me know what database software you're running and I'll find out what I can.
dickleyjones Posted September 8, 2014 Posted September 8, 2014 if you are not using it already: http://memcached.org/
SDS Posted September 8, 2014 Author Posted September 8, 2014 Scott - I just started a website and we are going to use an Amazon cloud server ($150/month and can add space immediately for additional charges when needed). I don't know how yours is set up, but if it is a matter of some of us kicking in some cash, I'm sure many members would be willing to contribute (including myself). Great board so keep me and the others up to date if you need anything ($) so we can keep it going. Thanks again. To be honest, I have not looked into these new server implementations. I assume Amazon knows how to do these things. I pay $310/month... The hardware is great, but I think we are just hitting the limit when too many requests come in at the same time. I assume RAM is the culprit here, but that's just my gut...
ExiledInIllinois Posted September 8, 2014 Posted September 8, 2014 You can dump all my posts... They piss off and confuse DCTom anyway... :-) LoL...
26CornerBlitz Posted September 8, 2014 Posted September 8, 2014 I am willing to make a contribution for additional memory, CPU cores, and hard drive space if any or all of those will improve performance.
ExiledInIllinois Posted September 8, 2014 Posted September 8, 2014 To be honest, I have not looked into these new server implementations. I assume Amazon knows how to do these things. I pay $310/month... The hardware is great, but I think we are just hitting the limit when too many requests come in at the same time. I assume RAM is the culprit here, but that's just my gut... Thanks for all the hard work and making the site gr8. Would it help if we strike up a collection?
ExiledInIllinois Posted September 8, 2014 Posted September 8, 2014 Just give me the keys - I'll fix it LMAO! Just do another PPP labotomy. It's not like it doesn't need it. This time go the ice pick route... It won't feel a thing! ;-)
BillsFan-4-Ever Posted September 8, 2014 Posted September 8, 2014 With over 3 million posts in the database, things are starting to creak, if not actually break down. Although I haven't officially diagnosed database size as being the reason it is slower these days, it seems to have gotten progressively worse as the seasons pass. So, that's an issue. I'm open to suggestions on how to handle this. Given that this is database size driven, I could possibly buy another license of IPB and offload a decade worth of posts, make it archive only. This will affect post counts, etc... on active board. I could just also just delete those posts. Not a big fan of that option. Or, perhaps there is a server tweak I have missed or there is something happening in the background that is causing the long page load times. Regardless, I'll need a plan of action for this season... Thoughts? do large embedded pictures hurt performance? I've noticed a lot of the images are much larger than in the past and I've been trying to post only the smaller ones. Can you omit images in quotes?
BringBackFergy Posted September 8, 2014 Posted September 8, 2014 To be honest, I have not looked into these new server implementations. I assume Amazon knows how to do these things. I pay $310/month... The hardware is great, but I think we are just hitting the limit when too many requests come in at the same time. I assume RAM is the culprit here, but that's just my gut... If you have an IT guy that helps you out, a cloud server through Amazon is an option (rather than continually upgrading a hard drive server, etc).
The Wiz Posted September 8, 2014 Posted September 8, 2014 Blocking threads with EJ,Manuel,manual in the title would increase performance. Seriously though reindexing normally helps when our db/server starts to lag. Requires you to go down for a bit though.
Fan in Chicago Posted September 8, 2014 Posted September 8, 2014 You can dump all my posts... They piss off and confuse DCTom anyway... :-) LoL... Or he could dump DC Tom - takes care of multiple problems in one shot.
boyst Posted September 8, 2014 Posted September 8, 2014 Or he could dump DC Tom - takes care of multiple problems in one shot. he has something like 100,000 posts in total. Its amazing how much he has posted and how little he said.
Recommended Posts