SDS Posted January 1, 2015 Author Posted January 1, 2015 If anyone has any ideas on what I can look at while under heavy load, I can snap screen shots.
justnzane Posted January 1, 2015 Posted January 1, 2015 (edited) If anyone has any ideas on what I can look at while under heavy load, I can snap screen shots. You could lock up Rosen before he talks our head coach into quitting the job Edited January 1, 2015 by justnzane
gordong Posted January 1, 2015 Posted January 1, 2015 (edited) I/O read and Writes Memory utilization Physical Disk\Avg. Disk Reads/sec Physical Disk\Avg. Disk Writes/sec machine processor time SQL processor time Would be a good place to start, Monitor every 5 minutes for a day and see where things settle out on first pass. don't add too much monitoring at once you will not get any numbers that are useful. Also location probably will not play much into performance unless one site has a much larger bandwidth. The Hard drives personally I would stick with 2 drive's in a raid one config, you can still backup but if one drive fails you're not down and the raid will rebuild the other drive once replaced, Backups are better for corruption than Hardware issues. Make sure your always installing service packs and security patches (Microsoft every 2nd Tuesday of the month) Oh the other part you can NEVER have too much memory, but the OS has to be able to support it, what OS is the server running/ what are you going to use on the new server? Edited January 1, 2015 by gordong
SDS Posted January 1, 2015 Author Posted January 1, 2015 07:40:01 PM CPU %user %nice %system %iowait %steal %idle 07:50:01 PM all 32.38 0.00 14.83 1.42 0.00 51.38 08:00:01 PM all 29.81 0.06 14.97 0.97 0.00 54.18 08:10:01 PM all 26.41 0.00 12.45 0.76 0.00 60.39 08:20:01 PM all 25.43 0.00 11.53 0.95 0.00 62.09 08:30:01 PM all 25.44 0.00 10.80 1.15 0.00 62.62 08:40:01 PM all 26.13 0.00 10.55 0.51 0.00 62.81 08:50:01 PM all 23.92 0.00 7.62 0.54 0.00 67.92 09:00:01 PM all 22.59 0.06 8.42 0.53 0.00 68.39 09:10:01 PM all 22.00 0.00 8.05 0.90 0.00 69.04 09:20:01 PM all 23.27 0.00 8.06 0.63 0.00 68.04 09:30:01 PM all 19.52 0.00 6.82 0.91 0.00 72.75 09:40:01 PM all 20.83 0.00 7.00 0.40 0.00 71.77 09:50:01 PM all 17.82 0.00 6.41 0.38 0.00 75.38 10:00:01 PM all 18.58 0.06 7.11 0.38 0.00 73.86 10:10:01 PM all 17.74 0.00 6.40 0.43 0.00 75.43 Average: all 23.46 0.01 9.40 0.72 0.00 66.40 load at 7:40 was around 15 or so. Several seconds to execute each page. top - 21:19:07 up 204 days, 19:27, 1 user, load average: 3.97, 3.18, 3.02 Tasks: 262 total, 6 running, 250 sleeping, 0 stopped, 6 zombie Cpu(s): 27.9%us, 18.5%sy, 0.0%ni, 52.2%id, 1.2%wa, 0.0%hi, 0.1%si, 0.0%st Mem: 12302712k total, 6826720k used, 5475992k free, 164572k buffers Swap: 2096472k total, 476868k used, 1619604k free, 3324168k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 3583 mysql 15 0 2653m 587m 3980 S 269.0 4.9 102021:54 mysqld 8684 apache 15 0 588m 59m 15m S 10.6 0.5 0:00.64 httpd 8658 apache 15 0 592m 63m 18m S 10.3 0.5 0:01.12 httpd 8678 apache 15 0 558m 38m 22m S 7.0 0.3 0:00.84 httpd 8566 apache 15 0 590m 78m 30m R 6.0 0.7 0:05.20 httpd 8353 apache 16 0 594m 105m 55m S 3.7 0.9 0:11.71 httpd
Repulsif Posted January 1, 2015 Posted January 1, 2015 load average: 3.97, 3.18, 3.02 Swap: 2096472k total, 476868k used, 1619604k free, 3324168k cached there is something wrong with your system
Jobu Posted January 4, 2015 Posted January 4, 2015 there is something wrong with your system Based on what? Care to elaborate? On a side note, this may have been brought up elsewhere but have you considered spinning up a box in Amazon Web Services, Google cloud or some other? Too much coin? Pretty easy to scale when demand warrants it.
Repulsif Posted January 4, 2015 Posted January 4, 2015 (edited) Based on what? Care to elaborate? Although I'm not a sysadmin, I'm in the web development business since 1999 as lead developer, and I can safely say there is a problem when your load average is more than 0.20 in normal conditions and swap used shows too something wrong. so, and assuming SDS posted normal conditions and not peak conditions : load average: 3.97, 3.18, 3.02 means for me that your CPU is too heavily used Tasks: 262 total, 6 running, 250 sleeping, 0 stopped, 6 zombie you should have 0 zombie process ; some proccess might have been wrongly terminated Cpu(s): 27.9%us, 18.5%sy, 0.0%ni, 52.2%id, 1.2%wa, 0.0%hi, 0.1%si, 0.0%st way too much on normal conditions Mem: 12302712k total, 6826720k used, 5475992k free, 164572k buffers for me, it's fine Swap: 2096472k total, 476868k used, 1619604k free, 3324168k cached means swap has been used (maybe once, aybe more), and generally it's a bad thing, server didn't have enough RAM, so went into swap (physical disk allocated memory), and then everything runs more slowly and everything become a vicious circle. 3583 mysql 15 0 2653m 587m 3980 S 269.0 4.9 102021:54 mysqld under normal conditions (no cron used by the time, no heavy requests on mysql by the time), that's way too much. that said, SDS said he changed something on mysql (internal stats disabled ?), it could have done the trick. Edited January 4, 2015 by Repulsif
SDS Posted January 4, 2015 Author Posted January 4, 2015 Ok, this is what I'm looking at: E3-1270v2 (for a one-time fee I can upgrade to version V3 of this)32GB DDR-3 ECC RAMRAID controller: LSI 9271-4i 4X 240GB SSD (m500 or intel520, RAID 10 for main OS and DB)1x 120GB SSD in ssd slot, directly connected to motherboard for backups100mbit unmetered or 30TB on 1gig port13x IPv4IPMI/KVM, firewalled This thing would fly I think.
SDS Posted January 5, 2015 Author Posted January 5, 2015 I'll right, I'm pulling the trigger today. Week from hell will commence starting tonight.
4BillsintheBurgh Posted January 6, 2015 Posted January 6, 2015 I'll right, I'm pulling the trigger today. Week from hell will commence starting tonight. Good luck, may the force be with you.
SDS Posted January 6, 2015 Author Posted January 6, 2015 Good luck, may the force be with you. Should be setup tomorrow. Currently researching any changes I want to make in back end. Nginx, mariadb, php-fpm, etc...
gordong Posted January 11, 2015 Posted January 11, 2015 (edited) All kind of trouble getting on tonight first time in hours I've been able to get to the forums Shoutbox not working and it thinks it's 7:02 PM (it's 12:02am)) Edited January 11, 2015 by gordong
thewildrabbit Posted January 11, 2015 Posted January 11, 2015 Sorry, but server is still bad. 7:30 and having extreme difficulty doing anything. Taking forever to load just like it used to do..
SDS Posted January 11, 2015 Author Posted January 11, 2015 Not sure what the difficulty was. Script execution was very high but load was low. Very odd.
fansince88 Posted January 11, 2015 Posted January 11, 2015 I was trying to get on 7am est and kept getting message server issue of some kind. Fine now. Last night shoutbox was taking forever to reload.
Recommended Posts