Fezmid Posted February 6, 2005 Share Posted February 6, 2005 Just wondering if anyone else is sharing my misery. I was supposed to have downtime on a couple of production servers from 6am-4pm Sunday morning... But Thursday morning they decided to make it 10pm Saturday - 8am Sunday. I'm definately not an evening person, so this really sucks for me... Just wondering if anyone else is miserable right now too? CW Link to comment Share on other sites More sharing options...
Tux of Borg Posted February 6, 2005 Share Posted February 6, 2005 I use to work the midnight to 8am shift at the NOC. Patched servers, removed warez from the Win2000 servers recycling bin, and monitored porn sites for illegal content. I liked it because everytime a server crashed, i got to call the CTO and wake him up. Link to comment Share on other sites More sharing options...
mcjeff215 Posted February 6, 2005 Share Posted February 6, 2005 I've never had to work the night shifts, but back when I did operational work I used to get to carry a pager. Nothing has ever sucked that bad. We also had those Nextel Direct-Connect deals for a while. There's nothing like hearing the following coming from your damn kitchen at 2:00 in the AM. :BEEP BEEP: "Hey Jeff? Server X is giving alerts on the monitor and I can't get it to..." There were cases where our NOC couldn't FIND servers and they'd call me asking "do you know where such and such a server is?" We've got a 60,000 square foot DC. No I have no damn idea where that server is. Start at row 1 and walk the floor you fool. I've spent more nights than I care to mention sleeping under a cubicle because I was too tired to attempt to drive home. I've nothing but respect for the poor folks that wind up working IT Operations! It's probably more important than us code/arch jockeys, and certainly less rewarded. I feel your pain. I've been there! I've spent a SB or two at a terminal. -Jeff Link to comment Share on other sites More sharing options...
Navy Chief Navy Pride Posted February 6, 2005 Share Posted February 6, 2005 Just wondering if anyone else is sharing my misery. I was supposed to have downtime on a couple of production servers from 6am-4pm Sunday morning... But Thursday morning they decided to make it 10pm Saturday - 8am Sunday. I'm definately not an evening person, so this really sucks for me... Just wondering if anyone else is miserable right now too? CW 229532[/snapback] No CW I am currently planning to get very tore up. Sorry to hear you gotta work. I will toss a couple down for you. Link to comment Share on other sites More sharing options...
Fezmid Posted February 6, 2005 Author Share Posted February 6, 2005 I liked it because everytime a server crashed, i got to call the CTO and wake him up. I must admit, I do enjoy waking up my boss at 2am when something crashes... It's like petty revenge (If I'm up, you are too!!!). Oh well. I've never had to work the night shifts, but back when I did operational work I used to get to carry a pager. Nothing has ever sucked that bad. Hey, that's me! And yes, it sucks... Luckily, our stuff doesn't go down in the middle of the night very often (maybe once or twice a month). I've nothing but respect for the poor folks that wind up working IT Operations! It's probably more important than us code/arch jockeys, and certainly less rewarded. I'm pretty sure at least one development group at my company considers the sysadmins to be computer janitors... But luckily, most of the groups seems to think highly of us. I've spent a SB or two at a terminal. I've never missed a football game I wanted to watch; we've had downtimes that would cut it close to me missing the start of a Bills game (I live about 30 minutes away from work), and I tell everyone, "I don't care if this ds done or not, at 11:20am CST, I'm leaving. I'll work on it again after the game's over." I've never had to walk out unfinished yet, so they probably think I"m just kidding around... CW Link to comment Share on other sites More sharing options...
mcjeff215 Posted February 6, 2005 Share Posted February 6, 2005 Hey, that's me! And yes, it sucks... Luckily, our stuff doesn't go down in the middle of the night very often (maybe once or twice a month). What industry are you in? I know what you do, but what type of company is it for? We've got the "dump it on ops" guys at our shop too. There's also that "bring 'em up to development, push 'em down to ops" mentality. I tell them over and over again... you want to ensure the people in ops are as good, and treated as well, as the developers and the architects. Link to comment Share on other sites More sharing options...
Matt in KC Posted February 6, 2005 Share Posted February 6, 2005 I just got pulled into a work outage.... One of our email and public web servers crashed. As these are managed remotely by a guy who reports to me, I'm getting tapped since he is not responig. We just started a project to bring this server into our data center (locally) too. I'm primarily an account manager now, so I'm not too happy about this.... Link to comment Share on other sites More sharing options...
mcjeff215 Posted February 6, 2005 Share Posted February 6, 2005 I just got pulled into a work outage.... One of our email and public web servers crashed. As these are managed remotely by a guy who reports to me, I'm getting tapped since he is not responig. We just started a project to bring this server into our data center (locally) too. I'm primarily an account manager now, so I'm not too happy about this.... 229690[/snapback] I used to base a lot of the work I did on one factor... whether or not I thought it was important enough to call me about. If I thought it was, it went into a cluster or some such H/A configuration. At least we know why he reports to you and not the other way around... Link to comment Share on other sites More sharing options...
Matt in KC Posted February 6, 2005 Share Posted February 6, 2005 I used to base a lot of the work I did on one factor... whether or not I thought it was important enough to call me about. If I thought it was, it went into a cluster or some such H/A configuration.At least we know why he reports to you and not the other way around... 229693[/snapback] Our company is merging with another, so we're pulling a problem server out of a place where it is maintained by force of will by one individual into our data center where we'll have proper load balancing / high availability for the website and a proper DR plan for the mail server. For now, we just have the monitoring set up with no real process changes (the replacement server is still being built and shipped) so we can see just how bad it is, but do not have much recourse when our force-of-will guy is AWOL. Link to comment Share on other sites More sharing options...
mcjeff215 Posted February 6, 2005 Share Posted February 6, 2005 Our company is merging with another, so we're pulling a problem server out of a place where it is maintained by force of will by one individual into our data center where we'll have proper load balancing / high availability for the website and a proper DR plan for the mail server. For now, we just have the monitoring set up with no real process changes (the replacement server is still being built and shipped) so we can see just how bad it is, but do not have much recourse when our force-of-will guy is AWOL. 229696[/snapback] So is it a case of false job security? He keeps this stuff in his silo as to become irreplaceable? I've been through a series of acquisitions (on both ends) and that type of stuff never ceases to amaze me. No matter what they think, there are very few people that can't be replaced. More often than not, those with years of institutional knowledge are more important. Trying to make yourself important by hoarding technical components simply flags you as a problem employee. -Jeff Link to comment Share on other sites More sharing options...
Fezmid Posted February 6, 2005 Author Share Posted February 6, 2005 What industry are you in? I know what you do, but what type of company is it for? That's a very good question... Our CEO recently said you'd be famous if you could find a one-sentance description of what we do as a company. At our core, we're a financial document company; we print prospectus, do SEC filings, etc. We also do other printings (you know those subway cards and stamps? We print one of those, and another company prints the other.) However, we've been growing quite a bit in the past 6 years (when I started, we had about 8 HP-UX servers, now we have about 90 HP-UX and Solaris servers...) So now we do document management for legal firms (ie: company one sues company two, and al of the documents for the case are sent to us; we scan them and put them in a big database for easy OCR searching). We also do stuff with real estate, but I'm not really sure what... There's also thiings like document translations, securities law (ie: paperwork when companies merge), etc. It's pretty broad. CW Link to comment Share on other sites More sharing options...
Matt in KC Posted February 6, 2005 Share Posted February 6, 2005 So is it a case of false job security? He keeps this stuff in his silo as to become irreplaceable? I've been through a series of acquisitions (on both ends) and that type of stuff never ceases to amaze me. No matter what they think, there are very few people that can't be replaced. More often than not, those with years of institutional knowledge are more important. Trying to make yourself important by hoarding technical components simply flags you as a problem employee. -Jeff 229700[/snapback] Yes, he is definitely holding the IT infrastructure hostage, and this has been his survival strategy for years. For the first time ever, though, he is no longer the top IT guy around and his approach is pretty transparent. He will come into the office and work all night to fix something (that should have been preventively addressed) and milk it for all it's worth. The longer the outage, the more clear how dedicated he is ("he fixed the mail server for 14 hours straight"). In contrast, our team has tight SLAs with the likes of GE where have financial penalties for under 99.9% uptime (about 43 minutes per month). We have tight monitoring, auto-task creation for resolution notes and escalation, and regularly drill DR, and have a formal review process for anything over a 15 min outage.... If he's willing to follow tier 2 NRT (network response team) processes, I think we could clearly measure and show his value. As it is, he's bitter and is sure he is losing his job. This has been a big challenge as his boss, and I am worried he might go out "with a bang." Thank you for listening, doc. How much do I owe you? Link to comment Share on other sites More sharing options...
mcjeff215 Posted February 6, 2005 Share Posted February 6, 2005 Yes, he is definitely holding the IT infrastructure hostage, and this has been his survival strategy for years. For the first time ever, though, he is no longer the top IT guy around and his approach is pretty transparent. He will come into the office and work all night to fix something (that should have been preventively addressed) and milk it for all it's worth. The longer the outage, the more clear how dedicated he is ("he fixed the mail server for 14 hours straight"). In contrast, our team has tight SLAs with the likes of GE where have financial penalties for under 99.9% uptime (about 43 minutes per month). We have tight monitoring, auto-task creation for resolution notes and escalation, and regularly drill DR, and have a formal review process for anything over a 15 min outage.... If he's willing to follow tier 2 NRT (network response team) processes, I think we could clearly measure and show his value. As it is, he's bitter and is sure he is losing his job. This has been a big challenge as his boss, and I am worried he might go out "with a bang." Thank you for listening, doc. How much do I owe you? 229706[/snapback] I've got a guy who can't show up prior to noon, but is the sole remaining programmer on one of our larger systems. I'm qualified to do the work, but I've got other responsibilities and the time crunch would kill me if I took over his duties. Once we can fill an open slot or two, the transition starts. EDIT: He's interpreting the lack of disciplinary action as security; it's really just a timing and capacity problem. Every org has 'em. -Jeff Link to comment Share on other sites More sharing options...
Recommended Posts