as it is a really really important development server.
Most people in IT will immediately understand this frustration. A common
simplified view of complex enterprise IT is that production servers are
only those that production data flows through.
In itself that's not a bad definition of a production server, but it is
incomplete. What about the servers that those "production servers" depend
on? Aren't those also production servers?
A reasonable view of configuration change management says that those support
or dependent servers are also production servers and together with the
"production servers" make up a production environment.
A Dave does not see this. He perceives only machines in a production data center
and only those through which production data flows as production servers.
The following story provides an example of what can happen (for those that cannot
already forsee it) when production servers are made dependent upon a "not so much a
production server".
Bob was a very bright developer. And motivated. He often went off on his own
time and implemented impressive solutions and tools. These tools far too often
quickly became standard tools for our group. As such they became heavily relied
upon.
All of this is good, but Bob had a habit of doing things his own way - and not documenting
much. In this case, although we were a Microsoft house he built a tool on Linux, for no other reason then Bob liked Linux. By the way Linux was not approved or authorized for use in the Production. This tool was very impressive in the way that it handled processing stories. Over time the
tool became a service that several soon to be production critical components depended on.
This sounds great and it should, but it soon won't. When these critical components did finally move into a production environment, no one bothered to ensure that Bobs service got hosted in a true production environment, as opposed to the "not so production environment" running under Bob's desk. Since there was no firewall between development, pre-production and production the service could
still be reached (on Bob's desktop). As long as Bob's computer ran, everything was
fine.
As a testament to linux (and Bob's capabilities) the service ran flawlessly for
several years. Bob eventually left the group and gave his hardware including this
"not so much a production server" desktop to a co-worker . This transition happened
at the end of the day Friday and by Monday was up and running. Production was
going along just fine.
Then it happened. The co-worker decided that since she had not needed to access any of the servers for development reasons, they could be retired and the desk uncluttered. Logical enough, so she turned off all the computers that Bob had given her for "safe" keeping.
Within moments, the calls started coming in, multiple "PRODUCTION" systems were failing, with out a obvious reason. After everyone associated with the failing systems are in panic mode racing around looking for the causes the outage, someone says "We should call Bob he knows alot about that system.". At the sound of the name Bob, a light goes off in the co-workers head, she strolls back to her office reaches under their desk and turns on the one computer
which Bob told her was pretty important moments later every thing is back up and running.
Later and reasonably concerned she calls and asks Bob why he gave her a production server. His response: "Its not so much a production server as a really important development server." After all
it didn't actually process production data and provided information to do so. And it was too
difficult to get a linux machine into our production environment.
In the end, lets remember one important thing. If a production server talks to it, its a
production server. There is no such thing as "not so much a production server".
DON'T BE A DAVE!
Tuesday, March 24, 2009
Subscribe to:
Posts (Atom)