Because I'm all about the "good enough."

Saturday, March 3, 2012

Going back to the stack.

In the spirit of trying to suggest solutions, here are a couple of thoughts about what an enterprise can do first off to make security a little better.

It's bothered me that infrastructure is being administered more horizontally than vertically these days.  Everyone specializes in a different layer: network, OS, utilities (such as Exchange), middleware, applications, etc.  And this gets worse when you outsource one or two layers to "the cloud" (think IaaS, PaaS and SaaS), so that you have to coordinate with a third party to troubleshoot something.

Back in the Pleistocene era, system administrators knew their systems like they were their babies.  They knew everything that was running on them, how they were configured, who they talked to, and they knew when something was "off."  I know sysadmins that would regularly help a developer debug code, and they were often better at it than the developer, because they also understood the underlying environment better.  They could troubleshoot all the way up and down the stack, and you went to one source to do it instead of having to get a conference call together with 3rd level engineers from four different companies.  (Seriously, I know of a data center that had five different networks owned by five separate entities.  Think you could figure out what happened to a packet?  Think again.)

So one thing that enterprises can do is simply to get control of their layers as much as possible.  Know what you have, know where it is, and be able to cause changes to it when you want to.  That sounds so obvious as to be not worth saying, but I don't know of any admins who know more than about 500 hostnames by heart, and many times the environment is so dynamic that boxes come and go without any centralized tracking keeping up with it.  (And I'm not even talking about VMs.)

If you already have parts of your infrastructure outsourced, go over your contracts and strengthen your relationships with your providers.  You want them to be able to give you logs, for example, within a few minutes of the request.  You also need to have the right technical level support people on call without having to fight your way through first-level script-readers.

And finally, go back to designating "stack admins," who are generalists rather than specialists in one particular technology.  It should be their job to know as much as possible about any given system.  You can fit this into DevOps if the developers truly know the lower layers.  A stack admin is your best hope for knowing what normal operation is, and for alerting you when something doesn't smell right; they're also the best at understanding the implications of any given planned change (such as changing the ports an application uses without creating the corresponding firewall rules).

Start with knowledge, and then work your way to control.  Notice we haven't really touched on security yet; that'll come later.  But knowledge and control are basic building blocks of security.