Comments on Black Fist Security: My first risk model

Sorry, Jay Jacobs here and on that last anonymous ...

2010-09-19T19:44:19.182-07:00

Sorry, Jay Jacobs here and on that last anonymous post.

I like this model, it makes sense to me (having ju...

2010-09-19T19:43:11.821-07:00

I like this model, it makes sense to me (having just got RiskAMP myself) and I agree with some of AThulin's comments, but I think what you've created is far more useful than asking likely/unlikely type questions.

While your description was a little vague, it looks like you've got a good trade off between time and accuracy. If you wanted to get a more accurate number on "CAS Failure" you could have broken that problem into further cause and effect. Meaning rather than ask what the probability was of failure, break down the various failure conditions, causes and probabilities of those. But like I said, I think your approach was a good trade off between time and accuracy. If you wanted more confidence in your results you just spend more time. ...and that's the way it should be!

This has got to the best start I've seen. Ever.

Oops, I mean AThulin not AHutton.

2010-09-14T20:08:23.515-07:00

Oops, I mean AThulin not AHutton.

@AHutton thanks for the comment. I'm sure I&#...

2010-09-14T20:07:45.766-07:00

@AHutton thanks for the comment. I'm sure I'll be able to work some of that into the next model I put together.

After reading Hubbard's first book, How to Measure Anything, I learned that an uncalibrated estimator is about 80% overconfident in some cases, so I could have set the chance of failure at 1.2% but I decided to round up to a whole number. It is still a gut feeling expressed as a number, but that's better than a gut feeling expressed as a word like medium or low. By saying 2% we have a testable hypothesis. If one of the four servers should fail then I know that the probability is most likely greater than 5% and my next model will be more accurate.

The output of this process probably IS garbage. Models are poor approximations of real life, but every estimate we make is based on some model. For many of us in the security field, those models live in our head and aren't open to scrutiny. I am more willing to put my faith in this model than the one in my head and the head of the overconfident sysadmins around me.

Looks like @RISK requires risk estimates to a degr...

2010-09-13T23:43:58.187-07:00

Looks like @RISK requires risk estimates to a degree that is unusual to get in infosec. For instance, I doubt that the 1% probability for Exchange server failure after applying a service pack is anything but a gut feeling expressed in numbers.

Your 'doubling' seems a pretty strange methodology -- if you had got 15% risk instead, would you have doubled that as well?

Do you have any error estimates for the original estimate, or the subsequent doubling?

If the various risk elements are not commensurable (and gut feelings usually aren't), you might want to try to estimate the risk that the data you get out of this process is garbage.

I find that it can be quite difficult to get even a 4-level estimate out of most IT-people (extremely unlikely, unlikely, likely, very likely), and a similar 4-level estimate of the damage. (I.e. error estimates are quite large.)

Also, it's one thing enumerating risks, and another to enumerate the right risks. Once you have defined the system for which you are trying to do a risk analysis, ask the right people involved with that system about the risks. IT support/helpdesk/customer support are often forgotten, yet they deal with see the effects of any failures, and even a small IT failure can mean long telephone queues for support -- which is another kind of damage.

I've worked a lot with mini-analyses: get everyone together in a room, brainstorm risks, evaluate -- i.e. place them into a 4x4 grid of probability vs damage --, decide which bins need to be addressed, and so on. As long as you get the right people together, this tend to work quite well -- failures happen when some area of the system in question was forgotten (like helpdesk operation). The main work is done in 6-8 hours, the documentation in another day.

@jth Sorry, @RISK is an excel add on that only wo...

2010-09-11T19:21:50.594-07:00

@jth Sorry, @RISK is an excel add on that only works in Windows. I am impressed with this style of risk modelling because it's pretty much impossible to create a risk model without knowing the system you're trying to model. Much like you can't make a model car without learning how the parts go together. That forced me to learn more about our email system than I knew before and I think I have a more realistic risk picture than if I just used a 50,000 foot view.

Seems pretty straightforward to me. I'd be int...

2010-09-11T07:48:41.520-07:00

Seems pretty straightforward to me. I'd be interested in seeing your copy, maybe working with you on a pet project or two to see if @RISK is something worth picking up for me as well.

Does that work on your mac? Or win only?

Awesome.

2010-09-08T17:03:46.203-07:00

Awesome.