Password cracking for statistical analysis

I went through a fun exercise today that yielded some interesting numbers about the password strength of our organization. We are going to be rolling out a new set of password requirements for all of our users in a few months. In anticipation of the changes, I decided that I would like to gather some statistics so that later on I can prove whether or not the change resulted in stronger passwords.

So the obvious solution is to run your password database through some kind of cracker and see what you get out of it. But it wasn't quite that simple, and I think that in the end we came up with a methodology that might be a better way to do things if your organization is particularly concerned about end user privacy. When I talked to my boss about this, he was concerned that people might feel violated because we had attacked their password. He also felt that doing such a thing might encourage our users to store their data on their own workstations or servers that we don't control if we were viewed to be abusing the trust that they place in us. After thinking about it for a while, I realized that he had some valid points. I'm not saying that I agreed with his decision to not analyze our passwords, but I did feel that he had some valid points.

I was able to address his concerns by changing my methodology a bit. First, I had to create some separation of duties. I was not going to gather the password hashes, it was going to be done by someone else. We were going to remove the account names from the file that was going to be analyzed so that the results were going to be anonymous. Finally, to minimize the amount of cracking that we had to do, we decided that we were only going to analyze a representative sample of the whole database.

Here is the exact procedure that we used. One of the Windows system administrators used the utility fgdump to create a text file containing usernames and hashed passwords. The resulting file was called 127.0.0.1.pwdump. Then we used a couple commands on a linux box to remove the usernames and machine accounts from the file. Machine accounts end with a dollar sign, so I stripped out any account name that had a dollar sign right before the separating colon. The second command strips off the username field in the file. The last line securely deletes the original file.

grep -v $: 127.0.0.1.pwdump > 127.0.0.1.pwdump.nomachines
cut -d: -f2-20 127.0.0.1.pwdump.nomachines > anonymous.txt
shred -n1 -u 127.0.0.1.*

The resulting file (anonymous.txt) was then given to me.

There had to be something in the username field for the file to be accepted by the password cracking program, so I whipped up a bit of python to put something there.

infile = open('anonymous.txt',r')
outfile = open('almostreadytocrack.txt','w')
listoflines = infile.readlines()

for i in range(1,len(listoflines)):
[tab]outfile.write(str(i))
[tab]outfile.write(':')
[tab]outfile.write(listoflines[i])

Now for the sampling. I went to one of the variety of web sites that you can find that offer sample size calculators. Based on the total number of passwords that I had gathered I needed to attack a random sample of about 1027 accounts to get the results that I wanted. Another quick bit of python would get me the chosen accounts.

import random
infile = open('almostreadytocrack.txt','r')
outfile = open('readytocrack.txt,'w')
hashlist = infile.readlines()
selectedhashes = random.sample(haslist,1027)
for eachline in selectedhashes:
[tab]outfile.write(eachline)

Finally I opened up my cracking program, Cain & Abel. We aren't going to be using the Abel part of this program, just the Cain. I went to the cracking tab and imported hashes from my new file, readytocrack.txt. Previously I downloaded the large wordlist from the John the Ripper web site. I didn't use the word list that you pay for because I want to simulate a script kiddie with no money attacking our accounts with a word list. I ran the cracker against the NTLM hashes with the default settings to mutate the words in the list.

And here are the statistics: We can be 95 percent certain that 30.5% of our passwords can be broken by a simple word list with basic mutation. The margin of error is +/- 3%.

After examining the broken passwords, I also found these tidbits. 43.5% of the cracked passwords had at least two character types, 7.5% of the cracked passwords had at least three character types, and 0% had four character types.
6.8% of the cracked password comply with the upcoming password standard.
Average password length was 7.527 characters in length with a standard deviation of 1.366 characters.

My plan is to repeat this process every month for a few months after the new password policy goes into effect. I expect that I should see the average password length to increase slightly and the standard deviation to decrease slightly. I also expect that the percentage of passwords with only 2 character types will drop and the number with three will increase. I also hope to see a trend where the number of passwords cracked by this method will decrease.

I have also downloaded a large rainbow table that I'm going to run my sample through. I expect that the rainbow table will break more passwords than the word list and will give me more insight into the state of our passwords as they exist right now.