In an interview with Bruce Schneier, TSA administrator Kip Hawley says:

We do many kinds of effectiveness tests at checkpoints daily. We use them to guide training and decisions on technology and operating procedures. We also do extensive and very sophisticated Red Team testing, and one of their jobs is to observe checkpoints and go back and figure out—based on inside knowledge of what we do—ways to beat the system.

One thing we’ve found in computer security is that this is not the only way to go about security testing. Sometimes an individual without inside knowledge of a system is more effective at breaking it than those with inside knowledge.

The successful hacker looks for patterns and anomalies, elicited from unexpected input. Inside knowledge of the system doesn’t necessarily help with this, and can lead the tester to go down the expected paths that have already been secured. They use the system in the way that the users are expected to use it. If, instead of using inside knowledge, they treat the system as a black box, then they can find unexpected paths that no one would have otherwise thought about. They can discover not just individual flaws, but huge holes in the original assumptions.

Inside-knowledge testing has a tendency to lead to a band-aid approach: find a problem, and put in a fix for that problem, without looking at the wider implications: what assumptions lead to this implementation and why did it lead to this flaw?

This isn’t to say that inside-knowledge penetration tests can’t also be useful, only that they shouldn’t be considered more useful. Sometimes they are, sometimes they’re not, but you can’t really know ahead of time. It’s always a good idea to look at your source code, of course. And having some knowledge of related fields will help identify patterns (just a few days ago I noticed that our “random” passwords consisted of two sets of numbers, six digits each; of course, they were a datestamp and a timestamp, just slightly mixed up; I’m not a hacker; it took me a few months using the system once or twice a week to notice this).

Now, it may be that Hawley simply didn’t discuss black-box testing; hopefully they are doing that as well, and using people with a hacker mindset.