I thought I understood that we don’t care about security at the University, but I didn’t truly understand the extent of the problem. Today, a Friday evening, just after I got home from the office, I got a call that no one could log in to our web server.

My first thought was, shit, I fucked up! Because of previous problems with our authentication system, I had partially decoupled the web server from our authentication system. Rather than have the web server become inaccessible whenever the authentication system became inaccessible, I had the web server cache authentication information locally. I had manually pushed out the cache early in the afternoon. Did I screw something up?

So I went and looked at the password file, and no, every account was there.

But every account also had the same password hash.

I knew that the LDAP system administrator had been editing LDAP earlier in the day to restore an accidentally deleted account. Did they mistakenly reset everyone’s password to the same one in the process?

No, it turns out that we did this on purpose. In preparation for a mail migration, we created a second password for every account that is exactly the same. My caching script was picking up on this second password. It was easy enough to fix this in the caching script, but this blew my mind.

Every single system at the University that requires a password now not only has a back door but they all have the same back door! Grades, shell access, web pages, homework, class registration, tuition payments, campus cash, and transcripts. Everything that requires a password now has a single backdoor that lets someone with knowledge of that back door view and change data.

The guy who tried to justify it did so by saying that our security was ironclad, and even if it wasn’t the change was only for the weekend. That’s like saying I didn’t kill the victim, your honor, and if I did it was self-defense.

“But I was only aware of two systems that use LDAP: the junk mail system and the mail system. What else uses it?” I mean, Christ, what is the purpose of a central authentication system? Our main server for personal web pages and file transfers uses it. Everyone at the University has an account there. The web server uses it. People use it to access their grades, to register for classes, and to update their transcripts. Every single system that requires a password uses it! Stupid little servers off in the art department that are managed by the art department use it!

“I can’t think of any other way to do it.” The obvious way is to program the systems to talk only to each other during the migration. Even if you have to use a secondary password in LDAP, systems nowadays run their own LDAP server to cache (for a few seconds at most) LDAP information. The secondary password can be entered on the necessary systems. In this case there were only two.

Even if the system somehow doesn’t run a configurable LDAP, a replication server (even we run replicated LDAP servers; we’re smart, just smart enough to shoot ourselves in the head) can be set up. The secondary password is entered on that server, and only the affected systems are pointed at it. It’s seamless and it gives us another backup authentication server. It’s probably still a bad idea, but it isn’t so grotesquely bad that it endangers every authenticated system we have. It only (in this case) endangers every mail store.

“And anyway, what could they do even if they got the password. It’s only temporary. We back everything up.” If any one of the systems that access our LDAP server has already been hacked, and the hacker notices that every account now has a second password that is the same for all accounts, what do you want to think they’re going to start trying to crack that hash? One single password that gives them access to every user-related service on campus? No, they’ll probably ignore it and go back to playing tetris on the command line.

Absolutely.

I wrote earlier about the Slashdot Test. This one fails it so badly I can’t imagine what the reaction would be. Setting up a single, universal back door to every authenticated service at the University is never the “only solution”. I used to think that a central authentication system was a good idea because it keeps people from having to post all of their passwords on sticky notes on their monitors. But if it also makes it more likely that IT is going to royally fuck up, we haven’t gained much.

I only found out about this one because (a) I already didn’t trust our LDAP server, (b) I was caching to a format that only allowed for one password, and (c) the script’s design caused it to choose the wrong one. God only knows how much other crap like this is on our system.

But none of this is even the funny part. It’s just sad. The funny part is that before I was called, the person who first noticed the problem went to the network group and asked them if there had been any changes that might affect logging in to the web server. “No, there were no major changes today.” I don’t know what constitutes a major change, but adding an alternate password to all 19,000 accounts sounds like one to me.