If your friends list typically includes lists of memes, and how you felt about the soup last night, just go ahead and skip this entry. Really, it’s ok. I’ve got a pent up rant and some UNIX geekery to get out of my system.
Still reading? Welcome UNIX geeks!
At work we run a ad-hoc network consisting of Mac clients, Windows clients, Windows servers, Linux boxen, virtual machines, etc, etc. Like all small companies, we started with a single server and a Good Idea which morphed into Failure By Committee and eventually Apathy By Admin, followed by Don’t Touch The Brittle, All Important Server That We Can’t Possibly Replace, and finally What Were We Thinking, Let’s Start Over.
Sound familiar? I get tired of these cycles, as they happen all too frequently. I’ve stared at the problem for a long time, and it seems the culprit is feature creep and committees. As an admin, you want your stuff to be flexible. So, you present your flexible idea to the ‘group’, which is usually full of folks who Don’t Care and just want to get back to work, or are Flag Wavers, making a huge stink about some corner condition. So, suddenly, you find yourself hacking out a kludge to calm the whiners. 3 months later you wake up screaming at 3am when the kludge falls apart for the nth time, and you wonder what it would be like to drive a truck instead.
Well, getting back on topic… my main annoyance with the network was that we had lots and lots of password sources. This meant that passwords rarely changed, because it was too much pain to track them all down. It also meant lots of places where an attacker could exploit weaknesses to get to a password hash. It pretty much all melted down when this actually happened. A quiet little mac mini running OSX Server was sitting outside the firewall, serving mail for a single domain and not being looked after… and managed to fall to a brute force attack. Of course it used to be the main mail server, and of course it had everyone’s password hash in there, and of course a year later we were all using the same passwords.
Ideas were tossed around in regards to the single sign on. We rallied around a Windows 2003 Server running Active Directory for a while, with our Linux boxen authenticating off it, roaming profiles, etc, but the problem became that we were not using Group Policy really at all, and our Mac clients were being totally overlooked. As push came to shove, we managed to buy a $3600 brute, a dual proc, 8 core Xserve with 16GB of RAM and 3TB of disk. Yum.
I’ve spent the last week or two setting it up and getting it to work. It has not always been point and click easy, but, I am pretty happy right now.
Things started out with the replacement 1TB disks we shoved in. The Western Digital drives were not showing up at all to OSX. Of course, we bought 80GB drives from Apple at a huge markup just so we could have the drive caddys, which were dumped out and replaced right away, so we were going out on a limb not using Apple Approved drives. After a lot of googling, I tried setting a random jumper on the drive, supposedly converting it from 3Gb/s to 1.5Gb/s. Suddenly, the Xserve liked the drives, and we were installed.
The next day dealt with me setting the box up as an OpenDirectory master, screwing up, demoting, and promoting over and over, trying to get Kerberos to start. Turns out that since we were using a .local domain interally, and Apple has decreed that .local is for Bonjour services, it was bailing on even configuring it. Switching the domain to a public .net fixed it.
Next up was the idea of using an certificate for LDAP. Buy an official GoDaddy cert (which of course needs an intermediate cert to work right), install on the server (pull hair for a day on this), and then… nothing. I should have known better from my old days of hacking out LDAP/Kerberos services what the problem would be, but I had thought better of Apple… but I was wrong. You see, LDAP doesn’t know how to look up root certs, and as such, won’t trust -anyone- unless you import the root cert into the server. So buying a real cert was a waste of money, as I’d have to install a root cert on each client anyway. Plus, Apple at least allows for encrypting the LDAP stuff using Kerberos (how I am not entirely sure), so… I ended up turning off SSL and now my clients could bind to the domain.
Next step was going through the loop of getting the gods happy so that Mobile Profiles would work for Mac Clients. This was harder than it needed to be, but half of it was probably myself going off half cocked, ie, ‘oh, I know what to do… clickity clickity FAIL’. Another wasted day and a half.
So now we have authenticated and bound clients, mobile profiles. I added in PDC duties to the server, so windows machines could bind to it and offer up roaming profiles. Surprisingly, this was pretty easy.
DNS came next. I setup the box to serve the new .net, and point upstream to the Linux box (running tinydns) to handle recursive queries and the old .local for the time being. Trust me, I’m a tad nervous about running BIND instead of TinyDNS, but… Brian likes his GUI, so… 
This step was pretty easy.
DHCP rolled in next. I wish Apple used ISC’s DHCP server instead of bootp, so I could do dynamic client DHCP->DNS, but… pipe dreams and all. I enabled this, and burned up what was left of the old Windows AD Domain. Ah well..
All this done, we discovered in the midst of converting user accounts that one user managed to install the one popular Apple trojan in the wild onto his machine, the DNSChanger trojan. Gah. I removed it, but we ended up deciding to go with a beefier firewall and doing something I had never advocated before… limiting what users could send out of the network. Now we block outbound DNS except for the recursive internal DNS, and are planning a web proxy. Yum! I pressed an old dual proc P3 450 into firewall duty, and it seems almost…. overkill… for the job.
I wanted remote access, so I installed poptop’s pptpd server on the firewall. Static password mapping was up and running in 5 minutes. Since OSX has a GUI managed RADIUS server, I figured, no sweat to make it the authentication back end… oh no. Sure, it is built on FreeRadius, but Apple has had some fun with the guts. Firstly, my pppd on the firewall was not liking the reply packet due to trailing chars in the MS-CHAPV2-SUCCESS packet. I patched pppd and recompiled. That worked, but now the Xserve was failing to give out the MPPE encryption keys. I spent a lot of time debugging this. It should Just Work, but… Apple. As near as I can tell, what they have done is strip that out of the standard module and now require that a special account in the LDAP server handle the MPPE keys, which, of course, the RADIUS server knows nothing about, as they didn’t port that feature over. I ended up going a different route, enabling the pptpd server on the Xserve itself and routing that protocol in from the firewall. Except it didn’t work. After a lot of debugging, I found the pppd server was using the wrong vpn_ user account to get the MPPE keys. I found some ways to fix this, but ended up chasing my tail until I got rid of all the old users. And then came the funny… I set a default policy of having to change the initial password on the server, which bit me in the ass. How do you change the password on a system account? I figured it out, and finally, it all works. Except for default routes that make sense. But that is for tomorrow…
Managing the server from a Mac is pretty easy, but managing it via the GUI from anything else is annoying. Sure, you can turn on the VNC server, but it’s slow slow slow over a DSL link. I see ping times of more than 2 seconds when I am using VNC from home, over my 16Mb/s cable link! How annoying. The upshot is that I am getting pretty good at getting around the command line on the OSX server, which is really just a wacky BSD variant with some interesting command line utils. I think I can handle most tasks now from just a command line, and this makes me happy.
All in all, I like where this is going. I think we will have a good setup once we can centrally manage all our computing assets. With the user data on a RAID1, the actual workstations become just terminals, and not all important tomes of important, irreplacable user data. This makes me much happier. An of course, I can finally realize my dream of Single Sign On with secure Kerberos passwords.
I’m sure you stopped reading a long time back, but I for one feel better for having ranted.
leave a reply