Google
Web BSDatwork.com
Implementing Security in FreeBSD UNIX System, Part One
This is part one of a two-part security series on DaemonNews. Part one describes security in general terms. Part two will drill down into specific strategies for securing common services.
The funny thing about security is that we actually have quite a lot of it in the UNIX paradigm. We have users, groups, chroot, secure levels, and jails. The only problem is that we don't use any of it by default. Most services are run as root - pop3, ftp, ssh, ident, sendmail, talkd, named, ntpd, and even the ones that aren't, such as apache, barely touch the first layer of security offered in FreeBSD: each runs as its own user and group but doesn't bother with anything else. Even when programs such as named have security-minded options, they tend to not use them by default.
How we managed to get ourselves into this mindset is an exercise in understanding human nature. You learn how to do something, you get used to it, and until someone bonks you over the head with a two-by-four, you don't deviate from that learning. The problem isn't 'unsafe languages', it's 'unsafe people'. After all, writing a safe C program is actually quite easy, you simply use asprintf() instead of sprintf(), and so forth. Running a service inside a chroot or a jail isn't too difficult: It would just work for most services if only the authors added the options. You can do it without building it into the program, but fewer programs are capable of operating that way. For example, in order for named to run as another user it must still first bind to a low numbered port, which it must do as root.
Many of the services we depend on are personal projects or university projects: Kerberos, popper, apache, and so forth. Even if we have learned to write safer programs, inexperienced college kids have not. So there is an issue of education here that can only be addressed by the teachers. Is sprintf() still being taught at UC Berkeley and MIT, I wonder?
And if you think the open-source world is bad, try dipping your toes into the commercial world. It's not just worse, it's much worse. Without peer review, commercial products tend to be chock full of bugs and security issues. Even some of the more enlightened companies, such as Sun, take an awful amount of time to adopt security-minded language features. It took many years for Sun to adopt the C snprintf() routine and as far as I know they still have not adopted asprintf(). And Microsoft? Forget it... their idea of security is to integrate everything under the sun into the operating system and then blame the user for not using a firewall if a machine is compromised.
The job of properly implementing security is still a major undertaking for the system administrator. Machines are only as secure as you make them, and security concerns are always competing against the human necessity for convenience. FreeBSD is considered reasonably secure out of the box (OpenBSD is considered the most secure), but even with the reputation, all FreeBSD really does is simply not turn on most services and provide minimal user/group sandboxing of the few remaining.
FreeBSD does offer some security advice in comments so when people look at, say, the rc.conf configuration variables for named they will notice an example showing how to run named with more secure options. But we are still a far cry from offering true security for the services that can be run under our operating system. If you turn something like samba on in a FreeBSD box, you are as much at risk as you are running it on a Linux box or a Solaris box. In FreeBSD-land we like to think that we offer plenty of kernel features to allow system administrators to implement a considerably greater degree of security on their systems then with other operating systems, and we do. But it is still up to the administrators to use those features. Most services do not use them by default.
Part of my work is to attempt to secure more services by default. About a year ago I did a simple first-pass on /etc/inetd.conf and user/group restricted ntalk and comsat, and I added instructions in /etc/defaults/rc.conf on how to run named in a user/group sandbox (though the default is not to, something that will soon be fixed). Those of us in control of system distributions cannot get the authors of many of these services to program defensively, but we certainly can add layers of security on top of those services to minimize the damage. This is just the tip of the iceberg of what we can do.
Security is of particular concern in mixed networks where running the most secure software across all platforms can be inconvenient and where managers and users are almost universally uneducated in even the most common-sense security practices. Most internal networks in businesses are so completely insecure that a single machine compromise can make the entire network vulnerable. Even worse, as yesterday's mainframes and mini-computers become today's desktops in terms of processing power and storage, individual workstations in your network wind up being capable of running hundreds, even thousands of processes and dozens of services all simultaneously without breaking a sweat. Security is an age-old problem because a computer is essentially useless when it cannot communicate the information it holds to the outside, and the very fact that it must communicate is what makes it ever-vulnerable to attack.
As a UNIX system administrator, your job does not end at the network port. Your job is also to educate higher level managers (or dictate policy if you happen to be one) in regards to good security practices to prevent excessive convenience-seepage from making your systems vulnerable, and your job is to be skeptical of everything.
Security is much more then protecting against the stereotypical root compromise. An attacker often does not need to break root to cause serious damage or steal information. For example, an attacker able to break into a webserver can access anything the webserver can, including potentially your database of credit card numbers if you happen to be an E-commerce house, or perhaps log files containing personal information, or perhaps business secrets being stored on an internal network. The biggest disaster that can befall an ISP is not an attacker wiping a shell machine's disks (that's what backups are for!), but instead an attacker stealing data or password information (whether encrypted or not) which results in continuing damage for the long term.
What-if and The Layered Onion
The best approach to solving security related issues is to simply assume that everything in your network can be compromised. Start asking 'what-if' questions of yourself, then solve them using a generalist, layered-onion approach. The key is to not assume that you have successfully protected yourself from a compromise, even when you are fairly sure you have. Pose the question 'what-if X is compromised?' then answer it with 'I can do Y', then pose the question 'well, what-if Y is also compromised?' and answer it with 'then do Z', and repeat until your what-ifs become absurd :-).
It is not possible to guarantee any solution, but this process forces you to think. It teaches you how to recognize the vulnerabilities in your system and if you can't fix them, at least you can monitor them (another huge strength of UNIX that something like MS Windows can't even come close to matching). Detection is just as important as prevention. What use are four layers of security if a hacker has free reign to eat through all of them before you manage to detect him? The layered onion not only protects your systems better, it gives you more time to deal with the intrusions (or intrusion attempts) that do occur. Here are some examples:
Despite your best efforts someone breaks into the user account your e-commerce webserver is running under. What if they want to deface the site? Well, make all the directories and static files owned and writable by some other user and only readable by the webserver.
What if they want to steal all my credit card numbers? Then encrypt the credit card numbers everywhere, including in any database backend you might have and don't give the webserver access to the decryption key. That mitigates the damage by preventing millions of credit card numbers from being compromised all at once, but might not prevent an attacker from intercepting new credit card numbers entered by users.
What-if they break root? Don't store the decryption key for the credit cards on this host at all, instead run your credit card charging code on a different machine entirely and only install the decryption key on that machine.
What if the network is being sniffed? Only allow encrypted connections between machines (e.g. ssh), even for things going across a LAN. What you end up with is a reasonably secure machine and, more importantly, if someone were able to break into it the amount of damage they could do would be limited (and here stealing a database holding thousands of credit cards is much more damaging then placing a backdoor on the machine!). This mitigation could have the effect of forcing the attacker to stay online or making easily auditable changes, making him more detectable, and could significantly reduce the amount of damage the attacker is able to affect.
Despite a magnificent effort on your part, one of your work-at-home employees with a cable modem connection is a little less diligent than usual. What if a hacker breaks into his machine and is able to steal his ssh keys to get to a company machine? Well, you can require that your ssh users password-protect their private key (e.g. when ssh-keygen asks for a password, type one in!).
What if the hacker figures out or steals the password? You can restrict ssh connections by IP or domain by using "from=" feature of the .ssh/authorized_keys file.
What if the hacker breaks in from your employee's machine directly and wants to steal other user's ssh keys and passwords? You could require that your users create ssh keys only on their workstations and never set up keys on the multi-user machines, limiting what could be stolen from the multi-user machine. FreeBSD provides some protection for the password file, but exploits exposing the encrypted password file have been found in the past and an attacker could then download the file and run a password breaker from the safety of his own machine. You could move the password file off-host through the use of kerberos or NIS+, or not have passwords at all ('*' them out), requiring ssh-key based authorization only.
Despite a supreme effort on your part, what if a visiting client plugs his laptop into your LAN and is able to sniff packets? The first line of defense is to use a switch instead of a hub. The second line of defense is to use a managed switch that completely isolates each ethernet port. The third line of defense is lock-in the MAC and IP addresses for each port. Of course, using a managed switch is relatively expensive compared to other solutions.
What if the user compromises the LAN anyway? You could encrypt all host-host communications with IPSEC, and/or you could require that only ssh (or telnet/rlogin in encrypted-mode) be used for all communications, even those staying within your LAN. As an example of this, all internal communication except for NFS is done over ssh, at work and even in my home network, and the only thing I use NFS for (since I'm not IPSEC-encrypting it) is for occasional read-only mounts for various things. In other words, I'm paranoid, but I'm not *that* paranoid. As a system administrator you have to balance convenience against the security. If you can find a sure fire all-encompassing solution that's really great, but you should never trust any solution to solve all your problems.
Always go for a generalist solution if you can. For example, if you want to protect yourself from network sniffing you might be able to use IPSEC to encrypt everything across the board. If you want to prevent someone from stealing your password file then the obvious answer is to move the password file off the 'weak' machine entirely. If you want to protect your ntp daemon the 'restrict' clause in ntp.conf may not be enough (it wasn't for the most recent exploit found for ntpd!), but ipfw or ipfilter could easily be used to truly stomp down on unauthorized access. If you want to protect against suid-based exploits, consider turning suid off using a mount option (nosuid,nosgid)) rather then chmoding program binaries. Remember that even these solutions aren't guaranteed... for example, ipfilter was recently found to have a fragment bug in it that a smart attacker could use to bypass the filter entirely. This is why a layered onion works: There might be little holes in each layer, but usually the holes in several layers don't line up. An attacker who gets through one hole is likely to hit a brick wall in the next layer.
You can't win them all
Sometimes you just can't win. An excellent example of this is email. Most of the world uses POP3 to get email, even though IMAP is potentially a better protocol. Our windows users and even some of our UNIX users use POP3. POP3 is not encrypted, passwords are only optionally encrypted, so the email spurting back and forth across the network is almost always in-the-clear. While it is possible to install software to encrypt email - PGP, for example, most people don't bother because it is usually just too cumbersome. Additionally, to be effective, email often must be accessible from home. Some of your users might even access it from a public terminal, virtually guaranteeing that their popbox will be sniffed.
This is a case where you might not be able to win. You might not be able to dictate a policy that has an acceptable level of security without excessive inconvenience to the employees. If you do wind up having to provide external visibility for your POP3 server you can still use the layered-onion approach to mitigate the damage that could occur from that visibility. You can certainly run the pop3 daemon as something other than root. You could probably run the POP3 daemon in a jail or chrooted, or use ipfw or ipfilter to reduce its exposure. You could isolate the mail machine from other services to avoid cross pollution if the mail server were to become compromised. You can require that POP3 passwords be changed every so often, and that they definitely be different from the windows logon passwords those people use to access file sharing from their laptop. In other words, you can severely limit the consequences of a stolen email account but you may not be able to prevent it.
Another example might be your DNS subsystem. The BIND distribution is notorious for containing holes. There are other name server packages available, such as DJBDNS, which are considerably more secure, but which also lack features which you might need. I personally still use BIND despite the fact that I know there are probably more holes yet to be found. I've given up on trying to run anything else... BIND is just too convenient, but that doesn't mean I haven't slopped on a couple of layers of security. In FreeBSD you can trivially run your named in a jail, in a chroot'ed environment, and as a different user and group (rather then as root). You can do all three together so if someone were able to crack BIND they wouldn't be able to do much beyond that.
All exposed subsystems have similar issues.
If there is one place where as a sysop you should dig your feet in and not compromise it would be with login access to your boxes. Simply put, there is no excuse to allow unencrypted telnet, rsh, or rlogin connections into your machines in this day and age. Just don't do it. This isn't to say that using something like kerberized telnet or ssh guarantees that you won't be rooted, but it does wipe out a whole slew of possible exploits and makes it far easier to detect what remains.
General Points on Securing a FreeBSD Machine
This is just a general list. In the second part of the series I will give explicit examples on the securing of various services.
* Turn off all unnecessary services. Many people turn off inetd entirely these days by setting inetd_enable="NO" in their /etc/rc.conf.
* Audit your security setup. For example, use sockstat to see what ports are exposed. Don't just assume that you've turned a service off. You might be surprised at what you see.
* Did you know you can bind services to specific IP addresses? For example, if you are running named on a host to serve as a recursive server only for that host, you can bind it to localhost (127.0.0.1) so only the local host can talk to it. If you are running a server with several network interfaces you can bind internal services only to the internal (usually a 10. network) interface(s).
* Use IPFW and/or IPFILTER. Just turning on the IPFIREWALL and IPFILTER kernel options offers some protection. For example, simply turning on IPFIREWALL will protect you from spoofed localhost packets. Learning how to create your own rules makes it even better. For example, I use IPFIREWALL (see man ipfw) to prevent anyone from outside a subnet from spoofing internal IP addresses. I do the same thing on our border routers. Learn how to use inclusive filters. Do not use an excluding filter. That is, use filters that say "allow A, B and C and trash everything else" rather then filters that say "disallow D, E, and F and allow everything else". If machines on a subnet are not supposed to have any external visibility, even to other subnets, then disallow incoming connections to those machines from outside their subnet. Don't be shy... it's much easier to make a network secure from the get-go then to make it permissive and then have to fight managers and users months later who began to 'leak' services and programs that you now want to block. FreeBSD is incredibly flexible when it comes to firewall operations.
* Jails. A Jail is a FreeBSD specific feature that allows untrusted programs to run as root with only limited access to certain aspects of the system. See the jail manual page for more information. Jails can be used to bind an environment (programs and services) to a specific IP address and are especially useful in this regard when used on a machine with many virtual hosts to create separate, isolated environments. Combined with chrooting, it is actually possible to give individual users their own 'virtual machines' on which they have 'root' without actually giving them the real root. More importantly, it is possible to segregate services so an exploit of one service is not able to bleed into others. Jails become much more important as machines get more powerful and run more and more services.
* Chrooting - The chroot command may be used to run a program inside a subdirectory and not give that program any access to the filesystem outside that subdirectory. Unfortunately chroot must generally be built into a program as an option (for example, named has the "-t" option to set itself up and then chroot to handle actual runtime operations). Attempting to chroot a shell can lead to a number of problems with programs which attempt to load shared libraries or access devices (such as /dev/null and /dev/zero), though it is possible to do.
* Security levels - FreeBSD implements a kernel security level feature (man securelevel). This feature can be used to disable access to raw devices, disable the ability to load kernel modules, and enforce chflags permissions on files. A higher securelevel also enforces chflags permissions on files, such as enforcing append-only operations on log files and disallowing any changes whatsoever to certain files or directories, even by root. Running at a higher securelevel is extremely inconvenient and most people don't do it, but it is well worth the effort. I believe that the FreeBSD securelevel mechanism can be made more useful in time if combined with jails. Such features are under discussion.
* Service isolation - for the services most exposed to the internet, such as your webserver, or for services that you do not entirely trust, the easiest first-onion-layer is to isolate that service on its own physical machine. If this is not practical (due to cost or space issues), definitely use the chroot, jail, and UNIX user and group features (run as something other then root) to mitigate any damage.
* Backups - What if all my efforts weren't enough and someone breaks in and wipes all my data? Answer: Make backups and make sure you can get to them when something fails. Don't just assume that a tape backup system will be able to restore a machine -- actually do it to see whether it works and how long it takes to do. Sometimes restoration is time-critical and tapes are just too slow. Consider packing a bunch of high capacity hard drives into a backup machine and backing up your systems to that instead of to tape, but beware that you have to protect the backup machine itself from exploitation too. When you take the online-backup approach, you still need an offline backup (such as DVD or tape) just in case the online backup is compromised. Another advantage of having an online incremental backup is that it is very easy to use it to perform independent audits of changes made to your systems.
* Logging - FreeBSD's default install runs a number of security-checking scripts from cron. Read the email they send root every day, week, and month. Even better, have your production machines syslog to some other machine rather then to local files in order to deny an attacker the ability to erase his tracks.
* Restrict automated remote access - Many sysops manage automated operations on remote machines by writing cron jobs which rsh to said remote machines, then runs a command on the remote machine (such as when doing backups or log file processing). This is a glaring security hole. You can't use NIS+ or Kerberos because you are trying to automate the operation, but there are some things you can do. You can use ssh for such accessed based on an ssh key pair that does not have a password, and on the server side you can run a specific command rather then run the command the client sends over by using the command="..." feature in the .ssh/authorized_keys file. This combined with IP/domain restrictions allows you to construct a reasonably secure infrastructure for automated jobs.
* Disallow unencrypted login services - Do not allow unencrypted logins. If using rlogind or telnetd, require kerberos authentication and encryption and disallow unencrypted links.
* Suid and sgid programs are a thorn in your side, always, but it is usually a hassle to go mess around with the permissions on binaries because the next installworld will restore them. Consider mounting filesystems with the nosuid and nosgid mount options.
* Monitor your system. Create a monitoring regimen and stick to it. By default FreeBSD generates daily, weekly, and monthly security reports, but they're useless if you don't read them. Syslog is a great tool but it's even more useful if you log to another host to prevent log edits. I could write a whole book on monitoring methodologies.
Detection (or: Script Kiddies are your Friends, but Blow Them Away anyway)
During my Best Internet years we had lots of hack attempts, and a few successes (many more successes with our IRIX boxes then our FreeBSD boxes later, mainly due to SGI never updating their utilities base and us having to learn the hard way). The hackers that went after BEST were typically either IRC weenies or script kiddies attracted to our good connectivity and bandwidth. IRC weenies break into accounts and run IRC BOTs. They try to break root and run IRC SERVERs if they can (as well as install suid root shells and the like). Script kiddies tended to break into machines solely for the purpose of breaking root and installing suid root shells and the like.
This is good because the kids almost never do anything destructive, their hacks are easy to detect (I mean, come on, running a program that listens on or connects to a socket is trivial to detect no matter what argv[0] says!), and there are so many of them that they tended to get into the machines before the hard core hackers could. We used them to hone our detection skills (these kids often used user accounts compromised due to the users giving away their password or logging in from public libraries and getting sniffed). But don't get friendly with them. If one happens your way, squeeze out as much information as you can and then throw him off the system.
The honed skill that I find the most valuable is the art of deception. Instead of trying to lock up things like log files, utmp, wtmp, and process accounting logs, one instead gets sneaky. Don't lock up log files with chflags or anything else, instead have syslog dup them in realtime to another machine and then diff them to see if anyone has modified the originals. Make a semi-realtime off-host backup of other system files and check for missing or inserted data. Instead of mounting / and /usr read-only, we instead synchronize them from a remote master and do whole filesystem comparisons nightly to see if anyone has made any unauthorized modifications.
The key to detection is to give the hacker plenty of rope to hang himself with. Most hackers are stupid (with a capital S), but even the smarter ones will try to 'clean up' after themselves by editing log files. The point here is to not completely limit the 'damage' a hacker can do if they do break in, just limit it enough so you don't have to spend hours restoring the machine and monitor the rest. Your first line of defense is your layered onion, but your insurance comes from timely detection and a comparative base.
More to come in part two of this series.
Original URL: http://www.daemonnews.org/200108/security_overv...
Content Copyright © Original Author