For servers I agree with Sesmar and go with Gentoo. PEQ and my internal test both run it without X and I couldn't be happier. Gentoo is simply awesome because you could very easily build your own work environment from the ground up.
Most of my boxes are Fedora (4 or 6) and they are used for development, some light server stuff, and even some personal desktop work. Basically, I use it as an all in one work horse. People complain that Fedora is bloated, but that is completely untrue if you only select the packages you want/need during your initial install. If you do that, not only with Fedora be as sleek, quick, and powerful as other forms of Linux, but also most of the stuff you will need is already installed, waiting for you. I once had a Fedora install that weighed in at 98MB and had everything I needed for that box. Bloated my ass
Being a musician, the fresh release of Ubuntu Studio has blown me away. I am finding that not only can I use my laptop for audio work without having to shell out thousands, but Ubuntu is also a very nice desktop environment. It isn't as good as Gentoo or Fedora when it comes to server based applications, but it crushes both in regards to my desktop needs (audio work, games, light development, and general internet stuff)