I think the number one pain that went away when I started using a build server was the just-before-the-release-panic where you learn all sorts of nitpicky details about what depenencies were never checked into source control, what projects stop compiling, what projects have tests that stop running, what recent changes broke the integration tests (the slow ones that actually hit the server or drive a web browser)
The time invested in using a build server (in my case, TeamCity), was paid back in the first release, easily.
-Number of Unit Tests
Well written unit tests execute in about the time it takes to compile & can be added as a post compilation step. So this isn't an argument for using builder server. The fact that devs don't usually set up unit tests as post compile step, is a good argument for it though. Also, if you have slow integration tests, a build server is happy to run these every night, overnight, but you won't get your investment in slow integration tests back unless a build server is running them for you-- people just don't manually kick of slow integration tests that often.
-How often changes are made
If the code changes infrequently, then I'll forget what unit test assemblies need to be run, etc, etc. If the code changes frequently, I'll want frequent compiles and test runs to make sure that I'm safely running with scissors (rapid changes = more human mistakes and the faster my build server can detect them for me, the faster I can run with scissors)
Before I had a source control, a build server, etc in place, I generally considered branching to be to much overhead to mentally keep track of anything. So I really don't know if it is easier or hard to do branch development w/o a build server. I think a team would be more likely to attempt branch development with a build server.
Having source control lets me be calm about devs checking in a disaster-- I can reverse it, but it is the build server that lets me find out about it, sometimes minutes or hours afterwards instead of weeks afterwards.