I'm currently working on an application which uses compiler directives to vary settings between environments. Would it be worth refactoring these to use settings in the web.config which will be modified with the config transforms?
Compiler directives are resolved at compile time, config is not. So its more about what you need to get out of it. Do you have to be able to change the configuration without recompiling?
Imagine you have a connectionstring to a database in your code. You do not want to connect to your production database when developing or testing. You can use the debug compiler directive to make sure you are automatically connecting to your test database when running in debug mode.
In another instance you change the database's location or connection settings. Now the running application doesn't connect to it anymore. You need to change the connection string in code and recompile it for it to work. This is when you want it to be in the config file.
You can also do both. Based on a compiler directive you fetch the debug or production connection string from the config.
If you go that route, there are more things you should consider. Having compiler directives all over your code is ugly (opinion). A better design would be to use dependency injection to resolve this connectionstring. I usually have an IEnvironmentSettings interface and multiple implementations of it for different purposes, prod, testing, unit testing.
Another thing you have to consider is that configs are plain text. Your connection string might contain a password, which will be readable to everyone with access to your application. This is not a problem with a server, but it is for client applications. You might want to encrypt the config in those instances or leave the sensitive information in code- you can easily decompile .Net executable so that isn't perfectly safe either.
I guess you have to decide what you want to get out of refactoring your code.