I added a short answer saying that the answer is the same as for the question "do you care if the feature has bugs in it". A moderator didn't like that answer and deleted it, so I'll elaborate.
Anytime you write code there is a chance it will have bugs. To help possibly uncover these bugs, you should test the code. Now, in some cases it doesn't matter if the code has bugs or not. For example, internal code such as that in the question often carries very low risk - low risk for bugs because it is uncomplicated code, and because the cost to fix bugs is low. Other software, such as for medical equipment, has a very high risk, and bugs can be costly.
To decide if you should test software --any software-- you need to ask yourself if it matters if the code has bugs. Will a bug cause irreparable harm, or harm that is expensive to fix? Is a bug simply a minor inconvenience? Where does your software lie on that continuium? Answering that goes a long way toward answering if you should test your code.
Of course, testing won't guarantee your software is bug free, but even simple tests can raise your level of confidence high enough to know if you can depend on the code.
So, in your specific case, what is the cost of the dynamic re-compiling failing? Will it cause developers to potentially introduce bugs because they are running old code? Or, is it just a minor inconvenience because they can always run a manual step?