This is a very bad situation, you need to refactor this ASAP - this is technical debt in it's worst - you don't even know how important the code *really* is - only speculate that it's important.

As to solutions ASAP:  
Something that can be done is adding a custom compilation step. If you use Maven that is actually fairly simple to do, other automated build systems are likely to cope with this as well. Write a file with a *different* extension than .java and add a custom step that searches your source for files like that and regenerates the actual .java. You may also want to add a **huge** disclaimer on the auto-generated file explaining not to modify it.

Pros vs using a once-generated file: Your developers will not get their changes to the .java working. If they actually run the code on their machine before committing they will find that their changes have no effect (hah). And then maybe they will read the disclaimer. You are absolutely right in not trusting your teammates and your future self with remembering that this particular file has to be changed in a different way. It also allows automatic testing as wel, as JUnit will compile your program before running tests (and regenerate the file as well)