This is a difficult problem: how can such user-provided code be executed securely, and how can you ensure a good user experience?
If you run code on behalf of someone else, you can't trust that code. The code may try to hog computing resources (thus incurring costs for you and denying computing resources to other bidders), may try to hack your system or the code of other bidders, or may simply use the computing resources you provide as part of a botnet … the possibilities for abuse are endless. So you need to sandbox this code very tightly. The Java security model is a good start, and allows you to deny reflection and other APIs to untrusted code. However, you may still find it difficult to reliably limit the available computing resources.
The next problem is user experience. How can users write, debug, and test their code? Will you provide a kind of IDE? Which APIs do you offer to that code? How is it documented? Is there a test environment where bidding strategies can be tried out? How will you gather and display error messages?
If you compile Java code, you are limited by the Java security model. This may actually be sufficient, assuming that you are proficient with that security model (I am not) and can configure it appropriately. You could easily cache the compiled code, so that it doesn't have to be re-compiled for each execution. There is mature tooling around Java, so it would be straightforward to offer a kind of IDE.
If you use any other JVM-based language such as Groovy, the same considerations apply. In the end, everything will end up as JVM bytecode and will therefore be reasonably fast.
If an in-process sandbox is insufficient for your needs, consider process-level or operating-system-level sandboxes: containerization and virtualization. You would run user code in a separate process, and communicate over a socket. This doesn't just improve security, it also enables more scalability. However, any input/output must be serialized, which can limit performance.
Instead of building a textual DSL, you could also design a rule engine. Basically, you would provide configurable blocks that users can combine in a GUI. This is more accessible to non-programmers. By offering or not offering certain blocks (such as loops, function declarations, …) you can influence how expensive the resulting programs will be, while still offering more flexibility than a set of pre-configured strategies. As you control all available blocks, you can also create a clear security boundary without having to use complicated isolation features.
Unless your bids have to be resolved within seconds, there is another option: offer a REST API. That way, you don't have to execute any untrusted code. There is very mature tooling for creating APIs. There is a very clear security boundary. You can easily rate-limit excessive users. With an API, users could subscribe to a feed of events and react by posting their actions to your API endpoints. They are then totally free in how they write their strategies. However, they will have to provide their own computing resources. Whether this is a viable alternative depends largely on your market: is your product intended for consumers, or is it B2B or for developers?
But whatever you choose, other considerations like offering a usable testing environment can be much more complicated than just integrating a scripting language or offering an API.
I would suggest to ask yourself and/or clarify with stakeholders:
- What kind of flexibility is needed, and what are the possibilities for abuse with that flexibility?
- Is a rule engine sufficient to provide the needed flexibility?
- If not, would an API be appropriate?