Doing code generation is not overly rare. In fact, it's a common approach for implementing REST APIs. But it's difficult to do this robustly when generating code from arbitrary user input. It's also more difficult to maintain that code.
Before thinking about generating code, it would be a good idea to take a step back and think about whether this is actually necessary. Common alternatives:
In a lot of cases, you don't have to generate code. You have to configure your software instead. You give an example where the user might define a value to be returned from a method. But instead of generating textual code
def some_function():
return {{value_to_be_returned}}
...
you might directly read that value from a configuration data structure
def configure(config):
def some_function():
return config["value_to_be_returned"]
...
In more complex cases, you might define a class that receives the user's configuration in the constructor.
Using configuration is an extremely good idea when the user merely provides values, but not necessarily when the user's configuration also describes the structure of the program, such as control flow or names of methods. For this, we might have to look into additional techniques.
Applying the interpreter pattern is a fairly DIY approach, but it affords you maximum flexibility.
Instead of creating Python code from the user's description, you transform the user's description into a data model that you can execute in your own “virtual machine”.
This isn't a very well-known approach but it can be very powerful.
It makes it possible for the user to safely provide fragments of business logic.
The main drawback is the implementation effort.
I've written a related answer about implementing a custom query language.
It's possible to implement a custom language from scratch, but you could also piggy-back on Python syntax, use the ast module, and then use the resulting syntax data model to drive your interpreter. In the case of that answer, the “interpreter” is actually just a method called evaluate().
Python is an extremely dynamic language. A lot of what you can achieve with code generation you can also achieve with reflection or meta-programming. For example, let's say we want to fill out the following template:
@app.get
def {{endpoint}}():
return {{value}}
This can also be achieved without any string interpolation, for example:
def create_endpoints(app, endpoint, value):
# Implement the function body.
def implementation():
return value
# Adjust the function's metadata to match the required name,
# compare "Callable types"
# in <https://docs.python.org/3/reference/datamodel.html>.
implementation.__name__ = endpoint
implementation.__module__ = __name__
implementation.__qualname__ = make_qualname(__name__, endpoint)
# Evaluate the decorator.
implementation = app.get(implementation)
def make_qualname(module, name):
if module == "__main__":
return name
return f"{module}.{name}"
By combining these various strategies – configuration, interpretation for custom behaviour, reflection – you can already achieve a lot of stuff.
Sometimes, this is not enough. There is nothing inherently wrong with code generation, it's just more difficult to do in a robust manner. In principle, you could use the ast module to generate the necessary syntax without any interpolation issues, but that might be tricky in practice.
When generating textual code, it might be a good idea to use a template engine. Instead of creating your own system with search and replace or with str.format, you might want to use a more feature-rich template engine like Jinja2.
One of the big problems with generating Python code is handling indentation correctly. Jinja2 can potentially help here because you can use “filters” to adjust values before they are interpolated.
For example, let's say that we have a template for a function:
def some_function():
x = 2
y = 3
{{user_defined_code}}
return result
The user might now provide the following multi-line input:
xy = x * y
result = xy + y
This would lead to the invalid syntax
def some_function():
x = 2
y = 3
xy = x * y
result = xy + y
return result
In contrast, the Jinja indent helper would fix this, but requires you to hardcode the indentation width:
def some_function():
x = 2
y = 3
{{ user_defined_code | indent(width=4) }}
return result
This snippet also highlights another problem with such templates: user-define variables and variables from the template might overwrite each other. If the template wants to use an internal variable that shouldn't be accessed by the user code, it might be necessary to use an unusual naming convention. For example, a variable x that is supposed to remain private might have to be called __rest_generator_x. Languages with first-class code generation capabilities (like Lisp or Rust) use a concept called “hygienic macros” that side-steps these issues: variables are not just identified by their name, but also by where they were declared (template or user defined code).
