1

I wish to develop an application that can generate code based on an user input.

Long story short: a user gives a formal description of a Resource (can be viewed as a REST resource) and based on this description I wish to generate an entire codebase that contains CRUD operations on that given resource.

I am currently having some "template" txt files in which I insert replaceable fields.

A very simple example would be (not related to my app):

template.txt

def generator():
    return {value_to_be_returned}

And in my main application i replace those {..} fields with the according values based on the user input.

My question is, are there common patterns that facilitate the generation of code .. with code? I am not sure whether my way is actually an acceptable Software Engineering practice.

1 Answer 1

3

Doing code generation is not overly rare. In fact, it's a common approach for implementing REST APIs. But it's difficult to do this robustly when generating code from arbitrary user input. It's also more difficult to maintain that code.

Before thinking about generating code, it would be a good idea to take a step back and think about whether this is actually necessary. Common alternatives:

  • In a lot of cases, you don't have to generate code. You have to configure your software instead. You give an example where the user might define a value to be returned from a method. But instead of generating textual code

    def some_function():
        return {{value_to_be_returned}}
    
    ...
    

    you might directly read that value from a configuration data structure

    def configure(config):
    
        def some_function():
            return config["value_to_be_returned"]
    
        ...
    

    In more complex cases, you might define a class that receives the user's configuration in the constructor.

    Using configuration is an extremely good idea when the user merely provides values, but not necessarily when the user's configuration also describes the structure of the program, such as control flow or names of methods. For this, we might have to look into additional techniques.

  • Applying the interpreter pattern is a fairly DIY approach, but it affords you maximum flexibility. Instead of creating Python code from the user's description, you transform the user's description into a data model that you can execute in your own “virtual machine”.

    This isn't a very well-known approach but it can be very powerful. It makes it possible for the user to safely provide fragments of business logic. The main drawback is the implementation effort.

    I've written a related answer about implementing a custom query language. It's possible to implement a custom language from scratch, but you could also piggy-back on Python syntax, use the ast module, and then use the resulting syntax data model to drive your interpreter. In the case of that answer, the “interpreter” is actually just a method called evaluate().

  • Python is an extremely dynamic language. A lot of what you can achieve with code generation you can also achieve with reflection or meta-programming. For example, let's say we want to fill out the following template:

    @app.get
    def {{endpoint}}():
        return {{value}}
    

    This can also be achieved without any string interpolation, for example:

    def create_endpoints(app, endpoint, value):
        # Implement the function body.
        def implementation():
            return value
    
        # Adjust the function's metadata to match the required name,
        # compare "Callable types"
        # in <https://docs.python.org/3/reference/datamodel.html>.
        implementation.__name__ = endpoint
        implementation.__module__ = __name__
        implementation.__qualname__ = make_qualname(__name__, endpoint)
    
        # Evaluate the decorator.
        implementation = app.get(implementation)
    
    def make_qualname(module, name):
      if module == "__main__":
        return name
      return f"{module}.{name}"
    

By combining these various strategies – configuration, interpretation for custom behaviour, reflection – you can already achieve a lot of stuff.

Sometimes, this is not enough. There is nothing inherently wrong with code generation, it's just more difficult to do in a robust manner. In principle, you could use the ast module to generate the necessary syntax without any interpolation issues, but that might be tricky in practice.

When generating textual code, it might be a good idea to use a template engine. Instead of creating your own system with search and replace or with str.format, you might want to use a more feature-rich template engine like Jinja2.

One of the big problems with generating Python code is handling indentation correctly. Jinja2 can potentially help here because you can use “filters” to adjust values before they are interpolated.

For example, let's say that we have a template for a function:

def some_function():
    x = 2
    y = 3
    {{user_defined_code}}
    return result

The user might now provide the following multi-line input:

xy = x * y
result = xy + y

This would lead to the invalid syntax

def some_function():
    x = 2
    y = 3
    xy = x * y
result = xy + y
    return result

In contrast, the Jinja indent helper would fix this, but requires you to hardcode the indentation width:

def some_function():
    x = 2
    y = 3
    {{ user_defined_code | indent(width=4) }}
    return result

This snippet also highlights another problem with such templates: user-define variables and variables from the template might overwrite each other. If the template wants to use an internal variable that shouldn't be accessed by the user code, it might be necessary to use an unusual naming convention. For example, a variable x that is supposed to remain private might have to be called __rest_generator_x. Languages with first-class code generation capabilities (like Lisp or Rust) use a concept called “hygienic macros” that side-steps these issues: variables are not just identified by their name, but also by where they were declared (template or user defined code).

1
  • This is such an interesting answer! One thing I forgot to mention is that I want to actually generate textual code. The idea behind my application is that based on the user input I will generate an entire codebase and will deliver it to the user so it can be used anywhere and anytime. Basically a CRUD REST generator. Therefore I think using Jinja2 might be my best chance! Feb 6 at 6:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.