0

I am thinking about how to build a language VM. I have been able to get some of the basic constructs right, including jumps to functions within the chunk of bytecode that is currently loaded.

But now I am thinking about how to handle (dynamic) calls to functions that are in other chunks of bytecode that have not yet been loaded.

After compilation, the calls to internal functions require only (in addition to parameters on the stack) a jump to a certain location.

But if the function being called is not in the current chunk, I assume I would need some textual representation (string) of the file and function name that I want to call, embedded in the bytecode somewhere.

Similarly, if functions in the current chunk need to be called from elsewhere, then presumably I need to have some lookup table in my bytecode to indicate the locations of functions so that external chunks can find them?

What additional information is typically stored in bytecode? / What are some strategies employed in language VMs to facilitate calling a function in a different program or chunk that is outside the current program?

2
  • 2
    It seems you are trying to create a (dynamic) linker, which has similar challenges to dispatching method calls in OOP systems. One potential strategy is to keep a global table of name→bytecode mappings and resolving external function calls by name. This can be optimized, e.g. by giving each chunk a table of the functions it requires, and filling that table during loading so that function calls can do an indirect jump through the table. This requires that chunks have metadata about required functions. In any case, chunks will provide metadata about their publicly accessible functions.
    – amon
    Commented Jul 6, 2023 at 13:23
  • 2
    Btw the Java linking process and bytecode format is well-specified. Its class files include full metadata about all methods in order to facilitate linking. However, since Java is a mature and evolved system, this linking process is more complicated than you probably need. WebAssembly's linking might offer a more modern and more C-oriented example.
    – amon
    Commented Jul 6, 2023 at 13:25

1 Answer 1

1

You are on the right track.

What you call a "chunk" is often called a "compilation unit" - the amount of source code that is compiled together. It seems you have already solved how to handle calls internally in a single compilation unit.

The simplest solution is of course to just have a single compilation unit for all the source code. But this may be impractical for larger programs (compilation too slow or requiring too much memory) and means you can't use precompiled libraries.

Therefore compilers allow code in a compilation unit to refer to functions defined in other compilation units. The output of a compilation is an intermediate format called object code (not related to objects in object-oriented code), which is almost-compiled code but with placeholders inserted for external references, and some metadata tables mapping identifiers to internal addresses.

Object code need at least two metadata tables: A table mapping names defined in the compilation unit to the address in the compiled code, and a table mapping placeholder to external names defined outside of the compilation unit.

Then a separate process "linking" takes all the object code as input and replaces the placeholders with the actual addresses. This can happen in different ways depending on the kind of language. In static linking, all the object files are combined into a single executable as part of the build process. In dynamic linking, they are kept as separate files, but at runtime are loaded into memory and the loader performs the linking. In the case of bytecode, it may be an JIT compiler performing the linking.

There is a problem when compiling multiple compilation units seperately: When source code uses a function defined in a different compilation unit, the compiler has to verify the external function exists and is called with the correct parameters - without having to actually parse the source code of the function in question, since this is what we wanted to avoid. In C this is solved by header files - files containing only the declaration of function signatures with no implementation. These header files can be included in multiple compilation units. In .net, the compiler instead looks op the external functions in the metadata tables in the object files (known as assemblies in .net) where the function is defined. The advantage is you avoid the redundancy of header files, the disadvantage is the compilation units have to be compiled in a certain order, and you can't have circular references between compilation units.

You might want to look into the C object file format and the .net assembly metadata format, to see different solutions to the linking problem.

1
  • Thanks. I think I have picked up the term "chunk" from some other literature or forums. "Compilation Unit" actually makes more sense. Commented Jul 11, 2023 at 13:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.