I am trying to build a static program analyzer for a proprietary progamming language for a school project, and am currently trying to implement the parser from scratch.

I was wondering, what are the pros and cons of

  1. Parsing in 2 passses (tokenization followed by parsing to create an AST) VS
  2. Parsing in 1 pass (tokenization + construction of AST done together)?

For 1, my current understanding of the benefits is that it follows the rule for seperation of concerns, making the code easier to extend, maintain and test over the course of the project. However are these the only benefits for tokenizing/lexing before parsing and constructing the AST?

1 Answer 1



  1. You can usually provide clearer error messages by separating lexing errors (which rarely stop processing) from parsing errors.
  2. Some language tooling (particularly string escaping/unescaping tools) needs part of the lexer, but not the parser. Separating them makes that work easier.

For a school project, it doesn’t much matter. Getting things working is more important than pristine design.

  • Hello! Thank you for the comment. The thing is for the school project I have to justify every single design decision that is why I am trying to get a clearer picture of what I am going to do, its pros and cons before I actually do it
    – Meowmi
    Mar 11, 2021 at 19:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.