I'm playing around since a while with building a little open source project based on the output of facebooks pfff tool, that can parse multiple programming languages and output some kind of unified AST.

For example, given a simple PHP file like this:


class FooBar {
    public function functionA(string $arg1)
        $localVar = 1;
        print $undefinedVar;
        print $arg1;


The output of the tool is something like this:

   {c_type=ClassRegular(i_1); c_name=Name(("FooBar", i_2)); c_tparams=None;
    c_extends=None; c_implements=None; c_attrs=None;
               {f_tok=i_4; f_type=MethodRegular; f_attrs=None;
                f_modifiers=[(Public, i_5)]; f_ref=None;
                f_name=Name(("functionA", i_6)); f_tparams=None;
                             {p_variadic=None; p_attrs=None; p_modifier=None;
                                            XName([QI(Name(("string", i_8)))]),
                              p_ref=None; p_name=DName(("arg1", i_9));
                              p_default=None; })],
                             IdVar(DName(("localVar", i_12)), Ref(NoScope)),
                             i_13, Sc(C(Int(("1", i_14))))), i_15);
                             IdVar(DName(("undefinedVar", i_17)),
                               Ref(NoScope))), i_18);
                             IdVar(DName(("arg1", i_20)), Ref(NoScope))),

Having a look at the output I thought it would be quite straightforward to create parser that can give me all class names or all methods or all functions etc.

I wrote a very ugly first parser that iterates over the output as a string, looks for keywords, like ClassDef, then tries to match the next brace and so on. I managed to extract some stuff and it works ok, but the code got convoluted and ugly very fast, to the point where I just don't want to work with it anymore and reconsider that approach. Adding more features to extract more data from the AST is hardly possible anymore.

So I did some more research and found basically two approaches:

  • Roll my own recursive descent parser, define a formal grammar for it. I read through the wiki page for it, but had a very hard time to get going as I don't have much experience with creating parsers. I read through some more tutorials, but I'm still not so sure how to start on parsing a complex string like the above.
  • Use some kind of parser framework like ANTLR. Apart from the fact that I would like to not use Java im my project, I'm also not so sure if that's the easier approach.

My ideal solution would be to either have an in-memory structure that allows me to query the parsed string like "give me all methods with classes" or so or convert the output from above to XML.

Currently I'm quite stuck at how to approach parsing the above structure. It would be much appreciated if someone with more experience in creating parsers could give me a hint in the right direction, maybe I'm also making it too complicated, but defining a grammar and then maybe writing the parser for it for the above structure seems like a big undertaking for me and I don't really know where to start.

  • Maybe the tool has alternative output formats like JSON? And the shown output format seems vaguely familiar – is it perhaps the default debug dump format for whatever languages was used in the tool? Then perhaps this language can also convert the output back to a data structure? I don't think it is sensible to develop a parser from scratch. – amon Jun 24 '17 at 8:52
  • @amon I also thought there might be an option for xml/json, but apparently not. You bring up a good point, it looks like some kind of dump. The tool itself is written in OCaml, maybe it's some kind of OCaml serialization format. I will have a look into that. – Max Jun 25 '17 at 10:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.