Questions tagged [lexer]

a lexer is a program performing lexical analysis: it converts a sequence of characters into a sequence of tokens.

Filter by
Sorted by
Tagged with
0
votes
4answers
190 views

Should my lexer allow what is obviously a syntax error?

This is kinda like a concrete version of the question Coming up with tokens for a lexer. I'm writing a lexer for a small subset of HTML. I'm wondering what should I do when the input stream ends and ...
0
votes
1answer
59 views

Differences between enumeration-based and hierarchical token typing

When writing a lexer/parser, why/when would an advised developer chose to define the tokens' types through an enumeration field/type hierarchy? The closest question I've found here so far was Lexing: ...
0
votes
1answer
246 views

Do any programming languages let you use other languages without restriction within them?

This may be a stupid question, and it would certainly take one Hell of a lexer, but do any extant programming languages allow you to do something like: c# (1.2) { // c# code } Perl (5) { # ...
-2
votes
1answer
119 views

Multiple variable declaration, multiple variable assignment, context-sensitive 'in' statement

Lately I've been playing with writing my own programming language, following the excellent Crafting Interpreters book but I've hit something of a snag. I'd like to extend the parser to accept ...
0
votes
1answer
255 views

In which way should I structure a compiler/Interpeter? [closed]

For a couple of months now Im writing a interpeter / compiler for a programming language in C#. I have encountered some issues recently which make the code feel incorrectly written Classes change a ...
3
votes
3answers
334 views

How to create simulator for distributed algorithms written in simple language

I started development of simulator for simulation of distributed algorithms in language C. My work consist of creating simple language for algorithm description and simulator which takes the described ...
53
votes
4answers
9k views

When to use a Parser Combinator? When to use a Parser Generator?

I've taken a deep dive into the world of parsers recently, wanting to create my own programming language. However, I found out that there exist two somewhat different approaches of writing parsers: ...
12
votes
1answer
1k views

What is the procedure that is followed when writing a lexer based upon a grammar?

While reading through an answer to the question Clarification about Grammars , Lexers and Parsers, the answer stated that: [...] a BNF grammar contains all the rules you need for lexical analysis ...
4
votes
1answer
114 views

Is there any need to have an evaluation stage for a lexer to properly work?

Wikipedia says that the lexical process is often divided into two phases. The scanning process, and the evaluation process. Wikipedia defines: The scanning process as: The first stage, the scanner,...
5
votes
1answer
358 views

How do parsers search for token patterns?

Could you explain how parsers search for token patterns like in markdown? I probably could come up with something matching only the braces pattern []() as soon as nested patterns are involved it ...
0
votes
3answers
133 views

What is a good strategy for reading XML like hiearchical text data?

I want to read data in a format like the following using Java. [scenario] id=my_first_scenario next_scenario=null name=_"My First Scenario." map_data="{~add-ons/my_first_campaign/...
5
votes
1answer
518 views

How should a lexer deal with multi-line statements(eg. Functions definitions, Control-Flow statements)?

tl;dr-ers: How does a lexer normally deal with none-inline statements. statements that do not end with a specified statement delimiter. Such as control flow statements? I believe that I have a ...
3
votes
2answers
4k views

Is it appropite for a tokenizer to use regex to gather tokens?

I have recently caught the 'Toy Language' bug, and have been experimenting with various simple tokenizer configurations. The most recent one, makes use of the boost.regex library to identify and get ...
21
votes
3answers
3k views

What should be the datatype of the tokens a lexer returns to its parser?

As said in the title, which data type should a lexer return/give the parser? When reading the lexical analysis article that Wikipedia has, it stated that: In computer science, lexical analysis is ...
0
votes
1answer
311 views

Is it possible to parse my grammar with multi-line productions without backtracking?

I'm playing around with creating a parser in PHP for my own flavor of BNF, to match strings against grammar in this BNF variant. It's still a work in progress and subject to change (I may even end up ...
-2
votes
1answer
559 views

How do I create my own Objective-C to Swift converter? [closed]

I'm really interested in writing my own converter. I know C++/Python/Objective-C/Swift and a little Haskell. There are website like objectivec2swift and iswift.org, which can convert OC to Swift ...
4
votes
6answers
2k views

Should I use a source-to-source or a traditional compiler in order to develop my own Programming Language?

I'm really interested in writing my own general-purpose high-level programming language, but I'm somewhat confused. I know that Python and Ruby were written in C, which makes me wonder that if I want ...
1
vote
2answers
238 views

Passing context around AST nodes

I have various objects inside my AST, such as IfBlock, FunctionBlock, LogicExpression, etc. All of those objects share a context, which is basically a hashmap with some variables. It's a very simple ...
6
votes
4answers
3k views

When to use ANTLR and when to use a parsing library

I've always wanted to learn how to write a compiler - I've decided to use ANTLR, and am currently reading through the book (its very good by the way) I'm pretty new to this, so go easy, but the jist ...
24
votes
6answers
4k views

Why implement a lexer as a 2d array and a giant switch?

I'm slowly working to finish my degree, and this semester is Compilers 101. We're using the Dragon Book. Shortly into the course and we're talking about lexical analysis and how it can be implemented ...
4
votes
2answers
509 views

Lexer/Parser for multidimensional Languages

How does Lexer/Parser work in a 2D programming language like Funciton in order to transform such an unusual source-code to the correct AST?
3
votes
1answer
904 views

Should a lexer un-escape strings?

Is it a lexer's job to undo any escaping done to a string literal? For example: "Me: \"Hello World!\"" Becomes: Me: "Hello World!" Should this conversion be done inside the lexer? I am guessing it ...
4
votes
4answers
1k views

Lexing: One token per operator, or one universal operator token?

When lexing, what would be the best way to tokenize operators? Would one just create a BinaryOperator token, or a separate token for every single binary operator? Examples: PlusOperator, MinusOperator,...
0
votes
2answers
378 views

Storing tokens during lexing stage

I am currently implementing a lexer that breaks XML files up into tokens, I'm considering ways of passing the tokens onto a parser to create a more useful data structure out of said tokens - my ...
0
votes
0answers
834 views

Parsing Razor-style Templates

I want to build a template engine (ITT not another template engine...) based on Razor. I've been at it for quite a long time not getting anywhere and quite frankly I'm at my limit. I've tried rolling ...
6
votes
2answers
1k views

Can you apply the same lexer rules to all programming languages?

I'm trying to understand the theory behind a lexer with the purpose of building one (just for my own fun and experience and to compensate for not taking proper CS courses :)). What I have yet to ...
8
votes
3answers
1k views

Clarification about Grammars , Lexers and Parsers

Background info (May Skip): I am working on a task we have been set at uni in which we have to design a grammar for a DSL we have been provided with. The grammar must be in BNF or EBNF. As well as ...
5
votes
3answers
1k views

What is the proper way to distinguish between keywords and identifiers?

I'm aware that most modern languages use reserved words to prevent things like keywords from being used as identifiers. Reserved words aside, let's assume a language that allows keywords to be used ...
4
votes
3answers
284 views

Is this a viable approach to resolving multiple matches in a lexer?

I'm writing a lexer in JavaScript. It's pretty typical - rules are specified with regular expressions and produce a token. I am unsure of the best way to handle when multiple rules are matched. The ...
5
votes
1answer
966 views

Chosing a parser for a code beautifier

I'm in the planning stage of making a code beautifier (similar to AStyle or Uncrustify) - originally I was going to just contribute to one of those projects, but reviewing their source led me to the ...
1
vote
4answers
1k views

Is it possible to create a single tokenizer to parse this?

This extends off this other Q&A thread, but is going into details that are out of scope from the original question. I am generating a parser that is to parse a context-sensitive grammar which can ...
-3
votes
2answers
326 views

What is the name of a grammar which can change its tokenizer in mid parse?

I was creating a language and discovered that my language tokenizer would have to change depending where in the parse it is. I.e. abc[1] would be parsed as 4 tokens (abc, [, 1, ]), where as { abc[1] }...
3
votes
1answer
744 views

Practical reference for learning about graph reduction

Are there any practical references (with actual examples) for getting started implementing a small, lazy functional programming language with graph reduction? A reference that included the lexing and ...
2
votes
1answer
654 views

How are “Json.org”-like specs graphs called and how can I generate them?

In http://www.json.org Douglas Crockford shows the specs of the JSON format in two interesting ways: In the right side column he lists a text spec that looks like a YACC or LEX listing. In the main ...
2
votes
1answer
266 views

What follows after lexical analysis?

I'm working on a toy compiler (for some simple language like PL/0) and I have my lexer up and running. At this point I should start working on building the parse tree, but before I start I was ...
2
votes
5answers
4k views

Understanding hand written lexers

I am going to make a compiler for C (C99; I own the standards PDF), written in C (go figure) and looking up on how compilers work on Wikipedia has told me a lot. However, after reading up on lexers ...
14
votes
5answers
4k views

Coming up with tokens for a lexer

I'm writing a parser for a markup language that I have created (writing in python, but that's not really relevant to this question -- in fact if this seems like a bad idea, I'd love a suggestion for a ...
9
votes
5answers
2k views

Lexical Analysis without regular expressions

I've been looking at a few lexers in various higher level langauges (Python, PHP, Javascript among others) and they all seem to use regular expressions in one form or another. While I'm sure regex's ...
17
votes
5answers
6k views

Are separate parsing and lexing passes good practice with parser combinators?

When I began to use parser combinators my first reaction was a sense of liberation from what felt like an artificial distinction between parsing and lexing. All of a sudden everything was just ...
16
votes
4answers
31k views

Writing a lexer in C++

What are good resources on how to write a lexer in C++ (books, tutorials, documents), what are some good techniques and practices? I have looked on the internet and everyone says to use a lexer ...