I'm trying to understand compilation and interpretation, step by step figuring out a total image. So I came up to a question while reading http://www.cs.man.ac.uk/~pjj/farrell/comp3.html this article
It says :
The next stage of the compiler is called the Parser. This part of the compiler has an understanding of the language's grammar. It is responsible for identifying syntax errors and for translating an error free program into internal data structures that can be interpreted or written out in another language.
But I couldn't figure out how tokenizer can properly tokenize the given stream which has the syntax error.
It should be stuck at there or give some wrong information to the parser. I mean isn't the tokenizing also a kind of translator?
So how it just overcome the lexical corrupted lines of code while tokenizing.
There is an example of token inside above link at The Tokenizer heading.
As I understand the form of the token seems like, if there is something wrong in the code token would be corrupted too.
Could you please clarify my misunderstanding?