9

From Java 5 language guide:

When you see the colon (:) read it as "in".

Why not use in in the first place then?

This has been bugging me for years. Because it's inconsistent with the rest of the language. For example, in Java there are implements, extends, super for relations between types instead of symbols like in C++, Scala or Ruby.

In Java colon used in 5 contexts. Three of which are inherited from C. And other two was endorsed by Joshua Bloch. At least, that was he sais during "The closures controversy" talk. This comes up when he criticises usage of a colon for mapping as inconsistent with for-each semantics. Which to me seems odd because it's the for-each abused expected patterns. Like list_name/category: elements or laberl/term: meaning.

I've snooped around jcp and jsr, but did not found no sign of mailing list. No discussions on this matter found by google. Only newbies confused by the meaning of colon in for.


Main arguments against in provided so far:

  • requires new keyword; and
  • complicates lexing.

Let's look at relevant grammar definitions:

statement
    :   'for' '(' forControl ')' statement
    |   ...
    ;

forControl
    :   enhancedForControl
    |   forInit? ';' expression? ';' forUpdate?
    ;

enhancedForControl
    :   variableModifier* type variableDeclaratorId ':' expression
    ;

Change from : to in don't bring additional complexity or requires new keyword.

1

2 Answers 2

8

Normal parsers as they are generally taught have a lexer stage before the parser touches the input. The lexer (also “scanner” or “tokenizer”) chops the input into small tokens that are annotated with a type. This allows the main parser to use tokens as terminal elements rather than having to treat each character as a terminal, which leads to noticeable efficiency gains. In particular, the lexer can also remove all comments and white space. However, a separate tokenizer phase means that keywords cannot also be used as identifiers (unless the language supports stropping which has somewhat fallen out of favour, or prefixes all identifiers with a sigil like $foo).

Why? Let's assume we have a simple tokenizer that understands the following tokens:

FOR = 'for'
LPAREN = '('
RPAREN = ')'
IN = 'in'
IDENT = /\w+/
COLON = ':'
SEMICOLON = ';'

The tokenizer will always match the longest token, and prefer keywords over identifiers. So interesting will be lexed as IDENT:interesting, but in will be lexed as IN, never as IDENT:interesting. A code snippet like

for(var in expression)

will be translated to the token stream

FOR LPAREN IDENT:var IN IDENT:expression RPAREN

So far, that works. But any variable in would be lexed as the keyword IN rather than a variable, which would break code. The lexer does not keep any state between the tokens, and cannot know that in should usually be a variable except when we are in a for loop. Also, the following code should be legal:

for(in in expression)

The first in would be an identifier, the second would be a keyword.

There are two reactions to this problem:

Contextual keywords are confusing, let's reuse keywords instead.

Java has many reserved words, some of which have no use except for providing more helpful error messages to programmers switching to Java from C++. Adding new keywords breaks code. Adding contextual keywords is confusing to a reader of the code unless they have good syntax highlighting, and makes tooling difficult to implement because they'll have to use more advanced parsing techniques (see below).

When we want to extend the language, the only sane approach is to use symbols that previously were not legal in the language. In particular, these can't be identifiers. With the foreach loop syntax, Java reused the existing : keyword with a new meaning. With lambdas, Java added a -> keyword which could not previously occur in any legal program (--> would still be lexed as '--' '>' which is legal, and -> might have previously been lexed as '-', '>', but that sequence would be rejected by the parser).

Contextual keywords simplify languages, let's implement them

Lexers are indisputably useful. But instead of running a lexer before the parser, we can run them in tandem with the parser. Bottom-up parsers always know the the set of token types that would be acceptable at any given location. The parser can then request the lexer to match any of these types at the current position. In a for-each loop, the parser would be at the position denoted by · in the (simplified) grammar after the variable has been found:

for_loop = for_loop_cstyle | for_each_loop
for_loop_cstyle = 'for' '(' declaration · ';' expression ';' expression ')'
for_each_loop = 'for' '(' declaration · 'in' expression ')'

At that position, the legal tokens are SEMICOLON or IN, but not IDENT. A keyword in would be entirely unambiguous.

In this particular example, top-down parsers wouldn't have a problem either since we can rewrite the above grammar to

for_loop = 'for' '(' declaration · for_loop_rest ')'
for_loop_rest =  · ';' expression ';' expression
for_loop_rest = · 'in' expression

and all the tokens necessary for the decision can be seen without backtracking.

Consider usability

Java has always tended towards semantic and syntactic simplicity. For example, the language doesn't support operator overloading because it would make code far more complicated. So when deciding between in and : for a for-each loop syntax, we have to consider which is less confusing and more apparent to users. The extreme case would probably be

for (in in in in())
for (in in : in())

(Note: Java has separate namespaces for type names, variables, and methods. I think this was a mistake, mostly. This does not mean later language design has to add more mistakes.)

Which alternative provides clearer visual separations between the iteration variable and the iterated collection? Which alternative can be recognized more quickly when you glance at the code? I've found that separating symbols are better than a string of words when it comes to these criteria. Other languages have different values. E.g. Python spells out many operators in English so that they can be read naturally and are easy to understand, but those same properties can make it quite difficult to understand a piece of Python at a glance.

17

The for-each loop syntax was added in Java 5. You'd have to make in a language keyword, and adding keywords to a language later on is something you avoid at all costs because it breaks existing code - suddenly all variables named in cause a parsing error. enum was bad enough in that regard.

10
  • 2
    That seems... inconvenient. It presupposes the language designers were good enough to forecast most of the required keywords from the start. I'm not sure it's even necessary; decent compilers can determine whether or not a keyword is a variable by its context. Mar 16, 2016 at 22:18
  • 2
    I don't think Java has contextual keywords like C# has. So, using in would have meant to either introduce a new keyword, thus breaking backwards compatibility (System.in, anyone?) or introducing a previously unknown brand-new concept (contextual keywords). All for what gain? Mar 16, 2016 at 22:49
  • 2
    What harm of the contextual keywords? Mar 16, 2016 at 22:55
  • 5
    @user2418306 Adding a keyword does not have to break existing code, provided that the language isn't parsed with a separate lexer phase. In particular, an “in” in for(variable in expression) can never be ambiguous with any legal code, even if “in” can be used for variables. However, a separate lexer phase is quite common in many compiler toolchains. This would make it impossible or at least far more difficult to parse Java with some common parser generators. Keeping a language's syntax simple is usually good for all involved; not everyone needs syntactic monstrosities like C++ or Perl.
    – amon
    Mar 17, 2016 at 7:37
  • 1
    @RobertHarvey: Don't forget that const and goto are both reserved words in Java, but aren't used (yet).
    – TMN
    Mar 17, 2016 at 10:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.