Scannerless parsing

Advantages

• Only one metalanguage is needed
• Non-regular lexical structure is handled easily
• "Token classification" is unneeded which removes the need for design accommodations such as "the lexer hack" and language reserved words
• Grammars can be compositional

  Disadvantages

• Since the lexical scanning and syntactic parsing are combined, the resulting parser tends to be harder to understand and debug for more-complex languages

  Required extensions

• Follow restrictions, a limited form of "longest match"
• Reject productions, a limited form of negative matching
• Preference attributes to handle the dangling else construct in C-like languages
• Per-production transitions rather than per-nonterminal transitions in order to facilitate:
• * Associativity attributes, which prevent a self-reference in a particular production of a nonterminal from producing that same production
• * Precedence/priority rules, which prevent self-references in higher-precedence productions from producing lower-precedence productions

  Implementations

• is a parser for the modular Syntax Definition Formalism SDF, and is part of the ASF+SDF Meta-Environment and the Stratego/XT program transformation system.
• , a pure Java implementation of SGLR, also based on SDF.
• TXL supports character-level parsing.
• generates ANSI C code for scannerless GLR parsers.
• Spirit allows for both scannerless and scanner-based parsing.
• SBP is a scannerless parser for boolean grammars, written in Java.
• is a two phase scannerless parser generator with support for mapping the grammar rules into objects, written in Java.
• The Raku Grammars feature of the general purpose programming language Raku.
• is a scannerless parser written in pure Python.
• META II Has built in token parsers functions.
• TREE-META Like META II also is scannerless having builtin lexer functions.
• CWIC Compiler for Writing and Implementing Compilers. Has token rules as part of its language. Rules in CWIC were compiled into boolean functions returning success or failure.