User:OrenBochman/ParserNG/WikiTable

An Antlr Spec for the WikiTable Markup

ANTLR spec

grammar wikiTable;

@header {
package p;

}

//@header {import org.antlr.test;} // not auto-copied to lexer
@lexer::header{
package p;
//import org.antlr.test;
//
}

@lexer::members {
//state check are deeply nested in a table are we?
int inTable=0;
List tokens = new ArrayList();
public void emit(Token token) {
        state.token = token;
    	tokens.add(token);
}
public Token nextToken() {
    	super.nextToken();
        if ( tokens.size()==0 ) {
            return Token.EOF_TOKEN;
        }
        return (Token)tokens.remove(0);
}
}

@members{
//int inTable=0;
//public void foo(){};
//int rows=0;
}

//Parser Rules
wikiTable 	: TBL_START  xml_attributes?  caption? head?  rows TBL_END;
caption		: CAPTION_START HS xml_attributes? captionText=TEXT+;
fragment
head		: (hCell hCellInLine*)+;	
rows 		: (firstRow|row) row*;
firstRow 	: cells;
row		: rowStart xml_attributes? cells;
rowStart	: ROW_START;

cells		:((cell|hCell) (cellInline|hCellInLine)*)+;
cell		: CELL_START xml_attributes? text=TEXT*;
cellInline	: CELL_INLINE_STRT xml_attributes? text=TEXT*;
hCell		: HEAD_START xml_attributes? text=TEXT*;
hCellInLine	: HEAD_INLINE_STRT xml_attributes? text=TEXT*;


//this is the recursive definition alowing table nesting
//cells		:( {input.LT(0)==CELL_START||input.LT(0)==HEAD_START}?=>(HEAD_START | CELL_START) XHTML_ATTRIBUTES? (TEXT|wikiTable)+ (CELL_INLINE_STRT XHTML_ATTRIBUTES? (TEXT|wikiTable)+)* )+  ;

//this needs to be in the parser for LT(2) to mean the second parser token
xml_attributes: {input.LT(2).getText().equals("=")}? xml_attribute+ PIPE? ;
xml_attribute: name=TEXT EQ DQUOTE value=TEXT* DQUOTE ;
//Lexer Rules
TBL_START	: {getCharPositionInLine()==0}?=> '{|'{inTable++; }	;
TBL_END		: {getCharPositionInLine()==0&&inTable>0}?=> '|}'{inTable--;}	;
HEAD_START      : {getCharPositionInLine()==0&&inTable>0}?=> '!';
HEAD_INLINE_STRT: {inTable>0}?=> '!!';

CELL_START  	: {getCharPositionInLine()==0&&inTable>0}?=> '|';	//this should only be recignised within a table
PIPE		: {getCharPositionInLine()>0||inTable==0}?=> '|';	//outside table or not at tart of line

CELL_INLINE_STRT: {inTable>0}?=> '||'; 					//this should only be recignised within a table
ROW_START 	: {getCharPositionInLine()==0&&inTable>0}?=> '|-' ;
CAPTION_START	: {getCharPositionInLine()==0&&inTable>0}?=> '|+' 	;


TEXT		: ('a'..'z'|'A'..'Z'|'0'..'9'|'.'|'-'|';'|':'|',')+;					//simplified

DQUOTE		: '"';
//WS 		:  (HS | VS)  ; //{ $channel = HIDDEN; } ;
HS		: ( ' ' | '\t'  )+ { $channel = HIDDEN; } ;
VS		: ( '\r' | '\n' )+ { $channel = HIDDEN; } ;
EQ		: '=';

Status

• This is a lexer + a parser.
• Tested against the examples in table.
• A tree grammar or a string template could be used to transform into XHTM etc.
• Does not support full unicode to simplify development - but the string could be changed with minimal impact.

Problems

The speck has a recognizer nondeterminism

1. Antlr is unabile to decide which path to take when meeting a HEAD_START symbol since it could belong to

• In the optional header.
• There is no optional header but the body starts with a header. (this is a mistake)

1. This is a warning and option #2 is discarded . How could this nondeterminsm be removed ?

1. adding a variable with a table wide scope

   boolean hasHead=TRUE;
   

2. use it in a predicate on the optional header

   {hasHead}?;
   

3. add an action after the optional header to flip it

   {hasHead=FALSE;}
   

• Antlr complains that the first non-header cell might belong

• In the (optional) first row, i.e. the one without a |- indicator.
• In the optional other rows after.

Table in Table Test

You type	You get
<!-- outer --> {| border="1" | Orange || Apple || align="right" | 12,333.00 |- | Bread || Pie || align="right" | 500.00 |- | Butter || Ice cream || align="right" | 1.00 <!-- inner --> {| border="1" | Orange || Apple || align="right" | 12,333.00 |- | Bread || Pie || align="right" | 500.00 |- | Butter || Ice cream || align="right" | 1.00 |} |}	Orange Apple 12,333.00 Bread Pie 500.00 Butter Ice cream 1.00 Orange Apple 12,333.00 Bread Pie 500.00 Butter Ice cream 1.00