To: Jan Kort <jan@research.techforce.nl>
Subject: Re: Flex
In-reply-to: Your message of Fri, 04 Sep 1998 12:18:43 +0200.
Date: Sat, 05 Sep 1998 00:59:49 PDT
From: Vern Paxson <vern>

> %%
>
> "TEST1\n"       { fprintf(stderr, "TEST1\n"); yyless(5); }
> ^\n             { fprintf(stderr, "empty line\n"); }
> .               { }
> \n              { fprintf(stderr, "new line\n"); }
>
> %%
> -- input ---------------------------------------
> TEST1
> -- output --------------------------------------
> TEST1
> empty line
> ------------------------------------------------

IMHO, it's not clear whether or not this is in fact a bug.  It depends
on whether you view yyless() as backing up in the input stream, or as
pushing new characters onto the beginning of the input stream.  Flex
interprets it as the latter (for implementation convenience, I'll admit),
and so considers the newline as in fact matching at the beginning of a
line, as after all the last token scanned an entire line and so the
scanner is now at the beginning of a new line.

I agree that this is counter-intuitive for yyless(), given its
functional description (it's less so for unput(), depending on whether
you're unput()'ing new text or scanned text).  But I don't plan to
change it any time soon, as it's a pain to do so.  Consequently,
you do indeed need to use yy_set_bol() and YY_AT_BOL() to tweak
your scanner into the behavior you desire.

Sorry for the less-than-completely-satisfactory answer.

		Vern