Next: EOF, Previous: Start Conditions, Up: Top [Contents][Index]
Some scanners (such as those which support “include” files) require
reading from several input streams. As flex
scanners do a large
amount of buffering, one cannot control where the next input will be
read from by simply writing a YY_INPUT()
which is sensitive to
the scanning context. YY_INPUT()
is only called when the scanner
reaches the end of its buffer, which may be a long time after scanning a
statement such as an include
statement which requires switching
the input source.
To negotiate these sorts of problems, flex
provides a mechanism
for creating and switching between multiple input buffers. An input
buffer is created by using:
which takes a FILE
pointer and a size and creates a buffer
associated with the given file and large enough to hold size
characters (when in doubt, use YY_BUF_SIZE
for the size). It
returns a YY_BUFFER_STATE
handle, which may then be passed to
other routines (see below).
The YY_BUFFER_STATE
type is a
pointer to an opaque struct yy_buffer_state
structure, so you may
safely initialize YY_BUFFER_STATE
variables to ((YY_BUFFER_STATE)
0)
if you wish, and also refer to the opaque structure in order to
correctly declare input buffers in source files other than that of your
scanner. Note that the FILE
pointer in the call to
yy_create_buffer
is only used as the value of yyin seen by
YY_INPUT
. If you redefine YY_INPUT()
so it no longer uses
yyin, then you can safely pass a NULL FILE
pointer to
yy_create_buffer
. You select a particular buffer to scan from
using:
The above function switches the scanner’s input buffer so subsequent tokens
will come from new_buffer
. Note that yy_switch_to_buffer()
may
be used by yywrap()
to set things up for continued scanning, instead of
opening a new file and pointing yyin at it. If you are looking for a
stack of input buffers, then you want to use yypush_buffer_state()
instead of this function. Note also that switching input sources via either
yy_switch_to_buffer()
or yywrap()
does not change the
start condition.
is used to reclaim the storage associated with a buffer. (buffer
can be NULL, in which case the routine does nothing.) You can also clear
the current contents of a buffer using:
This function pushes the new buffer state onto an internal stack. The pushed
state becomes the new current state. The stack is maintained by flex and will
grow as required. This function is intended to be used instead of
yy_switch_to_buffer
, when you want to change states, but preserve the
current state for later use.
This function removes the current state from the top of the stack, and deletes
it by calling yy_delete_buffer
. The next state on the stack, if any,
becomes the new current state.
This function discards the buffer’s contents,
so the next time the scanner attempts to match a token from the
buffer, it will first fill the buffer anew using
YY_INPUT()
.
is an alias for yy_create_buffer()
,
provided for compatibility with the C++ use of new
and
delete
for creating and destroying dynamic objects.
YY_CURRENT_BUFFER
macro returns a YY_BUFFER_STATE
handle to the
current buffer. It should not be used as an lvalue.
Here are two examples of using these features for writing a scanner
which expands include files (the
<<EOF>>
feature is discussed below).
This first example uses yypush_buffer_state and yypop_buffer_state. Flex maintains the stack internally.
/* the "incl" state is used for picking up the name * of an include file */ %x incl %% include BEGIN(incl); [a-z]+ ECHO; [^a-z\n]*\n? ECHO; <incl>[ \t]* /* eat the whitespace */ <incl>[^ \t\n]+ { /* got the include file name */ yyin = fopen( yytext, "r" ); if ( ! yyin ) error( ... ); yypush_buffer_state(yy_create_buffer( yyin, YY_BUF_SIZE )); BEGIN(INITIAL); } <<EOF>> { yypop_buffer_state(); if ( !YY_CURRENT_BUFFER ) { yyterminate(); } }
The second example, below, does the same thing as the previous example did, but manages its own input buffer stack manually (instead of letting flex do it).
/* the "incl" state is used for picking up the name * of an include file */ %x incl %{ #define MAX_INCLUDE_DEPTH 10 YY_BUFFER_STATE include_stack[MAX_INCLUDE_DEPTH]; int include_stack_ptr = 0; %} %% include BEGIN(incl); [a-z]+ ECHO; [^a-z\n]*\n? ECHO; <incl>[ \t]* /* eat the whitespace */ <incl>[^ \t\n]+ { /* got the include file name */ if ( include_stack_ptr >= MAX_INCLUDE_DEPTH ) { fprintf( stderr, "Includes nested too deeply" ); exit( 1 ); } include_stack[include_stack_ptr++] = YY_CURRENT_BUFFER; yyin = fopen( yytext, "r" ); if ( ! yyin ) error( ... ); yy_switch_to_buffer( yy_create_buffer( yyin, YY_BUF_SIZE ) ); BEGIN(INITIAL); } <<EOF>> { if ( --include_stack_ptr == 0 ) { yyterminate(); } else { yy_delete_buffer( YY_CURRENT_BUFFER ); yy_switch_to_buffer( include_stack[include_stack_ptr] ); } }
The following routines are available for setting up input buffers for
scanning in-memory strings instead of files. All of them create a new
input buffer for scanning the string, and return a corresponding
YY_BUFFER_STATE
handle (which you should delete with
yy_delete_buffer()
when done with it). They also switch to the
new buffer using yy_switch_to_buffer()
, so the next call to
yylex()
will start scanning the string.
scans a NUL-terminated string.
scans len
bytes (including possibly NUL
s) starting at location
bytes
.
Note that both of these functions create and scan a copy of the
string or bytes. (This may be desirable, since yylex()
modifies
the contents of the buffer it is scanning.) You can avoid the copy by
using:
which scans in place the buffer starting at base
, consisting of
size
bytes, the last two bytes of which must be
YY_END_OF_BUFFER_CHAR
(ASCII NUL). These last two bytes are not
scanned; thus, scanning consists of base[0]
through
base[size-2]
, inclusive.
If you fail to set up base
in this manner (i.e., forget the final
two YY_END_OF_BUFFER_CHAR
bytes), then yy_scan_buffer()
returns a NULL pointer instead of creating a new input buffer.
is an integral type to which you can cast an integer expression reflecting the size of the buffer.
Next: EOF, Previous: Start Conditions, Up: Top [Contents][Index]