Homework 1: Lexer

Due Tuesday, 29 January 2008, 6 PM

This assignment is to build a lexer for Appel's Tiger language using ML-Lex. The Tiger language is defined in Appendix A of the course textbook. There is limited documentation for ML-Lex in chapter two of the textbook, so you may also find the complete documentation for ML-Lex useful.

Skeleton files to get you started are available in the $TIGER/chap2/ directory, which you can find on Appel's textbook homepage. The provided sources.cm won't work with SML/NJ 110.67, so you may want to use this one instead.

You should submit:

A tiger.lex file, with the source for your lexer.
Any other source files you wrote to support your lexer.
An ASCII text file named README describing
- the members of your team,
- how you handled comments,
- how you handled errors,
- how you handled end-of-file, and
- anything else you think is of interest about your lexer.

You're expected to write clean code; just getting it to work is not enough.

Your lexer should use the error-reporting machinery in Appel's ErrorMsg module (see file $TIGER/chap2/errormsg.sml), or something equivalent that you write yourself. In particular, error messages should be reported using line-number/column offsets, not by simply specifying the character offset from the beginning of the file.

Caution: Relative to a parser, it's not hard to build a lexer, but:

Tiger's lexical grammar has some complexities that may take you more work to handle than you might initially suspect:
- Comments nest in Tiger.
- The string-literal syntax is especially complex, involving a lot of subcases, some of which are fairly complex in their own right.
- Having all of the above machinery interact with your line-number tracking machinery has its own complications.
Does your lexer do the right thing if eof occurs inside a comment or string-literal? How about illegal escape codes in string literals?

See the course text for more information, in particular chapter two and appendix A.

Last updated 18 February 2008.