CI test fest setup guide

1 What is Travis CI?

We will we using Travis CI to provide automatic feedback on assignments.

Travis is a service that monitors a GitHub repository; when it detects a newly pushed commit, it spins up a virtual machine and runs some scripts (that you write). Usually, Travis is used to build a project and run a test suite, so that teams can have an ongoing idea of their project's status while they work on it.

We will be using it in much the same way as is typical, but the status information you will get is your score for each assignment. On every push, Travis will run your submission against the tests submitted by every team and report the results.

2 Setup guide

2.1 Obtain a GitHub education account

If you don't already have one, go to this GitHub page and click "Get benefits".

2.2 Obtain a Travis education account

Go to Travis and click "Sign in with GitHub". As long as the account you sign in with has GitHub education, Travis will automatically detect it and provide student benefits as well.

If you have never used Travis before, then you will be sent to GitHub and asked if you want to give Travis access and if you want to install it on GitHub. Say yes. Then it will ask you for the set of repositories for which you want to enable Travis; for now, skip adding any. Now you should be able to access the Travis account page.

If you have used Travis before, you should go straight to the Travis account page.

2.3 Authorize Travis to access the class organization

On the left hand side of the Travis account page, underneath the "Sync account" button, should be a list entitled "Organizations". Underneath that is a button saying "Review and add your authorized organizations." Click it.

You will be sent to a GitHub settings page for Travis that has "Organization access" at the bottom. Underneath that, find the class organization ("NorthwesternSoftwareConstructionFall19") and click the "Request" button for it.

We will get an email to approve the request, and after we do so you will see the class organization under "Organizations" on the Travis account page. Click on the organization, and then click "Activate". You will be taken to a page to select which repositories to activate it for, and click "All repositories".

We will get another email to approve the request, and after we do so you will be able to see your repositories on the Travis account page. Travis is now enabled for your repositories.

2.4 Configure what Travis does on each push

Travis is configured with a special configuration file (named .travis.yml) containing commands to execute on every new push.

In the top level of your repository (the one containing the .git folder), create a file named .travis.yml with the following contents.

env:
 - ASSIGN_MAJOR=1 ASSIGN_MINOR=0 TEAM_NAME=dummy-team

script:
 - pushd Deliverables/$ASSIGN_MAJOR/$ASSIGN_MAJOR.$ASSIGN_MINOR
 - make
 - chmod u+x ./run
 - popd
 - git clone https://github.com/NorthwesternSoftwareConstructionFall19/oracle
 - chmod u+x ./oracle/run-test-fest.sh
 - ./oracle/run-test-fest.sh $ASSIGN_MAJOR $ASSIGN_MINOR $TEAM_NAME

The first section of this file specifies environment variables for the virtual machine. The only variables that we need to specify are which assignment should be tested, and which team we are. In the example, this is assignment "1.0" and a team named "dummy-team".

Important: As you start working on new assignments, you must change the assignment numbers here to specify which assignment you want to test.

Note that you only want a single one of the env bullets.

The next section specifies what Travis should do on the virtual machine. Each bullet is a command to execute in a shell. The example commands do the following:

Enter the directory for the assignment and build an executable
Clone the repository containing the oracle for every assignment
Run the test fest script for the assignment

To make Travis test your assignment with this configuration, all that you need to do is provide a makefile to build your executable (which must be named run).

2.5 Creating a makefile that builds your submission

2.5.1 Makefiles

We will build your submission using a makefile. A makefile is just a file that describes how to build your project.

Suppose that you have a project in hello-world.c that you want to compile. A makefile for that project might look like the one below.

The first line declares a rule with a name and then a colon, followed by a list of files that must exist to run the rule. That list can be empty. The following lines are shell commands to execute to build your project.

# This is a file named "makefile"
rule-name: hello-world.c
        echo "Each line is a shell command"
        echo "The commands must start with a TAB character (NOT SPACES)"
        gcc -Wall hello-world.c -o run

another-rule-with-no-dependencies:
        echo "I have no dependencies"

In the directory with this file, running

make

will execute the first rule in the file.

You can also specify which rule to run like so

make another-rule-with-no-dependencies

2.5.2 Submission makefiles

You must provide a makefile to build each of your assignments. The makefile you submit must be able to build your submission in a fresh environment (on Travis). Thus, the makefile needs to download and install any dependencies that it needs.

The Travis virtual machines run Ubuntu, so the Ubuntu software repositories are available to install packages easily. Otherwise, dependencies can be installed from source or from binary distributions.

For example, if you are using python, your makefile can do something like this to set up the environment.

rule-name:
        sudo apt-get update
        -sudo apt-get install python3 # dash allows failure if already installed

As an example of installing software from a distribution, suppose that the version of java available in the Ubuntu repositories isn't what you want. Instead, you can obtain the version you want manually from openJDK. With JDK 13, for example, that would look like this.

rule-name:
        wget "https://download.java.net/java/GA/jdk13/5b8a42f3905b406298b72d750b6919f6/33/GPL/openjdk-13_linux-x64_bin.tar.gz"
        tar -xf openjdk-13_linux-x64_bin.tar.gz
        echo "java binaries are now in ./jdk-13/bin/"

You should not use Travis's language support features to install dependencies (e.g. with language: python), because then we won't be able to build your assignment to grade it. Grading will happen in the same environment as the Travis jobs, so your travis job must be able to build the assignment.

2.5.3 The built executable should be able to just be run

We assume that your makefile builds an executable called run, and we should be able to execute it as-is (e.g. with ./run). It might be easiest to make the run executable be a script that runs the submission.

For example, if you are using Python then you might put a file named run (with execute permission) that looks like this:

#!/bin/bash

# This is the file "run": it runs the actual submission

python my-submission.py some arguments

2.6 Commit the Travis configuration and push

Go to the Travis account page and click on your repo. You will see an in-progress job, which you can click on to watch its progress.

2.7 Find your test results summary in the job log

The job will fail if you do not have a perfect score on the assignment specified in the .travis.yml file. This can happen in two ways:

Fewer than five of the submitted tests were found to be valid
The submission does not pass all of your peer's tests

If the job fails, first look at the end of the log for a summary of what went wrong. Then, scan the log for messages that look like this

fest: warning: something went wrong

fest: info: /path/to/run fails test /path/to/test

These will indicate what went wrong and how.

2.8 Debugging Travis CI

If you are having trouble with Travis, here are a few things you can try:

Modify the .travis.yml to temporarily add commands to look around
- E.g. if Travis can't find the assignment directory, try adding a line like
```
- ls -l Deliverables Deliverables/2 Deliverables/2/2.1
```
- Note that after resolving the issue you should go back to the original config like above to ensure that your assignment can build when we grade it
Modify your makefile to temporarily add debugging commands