CI test fest setup guide

1 What is Github Actions?

We will we using Github Actions CI to provide automatic feedback on assignments.

Github Actions is a service that monitors a GitHub repository; when it detects a newly pushed commit, it spins up a virtual machine and runs some scripts (that you write). Usually, Github Actions is used to build a project and run a test suite, so that teams can have an ongoing idea of their project’s status while they work on it.

We will be using it in much the same way as is typical, but the status information you will get is your score for each assignment. On every push, Github Actions will run your submission against the tests submitted by every team and report the results.

2 Setup guide

2.1 Obtain a GitHub education account

If you don’t already have one, go to this GitHub page and click "Get benefits".

2.2 Configure what Github Actions does on each push

Github Actions is configured with a special configuration file containing commands to execute on every new push.

The first section of this file specifies when the service should do something: in this case, whenever you push to github. Next, the file specifies the jobs to perform on every push: in this case, you just need one job that will test your submission against everyone’s tests (the job is called "test"). The job will run on an Ubuntu 20.04 virtual machine, and it has four environment variables which you need to fill in to specify which assignment to test, which team you are, and what log level you want from the testing system. When you create this file you need to change TEAM to be your team name, and you won’t need to change it again. The MAJOR and MINOR variables together specify the assignment to test, so you will need to change them for every assignment. Finally the LOGLEVEL variable affects how much output the testing system produces: error means to just report test failures, while info will report a little more information about how the tests are being run. In the example, this is assignment "3.1" and a team named "teamX".

Important: As you start working on new assignments, you must change the assignment numbers here to specify the new assignment to test.

Note that you only want a single one of these env blocks.

Important: The executable built from your submission must be called run, and it must be have execute permissions (e.g. by using chmod u+x run as part of building).

To make Github Actions test your assignment with this configuration, all that you need to do is provide a makefile in the right directory under Deliverables to build your executable (which must be named run).

You may not modify the yml file because the automated grading will use a stock one (exactly as above, except with an appropriate assignment number). But, that said, a temporary modification is okay. See Debugging GitHub Actions CI.

2.3 Creating a makefile that builds your submission

2.3.1 Makefiles

We will build your submission using a makefile. A makefile is just a file that describes how to build your project.

Suppose that you have a project in hello-world.c that you want to compile. A makefile for that project might look like the one below.

The first line declares a rule with a name and then a colon, followed by a list of files that must exist to run the rule. That list can be empty. The following lines are shell commands to execute to build your project.

will execute the first rule in the file.

2.3.2 Submission makefiles

You must provide a makefile to build each of your assignments. The makefile you submit must be able to build your submission in a fresh environment (on Github Actions). Thus, the makefile needs to download and install any dependencies that it needs. It’s also important that it only installs things not already installed, so that it won’t break if, for example, we build two assignments in a row.

The Github Actions virtual machines run Ubuntu, so the Ubuntu software repositories are available to install packages easily. Otherwise, dependencies can be installed from source or from binary distributions.

You should not use any special Github Actions features to install dependencies, because then we won’t be able to build your assignment to grade it. Important: Grading will happen in the same environment as the Github Actions jobs, so your job must be able to build the assignment. Specifically, we will be using a configuration file nearly identical to the example given earlier for grading.

2.3.3 The built executable should be able to just be run

We assume that your makefile builds an executable called run, and we should be able to execute it as-is (e.g. with ./run). It might be easiest to make the run executable be a script that runs the submission.

2.4 Commit the Github Actions configuration and push

Go to the github page for your repo and click on the tab named "Actions". You will see a list of "All workflows", at the top of which will be an in-progress job. Clock on it and click on the box that says "test" to watch the job’s progress.

2.5 Find your test results summary in the job log

These will indicate what went wrong and how.

2.6 Queue Times

GitHub has limits on the jobs that we all can run; they allow only six jobs at once across the whole organization (i.e., six jobs at once for all students in Software Construction), and we have a finite total amount of time that our jobs can use each month. If everyone is trying to run jobs at once, queue times may become long; we recommend trying to avoid using Actions close to the homework deadlines to avoid long waits.

If you do experience long queue times or jobs not starting at all, or you don’t want to deal with the possibility of it happening, you can run actions outside of the organization instead. As a student with a GitHub Education account, you get a large amount of time each month for free that you can use for Actions jobs. These are completely independent of the course organization, so you will not experience any queue times, and the amount of free time available is generous.

Of course, only pushes to the repository in the course organization count towards your grade.

2.7 Debugging GitHub Actions CI

Once you resolve the problem, remember to undo any changes you made to the .yml configuration, otherwise you might get out of sync with the configuration we will use for grading – this could allow tests to pass on your jobs but fail when we grade the same submission.