Took a few days off between Day 3 and Day 4 so just getting to write this post now. Turns out that what I wanted to do for Day 4 was actually simpler to do than I thought.
For my course, all assignments are in their own repo, but to avoid having many little repos, I put all the labs for the course into one repo and make them available within the first week. Students who wish to work ahead by reading the course notes are welcome to do so.
However as all labs are in the same repository, I do not want the submission of one lab to trigger a set of tests for every other lab. I was trying to figure out how to do this in the simplest way possible. It turns out, this is really simple to do. All I need to do is set up a separate workflow for each lab. For the labs, the starter files are all stored in separate folders. All that was necessary was to set up each work flow to trigger on changes made to files within a certain folder.
With this last piece, I think I am ready to use Github actions for testing my student’s submission. The next step is to go through my labs to ensure that the instructions are clear. This is not a short process but I believe the results will be well worth the effort. Not only will it make the assignment submission process be simplified, but it will also introduce students to CI process.
Today, what I wanted to add to my GitHub Actions Workflow was the ability to ensure that the tester used for testing is the official most up to date version of the tester. At the end of Day 2, I had added a bit of code to my tester that basically would cause a segmentation fault, just to see how Github Actions would react to such a situation. And sure enough it crashed. I am leaving this crash in the code within my current repository. This will ensure that if I do not copy the correct version of the tester in place, my program will not run properly.
Following this thread the changes I made to my workflow is as follows:
checkout the current repository into a different folder instead of current folder
checkout the tester repository
copy the tester files into the current folder
copy out the current repository files that are needed for the compilation into the current folder
So with those changes in mind I modified my yml file to the following:
# This is a basic workflow to help you get started with Actions
name: CI
# Controls when the workflow will run
on:
# Triggers the workflow on push or pull request events but only for the main branch
push:
branches: [ main ]
pull_request:
branches: [ main ]
# Allows you to run this workflow manually from the Actions tab
workflow_dispatch:
# A workflow run is made up of one or more jobs that can run sequentially or in parallel
jobs:
# This workflow contains a single job called "build"
build:
# The type of runner that the job will run on
runs-on: ubuntu-latest
# Steps represent a sequence of tasks that will be executed as part of the job
steps:
# Checks-out your repository under $GITHUB_WORKSPACE, so your job can access it
- uses: actions/checkout@v2
with:
path: ./assignment
- name: Checkout Tester
uses: actions/checkout@v2
with:
repository: catherine-leung/a2template
path: ./tester
- name: Copy cpp tester files
run: cp ./tester/*.cpp ./
- name: Copy header tester files
run: cp ./tester/*.h ./
- name: Copy assignment files
run: cp ./assignment/majelem.cpp ./
# Runs a single command using the runners shell
- name: Compile
run: c++ a1q3tester.cpp majelem.cpp timer.cpp -std=c++0x -o a1
# Runs a set of commands using the runners shell
- name: Run
run: ./a1
My first run of this was a failure… the reason that it showed was:
Of course this leads me down a rabbit hole trying to figure out why it couldn’t find a default branch name. The reality however was actually that it didn’t have proper access to the repository. Two options exist at this point.. one is to generate a PAT and use that, or make the repo public. I am not good at security, and I need to look much more carefully about how I would do this properly. Concern of course is that what I’m testing is something I would need my students to either do for themselves (so I would need to give clear advice on how to do things) or I would need to do it myself for them in a way that doesn’t open up more than expected. My testers aren’t exactly secret in any case. I think the safer bet is actually to just make my testers public. This is what I did for now.
I changed my tester’s repo to public and it worked!
So with this, I consider my plans for Day 3 completed. For Day 4, I need to look at how to deal with testing multiple things from the same repo. For example in an assignment there may be two different coding questions. How do we trigger the appropriate test based on what was modified. For example suppose assignment had section on programming linked list and a section on programming recursive functions, these pieces are not all the same. How do I set up actions to only test based on what files were changed.. so if student worked on linked list, how do we skip compiling and testing the recursion part but also do it in a way so that its clear that not the full assignment passed but just the one item. This is definitely going to need some thinking about. Let me know what you think.
Yesterday I set up my first github action workflow that would build and compile my program. Today, I wanted to see how I can get github action to recognize when testing fails. The testers for my class are in C/C++. I had done a big overhaul to how I write those a few years ago. At the bottom of my testers I typically have the following if/else block
if(numPassed == numTests){
std::cout << "Congratulations! You have passed testing for A1 part 1" << std::endl;
}
else{
std::cout << "Looks like you still have some work left to do" << std::endl;
}
I have been trying to find some docs about this online but much of what is written uses testing frameworks and is usually not C/C++. I wanted to see if I could take what I have and make it work with minimal changes. One of the first things we teach in C/C++ is that the return 0; at the end of the main() indicates that everything went well. Any other return value indicates an error so I figured.. lets see what would happen if I simply returned something other than 0. Now, since my program works.. I figured the easiest way to test it was to put a return 1; on success in the tester (thus if it works.. the test would fail).
if(numPassed == numTests){
std::cout << "Congratulations! You have passed testing for A1 part 1" << std::endl;
return 1;
}
else{
std::cout << "Looks like you still have some work left to do" << std::endl;
}
I made this change to main, added the file and pushed to github. This is the result:
And yay! it failed. Lets see the details:
And there we go… an exit code of 1, is all you need from your main to indicate that it was not working properly. This is wonderful! The fix would be to simply add some return statements.
What about crashes?
I wanted to see how github action would handle an outright crash. To test this, I added some very likely to crash code… a simple array overrun… this should crash:
if(numPassed == numTests){
std::cout << "Congratulations! You have passed testing for A1 part 1" << std::endl;
int array[10];
for(int i=0;i<100000;i++){
array[i]=i;
}
return 0;
}
else{
std::cout << "Looks like you still have some work left to do" << std::endl;
return 1;
}
And sure enough.. it crashes and indicates a failure state.
This is amazing. So far GitHub actions is doing what I need it to do in an expected manner. I can see how actions will work.
Next Step:
Day 2 goal accomplished. For Day 3, I would like to change my workflow to pull the tester from a different repo. This way, if there is a flaw in the tester when it was initially released, a fix can be applied to a single source and the build will use the most up to date and correct tester. I already think I have a good idea of how to do this.. just need to try it out.
Cathy's Blog 