BBGradebookOrganiser/README.md

# BBGradebookOrganiser

Blackboard Gradebook Organiser

## Description

**Blackboard Gradebook Organiser** is a tool for organising a downloaded gradebook with assignment submissions from [Blackboard Learn ⧉](https://en.wikipedia.org/wiki/Blackboard_Learn).  
The submission files are organised per student, by extracting the student number from the submission file names and creating a directory per student. Compressed files are extracted into the student's directory, and any remaining individually submitted files are also moved into the student's directory. Student comments from the submissions are also extracted into a single text file for convenient access and review.  
Optionally, after organising a gradebook, you can inspect the submissions to detect duplicated files from different submissions/students by generating and comparing SHA256 hashes. See section [Inspect submissions](#inspect-submissions-mag) for details.

### **Features**

- Extracts, and organises per student, the content of submitted compressed files with extensions: `.zip`, `.rar`, `.7z`

  - Detects invalid/corrupt files

  - Doesn't extract macOS system generated files (ignores directory *__MACOSX* inside the compressed file)

- Deletes each compressed file after successful extraction into student directory

- Organises per student any remaining individually submitted files

- Checks and extracts any comments from the student submission generated text files

- Checks if any compressed files (from the contents of the submitted compressed files) have been extracted and organised per student

  - The path of any extracted and organised compressed files will be displayed on the terminal - they need to be extracted manually

- [Inspect*s* submissions](#inspect-submissions-mag) by generating and comparing SHA256 hashes of submitted files

## Instructions

### **Download gradebook**

1. Go to the course page on Blackboard

2. Go to *Grade Centre -> Full Grade Centre*

3. Find the assignment and click on the arrow for more options, and select *Assignment File Download*

4. Select all (click *Show All* at the bottom first, to display all users) and click submit to generate the gradebook zip file

5. Wait for the generated download link to appear, and click to download

### **Extract gradebook**

Extract the downloaded gradebook in a new directory inside [*BB_gradebooks*](BB_gradebooks).

- e.g. for `AssignmentX` extract the gradebook in `BB_gradebooks\AssignmentX`

### **Organise gradebook**

Before running the script for the first time, install the required packages (*py7z*, *rarfile*):

```python
python -m pip install py7zr rarfile
```

Note: If running on Linux/Mac, you also need to have `unrar` installed in order to be able to extract *.rar* files.

- `sudo apt install unrar` for Linux

- `brew install rar` for Mac

   
To organise the gradebook run **`organise_gradebook.py`** and provide the name of the directory with the *extracted* gradebook (from section *Extract gradebook* above) as an argument.

- e.g. for gradebook `AssignmentX` (in [*BB_gradebooks*](BB_gradebooks)/`AssignmentX`) run:

```python
python organise_gradebook.py AssignmentX
```

While running, the script displays on the terminal information and stats about the gradebook submissions and files.

### **Post-run**

- All submission files can be found - organised in directories per student number - in directory [*BB_submissions*](BB_submissions), under the sub-directory named after the gradebook name provided when running the script

  - e.g. `organise_gradebook.py AssignmentX` creates the directory `AssignmentX` inside [*BB_submissions*](BB_submissions)

- Each student directory contains:

  - the extracted files from the submitted `.zip`, `.rar`, `.7z`
  
  - the individually submitted files
  
  - the text file generated by Blackboard for the submission (which also contains any comments left by the student)

- All comments found in the gradebook are extracted in a text file in [*BB_submissions*](BB_submissions), with the gradebook name as prefix

  - e.g. `AssignmentX_comments.txt` will be created for gradebook `AssignmentX`

- Compressed files are deleted after successfully extracting and organising the contents

  - Any invalid/corrupt compressed files are moved into folder `__BAD__` inside the gradebook directory

## **Inspect submissions** :mag:

### **Information**

- Generates SHA256 hashes for each submitted file, and outputs list to CSV file

  - Can exclude files from hashing, if provided with a CSV file listing the file names

- Compares the generated hashes and finds any duplicate hashes - ignores duplicates if they are by the same student/submission

- Finds all files with a duplicated hash and outputs them to CSV file with the following information: *Student ID*, *file path*, *file name* (without path), *SHA256 hash*

  - Further inspection and filtering needs to be done manually

### **Usage**

For this feature you also need to install the *pandas* package:

```python
python -m pip install pandas
```

   
To inspect the submissions run **`inspect_submissions.py`** and provide the name of the directory with the *organised* gradebook as an argument.

- e.g. for the organised gradebook `AssignmentX` (in [*BB_submissions*](BB_submissions)/`AssignmentX`) run:

```python
python inspect_submissions.py AssignmentX
```

**Note:** run ***after*** organising a gradebook with [*organise_gradebook.py*](organise_gradebook.py).

Generated CSV files can be found in directory `csv`, with the inspected submission's name as file name prefix - e.g. inspecting submissions for `AssignmentX` will create 2 CSV files:

- `AssignmentX_file_hashes_[datetime].csv` - all files and their hashes
  
- `AssignmentX_suspicious_[datetime].csv` - files with duplicate hashes

*(Optional)* In order to exclude files from hashing, create a CSV file in directory `csv` to provide the file names to be excluded - e.g. for `AssignmentX` create:

- `AssignmentX_excluded.csv` with a column named `exclude_filename` and list the file names

**Note:** the directory `csv` is automatically created when you run `inspect_submissions.py` - you need to create it manually if you want to exclude files before the first run.

## General notes

The Blackboard generated name for submission files must follow the pattern:
> ANYTHING_STUDENTNUMBER_attempt_DATETIME_FILENAME