Backing up my github repositories

1 min read

I spent today doing various ops tasks that have been postponed for a while. One of them was to re-enable backup of my github repositories. I require a local "mirror" with all the branches, tags etc. in case github has an outage or outrage against me ;)

I used to use a python script I found years ago. Unfortunately, it stopped working. After I installed a new backup server (an old, refurbished Lenovo ThinkCentre with external USB disks) with Debian "Bookworm", I noticed that the script required Python 2, so I just gave up on it.

I spent a few hours looking for a replacement. I found plenty of applications in Pyhton and go that promised to to the job. Looking closer at them, they all expected me to give them user credentials or a API key for my stuff on github. That is not an option for me. The backup server is mostly just a file-server with encrypted restic backups from other machines. It's always on, so I don't trust it very much - at least not with secrets. I also realized that python has turned into a horror show, where even simple things like this require venv to install the dependencies in Debian.

I ended up just writing a bash script, using a one-liner python command to parse the json from the github API. Unfortunately it returns "name": attributes for other tings than the repository names, so a grep would not work.

I also noted that github only return 30 repositories by default. Max is 100. I have 37 repositories at the moment, so I added a query parameter to ask for up to 100 repositories to the url. I'll have to do something else when they lower the max limit or I exceed 100 repositories ;)

The script that runs with bash, python3 and curl installed:

 1    #!/bin/bash
 2
 3    cd /mnt/data/pub/github/
 4
 5    user=$USER
 6
 7    repos=`curl -s "https://api.github.com/users/${user}/repos?per_page=100" | python3 -c '
 8    import json, sys
 9    obj=json.load(sys.stdin)
10    for x in obj:
11        print (x["name"])
12    '`
13
14    for r in $repos
15    do
16        echo "Backing up $r"
17
18        local_name=${r}.git
19        if [ -d "${local_name}" ]; then
20            echo "   updating..."
21            cd "${local_name}"
22            git remote update
23            cd ..
24        else
25            echo "   ** cloning **"
26            git clone --mirror "https://github.com/${user}/${r}.git"
27        fi
28    done

That's all it takes.