Backing up my github repositories

1 min read

I spent today doing various ops tasks that have been postponed for a while. One of them was to re-enable backup of my github repositories. I require a local "mirror" with all the branches, tags etc. in case github has an outage or outrage against me ;)

I used to use a python script I found years ago. Unfortunately, it stopped working. After I installed a new backup server (an old, refurbished Lenovo ThinkCentre with external USB disks) with Debian "Bookworm", I noticed that the script required Python 2, so I just gave up on it.

I spent a few hours looking for a replacement. I found plenty of applications in Pyhton and go that promised to to the job. Looking closer at them, they all expected me to give them user credentials or a API key for my stuff on github. That is not an option for me. The backup server is mostly just a file-server with encrypted restic backups from other machines. It's always on, so I don't trust it very much - at least not with secrets. I also realized that python has turned into a horror show, where even simple things like this require venv to install the dependencies in Debian.

I ended up just writing a bash script, using a one-liner python command to parse the json from the github API. Unfortunately it returns "name": attributes for other tings than the repository names, so a grep would not work.

I also noted that github only return 30 repositories by default. Max is 100. I have 37 repositories at the moment, so I added a query parameter to ask for up to 100 repositories to the url. I'll have to do something else when they lower the max limit or I exceed 100 repositories ;)

The script that runs with bash, python3 and curl installed:

    #!/bin/bash

    cd /mnt/data/pub/github/

    user=$USER

    repos=`curl -s "https://api.github.com/users/${user}/repos?per_page=100" | python3 -c '
    import json, sys
    obj=json.load(sys.stdin)
    for x in obj:
        print (x["name"])
    '`

    for r in $repos
    do
        echo "Backing up $r"

        local_name=${r}.git
        if [ -d "${local_name}" ]; then
            echo "   updating..."
            cd "${local_name}"
            git remote update
            cd ..
        else
            echo "   ** cloning **"
            git clone --mirror "https://github.com/${user}/${r}.git"
        fi
    done

That's all it takes.