Monthly update, February and March 2025
8 min readThese monthly updates may be a bit technical. They're written for my future self (to remember how I spent the month - and to motivate me to do at least something remotely interesting), for friends and colleagues (past and future) to give them an idea of what I work on, and of course, for potential clients of my C++ freelance business and fellow software developers.
The picture shows that spring is here, and that there will be delicious grapes in my garden in the fall.
Projects
NextApp
NextApp is an upcoming GTD/productivity application for desktop and mobile.
As usual, I fixed a number of bugs. One of them had been haunting me for months. Sometimes the backend would crash when a client connected and synced with the server. When the client reconnected, the server crashed again - in a loop. Oddly enough, if I killed the client and connected from another device (like my phone), everything worked perfectly again, even with the client that had previously crashed the server.
The server crashed on an assert in the gRPC libraries. I added a signal handler printing a stack trace using boost.stacktrace:
1// crash handlers
2__attribute__((noinline)) void signal_handler(int signum) {
3 cerr << "Signal (" << signum << ") received:\n";
4 cerr << "Symbol map(s): " << symbol_maps << "\n";
5 cerr << boost::stacktrace::stacktrace() << "\n";
6 exit(signum); // Exit with the signal number
7}
That gave me a stack trace from the crash in the container log, but it was stripped of symbols.
E0325 10:56:20.675773217 1 proto_buffer_writer.h:65] assertion failed: !byte_buffer->Valid()
Signal (6) received:
0# 0x0000558F51D19DE6 in /usr/local/bin/nextappd
1# 0x00007FC0BD28C330 in /lib/x86_64-linux-gnu/libc.so.6
2# pthread_kill in /lib/x86_64-linux-gnu/libc.so.6
3# gsignal in /lib/x86_64-linux-gnu/libc.so.6
4# abort in /lib/x86_64-linux-gnu/libc.so.6
5# 0x00007FC0BE938D68 in /lib/x86_64-linux-gnu/libgrpc++.so.1.51
6# 0x0000558F51DF1288 in /usr/local/bin/nextappd
7# 0x0000558F51DF0F4B in /usr/local/bin/nextappd
8# 0x0000558F51DF0EE9 in /usr/local/bin/nextappd
9# 0x0000558F51DB77EA in /usr/local/bin/nextappd
10# 0x0000558F51DBC5C5 in /usr/local/bin/nextappd
11# 0x0000558F51DF0C6E in /usr/local/bin/nextappd
12# 0x0000558F51D8D7B2 in /usr/local/bin/nextappd
13# 0x0000558F51D8E865 in /usr/local/bin/nextappd
14# 0x0000558F51ED4E4A in /usr/local/bin/nextappd
15# 0x0000558F51F265FF in /usr/local/bin/nextappd
16# 0x0000558F51D57541 in /usr/local/bin/nextappd
17# 0x0000558F51D68FB3 in /usr/local/bin/nextappd
18# 0x0000558F51D67F0B in /usr/local/bin/nextappd
19# 0x0000558F520A8F7E in /usr/local/bin/nextappd
20# 0x0000558F520B2C48 in /usr/local/bin/nextappd
21# 0x0000558F51D595B8 in /usr/local/bin/nextappd
22# 0x0000558F520B2779 in /usr/local/bin/nextappd
23# 0x0000558F520B21A5 in /usr/local/bin/nextappd
24# 0x0000558F51D53DE8 in /usr/local/bin/nextappd
25# 0x0000558F51D53891 in /usr/local/bin/nextappd
26# 0x0000558F51D4D614 in /usr/local/bin/nextappd
27# 0x0000558F51D2AF0F in /usr/local/bin/nextappd
28# 0x0000558F51D29FD1 in /usr/local/bin/nextappd
29# 0x0000558F51D1907E in /usr/local/bin/nextappd
30# 0x00007FC0BD2711CA in /lib/x86_64-linux-gnu/libc.so.6
31# __libc_start_main in /lib/x86_64-linux-gnu/libc.so.6
32# 0x0000558F51D15F15 in /usr/local/bin/nextappd
A stack trace like this isn't very useful on its own. Fortunately, Boost can produce an output with resolved symbols - if I'm willing to let the server run for another minute after one thread hits the assert. The problem is that the library takes about a minute to resolve the symbols, and during that time, the other threads have no idea anything went wrong. So, not really an option in a server app.
Another approach is to resolve the symbols post-mortem with addr2line. That works if you know the base address of your app while it was running, and you have a binary with debug symbols. I modified the build script to create a symbols file and container, then printed the base address above the stack trace to make it easy to access. Finally, I asked ChatGPT to write a script to take the binary (manually extracted from the container), the symbols file, the stack trace, and the base address- and output a stack trace with symbols.
Even with the resolved stack trace, I couldn’t pinpoint the problem. Then one day, while fixing unrelated bugs, I accidentally reproduced the assert in the debugger in my dev environment. I expected the root cause to be some kind of memory overwrite. I'd already analyzed the app with Valgrind, run static analyzers, and even asked various AIs to review the code.
Turns out, it was a trivial wrong enum state in a function that publishes update messages to users from a queue. Usually the queue is empty or has one update, but sometimes a single change produces multiple updates. So it was kind of a memory overwrite - but not in the way I expected.
Qt is still evolving their gRPC API, so I had to update my code to follow the latest changes in Qt 6.8.2.
One long-planned feature I finally implemented is a popup log viewer inside the app. I'm using logfault for logging, so it was trivial to add a log handler to push messages into a circular buffer inside a simple Qt data model. This lets me see the latest 1000 messages at any time inside the app. I also added a field at the bottom right of the desktop window that displays the latest warning or error for a few seconds. Clicking it opens the full log window.
Another big feature was multi-region support on the server side. I can now deploy regional backends for performance and legal reasons. The signup server can handle any number of regions, each with any number of backend instances. Each user always connects to one specific instance (sharding), but with regional clusters, the system can scale to millions of users globally without changing a line of code.
I also removed the dependency on QtMultimedia, a huge Qt library with lots of dependencies and build complexity. It was only used to play sounds. I replaced it with MiniAudio, which was much simpler and easier to integrate.
Then came time to prepare release binaries. I’d already paid the two-week “tax of frustration and horror” to figure out how to build Qt statically on Linux. I now have a container that builds Qt on Ubuntu 22.04. I couldn't get full static linking - many shared libs are still needed, some unavailable on newer Ubuntu versions. So I explored packaging systems like AppImage.
AppImage suggests building with the oldest Linux version possible for compatibility - which I already do. But their tools are distributed as AppImages, and those don't work well inside containers without serious Docker and host OS tweaks - not great for CI/CD with GitHub Actions.
I tried building the tools from source but ran into build failures due to sloppy C code (snprintf() with no buffer size!). After fixing and cleaning up their code (commit), it still didn’t work. After two days, I gave up on AppImage and switched to Flatpak. Within a day, I had a working Flatpak using a GitHub Workflow. The app runs fine under Debian 12 and Ubuntu 24.04.
(Snap was my last resort. I really despise Snap. One of the first things I do on a new Ubuntu install is to remove Snap and reinstall Firefox and other "snapped" apps with native packages. So no Snap for NextApp.)
Windows Build
Then I moved on to Windows. Static Qt builds there are just as painful. After about a week, I had one - only to find that the app linked both 32 and 64 bit libraries. The culprit was a rogue vcpkg from Visual Studio 2022. Even though I told CMake to use C:\src\vcpkg\vcpkg.exe, it sometimes ignored my settings. Removing the cursed VS version helped - now everything linked using the same bitsize.
Still, QtGrpc and others were dynamically linked. So I rebuilt Qt dynamically - got a working app - but it requires users to download lots of shared Qt libraries. Then I noticed that vcpkg now supports Qt 6.8.2, so I tried that. I started with a dynamic build, but it failed an error like this:
C:\build\qtstuff>cmake --build .
MSBuild version 17.13.19+0d9f5a35a for .NET Framework
1>Checking Build System
Generating QtProtobuf grpc_stuff sources for qtprotobufgen...
CUSTOMBUILD : CMake warning : [C:\build\qtstuff\qtclient\grpc_stuff_qtprotobufgen_deps_0.vcxproj]
No source or binary directory provided. Both will be assumed to be the
same as the current working directory, but note that this warning will
become a fatal error in future CMake releases.
CUSTOMBUILD : CMake error : The source directory "C:/build/qtstuff/qtclient" does not appear to contain CMakeLists.txt. [C:\build\qtstuff\qtclient\grpc_stuff_qtprotobufgen_deps_0.vcxproj]
Specify --help for usage, or press the help button on the CMake GUI.
At this point I was pissed off. So I created a new minimal Qt project, qtstuff, to figure out how to build with QtGrpc and QML. NextApp is too complex for debugging these kinds of issues.
I built QT using vcpkg. Under Windows, I got the same error as I got with NextApp when I tried to compile qtstuff.
Under Linux, it compiled. But nothing happened when I ran it. It turned out that unlike the Windows version, under Linux, I had to specifify the QT desktop driver to use. Interestingly, I was unable to deselect the SQL dependency chain, but I had to specify that I wanted a visible UI to the QtCore module (I already specified the QML module).
My vcpkg.json for Linux now looks like:
1{
2 "name": "qtstuff",
3 "dependencies": [
4 "boost-program-options",
5 "grpc",
6 "protobuf",
7 "pcre2",
8 "double-conversion",
9 "libpng",
10 "libjpeg-turbo",
11 "brotli",
12 {
13 "name": "qtbase",
14 "version>=": "6.8.2",
15 "features": [
16 "concurrent",
17 "testlib",
18 "dnslookup",
19 "network",
20 "xcb",
21 "dbus",
22 "gui",
23 "icu",
24 "jpeg",
25 "opengl",
26 "openssl",
27 "xcb-xlib"
28 ]
29 },
30 {
31 "name": "qtdeclarative",
32 "version>=": "6.8.2"
33 },
34 {
35 "name": "qtcharts",
36 "version>=": "6.8.2"
37 },
38 {
39 "name": "qtsvg",
40 "version>=": "6.8.2"
41 },
42 {
43 "name": "qtgrpc",
44 "version>=": "6.8.2"
45 }
46 ],
47
48 "builtin-baseline": "acd5bba5aac8b6573b5f6f463dc0341ac0ee6fa4"
49}
50
I added dbus because NextApp use it. It did not compile out of the box, as it tried to install the libraries in /var/opt/vcpkg/... rather than /opt/vcpkg/....
A pleasant surprise: vcpkg links everything statically under Linux (except for glibc etc.). I’ll likely switch to this instead of Docker for Qt builds. The downside: GitHub Actions offers too little disk space to cache vcpkg's build tree, so building Qt on every commit will likely take 6+ hours. I haven’t tested that yet.
SafeKeeping
SafeKeeping is a simple C++ library for securely storing secrets using the OS's or desktop manager's vault.
While implementing cluster features in the NextApp backend, I wrote a CLI to access the new gRPC cluster interfaces. I needed a way to store certificates and passwords securely, so I created this small library for Linux (KDE/GNOME), Windows, and macOS. Under KDE, secrets end up in KWallet.
Shinysocks
Shinysocks is a small, ultra-fast SOCKS proxy server that just works.
I had a long-standing to-do item to automate GitHub Actions releases for Shinysocks. Since it’s a simple project, it was perfect for experimenting. It now works.
OpenValify
OpenValify is a C++ library that validates TLS certs for a list of domain names.
Let’s Encrypt announced they’ll stop sending expiry emails. I use TLS certs in various places - gRPC interfaces, development, email - where auto-renewals aren’t set up. So I wrote a C++ library and CLI to directly check cert expiration. It was a fun weekend project.
qtstuff
qtstuff is a very simple project for testing Qt and Grpc builds with vcpkg on various platforms.
Mentioned above in the NextApp section. If you're working with Qt, QML, or QtGrpc, you might find it handy.
Reflections
vcpkg
Once you learn how to use it, vcpkg is quite nice. I was pretty frustrated at first - being forced into manifest mode (having to DuckDuck that very consept), dealing with confusing errors about missing GitHub hashes, etc.
The GitHub hash is meant for reproducible builds, but in practice, it often ends up being ignored. People set it once and forget it, potentially shipping insecure dependencies. It might be better if projects was allowed to and defaulted to not setting the hash. Then teams that care about strict reproducibility could opt-in.
Another pain point: vcpkg doesn’t manage system dependencies. I lost a whole day compiling qtstuff due to missing system libs. Even after installing Qt’s required dependencies, I still had to manually install things like bison, flex, autoconf, zip, libtool, autoconf-archive, python3-jinja2, and libxcb-xinput-dev - each revealed only after a build failure, with varying clarity about why.
I would love if the various vcpkg components had a list of hard system dependencies, so that vcpkg could check if the required libraries are installed before starting the build. If something is missing, it should explain how to install the missing dependencies on the build platform, for example by using apt or dnf.
Personal
I mentioned last fall that one of my dogs, Ares, got very ill, but fortunately recovered. At the end of February, he fell ill again and passed away 10 days later, despite intense and eventually desperate treatment. He lost that last fight.
Now, a month later, I still feel a deep void in my life.
(Ares, in better times)