Debugging Neomutt

About a month ago, I was checking my emails in Neomutt. When I opened a particular email, suddenly, Neomutt core dumped.

I thought maybe this was a one-off. I started Neomutt again and tried to load the same email. It crashed again. Because the crash occurred when trying to open the contents of an email, I was concerned that it may be exploitable. So I started investigating.

I was using the fake system time option in GPG for privacy, which caused problems in other applications. Since the email that crashed my client was encrypted, I suspected the bug had something to do with my esoteric GPG configuration. I found that if I disabled gpgme in Neomutt, the crash went away.

I wasn’t sure how to further proceed in debugging, so I joined the Neomutt channel on Libera Chat, asking for help. I got in touch with Flatcap, the creator of Neomutt. He had me send him the raw email data for the email in question. Since he didn’t possess my private key for decrypting it, he could not reproduce the bug.

I wasn’t about to send him my private key. Someone else suggested I could decrypt the email with my private key, then reencrypt it with a new private key I wouldn’t mind divulging. Luckily I knew of a better way. GPG has the “–show-session-key” option. It’s used to allow others to decrypt specific messages intended for you without giving them your private key.

The Neomutt developers edited their GPG options inside the Neomutt configuration so it would successfully decrypt the original email. They used “–override-session-key [string]”. Their clients did not crash when opening the email. At that point, I knew the cause was my GPG configuration. Still, an esoteric configuration should not cause Neomutt to core dump. Core dumps should never happen. So I was determined to find the root cause of the bug.

Since Flatcap couldn’t reproduce the crash, the only option left was to debug it myself. Flatcap helped me to compile Neomutt from source so I’d have the debugging symbols. Then he explained how to attach GDB to the Neomutt process. I proceeded to purposely crash Neomutt.

With some more assistance, I then used GDB to check the variable values and found the error was coming from a print statement. It was a null pointer error caused by a partially defined key. In my case, the key was partially defined because the faked system time on GnuPG was dated before the key in question was created. It appeared to GPG that the key was created in the future. That’s obviously impossible, which is why it was partially defined.

Now that the cause of the crash was understood, Flatcap patched gpgme so it could handle partially defined keys. I pulled the patched branch, recompiled, and tried to open the buggy email again. No crash!

Flatcap invited me to open a Github issue so I could take credit for finding the bug. I informed him I couldn’t do that because I don’t use Github. So he just opened the issue and mentioned me instead. I looked over it for review and approved.

I thanked Flatcap for creating Neomutt. I really enjoy using it. I’m glad I was able to contribute to such a fantastic email client. The only reason this interaction was possible is because Neomutt is free software. If it were proprietary, I couldn’t have debugged it with GDB. I probably wouldn’t have gotten to interact directly with the developer who writes the software I use. I’d probably be stuck waiting days or weeks on a response from an opaque company where I couldn’t even review the patch or get credit for reporting the bug.

In free software communities, interactions like these are happening all the time. When people use proprietary software, they’re not just missing out on good software. They’re missing out on being part of a community. Free software isn’t just about writing better code. That’s open source. Neither is it fundamentally about money. That’s proprietary software. Free software is about community.

Join us now and share the software;
You'll be free, hackers, you'll be free
Join us now and share the software;
You'll be free, hackers, you'll be free

Hoarders can get piles of money
That is true, hackers, that is true
But they cannot help their neighbors;
That's not good, hackers, that's not good

When we have enough free software
At our call, hackers, at our call
We'll kick out those dirty licenses
Ever more, hackers, ever more

Join us now and share the software;
You'll be free, hackers, you'll be free
Join us now and share the software;
You'll be free, hackers, you'll be free