Yet Another Blog

Managing e-mail is a science. Filtering hundreds of incoming messages and still not missing important ones, correct formatting and addressing can be problematic. Add a huge number of available tools and e-mail providers to the mix. I was in a need to rethink my current e-mail setup because it just didn’t work. I am subscribed to a number of mailing lists. Because of that my mailbox was full of less important e-mails coming from mailing lists. I enjoy reading them although I consider unsubscribing from most of them and read them vi Gmane NNTP interface. It became harder and harder to find a reply for an important e-mail I expected. I also maintain several folders for specific purposes, for example a folder name TRAVEL for all e-mails related to traveling or BANKS for all e-mails sent by my bank.

I had a specific set of requirements that a new setup must meet:

1) make it possible to access all e-mails, also sent items both on PC and Android mobile devices
2) keep a local backup of all e-mails except ones sent to mailing lists on the PC for a peace of mind
3) filter incoming e-mails directly on the server and not locally. There is no procmail for Android, and even if there was it would probably cause battery drainage and it would require me to maintain procmail filters on several devices

But first things first – I needed to clean my current inbox. All e-mails sent to mailing lists should go to the respective folders. I had already thousands of them so moving them manually was out of question. I have my e-mail account at Hostgator so I can access it by SSH and run procmail directly inside e-mail directory. I created a ~/.procmailrc that looks like this:

MAILDIR=$HOME/mail/drabczyk.org/arkadiusz
DEFAULT=$MAILDIR

LOGFILE=$HOME/.procmail.log
#Uncomment below for troubleshooting
VERBOSE=YES
LOGABSTRACT=YES

:0:
* ^TO_git@vger.kernel.org
.git_list/

:0:
* ^TO_fxos-feedback@lists.mozilla.org
.fx_os_list/

:0:
* ^TO_gdb@sourceware.org
.gdb_list/

(...)

# all other mails will be moved to the inbox
:0:
$DEFAULT/

and a following simple CLEANUP.sh script similar to this:

#!/usr/bin/env sh

set -e

for dir in cur new; do
    find $INPUTMAILDIR/$dir -type f | while read file; do
        procmail < $file
        rm $file
    done
done

It took some time to process all e-emails but finally my mailbox was clean. All other e-mails required maunal work, they were either spam sent to the mailing list using Bcc: or advertisements and special offers from various legitimate companies. The only thing I didn’t predict is that they were several e-mails, reddit registration e-mail amongst them that ended up first on the e-mail list in SquirrelMail because they lacked a date header but it’s not a big problem.

Next setup was to setup procmail to filter all incoming e-mails on the server. It wasn’t so simple as I thought it would be because Hostgator uses CPanel, CPanel uses Exim and Exim has its owner filtering capabilities. I didn’t feel like I want to rewrite all procmail rules to Exim format and learn another tool that would do the same. Luckily it’s possible to convince Exim to hand e-mail filtering to procmail by modifying /etc/vfilter/<USERNAME> file, for example to this:

# Exim filter

# Auto Generated by cPanel.  Do not manually edit this file as your changes will be overwritten.  If you must edit this filter, edit the corresponding .yaml file as well.

if not first_delivery and error_message then finish endif

#filter
if
 $header_from: matches ".*"
then
 pipe "/usr/bin/procmail /home1/rkumvbrh/.procmailrc"
endif

Note that /etc/vfilters/ directory itself has no +r for anybody except root so it’s not possbile to list its contents but it has +x so it’s possbile to modify existing files inside the directory if you know their names.

So far, so good. My mailbox was now clean and it will always be because all e-mails are now filtered on the server. Next step was to synchronize mailbox state between various devices. For example, if I move a given e-mail to TRAVEL folder I want to see this e-mail in TRAVEL folder on all devices. Of course IMAP is a way to go here. As I said, I also want to keep a backup of all incoming e-mails except for e-mails sent to the mailing lists. I used to use getmail for downloading e-mails but this time I decided to give OfflineIMAP a try. It turned out it works really well although some people don’t favor it. It does 2-way synchronization, makes it possible to exclude certain folders from being synchronized and can transpose folder names on the fly. In my case procmail puts all e-mails sent to the mailing list in folders named <LIST_NAME>.list on the server. I can easily tell Offlineimap not to download them without specifying all folders names:

folderfilter = lambda folder: not re.search('(_list$)', folder)

I got used to box1 as the main folder name instead of INBOX and sent for a folder with sent items instead of INBOX.sent. No problem for Offlineimap:

nametrans = lambda folder: {'box1':   'INBOX',
                            'sent':   'INBOX.Sent',
                            }.get(folder, folder)

I added Offlineimap to my crontab to start every 5 minutes. As I said, Offlineimap does 2-way synchronization. It means that when I delete an e-mail or move it e-mail from box1 to TRAVEL folder it will be reflected on the server and therefore visible in all other clients such as SquirrelMail or Android mail client. And when it comes to Android, the best client I found is k9mail. It can be healed from top posting disease, can send all e-mails in plain text and is open source. Setting up k9mail on Android doesn’t require any special attention apart from not marking mailing lists folder to be synchronized. As all e-mails are filtered on the server, I don’t have to care about synchronization here. I can access all sent items any time on any device I want because they’re synchronized, all other folders are too. I can move a message to a different folder on my phone on the go and then sit at my PC, open up mutt and see the same thanks to Offlineimap.

Last thing I did was to unsubscribe from multiple newsletters. I didn’t need most of them anyway and as they were not automatically moved to their folders because it was hard to filter them or I didn’t even see them before but they caused my phone to notify me about the catching my attention unnecessarily.

Now I can finally stop worrying and love e-mail again.