Complete Computing Environment: Email and News with Gnus

Table of Contents

Email, News and the Art of Expansion

Despite everyone's worst intentions, Email still is the primary form of communication and management that we use in the real world. Your email address is your identity, your email address is how strangers reach out to you and how family sends you fucking obnoxious chain letters. It's as ubiquitous as a phone number, and far less meaningless1.

Email, however, is technically difficult. Managing an email system is damn near impossible and as a result, people centralize their data. Google and Yahoo have their main products built around this core idea, and have used this position to collect the entire lives of people. Google has made this a core part of their business, the collection of users' information so that it can be retargeted back to them in the form of advertisements and, as recent news has led me to believe2, 3, as a tool for governments tracking users and supressing free speech. The government doesn't even believe you should have privacy if you dare use these providers as anything more than braindead proxies4.

These beliefs have led me to withdraw, to move my communications out of the hands of these hulking goliaths and to embrace a more self-respecting, if more difficult mode of storage. My private life, years of emails, chat logs and messages, exist in these confines, and god damnit they are mine and live on a server that I trust.

End rant. Sorry about that.

As such, I put a lot of effort in to running a mailserver, and am in the process of rebuilding that right now. That's out of scope for this document though, the important parts are that it's going to give me an IMAP interface, the backend doesn't matter in this scope.

For the frontend, for the client, I've fallen in love with a piece of kit called Gnus 5, which comes standard with Emacs. It bills itself as an NNTP reader, but can handle way more backends than just that including IMAP. I have a fairly interesting multi-account setup, since my work and personal email should both work in this system, as well as a bunch of RSS feeds.

(provide 'cce-gnus)

Core Thoughts and Requirements:

  • Mail is ridiculously hard but still necessary
  • Mail clients need to work with the idea of multiple handles with nearly-separate encapsulations
  • Open formats such as RSS are the best consumption medium for how I work.
  • Fuck black magic and opaque systems.

Syncing to Maildirs

I have been struggling with the state of nnrss being slow, and have been looking at other solutions to handling my mail situation, and have landed on the good-old local dovecot setup. I have had a fair amount of trouble with nnimap not playing well with

The easy solution is to put the mail in to Maildirs, which my system can easily handle, and probably handle much better. Syncing them down to the clients is a little bit tougher, but I can work with that pretty easily with a custom Unison profile like I do for my Org-mode files and projects.

mail-sync.png

Syncing Mail

The low-hanging fruit in all of this is mail, since that's what Maildir is for. I used to use offlineIMAP, but these days everyone seems keen to tell me to switch to mbsync so that is what I did.

deb-install isync
rpm-install isync
rpm-install gnutls-bin msmtp
deb-install gnutls-bin msmtp

The mbsync configuration file is pretty well laid out, you can essentially define various stores and link them with channels, group them, etc.

IMAPAccount gmail
# Address to connect to
Host imap.gmail.com
User rrix@uber.com
PassCmd "cat /home/rrix/.authinfo | sed -n 's,^machine imap.gmail.com .*password \\([^ ]*\\).*,\\1,p'"
UseIMAPS yes
RequireSSL yes
CertificateFile /etc/ssl/certs/ca-certificates.crt

IMAPAccount fastmail
Host mail.messagingengine.com
Port 992
User "rrix@fastmail.com"
PassCmd "cat /home/rrix/.authinfo | sed -n 's,^machine mail.messagingengine.com .*password \\([^ ]*\\) port 993,\\1,p'"
UseIMAPS yes
CertificateFile /etc/ssl/certs/ca-certificates.crt

# ------------------------------ Gmail Sync
IMAPStore gmail-remote
Account gmail

MaildirStore gmail-local
Path /srv/files/files/rrix/org/maildir/gmail/
Inbox /srv/files/files/rrix/org/maildir/gmail/Inbox

Channel gmail
Master :gmail-remote:
Slave :gmail-local:
Patterns * ![Gmail]* "[Gmail]/Sent Mail" "[Gmail]/Starred"
Create Both
SyncState *
# ------------------------------

# ------------------------------ Gmail All Mail Sync
Channel gmail_all
Master :gmail-remote:
Slave :gmail-local:
Patterns "[Gmail]/All Mail"
Create Both
SyncState *
# ------------------------------

# ------------------------------ Fastmail sync
IMAPStore fastmail-remote
Account fastmail

MaildirStore fastmail-local
Path /srv/files/files/rrix/org/maildir/fastmail/
Inbox /srv/files/files/rrix/org/maildir/fastmail/INBOX
Flatten "_"

Channel fastmail_in
Master :fastmail-remote:
Slave :fastmail-local:
Patterns "INBOX" "TraSh" "Drafts" "newsletters" "social"
Sync All
Create Slave
SyncState *

Channel fastmail_sent
Master :fastmail-remote:
Slave :fastmail-local:
Patterns "Sent Items"
Sync All
Expunge Both
Create Slave
SyncState *

Channel fastmail_lists
Master :fastmail-remote:
Slave :fastmail-local:
Patterns "ableconf" "ansible" "asu" "emacsconf" "fedora/*" "github" "gsoc" "hackerspaces/*" "ingress/*" "kde/*" "lsdj" "org-mode" "phoenix*" "rocketry/*" "CHIP" "RecruitingSpam"
Sync All
Create Slave
SyncState *

Group fastmail
Channels fastmail_in fastmail_sent fastmail_lists
# ------------------------------

This dumps my mail in to two Maildirs on my encrypted disk, one for personal email and one for work email. I can then add them to my crontab on my servers, to run every so often.

*/5  * * * * mbsync gmail
*/15 * * * * mbsync fastmail
5    0 * * * mbsync gmail_all

Syncing Feeds

RSS is a little bit more complex. My first instinct is to use rss2email by the late Aaron Swartz, but that is set up to send the feeds as email – I only want them on disk as emails. I eventually found Universal Aggregator6 which does that very thing, and I hacked together a Dockerfile which runs UA and outputs the Maildir to a persistent volume under the same /srv location as my e-mail.

Getting it local

Because I run the dovecot on the local agent, I need to get the mail down to the machine, which I use Unison to handle. The case for Qubes is a little interesting, I use Unison to pull all of the mail in my data domain, and then sync it out of their, like I do in the case of the regular ~/sync directory. I realize this is hugely against the threat model of Qubes, but it's how my particular threat model operates given that the source-of-truth for these files is a single-domain server. I use Unison ignore-filters to keep certain classes of mail out of those domains; no sense in getting personal stuff in to the Qubes work domain since I can just start up the main domain to get to those when I need them.

root = /home/user/Maildir
root = ssh://rrix@hypervisor01.pss9.kickass.systems//srv/files/files/rrix/org/maildir

servercmd = /usr/bin/unison
root = /home/user/Maildir
root = ssh://user@10.137.2.13//home/user/Maildir

servercmd = /usr/bin/unison

ignore = Path gmail
root = /home/user/Maildir
root = ssh://user@10.137.2.13//home/user/Maildir

servercmd = /usr/bin/unison

ignore = Path fastmail
ignore = Path Maildir-feeds
ignore = Path rss
if [[ "x" != "x$QUBES_ENV_SOURCED" ]] && [[ "x$(hostname)" = "xmanager" ]]; then
    for domain in work main; do
        domain-copy $domain $PWD/out/mail-$domain.prf
        domain-run $domain mkdir -p ~/.unison
        domain-run $domain mkdir -p ~/sync
        domain-run $domain cp QubesIncoming/manager/mail-$domain.prf ~/.unison/mail.prf
    done
else
    cat ./out/mail-data.prf | sed -e "s/user/rrix/g" > ~/.unison/main.prf
fi
(cce/with-any-domain
 '("work" "main")
(add-hook 'after-init-hook (lambda () (let ((elapsed (cce/time-since-start))) (message "(%.3fs) [unison mail]" elapsed))))
 (add-hook 'after-init-hook (lambda () (cce/kick-off-unison "mail"))))

Making it all visible

My original plan here was to simply use nnmaildir to handle this, but it turns out nnmaildir is fairly insane7 and so I am back to the good-old local dovecot set up. We shall see how well this plays out, but for now, a working configuration.

deb-install dovecot
rpm-install dovecot
domain-install main dovecot
domain-install work dovecot

Because I run Dovecot locally, I can get away with only using IMAP, and not enabling authentication – if you can connect to localhost:143 you probably can do other bad things.

mail_location = maildir:~/Maildir:LAYOUT=fs
protocols = imap

protocol imap {
    listen = 127.0.0.1:143
}

auth_mechanisms = anonymous
if [[ "x" != "x$QUBES_ENV_SOURCED" ]] && [[ "x$(hostname)" = "xmanager" ]]; then
    for domain in work main; do
        domain-copy $domain $PWD/out/dovecot.conf
        domain-run $domain cp QubesIncoming/manager/dovecot.conf /etc/dovecot/dovecot.conf
    done
else
    sudo cp $PWD/out/dovecot.conf /etc/dovecot/dovecot.conf
fi

Gnus Set Up

Who the hell I am?

(setq user-mail-address "ryan@whatthefuck.computer")
(setq user-full-name    "Ryan Rix")

Gnus has 5 "levels of subscribedness"8, which I sort of equate to as the priority of the group. I'm subscribed to a shitload of groups, and this is really the only sane way to manage this without wailing. On Gnus startup, I activate the top 2 of 5 group levels, which give me my important inboxes, and some interesting personal folders and RSS feeds.

(setq gnus-activate-level 2)

Sometimes when I split, the mail will duplicate itself, and I suppress those using these two variables.

(setq gnus-suppress-duplicates t
      gnus-summary-ignore-duplicates t
      gnus-list-groups-with-ticked-articles nil)

We display images in the Article buffers.

(setq mm-inline-text-html-with-images t)
(setq mm-text-html-renderer 'shr)
(setq mm-inline-text-html-with-images t)

Enable ace-link in Gnus

(eval-after-load 'gnus (lambda ()
                         (define-key gnus-summary-mode-map (kbd "M-o") 'ace-link-gnus)
                         (define-key gnus-article-mode-map (kbd "M-o") 'ace-link-gnus)))

Make threading kind of pretty:

(setq-default
 gnus-summary-line-format "%U%R%z %(%&user-date; %-15,15f  %B%s%)\n"
 gnus-summary-thread-gathering-function 'gnus-gather-threads-by-references
 gnus-thread-sort-functions '(gnus-thread-sort-by-score)
 gnus-sum-thread-tree-false-root ""
 gnus-sum-thread-tree-indent " "
 gnus-sum-thread-tree-leaf-with-other "├► "
 gnus-sum-thread-tree-root ""
 gnus-sum-thread-tree-single-leaf "╰► "
 gnus-sum-thread-tree-vertical "│")

Discourage showing HTML email.

(setq mm-discouraged-alternatives
      '("text/html" "text/richtext")
      mm-automatic-display
      (remove "text/html" mm-automatic-display))

Split the layout of the Gnus interface to be a three-panel setup with a vertical group selector on the left, and then a horizontal split with group summary on top and the active article on the bottom.

(add-to-list 'gnus-buffer-configuration
             '(article (horizontal 1.0
                                   (vertical 30
                                             (group 1.0))
                                   (vertical 1.0
                                             (summary 0.25 point)
                                             (article 1.0)))))

Splitting and Filtering

For the longest time, I've used Sieve to get my email organized. It's, well, a shit show. A bloody mess. It regularly stops working, it commonly shits the bed, hell my mail setup refuses to deliver mails from Steam, even though I don't have any filters for it!

Gnus has ways to filter mail, built in of course to the system. Like damn near everything in Gnus, it's incredibly powerful, if a little obtuse. Sacha Chua has a great piece on her blog about Splitting, which I base my configuration off of9, and of course the Gnus documentation10.

I have these loosely groupded in to their categories, and they're included in the section below, Multiple Accounts and Multiple Backends.

Fedora mailing lists:

Hackerspaces mailing lists:

KDE mailing lists:

Social media sites and crap like that:

Newsletters and recurring crap:

Some random tech lists and other things:

Multiple Accounts and Multiple Backends

My life is layered in multiple encapsulations. I have a work life, I have a personal life, and they don't really overlap. My workflow needs to accept that and work within those bounds, my friends and newsletters and such don't connect to me via my work email and my work contacts don't even have my personal email. That's just how I work. As a part of this, though, Gnus needs to speak both. I have borrowed and crafted a methodology for this, based on the From email address in Gnus mails and msmtp, and it works pretty decently. I'd love to clean it up to be a bit more verbose about what it's doing, maybe with an identity chooser dialog or something, but for now this works.

rpm-install gnutls-bin msmtp
deb-install gnutls-bin msmtp

For starters, we define a primary select method; for me this is gmane, the typical FOSS mailing list frontend. It's nice and generic and pure NNTP.

(setq gnus-select-method '(nntp "news.gmane.org"))

And then, more importantly, we set our secondary select methods, which is where most of our stuff actually lives. These are a pair of NNIMAP11 backends and an NNRSS12 backend. All authentication lives in the authinfo file, of course. This includes all of the split information detailed above in Splitting and Filtering due to noweb silliness.

(setq gnus-secondary-select-methods
      `((nnimap ,(if (equal "main" (getenv "HOSTNAME"))
                     "maildir" "work")
                (nnimap-stream shell)
                (nnimap-inbox ,(cond ((equal "work" (getenv "HOSTNAME"))
                                      "gmail/Inbox")
                                     ((equal "main" (getenv "HOSTNAME"))
                                      "fastmail/INBOX")))
                (nnir-search-engine imap)
                (nnimap-shell-program ,(cond ((file-exists-p "/usr/lib/dovecot/imap") "/usr/lib/dovecot/imap")
                                             ((file-exists-p "/usr/libexec/dovecot/imap") "/usr/libexec/dovecot/imap")))
                (nnimap-split-methods ,(if (equal "main" (getenv "HOSTNAME"))
                                           'nnmail-split-fancy))
                (nnimap-split-fancy
                 (|
                  (from ".*@.*nextdoor.com.*" "fastmail/social")
                  (from ".*google-plus@google.com.*" "fastmail/social")
                  (from ".*@facebookmail.com.*" "fastmail/social")
                  (from ".*@linkedin.com.*" "fastmail/social")
                  (from ".*@e.linkedin.com.*" "fastmail/social")
                  (from ".*info@meetup.com.*" "fastmail/social")
                  (from ".*@meetup.com.*" "fastmail/social")
                  (from ".*@meetup.com.*" "fastmail/social")
                  (from ".*@eventbritemail.com.*" "fastmail/social")
                  (from ".*@angel.co.*" "fastmail/social")
                  (from ".*@useqwitter.com.*" "fastmail/social")
                  (from ".*@youtube.com.*" "fastmail/social")
                  (from ".*@tripit.com.*" "fastmail/social")
                  (from ".*@instagram.com.*" "fastmail/social")
                  (from ".*@twitter.com.*" "fastmail/social")
                  (from ".*@digg.com.*" "fastmail/social")
                  (from ".*@pinterest.com.*" "fastmail/social")
                  (from ".*@email.app.net.*" "fastmail/social")
                  (any ".*goodreads.com.*" "fastmail/social")
                  ("List-Id" ".*advisory-board.lists.fedoraproject.org.*" "fastmail/fedora_advisory-board")
                  ("List-Id" ".*council.*.lists.fedoraproject.org.*" "fastmail/fedora_advisory-board")
                  ("List-Id" ".*ambassadors.lists.fedoraproject.org.*" "fastmail/fedora_ambassadors")
                  ("List-Id" ".*announce.lists.fedoraproject.org.*" "fastmail/fedora_announce")
                  ("List-Id" ".*bugzilla.redhat.com.*" "fastmail/fedora_bugs")
                  ("List-Id" ".*cloud.lists.fedoraproject.org.*" "fastmail/fedora_cloud")
                  ("List-Id" ".*design-team.lists.fedoraproject.org.*" "fastmail/fedora_design-team")
                  ("List-Id" ".*devel.lists.fedoraproject.org.*" "fastmail/fedora_devel")
                  ("List-Id" ".*docs.lists.fedoraproject.org.*" "fastmail/fedora_docs")
                  ("List-Id" ".*famna-regionals.lists.fedoraproject.org.*" "fastmail/fedora_ambassadors")
                  ("List-Id" ".*freemedia.lists.fedoraproject.org.*" "fastmail/fedora_freemedia")
                  ("List-Id" ".*fudcon-planning.lists.fedoraproject.org.*" "fastmail/fedora_fudcon-planning")
                  ("List-Id" ".*kde.lists.fedoraproject.org.*" "fastmail/fedora_kde")
                  ("List-Id" ".*marketing.lists.fedoraproject.org.*" "fastmail/fedora_marketing")
                  ("List-Id" ".*metrics-wg.theopensourceway.org.*" "fastmail/fedora_tosw")
                  ("List-Id" ".*news.lists.fedoraproject.org.*" "fastmail/fedora_news")
                  ("List-Id" ".*packaging.lists.fedoraproject.org.*" "fastmail/fedora_packaging")
                  ("List-Id" ".*.rpmfusion.org.*" "fastmail/fedora_rpmfusion")
                  ("List-Id" ".*summer-coding-discuss.lists.fedoraproject.org.*" "fastmail/fedora_summer-coding")
                  ("List-Id" ".*summer-coding.lists.fedoraproject.org.*" "fastmail/fedora_summer-coding")
                  ("List-Id" ".*tos.teachingopensource.org.*" "fastmail/fedora_tos")
                  ("List-Id" ".*tosw.lists.fedorahosted.org.*" "fastmail/fedora_tosw")
                  ("List-Id" ".*web-members.*" "fastmail/fedora_websites")
                  ("List-Id" ".*websites.lists.fedoraproject.org.*" "fastmail/fedora_websites")
                  ("to" "gitbeefymiracle-members@fedoraproject.org" "fastmail/fedora")
                  ("to" "web-members@fedoraproject_org" "fastmail/fedora_websites")
                  (any "bugzilla@redhat.com" "fastmail/fedora_bugs")
                  (any "buildsys@fedoraproject.org" "fastmail/fedora_buildsys")
                  (any "root@fedoraproject.org" "fastmail/fedora_websites")
                  (any "scm-commits@lists.fedoraproject.org" "fastmail/fedora_buildsys")
                  (any "updates@fedoraproject.org" "fastmail/fedora_buildsys")
                  (any "*@rpmfusion.org" "fastmail/fedora_rpmfusion")
                  ("List-Id" ".*bay-area-reprap.googlegroups.com.*" "hackerspaces_bay-fastmail/area-reprap")
                  ("List-Id" ".*board.lists.acemonstertoys.org.*" "hackerspaces_fastmail/amt-board")
                  ("List-Id" ".*discuss.lists.acemonstertoys.org.*" "fastmail/hackerspaces_amt")
                  ("List-Id" ".*discuss.lists.hackerspaces.org.*" "hackerspaces_fastmail/hackerspaces-discuss")
                  ("List-Id" ".*discuss.synhak.org.*" "fastmail/hackerspaces_synhak")
                  ("List-Id" ".*dn42.*" "fastmail/hackerspaces_dn42")
                  ("List-Id" ".*heatsynclabs+owner.googlegroups.com.*" "fastmail/hackerspaces_heatsynclabs")
                  (any ".*heatsynclabs+msgappr@googlegroups.com.*" "fastmail/hackerspaces_heatsynclabs")
                  ("List-Id" ".*heatsynclabs.googlegroups.com.*" "fastmail/hackerspaces_heatsynclabs")
                  ("List-Id" ".*lists.sudoroom.org.*" "fastmail/hackerspaces_sudo")
                  ("List-Id" ".*makerbot.googlegroups.com.*" "fastmail/hackerspaces_makerbot")
                  ("List-Id" ".*noisebridge-discuss.lists.noisebridge.net.*" "fastmail/hackerspaces_noisebridge")
                  ("List-Id" ".*open-access-control.googlegroups.com.*" "fastmail/hackerspaces_oac")
                  ("List-Id" ".*spaceapi-devel.lists.hackerspaces.org.*" "fastmail/hackerspaces_spaceapi")
                  ("List-Id" ".*synshop.googlegroups.com.*" "fastmail/hackerspaces_synshop")
                  ("List-Id" ".*ultimaker.googlegroups.com.*" "fastmail/hackerspaces_ultimaker")
                  ("List-Id" ".*vhs-general.lists.hackspace.ca.*" "fastmail/hackerspaces_vhs")
                  (any "Norton Imperial Laboratories" "fastmail/hackerspaces_sfhackerspace")
                  ("List-Id" ".*phxdata.googlegroups.com.*" "fastmail/hackerspaces_phxdata")
                  ("List-Id" ".*lists.acemonstertoys.org.*" "fastmail/hackerspaces_amt")
                  ("List-Id" ".*sfhackerspace.googlegroups.com.*" "fastmail/hackerspaces_sfhackerspace")
                  ("List-Id" ".*nortonimperiallabs.github.com.*" "fastmail/hackerspaces_sfhackerspace")
                  ("List-Id" ".*c.h.i.p..bbs.nextthing.cofastmail/.*" "fastmail/CHIP")
                  ("List-Id" ".*android-qt.googlegroups.com.*" "fastmail/kde_android-qt")
                  ("List-Id" ".*bugsquad.kde.org.*" "fastmail/kde_bugs")
                  ("List-Id" ".*kde-announce.kde.org.*" "fastmail/kde_announce")
                  ("List-Id" ".*kde-community.kde.org.*" "fastmail/kde_community")
                  ("List-Id" ".*kde-core-devel.kde.org.*" "fastmail/kde_core-devel")
                  ("List-Id" ".*kde-cvs-announce.kde.org.*" "fastmail/kde_announce")
                  ("List-Id" ".*kde-devel.kde.org.*" "fastmail/kde_devel")
                  ("List-Id" ".*kde-ev-membership.kde.org.*" "fastmail/kde_ev-membership")
                  ("List-Id" ".*kde-games-devel.kde.org.*" "fastmail/kde_games")
                  ("List-Id" ".*kde-linux.kde.org.*" "fastmail/kde_linux")
                  ("List-Id" ".*kde-pim.kde.org.*" "fastmail/kde_pim")
                  ("List-Id" ".*kde-promo.kde.org.*" "fastmail/kde_promo")
                  ("List-Id" ".*kde-scm-interest.kde.org.*" "fastmail/kde_scm")
                  ("List-Id" ".*kde-soc.kde.org.*" "fastmail/kde_soc")
                  ("List-Id" ".*kde-usability.kde.org.*" "fastmail/kde_usability")
                  ("List-Id" ".*kde.mail.kde.org.*" "fastmail/kde_support")
                  ("List-Id" ".*kdepim-users.kde.org.*" "fastmail/kde_pim-users")
                  ("List-Id" ".*owncloud.kde.org.*" "fastmail/kde_owncloud")
                  ("List-Id" ".*plasma-devel.kde.org.*" "fastmail/kde_plasma")
                  (any ".*bugzilla_noreply@kde.org.*" "fastmail/kde_bugs")
                  (any ".*aclu.org.*" "fastmail/newsletters")
                  (any ".*aclunc.org.*" "fastmail/newsletters")
                  (any ".*tinycircuits.com.*" "fastmail/newsletters")
                  (any ".*kitsch-bent.com.*" "fastmail/newsletters")
                  (any ".*lifx.co.*" "fastmail/newsletters")
                  (any ".*@acm.org.*" "fastmail/newsletters")
                  (any ".*@ACM.ORG.*" "fastmail/newsletters")
                  (any ".*ahoyahoy@nextthing.co.*" "fastmail/newsletters")
                  (any ".*andrea@phxfilm.com.*" "fastmail/newsletters")
                  (any ".*answers@amazon.com.*" junk)
                  (any ".*auntritas.org.*" junk)
                  (any ".*bandcamp.com.*" "fastmail/newsletters")
                  (any ".*@bandsintown.com.*" "fastmail/newsletters")
                  (any ".*barnesandnoble@m.bn.com.*" "fastmail/newsletters")
                  (any ".*bluebottlecoffee.com.*" "fastmail/newsletters")
                  (any ".*cardsagainsthumanity.com.*" "fastmail/newsletters")
                  (any ".*@coderwall.com.*" "fastmail/newsletters")
                  (any ".*codeword-sf.com.*" "fastmail/newsletters")
                  (any ".*community@lyft.com" "fastmail/newsletters")
                  (any ".*comp.compilers@googlegroups.com.*" "fastmail/newsletters")
                  (any ".*crowdsupply.com.*" "fastmail/newsletters")
                  (any ".*dbweekly@cooperpress.com.*" "fastmail/newsletters")
                  (any ".*dieselsweeties.com.*" "fastmail/newsletters")
                  (any ".*digest-noreply@quora.com.*" "fastmail/newsletters")
                  (any ".*discountcab.com.*" junk)
                  (any ".*dnalounge.com.*" "fastmail/newsletters")
                  (any ".*@drdiabolicshow.com.*" "fastmail/newsletters")
                  (any ".*drippler.com.*" "fastmail/newsletters")
                  (any ".*e.asu.edu.*" "fastmail/newsletters")
                  (any ".*eazeup.com.*" "fastmail/newsletters")
                  (any ".*ebay@ebay.com.*" "fastmail/newsletters")
                  (any ".*@e.bevmo.com.*" "fastmail/newsletters")
                  (any ".*@e.com.*.bankofamerica.com" "fastmail/newsletters")
                  (any ".*@e.lego.com.*" "fastmail/newsletters")
                  (any ".*emailmarketing@rackspace.com.*" "fastmail/newsletters")
                  (any ".*@email.overnightprints.com.*" "fastmail/newsletters")
                  (any ".*@email.vzwshop.com.*" "fastmail/newsletters")
                  (any ".*@email.zazzle.com.*" "fastmail/newsletters")
                  (any ".*eric@lispcast.com.*" "fastmail/newsletters")
                  (any ".*evening-edition@muledesign.com.*" "fastmail/newsletters")
                  (any ".*ezine@apogeerockets.com.*" "fastmail/newsletters")
                  (any ".*fh.org.*" junk)
                  (any ".*flippfly.com.*" junk)
                  (any ".*footcardigan.com.*" junk)
                  (any ".*@gaffta.org.*" "fastmail/newsletters")
                  (any ".*gareth@morethanseven.net.*" "fastmail/newsletters")
                  (any ".*@geekli.st.*" "fastmail/newsletters")
                  (any ".*getpocket.com.*" "fastmail/newsletters")
                  (any ".*@github.com.*" "fastmail/github")
                  (any ".*groupsupdates@yahoo.com.*" "fastmail/newsletters")
                  (any ".*@heroku.com.*" "fastmail/newsletters")
                  (any ".*hobbico.com.*" "fastmail/newsletters")
                  (any ".*hulumail.com.*" "fastmail/newsletters")
                  (any ".*@humblebundle.com.*" "fastmail/newsletters")
                  (any ".*humblebundle.com.*" "fastmail/newsletters")
                  (any ".*iam8bit.*" "fastmail/newsletters")
                  (any ".*info@liveheroes.com.*" "fastmail/newsletters")
                  (any ".*info@victorlove.com.*" "fastmail/newsletters")
                  (any ".*ingress-support@google.com.*" "fastmail/newsletters")
                  (any ".*nianticlabs.com.*" "fastmail/newsletters")
                  (any ".*betabrand.com.*" "fastmail/newsletters")
                  (any ".*maximumfun.org.*" "fastmail/newsletters")
                  (any ".*insymmetry.*" "fastmail/newsletters")
                  (any ".*@klm-email.com.*" "fastmail/newsletters")
                  (any ".*laffairemusicale.com.*" "fastmail/newsletters")
                  (any ".*launch@rackspace.com.*" "fastmail/newsletters")
                  (any ".*@leapmotion.com.*" "fastmail/newsletters")
                  (any ".*lex@sreweekly.com.*" "fastmail/newsletters")
                  (any ".*lulu.com.*" "fastmail/newsletters")
                  (any ".*@lwn.net.*" "fastmail/newsletters")
                  (any ".*mail2.ncaa.com.*" "fastmail/newsletters")
                  (any ".*marketplace-messages@amazon.com.*" "fastmail/newsletters")
                  (any ".*massdrop.com.*" "fastmail/newsletters")
                  (any ".*matadorcoffee@gmail.com.*" "fastmail/newsletters")
                  (any ".*@mesaartscenter.com.*" "fastmail/newsletters")
                  (any ".*messages.squareup.com.*" "fastmail/newsletters")
                  (any ".*misfit.com.*" "fastmail/newsletters")
                  (any ".*@newegg.com.*" "fastmail/newsletters")
                  (any ".*newscomms.squareup.com" "fastmail/newsletters")
                  (any ".*news-googleplay@google.com.*" "fastmail/newsletters")
                  (any ".*@news.hpwebosnewsletters.com.*" "fastmail/newsletters")
                  (any ".*newsletter@bad-dragon.com.*" "fastmail/newsletters")
                  (any ".*@newsletters.sourceforge.net.*" "fastmail/newsletters")
                  (any ".*noreply@blizzard.com.*" "fastmail/newsletters")
                  (any ".*noreply-googleplay@google.com.*" "fastmail/newsletters")
                  (any ".*no-reply@kickstarter.com.*" "fastmail/newsletters")
                  (any ".*noreply@messaging.squareup.com.*" "fastmail/newsletters")
                  (any ".*noreply@musicbrainz.org.*" "fastmail/newsletters")
                  (any ".*noreply@ultimaker.com.*" "fastmail/newsletters")
                  (any ".*notification@mention.com.*" "fastmail/newsletters")
                  (any ".*pebble.com.*" "fastmail/newsletters")
                  (any ".*peter@webopsweekly.com.*" "fastmail/newsletters")
                  (any ".*ponoko.com.*" "fastmail/newsletters")
                  (any ".*postmates.com.*" "fastmail/newsletters")
                  (any ".*rackspace-feedback@noreply.jobs2web.com.*" "fastmail/newsletters")
                  (any ".*redbubble.com.*" "fastmail/newsletters")
                  (any ".*@rickshawstop.com.*" "fastmail/newsletters")
                  (any ".*@roku.com.*" "fastmail/newsletters")
                  (any ".*sales@xheli.com.*" "fastmail/newsletters")
                  (any ".*scottevest.com.*" "fastmail/newsletters")
                  (any ".*shop.kobo.com.*" "fastmail/newsletters")
                  (any ".*@sosh.com.*" "fastmail/newsletters")
                  (any ".*@sparkfun.com.*" "fastmail/newsletters")
                  (any ".*stackexchange.com.*" junk)
                  (any ".*@stacksocial.com.*" "fastmail/newsletters")
                  (any ".*statesidepresentsinaz@yahoo.com.*" "fastmail/newsletters")
                  (any ".*@statusnetondemand.org.*" "fastmail/newsletters")
                  (any ".*store-news@amazon.com.*" "fastmail/newsletters")
                  (any ".*Store.*steampowered.com.*" "fastmail/newsletters")
                  (any ".*@storquest.com.*" "fastmail/newsletters")
                  (any ".*support@linode.com.*" "fastmail/newsletters")
                  (any ".*teespring.com.*" "fastmail/newsletters")
                  (any ".*@theaterleague.org.*" "fastmail/newsletters")
                  (any ".*@theatreleage.org.*" "fastmail/newsletters")
                  (any ".*thefilmbarphx.com.*" junk)
                  (any ".*@theroyaleaz.com.*" "fastmail/newsletters")
                  (any ".*this-week-in-rust@webstream.io.*" "fastmail/newsletters")
                  (any ".*this-week-in-rust@webstream.io.*" "fastmail/newsletters")
                  (any ".*ticketmaster.com.*" "fastmail/newsletters")
                  (any ".*tindie.com.*" "fastmail/newsletters")
                  (any ".*@tokyoflash.com.*" "fastmail/newsletters")
                  (any ".*trycaviar.com.*" "fastmail/newsletters")
                  (any ".*typeamachines.com.*" "fastmail/newsletters")
                  (any ".*updates@trulia.com.*" "fastmail/newsletters")
                  (any ".*vectr.com.*" "fastmail/newsletters")
                  (any ".*@verizonwireless.com.*" "fastmail/newsletters")
                  (any ".*vfe-campaign-response@amazon.com.*" "fastmail/newsletters")
                  (any ".*@wolfram.com.*" "fastmail/newsletters")
                  ("Subject" ".*mailing list membership reminder.*" junk)
                  (any ".*@glassdoor.com.*" "fastmail/newsletters")
                  (any ".*@brainstormer.com.*" "fastmail/newsletters")
                  (any ".*@elastic.co.*" "fastmail/newsletters")
                  (any ".*@raggedwing.org.*" "fastmail/newsletters")
                  (any ".*rye51.com.*" "fastmail/newsletters")
                  (any ".*e.grubhub.com.*" "fastmail/newsletters")
                  (any ".*1015.com.*" "fastmail/newsletters")
                  (any ".*theinformation.com.*" "fastmail/newsletters")
                  (any ".*mezzaninesf.com.*" "fastmail/newsletters")
                  (any ".*xheli.com.*" "fastmail/newsletters")
                  (any ".*purelyfunction.tv.*" "fastmail/newsletters")
                  (any ".*goldenvoice.com.*" "fastmail/newsletters")
                  (any ".*editor@eff.org.*" "fastmail/newsletters")
                  (any ".*action@eff.org.*" "fastmail/newsletters")
                  (any ".*zipcar.com.*" junk)
                  (any ".*heroforge.com.*" "fastmail/newsletters")
                  ("List-ID" ".*orgzly-android-beta.googlegroups.com.*" "fastmail/org-mode")
                  ("List-ID" ".*emacsconf.*" "fastmail/emacsconf")
                  ("List-Id" ".*emacsconf.*" "fastmail/emacsconf")
                  ("List-ID" ".*ansible-project.googlegroups.com.*" "fastmail/ansible")
                  ("List-Id" ".*enyo-development.googlegroups.com.*" "fastmail/webos")
                  ("List-Id" ".*google-summer-of-code-students-list.googlegroups.com.*" "fastmail/gsoc")
                  ("List-Id" ".*lsdj.yahoogroups.com.*" "fastmail/lsdj")
                  ("List-Id" ".*lunar-general.googlegroups.com.*" "fastmail/rocketry_lunar")
                  ("List-Id" ".*phoenix-gdg.googlegroups.com.*" "fastmail/phoenix-gdg")
                  ("List-Id" ".*plug-discuss.lists.phxlinux.org.*" "fastmail/phoenix-lug")
                  ("List-Id" ".*sssmembers.yahoogroups.com.*" "fastmail/rocketry.sss")
                  ("to" ".*ASULUG@ASU.EDU.*" "fastmail/asu")
                  ("to" ".*google-summer-of-code-mentors-list@googlegroups.com.*" "fastmail/gsoc")
                  ("to" ".*lunar-general@googlegroups.com.*" "fastmail/rocketry_lunar")
                  ("to" ".*sssmembers@yahoogroups.com" "fastmail/rocketry_sss")
                  (any ".*nextthing@discoursemail.com.*" "fastmail/CHIP")
                  ("from" ".*root@hypervisor01.pss9.kickass.systems.*" junk)
                  ("from" ".*info@extremefliers.co.uk*" junk)
                  "fastmail/INBOX")))
        (nntp "nntp.olduse.net")))

Those handle receiving mail; sending it is an entirely different story. We have to do a bit of magic janking to make that work decently; we do it in a few parts. The core of it is a tool called msmtp13, which is a sendmail-ish thing that supports multiple SMTP server profiles.

We start out by enabling msmtp as the 'sendmail' program for Gnus

(setq message-sendmail-f-is-evil 't)
(setq message-sendmail-extra-arguments '("--read-envelope-from"))
(setq message-send-mail-function 'message-send-mail-with-sendmail)

(setq sendmail-program (cond
                        ((file-exists-p "/usr/bin/msmtp") "/usr/bin/msmtp")
                        ((file-exists-p "/usr/local/bin/msmtp") "/usr/local/bin/msmtp")))

Default to my personal mail account.

(setq message-sendmail-extra-arguments '("-a" "fastmail"))
(setq mail-host-address "n.rix.si")

And then we set group parameters for my work and personal accounts. These group parameters will munge the from address in the posting-style, and will set the proper mail account for msmtp.

(setq gnus-parameters
      '((".*fastmail.*"
         (posting-style
          (address "ryan@whatthefuck.computer")
          (name "Ryan Rix")
          (body "")
          (eval (setq message-sendmail-extra-arguments '("-a" "fastmail")))
          (signature-file "~/sync/personal-sig")
          (user-mail-address "ryan@whatthefuck.computer")))
        (".*gmail.*"
         (posting-style
          (address "rrix@uber.com")
          (name "Ryan Rix")
          (body "")
          (signature-file "~/sync/work-sig")
          (eval (setq message-sendmail-extra-arguments '("-a" "work")))
          (user-mail-address "rrix@uber.com")))
        (".*fastmail/social"
          (gnus-thread-sort-functions '(gnus-thread-sort-by-score gnus-thread-sort-by-author)))
        (".*fastmail/newsletters"
          (gnus-thread-sort-functions '(gnus-thread-sort-by-score gnus-thread-sort-by-author)))))

(defun cg-feed-msmtp ()
  (if (message-mail-p)
      (save-excursion
        (let* ((from
                (save-restriction
                  (message-narrow-to-headers)
                  (message-fetch-field "from")))
               (account
                (cond
                 ((string-match "ryan@whatthefuck.computer" from) "fastmail")
                 ((string-match "rrix@uber.com" from) "work")
                 ((string-match "ry@n.rix.si" from) "fastmail"))))
          (setq message-sendmail-extra-arguments (list '"-a" account))))))

(setq message-sendmail-envelope-from 'header)
(add-hook 'message-send-mail-hook 'cg-feed-msmtp)

Adaptive Scoring

To replicate the most interesting part of GMail, its Priority Inbox 14, you can use Gnus' awesome automated scoring system, which you can augment with manual scores. This provides a really wonderful interface to weight your mail based on what you want to read, based on what you've read (or ignored) in the past.

Firstly, make sure you're sorting threads by score.

(setq gnus-thread-sort-functions
      '(gnus-thread-sort-by-subject
        gnus-thread-sort-by-date
        gnus-thread-sort-by-total-score))

Then we enable the adaptive scoring, both for the entire line (full From, full Subject, etc) as well as for each word, we'll add scoring.

(setq gnus-use-adaptive-scoring '(word line))

Don't add scoring filters for short words.

(setq gnus-adaptive-word-length-limit 5)

Some email lists mangle the Subject header to add the list name. This is unholy, but we can account for it, so that it doesn't fuck up the scoring.

(setq gnus-adaptive-word-no-group-words t)

Here we define our scores; these are the defaults, and should probably be tweaked a little bit, but basically if I read it or mark it as important, they should be pulled up, and if I expire or kill them or delete them, they should be pulled down. Nice and simple.

(setq gnus-default-adaptive-score-alist
      '((gnus-unread-mark)
        (gnus-ticked-mark (from 4))
        (gnus-dormant-mark (from 5))
        (gnus-del-mark (from -4) (subject -1))
        (gnus-read-mark (from 4) (subject 2))
        (gnus-expirable-mark (from -1) (subject -1))
        (gnus-killed-mark (from -1) (subject -3))
        (gnus-kill-file-mark)
        (gnus-ancient-mark)
        (gnus-low-score-mark)
        (gnus-catchup-mark (from -1) (subject -1))))

I used to use owncloud15 to sync my Gnus score files between my various machines; for a while I couldn't understand why it wouldn't work, why Owncloud just silently refused to sync them. Low and behold, it turns out they silently were ignoring files with these characters in them16. Strip them out, replace them with underscores.

(setq nnheader-file-name-translation-alist '((?: . ?_) (?[ . ?_) (?] . ?_)) )

We can also score Groups by how often we enter them, using gnus-summary-bubble-group, and then can sort that way by hitting G S v.

(add-hook 'gnus-summary-exit-hook 'gnus-summary-bubble-group)

Expansion

Gnus exposes an RSS feed as a list of messages, much like any other folder, and displays the bodies of the RSS item inside where the mails usually go. This is neat, except in the case where an RSS feed shows redacted items with only a summary or the first paragraph of the body or, hell, nothing at all. So I could read some of the articles directly in Gnus, and some not. Well, boo, I'd rather have a unified workflow even if it is slightly less efficient for certain idle streams.

So what I do is I expand the RSS feed in to my browser, and then backfill in to my idle queue using my normal Org capture templates. This makes all of my items exist in the same place with the same viewing bandwidth attached to them.

I use EWW for my browser in this case, to make expansion and consumption easier.

Footnotes:

Author: Ryan Rix

Created: 2017-07-02 Sun 11:26

Validate XHTML 1.0