TITLE: Gmail on macOS command line: neomutt + offlineimap + notmuch + pass + vim + launchd (brew services) + w3m
DATE: 2018-07-15
AUTHOR: John L. Godlee
====================================================================


I’ve been experimenting with a new email client on the command line.
I used alpine for a long time, but I’ve heard a lot about mutt as
well. It seems to be the standard terminal email program, but my
past attempts to set it up have always failed, so I set aside a
couple of hours over the last few days to really get into the
subject. This is the setup I have so far, which uses the neomutt
fork. It’s an amalgamation of lots of different guides and snippets
that I’ve found scattered around on the internet. I’ve referenced
the important ones at the end.

y setup is fairly particular to me and my needs. I use macOS rather
than Linux which produces some interesting idiosyncrasies, and I
wanted to integrate my existing command line programs where
possible, notably vim (text editor) and pass (GPG enabled password
manager), and w3m (web browser). I also use Gmail, so interacting
with that requires understanding their slightly odd IMAP
configuration.

Offlineimap

For now we won’t even install neomutt, there isn’t anything to see
yet anyway and much of the config won’t make sense until we’ve set
up the back end stuff. The first thing to set up is the syncing of
emails with Gmail’s IMAP system.

neomutt can handle IMAP email, but grabbing from a remote server to read an email requires that there be an internet connection everytime you open neomutt. To keep an offline database of my Gmail account, so I can search it when travelling, I use offlineimap, which can also be installed using [Homebrew]:

    

  [Homebrew]: https://brew.sh/

    brew install offlineimap

[Steve Losh] says that offlineimap tends to include script
breaking changes between updates, and they recommend installing an
earlier version, but I used v7.2.1, which is current as of 27th Mar
2018 and it worked fine.

  [Steve Losh]: http://stevelosh.com/blog/2012/10/the-homely-mutt/

offlineimap takes its configuration from ~/.offlineimaprc. This is
my config:

    [general]
    ui = TTY.TTYUI
    accounts = johngodlee@gmail.com 
    pythonfile = ~/.offlineimap.py
    fsync = False
    ssl = False

    [Account johngodlee@gmail.com]
    localrepository = johngodlee-local
    remoterepository = johngodlee-remote

    [Repository johngodlee-local]
    type = Maildir
    localfolders = ~/.mail/johngodlee@gmail.com
    nametrans = lambda folder: {'drafts': '[Google Mail]/Drafts',
        'sent': '[Google Mail]/Sent Mail',
        'flagged': '[Google Mail]/starred',
        'trash': '[Google Mail]/Bin',
        }.get(folder, folder)

    [Repository johngodlee-remote]
    maxconnections = 3
    type = Gmail
    remoteuser = johngodlee@gmail.com
    remotepasseval = get_pass()
    realdelete = no
    ssl=true
    sslcacertfile = /usr/local/etc/openssl/cert.pem
    nametrans = lambda folder: {'[Google Mail]/Drafts':    'drafts',
        '[Google Mail]/Sent Mail': 'sent',
        '[Google Mail]/Starred':   'starred',
        '[Google Mail]/Bin':     'trash',
        }.get(folder, folder)

    folderfilter = lambda folder: folder not in ['[Google Mail]/Bin',
        '[Google Mail]/Important',
        '[Google Mail]/Spam',
        '[Google Mail]/Chats',
        '[Google Mail]/All Mail',
        ]

There is a lot there, so here is a line by line breakdown:

ui = TTY.TTYUI signals which user interface to use. TTY.TTYUI makes
sure that the process runs quietly in the background, there are also
other options which may be better, but this one works for me.

accounts = johngodlee@gmail.com is the email account to be checked.

pythonfile = ~/.offlineimap.py is a python script which offlineimap
calls to get the password for my email account from pass. I’ll go
through .offlineimap.py later.

fsync = False just tells offlineimap that it doesn’t need to make
sure a full sync is completed every time, if a few emails get lost,
no bother, they will just get synced next time

    [Account johngodlee@gmail.com]
    localrepository = johngodlee-local
    remoterepository = johngodlee-remote

The section above defines the names of repositories to use for this
email account.

    [Repository johngodlee-local]
    type = Maildir
    localfolders = ~/.mail/johngodlee@gmail.com

The section above defines the location and mailbox type of the local
repository for the email from my account

nametrans = lambda folder: {'drafts': '[Google Mail]/Drafts', ...}.get(folder, folder)
translates Gmail’s IMAP folders into folders in my local mail
directory. I think this is necessary as Gmail’s folders contain
forward slashes, which would mess up a normal file system. I only
translated the system IMAP folders, as custom folders (labels) don’t
use the [Google Mail]/ prefix and are synced automatically as is. I
had some trouble getting offlineimap to recognise my system folders,
as many other guides recommend using the [Gmail]/ prefix, but for
whatever reason my account uses [Google Mail]/.

    [Repository johngodlee-remote]
    maxconnections = 3
    type = Gmail
    remoteuser = johngodlee@gmail.com
    remotepasseval = get_pass()
    realdelete = no
    ssl=true
    sslcacertfile = /usr/local/etc/openssl/cert.pem

This section configures the remote repository and how to interact
with it.

maxconnections = 3 defines the number of parallel connections
offlineimap can make when syncing emails. 3 is low enough that Gmail
won’t enforce rate limits and break the connection.

type = Gmail tells offlineimap that the account is a Gmail account,
and so it should take that into account when dealing with their
weird IMAP setup.

remoteuser = johngodlee@gmail.com simply tells us what account is
being accessed.

remotepasseval = get_pass() tells offlineimap where to get the
password for the account. get_pass() is the python function that is
created in ~/.offlineimap.py

realdelete = no tells offlineimap not to totally delete an email
when you press delete in neomutt, instead it will keep it in All
Mail.

ssl = true says to always use ssl encryption when syncing

sslcacertfile = /usr/local/etc/openssl/cert.pem gives the location
of a security certificate, which I think helps ssl to further
prevent man in the middle attacks when syncing.

nametrans = lambda folder: {'[Google Mail]/Drafts':    'drafts', ...}.get(folder, folder)
is just like the nametrans function earlier, only it goes in the
other direction.

folderfilter = lambda folder: folder not in ['[Google Mail]/Bin', ...]
gives a list of folders that should not be synced. In my case, I
don’t want to sync the trash (Bin because British), Spam, Chats, and
All Mail.

Later on I’ll look at how to get offlineimap to run in the
background using launchd, a macOS alternative to crontab.

Pass and .offlineimap.py

    #!/usr/bin/env python

    from subprocess import check_output

    def get_pass():
      return check_output("/usr/local/bin/pass email/johngodlee@gmail.com", shell=True).splitlines()[0]

This script looks in pass for the entry email/johngodlee@gmail.com,
which contains the password for my email account, and then takes the
first line (splitlines()[0]) of that entry to store it in
get_pass(). This function is then called by remotepasseval in
.offlineimaprc. The script should be saved as ~/.offlineimap.py.

Other programs

notmuch

notmuch provides very fast email searching. It has other
capabilities such as tagging and indexing email, but I use Gmail’s
IMAP folders (labels) for this, so I only use it for searching.
notmuch can be installed via Homebrew:

    brew install notmuch

When installed, neomutt allows searching using notmuch by defining a
custom macro. You can then use all the notmuch search syntax to
quickly find emails within neomutt. I’ll go through that in the next
section where I define the muttrc.

w3m

Neomutt doesn’t know how to display emails that are encoded as HTML
only. To render HTML as something legible in the terminal I use w3m.
This is a text based web browser. Once again, this can be installed
using Homebrew:

    brew install w3m

To tell neomutt to forward HTML emails to w3m for plain text
encoding, then send it back to neomutt, we need to create a file
called .mailcap. Keep it in the home directory as convention:

    touch ~/.mailcap

Ten add the following snippet to the file:

    text/html; w3m -dump -o document_charset=%{charset} '%s'; nametemplate=%s.html; copiousoutput

.mailcap is then referenced later in the muttrc.

Neomutt

Now that we’ve set up all the background, we can install the central
part of the workflow, mutt. Specifically, I’m using neomutt, which
is a fork of the original mutt which incorporates some of the most
widely used plugins.

neomutt can be installed using Homebrew:

    brew install neomutt

Then we need to make a configuration file, I like to keep mine in
~/.mutt/muttrc but see man neomutt for more locations where neomutt
will look for a config file:

    mkdir ~/.mutt

    touch ~/.mutt/muttrc

The first thing to add to the configuration is some IMAP settings:

    set imap_user = "johngodlee@gmail.com"

    ## Call pass from within 
    set my_pass = "`pass email/johngodlee@gmail.com`" 
    set imap_pass = $my_pass

    set from = "johngodlee@gmail.com"
    set realname = "John Godlee"

    set folder = "~/.mail/johngodlee@gmail.com"
    set spoolfile = +INBOX
    set postponed = +drafts
    set record = +sent

    ## Don't automatically move messages after reading
    set move = no

    ## Max time mutt should wait before polling IMAP connections, in minutes
    set imap_keepalive = 30  

    ## Set smtp URL for replying 
    set smtp_url = "smtp://johngodlee@smtp.gmail.com:587/"

    ## Set the login method for smtp replying
    set smtp_pass = $my_pass
    set smtp_authenticators = 'login'

    ## Allow mutt to open new imap connections to test for new mail 
    unset imap_passive

    ## Set cache locations so IMAP polling is quicker on startup
    set header_cache = ~/.mutt/johngodlee/headers
    set message_cachedir = ~/.mutt/johngodlee/bodies
    set certificate_file = ~/.mutt/certificates

    ## If stuck in a prompt, abort after n minutes to check IMAP (timeout), n minutes between checks (mail_check)
    set timeout = 10
    set mail_check = 5

Some of this is self explanatory, so I’ll only go through the
important bits

set my_pass ... calls pass to create the variable my_pass which is
then sent to imap_pass to set the password for my Gmail account,
which is needed for sending mail.

set folder ... sets the location of the mailbox which offlineimap
syncs email to. The value of folder also acts as the prefix for
set spoolfile, set postponed and set record, as indicated by putting
+ before the folder names.

set spoolfile ... should be the location of your inbox, where new
mail arrives

set postponed ... is where unfinished new emails are kept if they
are aborted before being sent

set record ... should be a folder where a copy of sent messages are
kept

Note that set smtp_pass ... also calls the my_pass variable

As neomutt comes with the sidebar patch, I wanted to make use of it
by displaying any folders that are synced from offlineimap:

    ## Mailboxes to display in sidebar and check regularly 
    mailboxes +INBOX +'starred' +'trash' +'drafts' +'sent' +'Archived' +'coding_club' +'diss_manuscript' +'personal' +'PhD' +'SEOSAW_contacts' +'SEOSAW_logos' +'STEB_2018' +'urgent' +'hemi_lens_proposal' +kew_taxonomy_course +'angola_botanic_garden'

    ## Sort sidebar folders alphabetically
    set sidebar_sort_method = name

    ## Sidebar visible by default
    set sidebar_visible = yes

    ## Don't abbreviate folders in sidebar
    set sidebar_short_path = no

I use a stripped down .vimrc for writing email, as I don’t need a
lot of the plugins that I use for programming. I can tell neomutt to
write emails using vim with this alternative .vimrc (.vimrc_alpine)
by adding the following to the neomuttrc:

    set editor = 'vim -u ~/.vimrc_alpine'

Here is my .vimrc_alpine:

    set nocompatible              " be iMproved, required
    filetype off                  " required

    " enable syntax highlighting
    syntax on

    " Stop creating swp and ~ files
    set nobackup
    set noswapfile

    " Ignore case of searches
    set ignorecase

    " Don’t reset cursor to start of line when moving around
    set nostartofline

    " Preserve indentation on wrapped lines
    set breakindent

    " Disable folding in markdown
    let g:vim_markdown_folding_disabled = 1

    " Disable syntax conceal in markdown
    let g:vim_markdown_conceal = 0

    " Normal backspace behaviour
    set backspace=2

    " map A (append at end of line) to a (append in place)
    nnoremap a A

    " Move by visual lines rather than actual lines with k,j
    nnoremap k gk
    nnoremap j gj
    nnoremap gk k
    nnoremap gj j

    " Easier save and quit with ;
    noremap ;w :w<CR>
    noremap ;q :q<CR>

    " Copy and paste from `+` register for interacting with mac clipboard
    vnoremap y "+y 
    vnoremap p "+p
    nnoremap p "+gp
    vnoremap d "+d
    nnoremap dd "+dd

    " Map of modes and their codes for statusline
    let g:currentmode={
        \ 'n'  : 'N ',
        \ 'no' : 'N·Operator Pending ',
        \ 'v'  : 'V ',
        \ 'V'  : 'V·Line ',
        \ '^V' : 'V·Block ',
        \ 's'  : 'Select ',
        \ 'S'  : 'S·Line ',
        \ '^S' : 'S·Block ',
        \ 'i'  : 'I ',
        \ 'R'  : 'R ',
        \ 'Rv' : 'V·Replace ',
        \ 'c'  : 'Command ',
        \ 'cv' : 'Vim Ex ',
        \ 'ce' : 'Ex ',
        \ 'r'  : 'Prompt ',
        \ 'rm' : 'More ',
        \ 'r?' : 'Confirm ',
        \ '!'  : 'Shell ',
        \ 't'  : 'Terminal '
        \}

    " Change statusline based on colour
    function! ChangeStatuslineColor()
      if (mode() =~# '\v(n|no)')
        exe 'hi! StatusLine ctermfg=112'
      elseif (mode() =~# '\v(v|V)' || g:currentmode[mode()] ==# 'V·Block' || get(g:currentmode, mode(), '') ==# 't')
        exe 'hi! StatusLine ctermfg=172'
      elseif (mode() ==# 'i')
        exe 'hi! StatusLine ctermfg=044'
      else
        exe 'hi! StatusLine ctermfg=007'
      endif
      return ''
    endfunction

    " Make statusline always show
    set laststatus=2

    " statusline
        " left side
        set statusline=%{ChangeStatuslineColor()}   " Change colour
        set statusline+=%0*\ %{toupper(g:currentmode[mode()])}  " Current mode
        set statusline+=\
        set statusline+=%1*%m%r " Modified and read only flags
        set statusline+=%1*%=   " Switch to right side

        " right side
        set statusline+=%0*\        "Space
        set statusline+=%0*%y   " File syntax
        set statusline+=%0*\|   " Vert-line
        set statusline+=%0*%p%% " Percentage through file
        set statusline+=%0*\|\  " Vert-line and Space
        set statusline+=%0*%l:%c    " Line and column number
        set statusline+=%0*\        " Space

    " Set colours for statusline middle section
    hi User1 ctermfg=255 ctermbg=240

    " Ragged right line breaks
    set linebreak

I want to read emails in plain text if possible, with the provided
plain text being used preferentially, and a converted HTML version
if plain text isn’t available, which I can set using
alternative_order .... I also want to tell neomutt how to deal with
forced HTML emails, which is to automatically view them and also to
give the path to the .mailcap file we created earlier, which uses
w3m to parse HTML as plain text:

    # Read in plain text if possible
    alternative_order text/plain text/html
    auto_view text/html
    set mailcap_path = ~/.mailcap

    # Enforce encoding in utf8
    set send_charset="utf-8"

To allow notmuch to work I need to allow neomutt to create virtual
folders when searching and also to give the location of the mailbox
to search (set nm_default_uri ...)

I also set up a keybinding to initiate the notmuch search from
within neomutt as \\

    set virtual_spoolfile = yes   
    set nm_default_uri = "notmuch:///Users/johngodlee/.mail/johngodlee@gmail.com"
    macro index,pager \\  "<vfolder-from-query>"  

Launchd

Other guides that I read recommended that I use crontab to schedule
offlineimap updates. But on macOS launchd is the proper way to do
things. macOS uses .plist scripts stored in ~/Library/LaunchAgents
to designate jobs, which use an XML format. Helpfully, Homebrew
services can generate .plist scripts for many programs, which can
then be amended later.

To start running Homebrew services, first see what services are
listed:

    brew services list

offlineimap should be listed in the output.

To generate the script to start the service at login:

    brew services start offlineimap

To ensure that notmuch follows behind offlineimap and updates its
database after every pull from Gmail, I added some extra commands to
the .plist file which Homebrew services generated:

    vim ~/Library/LaunchAgents/homebrew.mxcl.offlineimap.plist

Then amend the <array> section so it looks like this:

        <array>
          <string>/usr/local/opt/offlineimap/bin/offlineimap</string>
          <string>-u</string>
          <string>quiet</string>
          <string>;</string>
          <string>notmuch</string>
          <string>new</string>
        </array>

This runs a notmuch new everytime offlineimap runs. The other flags
-u and quiet tell offlineimap not to print anything to the terminal
which could disrupt the job running in the background.

Guides I used

With all this setup you should be able to run offlineimap, then open
neomutt and start reading and writing email, though it’s entirely
possible that I may have missed something. In any case, here are the
guides that I learnt from and used to write this setup:

-   https://wiki.archlinux.org/index.php/mutt
-   http://stevelosh.com/blog/2012/10/the-homely-mutt/
-   https://smalldata.tech/blog/2016/09/10/gmail-with-mutt
-   https://gist.github.com/amandabee/cf7faad0a6f2afc485ee
-   https://github.com/cbracken/mutt
-   https://www.farces.com/wikis/naked-server/homebrew/brew-services/
-   https://bbs.archlinux.org/viewtopic.php?id=142377
-   https://pbrisbin.com/posts/mutt_gmail_offlineimap/
-   https://baptiste-wicht.com/posts/2014/07/a-mutt-journey-my-mutt-configuration.html

y full setup can be explored in my [dotfiles]

  [dotfiles]: https://github.com/johngodlee/dotfiles

  {IMAGE}