TITLE: Gmail on macOS command line: neomutt + offlineimap + notmuch
+ pass + vim + launchd (brew services) + w3m
DATE: 2018-07-15
AUTHOR: John L. Godlee
====================================================================
I've been experimenting with a new email client on the command
line. I used alpine for a long time, but I've heard a lot about
mutt as well. It seems to be the standard terminal email program,
but my past attempts to set it up have always failed, so I set
aside a couple of hours over the last few days to really get into
the subject. This is the setup I have so far, which uses the
neomutt fork. It's an amalgamation of lots of different guides and
snippets that I've found scattered around on the internet. I've
referenced the important ones at the end.
y setup is fairly particular to me and my needs. I use macOS rather
than Linux which produces some interesting idiosyncrasies, and I
wanted to integrate my existing command line programs where
possible, notably vim (text editor) and pass (GPG enabled password
manager), and w3m (web browser). I also use Gmail, so interacting
with that requires understanding their slightly odd IMAP
configuration.
Offlineimap
For now we won't even install neomutt, there isn't anything to see
yet anyway and much of the config won't make sense until we've set
up the back end stuff. The first thing to set up is the syncing of
emails with Gmail's IMAP system.
neomutt can handle IMAP email, but grabbing from a remote server to
read an email requires that there be an internet connection
everytime you open neomutt. To keep an offline database of my Gmail
account, so I can search it when travelling, I use offlineimap,
which can also be installed using Homebrew:
[Homebrew]: https://brew.sh/
brew install offlineimap
Steve Losh
offlineimap takes its configuration from ~/.offlineimaprc. This is
my config:
general
ui = TTY.TTYUI
accounts = johngodlee@gmail.com
pythonfile = ~/.offlineimap.py
fsync = False
ssl = False
Account johngodlee@gmail.com
localrepository = johngodlee-local
remoterepository = johngodlee-remote
Repository johngodlee-local
type = Maildir
localfolders = ~/.mail/johngodlee@gmail.com
nametrans = lambda folder: {'drafts': '[Google Mail]/Drafts',
'sent': '[Google Mail]/Sent Mail',
'flagged': '[Google Mail]/starred',
'trash': '[Google Mail]/Bin',
}.get(folder, folder)
Repository johngodlee-remote
maxconnections = 3
type = Gmail
remoteuser = johngodlee@gmail.com
remotepasseval = get_pass()
realdelete = no
ssl=true
sslcacertfile = /usr/local/etc/openssl/cert.pem
nametrans = lambda folder: {'[Google Mail]/Drafts': 'drafts',
'[Google Mail]/Sent Mail': 'sent',
'[Google Mail]/Starred': 'starred',
'[Google Mail]/Bin': 'trash',
}.get(folder, folder)
folderfilter = lambda folder: folder not in ['[Google
Mail]/Bin',
'[Google Mail]/Important',
'[Google Mail]/Spam',
'[Google Mail]/Chats',
'[Google Mail]/All Mail',
]
There is a lot there, so here is a line by line breakdown:
ui = TTY.TTYUI signals which user interface to use. TTY.TTYUI makes
sure that the process runs quietly in the background, there are
also other options which may be better, but this one works for me.
accounts = johngodlee@gmail.com is the email account to be checked.
pythonfile = ~/.offlineimap.py is a python script which offlineimap
calls to get the password for my email account from pass. I'll go
through .offlineimap.py later.
fsync = False just tells offlineimap that it doesn't need to make
sure a full sync is completed every time, if a few emails get lost,
no bother, they will just get synced next time
Account johngodlee@gmail.com
localrepository = johngodlee-local
remoterepository = johngodlee-remote
The section above defines the names of repositories to use for this
email account.
Repository johngodlee-local
type = Maildir
localfolders = ~/.mail/johngodlee@gmail.com
The section above defines the location and mailbox type of the
local repository for the email from my account
nametrans = lambda folder: {'drafts': '[Google Mail]/Drafts',
...}.get(folder, folder) translates Gmail's IMAP folders into
folders in my local mail directory. I think this is necessary as
Gmail's folders contain forward slashes, which would mess up a
normal file system. I only translated the system IMAP folders, as
custom folders (labels) don't use the [Google Mail]/ prefix and are
synced automatically as is. I had some trouble getting offlineimap
to recognise my system folders, as many other guides recommend
using the [Gmail]/ prefix, but for whatever reason my account uses
[Google Mail]/.
Repository johngodlee-remote
maxconnections = 3
type = Gmail
remoteuser = johngodlee@gmail.com
remotepasseval = get_pass()
realdelete = no
ssl=true
sslcacertfile = /usr/local/etc/openssl/cert.pem
This section configures the remote repository and how to interact
with it.
maxconnections = 3 defines the number of parallel connections
offlineimap can make when syncing emails. 3 is low enough that
Gmail won't enforce rate limits and break the connection.
type = Gmail tells offlineimap that the account is a Gmail account,
and so it should take that into account when dealing with their
weird IMAP setup.
remoteuser = johngodlee@gmail.com simply tells us what account is
being accessed.
remotepasseval = get_pass() tells offlineimap where to get the
password for the account. get_pass() is the python function that is
created in ~/.offlineimap.py
realdelete = no tells offlineimap not to totally delete an email
when you press delete in neomutt, instead it will keep it in All
Mail.
ssl = true says to always use ssl encryption when syncing
sslcacertfile = /usr/local/etc/openssl/cert.pem gives the location
of a security certificate, which I think helps ssl to further
prevent man in the middle attacks when syncing.
nametrans = lambda folder: {'[Google Mail]/Drafts': 'drafts',
...}.get(folder, folder) is just like the nametrans function
earlier, only it goes in the other direction.
folderfilter = lambda folder: folder not in ['[Google Mail]/Bin',
...] gives a list of folders that should not be synced. In my case,
I don't want to sync the trash (Bin because British), Spam, Chats,
and All Mail.
Later on I'll look at how to get offlineimap to run in the
background using launchd, a macOS alternative to crontab.
Pass and .offlineimap.py
#!/usr/bin/env python
from subprocess import check_output
def get_pass():
return check_output("/usr/local/bin/pass
email/johngodlee@gmail.com", shell=True).splitlines()[0]
This script looks in pass for the entry email/johngodlee@gmail.com,
which contains the password for my email account, and then takes
the first line (splitlines()[0]) of that entry to store it in
get_pass(). This function is then called by remotepasseval in
.offlineimaprc. The script should be saved as ~/.offlineimap.py.
Other programs
notmuch
notmuch provides very fast email searching. It has other
capabilities such as tagging and indexing email, but I use Gmail's
IMAP folders (labels) for this, so I only use it for searching.
notmuch can be installed via Homebrew:
brew install notmuch
When installed, neomutt allows searching using notmuch by defining
a custom macro. You can then use all the notmuch search syntax to
quickly find emails within neomutt. I'll go through that in the
next section where I define the muttrc.
w3m
Neomutt doesn't know how to display emails that are encoded as HTML
only. To render HTML as something legible in the terminal I use
w3m. This is a text based web browser. Once again, this can be
installed using Homebrew:
brew install w3m
To tell neomutt to forward HTML emails to w3m for plain text
encoding, then send it back to neomutt, we need to create a file
called .mailcap. Keep it in the home directory as convention:
touch ~/.mailcap
Ten add the following snippet to the file:
text/html; w3m -dump -o document_charset=%{charset} '%s';
nametemplate=%s.html; copiousoutput
.mailcap is then referenced later in the muttrc.
Neomutt
Now that we've set up all the background, we can install the
central part of the workflow, mutt. Specifically, I'm using
neomutt, which is a fork of the original mutt which incorporates
some of the most widely used plugins.
neomutt can be installed using Homebrew:
brew install neomutt
Then we need to make a configuration file, I like to keep mine in
~/.mutt/muttrc but see man neomutt for more locations where neomutt
will look for a config file:
mkdir ~/.mutt
touch ~/.mutt/muttrc
The first thing to add to the configuration is some IMAP settings:
set imap_user = "johngodlee@gmail.com"
## Call pass from within
set my_pass = "`pass email/johngodlee@gmail.com`"
set imap_pass = $my_pass
set from = "johngodlee@gmail.com"
set realname = "John Godlee"
set folder = "~/.mail/johngodlee@gmail.com"
set spoolfile = +INBOX
set postponed = +drafts
set record = +sent
## Don't automatically move messages after reading
set move = no
## Max time mutt should wait before polling IMAP connections,
in minutes
set imap_keepalive = 30
## Set smtp URL for replying
set smtp_url = "smtp://johngodlee@smtp.gmail.com:587/"
## Set the login method for smtp replying
set smtp_pass = $my_pass
set smtp_authenticators = 'login'
## Allow mutt to open new imap connections to test for new mail
unset imap_passive
## Set cache locations so IMAP polling is quicker on startup
set header_cache = ~/.mutt/johngodlee/headers
set message_cachedir = ~/.mutt/johngodlee/bodies
set certificate_file = ~/.mutt/certificates
## If stuck in a prompt, abort after n minutes to check IMAP
(timeout), n minutes between checks (mail_check)
set timeout = 10
set mail_check = 5
Some of this is self explanatory, so I'll only go through the
important bits
set my_pass ... calls pass to create the variable my_pass which is
then sent to imap_pass to set the password for my Gmail account,
which is needed for sending mail.
set folder ... sets the location of the mailbox which offlineimap
syncs email to. The value of folder also acts as the prefix for set
spoolfile, set postponed and set record, as indicated by putting +
before the folder names.
set spoolfile ... should be the location of your inbox, where new
mail arrives
set postponed ... is where unfinished new emails are kept if they
are aborted before being sent
set record ... should be a folder where a copy of sent messages are
kept
Note that set smtp_pass ... also calls the my_pass variable
As neomutt comes with the sidebar patch, I wanted to make use of it
by displaying any folders that are synced from offlineimap:
## Mailboxes to display in sidebar and check regularly
mailboxes +INBOX +'starred' +'trash' +'drafts' +'sent'
+'Archived' +'coding_club' +'diss_manuscript' +'personal' +'PhD'
+'SEOSAW_contacts' +'SEOSAW_logos' +'STEB_2018' +'urgent'
+'hemi_lens_proposal' +kew_taxonomy_course +'angola_botanic_garden'
## Sort sidebar folders alphabetically
set sidebar_sort_method = name
## Sidebar visible by default
set sidebar_visible = yes
## Don't abbreviate folders in sidebar
set sidebar_short_path = no
I use a stripped down .vimrc for writing email, as I don't need a
lot of the plugins that I use for programming. I can tell neomutt
to write emails using vim with this alternative .vimrc
(.vimrc_alpine) by adding the following to the neomuttrc:
set editor = 'vim -u ~/.vimrc_alpine'
Here is my .vimrc_alpine:
set nocompatible " be iMproved, required
filetype off " required
" enable syntax highlighting
syntax on
" Stop creating swp and ~ files
set nobackup
set noswapfile
" Ignore case of searches
set ignorecase
" Don’t reset cursor to start of line when moving around
set nostartofline
" Preserve indentation on wrapped lines
set breakindent
" Disable folding in markdown
let g:vim_markdown_folding_disabled = 1
" Disable syntax conceal in markdown
let g:vim_markdown_conceal = 0
" Normal backspace behaviour
set backspace=2
" map A (append at end of line) to a (append in place)
nnoremap a A
" Move by visual lines rather than actual lines with k,j
nnoremap k gk
nnoremap j gj
nnoremap gk k
nnoremap gj j
" Easier save and quit with ;
noremap ;w :w<CR>
noremap ;q :q<CR>
" Copy and paste from `+` register for interacting with mac
clipboard
vnoremap y "+y
vnoremap p "+p
nnoremap p "+gp
vnoremap d "+d
nnoremap dd "+dd
" Map of modes and their codes for statusline
let g:currentmode={
\ 'n' : 'N ',
\ 'no' : 'N·Operator Pending ',
\ 'v' : 'V ',
\ 'V' : 'V·Line ',
\ '^V' : 'V·Block ',
\ 's' : 'Select ',
\ 'S' : 'S·Line ',
\ '^S' : 'S·Block ',
\ 'i' : 'I ',
\ 'R' : 'R ',
\ 'Rv' : 'V·Replace ',
\ 'c' : 'Command ',
\ 'cv' : 'Vim Ex ',
\ 'ce' : 'Ex ',
\ 'r' : 'Prompt ',
\ 'rm' : 'More ',
\ 'r?' : 'Confirm ',
\ '!' : 'Shell ',
\ 't' : 'Terminal '
\}
" Change statusline based on colour
function! ChangeStatuslineColor()
if (mode() =~# '\v(n|no)')
exe 'hi! StatusLine ctermfg=112'
elseif (mode() =~# '\v(v|V)' || g:currentmode[mode()] ==#
'V·Block' || get(g:currentmode, mode(), '') ==# 't')
exe 'hi! StatusLine ctermfg=172'
elseif (mode() ==# 'i')
exe 'hi! StatusLine ctermfg=044'
else
exe 'hi! StatusLine ctermfg=007'
endif
return ''
endfunction
" Make statusline always show
set laststatus=2
" statusline
" left side
set statusline=%{ChangeStatuslineColor()} " Change colour
set statusline+=%0*\ %{toupper(g:currentmode[mode()])} "
Current mode
set statusline+=\
set statusline+=%1*%m%r " Modified and read only flags
set statusline+=%1*%= " Switch to right side
" right side
set statusline+=%0*\ "Space
set statusline+=%0*%y " File syntax
set statusline+=%0*\| " Vert-line
set statusline+=%0*%p%% " Percentage through file
set statusline+=%0*\|\ " Vert-line and Space
set statusline+=%0*%l:%c " Line and column number
set statusline+=%0*\ " Space
" Set colours for statusline middle section
hi User1 ctermfg=255 ctermbg=240
" Ragged right line breaks
set linebreak
I want to read emails in plain text if possible, with the provided
plain text being used preferentially, and a converted HTML version
if plain text isn't available, which I can set using
alternative_order .... I also want to tell neomutt how to deal with
forced HTML emails, which is to automatically view them and also to
give the path to the .mailcap file we created earlier, which uses
w3m to parse HTML as plain text:
# Read in plain text if possible
alternative_order text/plain text/html
auto_view text/html
set mailcap_path = ~/.mailcap
# Enforce encoding in utf8
set send_charset="utf-8"
To allow notmuch to work I need to allow neomutt to create virtual
folders when searching and also to give the location of the mailbox
to search (set nm_default_uri ...)
I also set up a keybinding to initiate the notmuch search from
within neomutt as \\
set virtual_spoolfile = yes
set nm_default_uri =
"notmuch:///Users/johngodlee/.mail/johngodlee@gmail.com"
macro index,pager \\ "<vfolder-from-query>"
Launchd
Other guides that I read recommended that I use crontab to schedule
offlineimap updates. But on macOS launchd is the proper way to do
things. macOS uses .plist scripts stored in ~/Library/LaunchAgents
to designate jobs, which use an XML format. Helpfully, Homebrew
services can generate .plist scripts for many programs, which can
then be amended later.
To start running Homebrew services, first see what services are
listed:
brew services list
offlineimap should be listed in the output.
To generate the script to start the service at login:
brew services start offlineimap
To ensure that notmuch follows behind offlineimap and updates its
database after every pull from Gmail, I added some extra commands
to the .plist file which Homebrew services generated:
vim ~/Library/LaunchAgents/homebrew.mxcl.offlineimap.plist
Then amend the <array> section so it looks like this:
<array>
<string>/usr/local/opt/offlineimap/bin/offlineimap</string>
<string>-u</string>
<string>quiet</string>
<string>;</string>
<string>notmuch</string>
<string>new</string>
</array>
This runs a notmuch new everytime offlineimap runs. The other flags
-u and quiet tell offlineimap not to print anything to the terminal
which could disrupt the job running in the background.
Guides I used
With all this setup you should be able to run offlineimap, then
open neomutt and start reading and writing email, though it's
entirely possible that I may have missed something. In any case,
here are the guides that I learnt from and used to write this setup:
- https://wiki.archlinux.org/index.php/mutt
- http://stevelosh.com/blog/2012/10/the-homely-mutt/
- https://smalldata.tech/blog/2016/09/10/gmail-with-mutt
- https://gist.github.com/amandabee/cf7faad0a6f2afc485ee
- https://github.com/cbracken/mutt
-
https://www.farces.com/wikis/naked-server/homebrew/brew-services/
- https://bbs.archlinux.org/viewtopic.php?id=142377
- https://pbrisbin.com/posts/mutt_gmail_offlineimap/
-
https://baptiste-wicht.com/posts/2014/07/a-mutt-journey-my-mutt-conf
iguration.html
y full setup can be explored in my dotfiles
[dotfiles]: https://github.com/johngodlee/dotfiles
