Backup gmail with offlineimap: folders and labels.

Here are a couple details about backing up a gmail account through the IMAP protocol using offlineimap.

Gmail uses IMAP folders, that we can consider as normal filesystem folders, and also labels, often called tags in other applications. The problem is that the IMAP protocol (or at least Maildir) does not know anything about labels. So this is how gmail deals with this discrepancy:

Gmail uses only 4 real IMAP folders*:

  • [Gmail]/All Mail
  • [Gmail]/Drafts
  • [Gmail]/Spam
  • [Gmail]/Trash

This means that every email in your account, apart from trashed ones, spam or drafts, are stored in the IMAP folder [Gmail]/All Mail. If we synchronize our Gmail account (e.g. with offlineimap), the Gmail server will transorm the labels as real IMAP folders. For example, synching with offlineimap without any folderfilter, we get the following directory structure:

[Gmail]
    All Mail
    Drafts
    Important
    Sent Mail
    Spam
    Starred
    Trash
INBOX
Personal
Receipts
Travel
Work

Every labeled email (note that even [Gmail]/Sent Mail is a label), will be duplicated because any email will always be stored in All Mail and will also be stored in the folder(s) corresponding to its label(s).

If you have a crappy internet connection, for backup purposes, we can fetch only the [Gmail]/All Mail folder. This can be achieved setting folderfilter in the .offlineimaprc file as:

folderfilter = lambda foldername: foldername in ['[Gmail]/All Mail']

Note, however, that in this way we lose the label structure.

If instead we want to use the downloaded Maildir for bidirectional synchronization, i.e. we want to use a local mail client like mutt, then we have to fetch everything, thus downloading and storing duplicated emails. When synching back, the Gmail IMAP server will know how to handle IMAP folders as labels, and will not duplicate anything server-side.

A sample offlineimaprc file is:

## For a complete offlineimaprc example, see: https://github.com/OfflineIMAP/offlineimap/blob/master/offlineimap.conf

[general]
## Possible options: Quiet Basic TTYUI Blinkenlights
## NB: only TTYUI and Blinkenlights are capable of prompting for the password
ui = TTYUI
accounts = gmail

[Account gmail]
localrepository = gmail-local
remoterepository = gmail-remote

[Repository gmail-local]
type = Maildir
localfolders = ~/.gmail
## This will locally replicate the IMAP folder structure, instead of the default separator '.' that flattens it.
sep = /

[Repository gmail-remote]
type = Gmail
remoteuser = marco.rucci@gmail.com
# remotepass = <insert your password or you will be prompted>

## One-way synching.  Perfect for backups.
readonly = True

ssl = yes
cert_fingerprint = f3043dd689a2e7dddfbef82703a6c65ea9b634c1

## Sync only 'All Mail': we lose the folder/labels structure, but this is the only way to not duplicate emails.
# folderfilter = lambda foldername: foldername in ['[Gmail]/All Mail']

More information at the following pages:

http://askubuntu.com/questions/23287/what-can-i-use-to-automate-backups-of-gmail
http://pbrisbin.com/posts/mutt_gmail_offlineimap
https://github.com/OfflineIMAP/offlineimap/
http://support.google.com/mail/bin/answer.py?hl=en&answer=82367&topic=1668961&ctx=topic

* I did not find this explicitly anywhere but this is what I understood reading this google IMAP support page.

Using vimdiff with mercurial

Originally posted on 12 Feb 2010

I recently started using mercurial in order to version control my projects and one of the first needs I encountered was to analyze the output of an hg diff command to view the differences between revisions of the same file. Even using the color extension, interpreting the output of hg diff is unnecessarily complicated so I developed a simple script to confortably invoke vimdiff.

This small script, called hgvimdiff, can be used in the following way:

$ hgvimdiff [-r|--rev [-r|--rev]] file

For example:

Let’s see it in action in a simple case. The following is the output of an hg diff command executed in a test repository.

diff -r 318572579991 a
--- a/a Fri Feb 12 18:39:32 2010 +0100
+++ b/a Fri Feb 12 18:46:09 2010 +0100
@@ -1,3 +1,5 @@
 first line

-second line after first commit
+
+
+added a third line

while the following is the vimdiff window created invoking “hgvimdiff a” that will compare the file “a” in the woring directory with the file “a” in the parent revision.

images/hgvimdiff-example.png

Show gmail unread messages in wmii status bar

Originally posted on 20 Jun 2009

Straight to the point. If you want to see the number of unread messages of your gmail account on your wmii status bar you can modify your wmiirc as follows:

  1. Add the following Action (substituting <user>, <password> and <refreshTime> to your needings):

  1. Ask to launch such Action on startup by inserting:

    action gmailRefresh &

  2. Update your status() function to something like:

    status() { echo -n "... <b>M:$(cat /tmp/gmail-unread-count)</b> ..." }

The idea is to write every <refreshTime> in a temporary file the number of unread messages obtained by reading the gmail atom feed via wget.

Fix visual mode paste command in vim

Originally posted on 10 Jun 2009

The visual mode put command can be used to paste previously copied text onto a visual selection, i.e. to replace selected test with previously yanked text.

I find the default behavior of this command quite incongruous because this paste command has a side effect: It also copies the text that is being changed/replaced.

The problem with that arises when I find myself in the following example situation:

After a furious copy’n’paste moment, I have the following sequence of lines in a text (usually c source code) file:

Line1
Line2
Line3

Now my objective is to replace the second and third line with the first by yanking (yy) the first line and then selecting (V) separately the second and third line and then pasting/replacing using the infamous visual-mode-put command (p) (there are other methods to do this, I know, but this is just an example).

The default command behaviour produces this: 

Line1
Line1
Line2

While obviously the expected result is:

Line1
Line1
Line1

The internal reason for this behavior can be easily spotted on the help for the visual paste command.

…Implementation detail: it actually works by first putting the register after the selection and then deleting the selection.

So, it is not different than just pressing ‘dP’ after visual selecting the text to replace.

This suggest where the problem is by recalling that the delete (d) command also stores the deleted text into the unnamed default register, overwriting the current content, so, when we perform the next paste command we will paste not the original text, but the last replaced text.

The solution to fix this incongruence, is to just remap the visual paste command with:

vnoremap p "_dP

in which we discard the deleted text by storing it into the black hole register.

I’ve not tested this mapping extensively but it seems to work in “normal” situations.