Wednesday, October 22, 2008

Exporting emails from Gmail to another account.

I've decided to create a separate gmail account for my work related stuff. Actually I used my account on gazeta.pl. It is powered by gmail but has an advantage - it uses separate cookies for the authentication. This means that I can be logged on both accounts at the same time in one browser. The gazeta.pl is a polish portal so the registration is in polish language. Still you can set the gmail interface to be displayed in your preferred language and google translate should help with the registration.

So I have my two separate gmail accounts and I can be logged onto both of them at the same time. But what to do with all my 12k emails from which most are work related. I searched the web if there is a way to copy all those emails from one gmail account to the other. The only thing that I found that was reasonable was to use the imap access to gmail accounts to copy the emails. I tried to do it with Evolution and Thunderbird - it is doable with few hundred emails. I got an imap error on average every 100 emails copied. Searching for the email were the error occurred and restarting from that point was a real PITA.

As recently I started using python for my daily work stuff I came up with an idea to write a script that will do the copying for me and will be more graceful about imap errors. After few evenings I have a solution that allowed me to copy all my 12k emails with all their tags. It should also help with exporting gmail emails to another imap server.

There are two scripts:
gmail.copy.mail.py - copies emails from "All Mail" folder to new account.
gmail.copy.tags.py - when we have all our emails on the new account it's time to propagate tags. This scripts goes through all emails that have "TagA" on initial gmail account, searches for the same emails in the new account and applies the "TagA" to them (though it can be set to be a "TagB").

So how to do it:
  1. Copy all the emails
    • First we should check how the "All Mail" folder is actually called:
      - This can be done by using Thunderbird or other email client. In case of my gmail.com account it was: '[Gmail]/All Mail', and in case of my gazeta.pl account it was: '[Google Mail]/All Mail'
      - Or use the script for this. There are three lines just after the config section which can be uncommented. They will list all folder names on the first account. So uncomment the lines. As first account specify the old account. Run the script. As the first account specify the new account. Run the script. Comment out the lines.
    • Edit the Config section in the script. Put your account names and the passwords. Also edit the names of the "All Mail" folders if needed.
    • Start the script:
      $ python gmail.copy.mail.py
    • If the script would fail on some email:
      - First check with Thunderbird if the mail is not replicated on the new account. It sometimes happened to me that I got several copies of the one mail in my new account. This didn't happen for the final version of the script on any of my 12k emails but who knows if it is bullet proof...
      - Copy the email by hand using Thunderbird
      - Restart the script from the next email:
      $ python gmail.copy.mail.py 1013
      Here I assumed that the script failed for emaill 1012. The uid will be printed for problematic emails.
  2. Copy the tags
    • This time we edit the gmail.copy.tags.script. As previously put your accounts and passwords.
    • As in the previous script there are three lines that let you to check all tag names in your gmail account. This is especially useful for Inbox, Drafts and other standard Gmail folders.
    • Before running the script go to the gmail interface. Select the tag you want to propagate. Select all emails with this tag and apply the tag to them once again. The reason for this is following. Gmail exposes the tags as imap folders. Each email that has a tag will be in an appropriate imap folder. But Gmail uses a concept of conversations, it is enough for one email to have a tag for the whole conversation to have a tag. This usually happens when one of the emails triggered a filter and others didn't. When coppying to other gmail account this probably will not matter as gmail will rebulid the conversations. Still to be on the safe side...
    • Run the script:
      $ python gmail.copy.tags.py MyTag MyTag
      Tag can be renamed during the copying:
      $ python gmail.copy.tags.py MyOldTag MyNewTag
    • There will be some emails on which the script will fail. It will return their date and uid. Based on date find them in Thunderbird and copy manually. Then start the script from next uid:
      $ python gmailcopy.tags.py MyTag MyTag 124
      Above I assumed that script failed on email 123. This is actually quite rare. In my 12k case I had 2 of such emails.
    • Repeat this for all tags you want to copy.
    • The script can be run in parallel.
    • If in the mean time new emails arrive on your old account the script will of course fail to copy their tags. Copy such emails manually using Thunderbird.
Please keep in mind that I provide those scripts as is. If they will mess up your account don't blame me :-).

Tuesday, January 08, 2008

SockSniffer v0.3

New version of SockSniffer (v0.3)
  • makejail and mountjail scripts were modified to work in 64bit environment.
  • Fixed some nasty race conditions in socksniff.
SockSniffer.0.3.tar.gz