Wednesday, October 22, 2008

Exporting emails from Gmail to another account.

I've decided to create a separate gmail account for my work related stuff. Actually I used my account on gazeta.pl. It is powered by gmail but has an advantage - it uses separate cookies for the authentication. This means that I can be logged on both accounts at the same time in one browser. The gazeta.pl is a polish portal so the registration is in polish language. Still you can set the gmail interface to be displayed in your preferred language and google translate should help with the registration.

So I have my two separate gmail accounts and I can be logged onto both of them at the same time. But what to do with all my 12k emails from which most are work related. I searched the web if there is a way to copy all those emails from one gmail account to the other. The only thing that I found that was reasonable was to use the imap access to gmail accounts to copy the emails. I tried to do it with Evolution and Thunderbird - it is doable with few hundred emails. I got an imap error on average every 100 emails copied. Searching for the email were the error occurred and restarting from that point was a real PITA.

As recently I started using python for my daily work stuff I came up with an idea to write a script that will do the copying for me and will be more graceful about imap errors. After few evenings I have a solution that allowed me to copy all my 12k emails with all their tags. It should also help with exporting gmail emails to another imap server.

There are two scripts:
gmail.copy.mail.py - copies emails from "All Mail" folder to new account.
gmail.copy.tags.py - when we have all our emails on the new account it's time to propagate tags. This scripts goes through all emails that have "TagA" on initial gmail account, searches for the same emails in the new account and applies the "TagA" to them (though it can be set to be a "TagB").

So how to do it:
  1. Copy all the emails
    • First we should check how the "All Mail" folder is actually called:
      - This can be done by using Thunderbird or other email client. In case of my gmail.com account it was: '[Gmail]/All Mail', and in case of my gazeta.pl account it was: '[Google Mail]/All Mail'
      - Or use the script for this. There are three lines just after the config section which can be uncommented. They will list all folder names on the first account. So uncomment the lines. As first account specify the old account. Run the script. As the first account specify the new account. Run the script. Comment out the lines.
    • Edit the Config section in the script. Put your account names and the passwords. Also edit the names of the "All Mail" folders if needed.
    • Start the script:
      $ python gmail.copy.mail.py
    • If the script would fail on some email:
      - First check with Thunderbird if the mail is not replicated on the new account. It sometimes happened to me that I got several copies of the one mail in my new account. This didn't happen for the final version of the script on any of my 12k emails but who knows if it is bullet proof...
      - Copy the email by hand using Thunderbird
      - Restart the script from the next email:
      $ python gmail.copy.mail.py 1013
      Here I assumed that the script failed for emaill 1012. The uid will be printed for problematic emails.
  2. Copy the tags
    • This time we edit the gmail.copy.tags.script. As previously put your accounts and passwords.
    • As in the previous script there are three lines that let you to check all tag names in your gmail account. This is especially useful for Inbox, Drafts and other standard Gmail folders.
    • Before running the script go to the gmail interface. Select the tag you want to propagate. Select all emails with this tag and apply the tag to them once again. The reason for this is following. Gmail exposes the tags as imap folders. Each email that has a tag will be in an appropriate imap folder. But Gmail uses a concept of conversations, it is enough for one email to have a tag for the whole conversation to have a tag. This usually happens when one of the emails triggered a filter and others didn't. When coppying to other gmail account this probably will not matter as gmail will rebulid the conversations. Still to be on the safe side...
    • Run the script:
      $ python gmail.copy.tags.py MyTag MyTag
      Tag can be renamed during the copying:
      $ python gmail.copy.tags.py MyOldTag MyNewTag
    • There will be some emails on which the script will fail. It will return their date and uid. Based on date find them in Thunderbird and copy manually. Then start the script from next uid:
      $ python gmailcopy.tags.py MyTag MyTag 124
      Above I assumed that script failed on email 123. This is actually quite rare. In my 12k case I had 2 of such emails.
    • Repeat this for all tags you want to copy.
    • The script can be run in parallel.
    • If in the mean time new emails arrive on your old account the script will of course fail to copy their tags. Copy such emails manually using Thunderbird.
Please keep in mind that I provide those scripts as is. If they will mess up your account don't blame me :-).

Tuesday, January 08, 2008

SockSniffer v0.3

New version of SockSniffer (v0.3)
  • makejail and mountjail scripts were modified to work in 64bit environment.
  • Fixed some nasty race conditions in socksniff.
SockSniffer.0.3.tar.gz

Friday, December 21, 2007

NETDEV WATCHDOG: eth0: transmit timed out

Recently I encountered a problem with my desktop computer running Ubuntu 7.10 Gutsy Gibbon.

After some time network would die and the only way to bring it back was to restart the computer.
Nothing worked:
  • NetworkManager
  • sudo /etc/init.d/networking restart
  • reloading kernel modules
Looking into the /var/syslog I found this:
Dec 7 08:09:36 moonbiter kernel: [ 3751.383550] NETDEV WATCHDOG: eth0: transmit timed out
Dec 7 08:09:36 moonbiter kernel: [ 3751.383626] eth0: Transmit timed out, status 0003, PHY status 786d, resetting...
Dec 7 08:09:36 moonbiter kernel: [ 3751.383939] eth0: link up, 100Mbps, full-duplex, lpa 0x45E1
I have two ethernet cards: one builtin and one in a PCI slot (lspci comes in handy):
00:12.0 Ethernet controller: VIA Technologies, Inc. VT6102 [Rhine-II] (rev 7c)
05:07.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL-8169 Gigabit Ethernet (rev 10)

Googling around provided with a lot of contradicting solutions. None of them worked. Finally, on one of the forums I found an advice to check IRQs. Indeed this could have been the problem. I recently bought nVidia GeForce 7300 GT and to fit it in I had to move my Realtek ethernet adapter to another PCI slot.

Looking into /proc/interrupts I found:
CPU0
0: 46332 IO-APIC-edge timer
1: 155 IO-APIC-edge i8042
6: 5 IO-APIC-edge floppy
7: 0 IO-APIC-edge parport0
8: 0 IO-APIC-edge rtc
9: 0 IO-APIC-fasteoi acpi
12: 1617 IO-APIC-edge i8042
14: 15664 IO-APIC-edge ide0
15: 1242 IO-APIC-edge ide1
17: 717 IO-APIC-fasteoi eth1, HDA Intel
20: 0 IO-APIC-fasteoi uhci_hcd:usb1
21: 0 IO-APIC-fasteoi sata_via, uhci_hcd:usb3, ehci_hcd:usb5
22: 0 IO-APIC-fasteoi uhci_hcd:usb2
23: 13202 IO-APIC-fasteoi uhci_hcd:usb4, eth0
24: 13183 IO-APIC-fasteoi nvidia
NMI: 0
LOC: 46223
ERR: 0

And in /var/syslog
Dec 7 06:59:06 moonbiter kernel: [ 1637.162744] irq 23: nobody cared (try booting with the "irqpoll" option)
Dec 7 06:59:06 moonbiter kernel: [ 1637.162748]
Dec 7 06:59:06 moonbiter kernel: [ 1637.162749] Call Trace:
Dec 7 06:59:06 moonbiter kernel: [ 1637.162751] [__report_bad_irq+30/128] __report_bad_irq+0x1e/0x80
Dec 7 06:59:06 moonbiter kernel: [ 1637.162770] [note_interrupt+643/704] note_interrupt+0x283/0x2c0
Dec 7 06:59:06 moonbiter kernel: [ 1637.162777] [handle_fasteoi_irq+221/272] handle_fasteoi_irq+0xdd/0x110
Dec 7 06:59:06 moonbiter kernel: [ 1637.162897] [_end+129724926/2130332920] :nvidia:_nv003707rm+0x1f/0x27
Dec 7 06:59:06 moonbiter kernel: [ 1637.162904] [do_IRQ+123/256] do_IRQ+0x7b/0x100
Dec 7 06:59:06 moonbiter kernel: [ 1637.162909] [ret_from_intr+0/10] ret_from_intr+0x0/0xa
Dec 7 06:59:06 moonbiter kernel: [ 1637.162914] [pci_conf1_read+0/272] pci_conf1_read+0x0/0x110
Dec 7 06:59:06 moonbiter kernel: [ 1637.162921] [__do_softirq+84/224] __do_softirq+0x54/0xe0
Dec 7 06:59:06 moonbiter kernel: [ 1637.162929] [call_softirq+28/48] call_softirq+0x1c/0x30
Dec 7 06:59:06 moonbiter kernel: [ 1637.162933] [do_softirq+53/144] do_softirq+0x35/0x90
Dec 7 06:59:06 moonbiter kernel: [ 1637.162937] [do_IRQ+128/256] do_IRQ+0x80/0x100
Dec 7 06:59:06 moonbiter kernel: [ 1637.162942] [ret_from_intr+0/10] ret_from_intr+0x0/0xa
Dec 7 06:59:06 moonbiter kernel: [ 1637.162944] [_end+127707073/2130332920] :processor:acpi_processor_idle+0x25f/0x456
Dec 7 06:59:06 moonbiter kernel: [ 1637.162964] [_end+127707063/2130332920] :processor:acpi_processor_idle+0x255/0x456
Dec 7 06:59:06 moonbiter kernel: [ 1637.162971] [_end+127706466/2130332920] :processor:acpi_processor_idle+0x0/0x456
Dec 7 06:59:06 moonbiter kernel: [ 1637.162976] [cpu_idle+112/192] cpu_idle+0x70/0xc0
Dec 7 06:59:06 moonbiter kernel: [ 1637.162982] [start_kernel+645/784] start_kernel+0x285/0x310
Dec 7 06:59:06 moonbiter kernel: [ 1637.162987] [x86_64_start_kernel+286/352] _sinittext+0x11e/0x160
Dec 7 06:59:06 moonbiter kernel: [ 1637.162991]
Dec 7 06:59:06 moonbiter kernel: [ 1637.162992] handlers:
Dec 7 06:59:06 moonbiter kernel: [ 1637.162994] [_end+128326184/2130332920] (usb_hcd_irq+0x0/0x60 [usbcore])
Dec 7 06:59:06 moonbiter kernel: [ 1637.163010] [_end+128535048/2130332920] (rhine_interrupt+0x0/0xc70 [via_rhine])
Dec 7 06:59:06 moonbiter kernel: [ 1637.163017] Disabling IRQ #23

So irq assigned to eth0 was disabled ...
Following advice from the forum I added "noapic" boot option to my /boot/grub/menu.lst
And now network is solid stable again :D.

Just to see how /proc/interrupts looks now:
CPU0
0: 145441 XT-PIC-XT timer
1: 1186 XT-PIC-XT i8042
2: 0 XT-PIC-XT cascade
4: 0 XT-PIC-XT uhci_hcd:usb3, ehci_hcd:usb5
5: 24510 XT-PIC-XT uhci_hcd:usb1, uhci_hcd:usb4, eth0
6: 5 XT-PIC-XT floppy
7: 47352 XT-PIC-XT parport0
8: 0 XT-PIC-XT rtc
9: 0 XT-PIC-XT acpi
10: 48019 XT-PIC-XT nvidia
11: 740 XT-PIC-XT sata_via, uhci_hcd:usb2, eth1, HDA Intel
12: 7125 XT-PIC-XT i8042
14: 51693 XT-PIC-XT ide0
15: 5002 XT-PIC-XT ide1
NMI: 0
LOC: 145342
ERR: 4

I'm not too happy with this solution but it works...

Sunday, October 21, 2007

Sniffing the Google Desktop for Linux traffic.

I promised to present how to sniff communication between clients and server of Google Desktop for Linux.

So I'll be using SockSniffer scripts I prepared and wrote about [1], [2].
$ tar xvzf SockSniffer.0.2.tar.gz
$ mkdir jail
$ cd jail
$ $HOME/SockSniffer.0.2/makejail.sh
$ sudo $HOME/SockSniffer.0.2/mountjail.sh
$ sudo chroot $PWD
$ su my_user_name
$ cd
$ gdlinux

No we should have two Google Desktop icons in taskbar.

In chrooted console:
$ cd .google/desktop
$ rm a1_sock
In Another console:
$ mkdir socksniff_logs
$ cd socksniff_logs
$ $HOME/SockSniffer.0.2/socksniff $HOME/.google/desktop/a1_sock $HOME/jail$HOME/.google/desktop/a1_sock

Now use the Google Desktop for a while.
Close it.
Kill the socksniff process (e.g. using ^c).

In socksniff console:
$ $HOME/SockSniffer.0.2/sockmerge

Open $HOME/socksniff_logs/dumps/merge.pcap in wireshark.

That's it. Now we can analyze the traffic.

And here's an example dump.

SockSniffer v.0.2

New version of SockSniffer (v0.2):
  • makejail.sh and mountjail.sh scripts added.
    These scripts are used to prepare chroot jail for client to be sniffed.
    They are prepared with GoogleDesktop in mind.
  • Added some comments in the code.
  • The socksniff script was corrected: The -n option didn't work for server traffic.
SockSniffer.0.2.tar.gz