Ubuntu Linux on the Move Legacy page for Dapper Drake

Introduction

 The main page covering my experiences in making the transition from Microsoft Windows to Ubuntu Linux - Fun with Ubuntu Linux quickly become excessively long.  This is the first, and for me one of the most important, specialised pages in what will probably become a series  covering specific aspects of the transition. I have just started the next new page covering what is perhaps the second most important activity after communications for inveterate travelers like ourselves, that is handling all the photographs we take with digital cameras, viewing them and ultimately putting them into a form that can be used on our web site. Retaining the fun approach I have called it Ubuntu Linux on the Take. Logically the third will be "Ubuntu Linux on the Make" covering Web Authoring!

One of my main reasons for making the transition were the huge flows of data with little control whilst on the move when using Microsoft Windows XP. I got so tired of the continual updates to Windows XP and the associated Virus checkers, Firewalls and Malware detectors. It is almost impossible to prevent a large data flow when one first connects and many programs seek information updates etc. There are dozens of processes running in the background under Windows XP and my hard drive is in continuous use minutes after the last user activity has finished. This all makes Windows XP difficult to use securely whilst traveling and costly on a mobile connection. Using the firewall to restrict the programs accessing the internet used to work but now many share common service interfaces and it has become less easy. In contrast Ubuntu uses under a 100 bytes (bytes not Kbytes) when make an average GPRS connection and will sit there for 10 minutes and use no more. Furthermore it has no virus checkers or firewalls which need updating and no unstoppable or essential automatic system updates.

Effective communications was the real test of whether I could make a permanent swap to Ubuntu Linux. It turned out to be easier in many ways than I expected but progress was marred by a number of little irritations, many caused by my ignorance and I went down enough dark if not blind alleys that I have decided to make it the subject of a separate page and only leave some general considerations and the overall outcome in Fun with Ubuntu Linux .

This page has become a record of  what I have done with details of all the changes I have made (and how) to the systems so I can repeat them in the future. It is a bit difficult to know where to start as there are hardware aspects (modems, phones, specialist cards etc) connection/interface aspects (Bluetooth, USB cables, serial cables etc) and low (Terminal) and high level Graphical User Interface (GUI) software. It is still in a sort of chronological order although considerable backfilling has occurred in many sections and I have done some re-ordering to avoid swapping back and forwards between different aspects. I have also included many of the links I found useful at the time. I hope it will be useful to others and perhaps avoid yet more reinventions of the wheel. 

The proof of what I have written here will be how we get on with the MSI Wind U100 Netbook which is more powerful than a Toshiba Satellite yet only weighs in at 1.1 Kgs - this is going to be entirely used with Linux and I am fully covering the transition to Ubuntu on the MSI Wind U100 for Global Communications so far it is all looking very good and Wifi and mobile internet via Bluetooth and Mobile phone seem to be fully operational - the only lack is a built in modem for landline.

'Classic' Dial-up Modems on a Land-line. 

I will start with what ought to be the simplest and certainly the most common type of connection to the internet used away from home. In practice it is not that simple and it is worth noting from the start that support for some types of modems is poor in all versions of Linux. The reason is simple - the type of modem which is used in many machines and built into many recent cards are often referred to as Winmodems or soft modems because they use Windows and the processor to do much of the hard work in software rather than do it on the chip or card. This software is often proprietary and Linux is poorly supported by most manufacturers. 

In my case Intel does support the chip set on the very cheap (£6.00) modem card in my AMD based desktop machine under some versions of Linux but  I could not really face a long battle at an early stage in Linux. I therefore got out my old but very good US Robotics V90 external serial interface Fax modem for the desktop.

The laptop also has a soft modem and the information I have gathered shows that relatively easy driver support is likely in the near future but at present it is a nightmare to install. I then remembered that I had a Xircom PCMCIA modem and Ethernet card which I plugged in - it was recognised immediately and I had a telephone connect working a few minutes later. 

Basic Modem Connection Software

The starting point for testing a modem connection is to use System  -> Administration -> Networking which allows a dial-up connection to be configured  and provides simple and consistent swapping between a broadband Ethernet connection, WiFi connection and a modem. This allows setting up a basic connection on a serial, or specified device, and needed no editing of files for my Serial Modem - it even detected the port it was connected to. There is however one very important trick to swapping - you MUST use a tick box to set the modem as the default route to the Internet before connecting and un-check the box before switching back to Ethernet. If you do not do that you may find that there is no Internet functionality when you go back to broadband although one can still see other machines over the network!  This switching of the default route took me some time to work out when I carried out my initial experiments in accessing my mobile phone. A disadvantage of is this approach is that you need to enter the superuser password every time you use Networking  as it is a administrative function needing root privileges. 

The Modem Monitor

At an early stage you should add a connection icon to the top panel by right clicking on the top  panel -> Add to Panel and dragging the Modem Monitor onto the Panel - it needs to be set by a double clicking and being set to monitor ppp0 for modems. It will also monitor the Ethernet connection eth0 and the Wifi connection ath0 which usefully shows the signal strength. 

Allowing extra users to use sudo.

Many of the modem related activities need administrative privileges. When you set up Ubuntu, it automatically adds the first user to the sudo group, allowing that user to make changes as the super user (root) by typing in their password. If you have several users on the machine it is useful to know how to give them the ability to use sudo and system programs such as Network Manager to manage connections. I found this suggestion at Linux.com | Ten tips for new Ubuntu users.  Go to System -> Administration -> Users and Groups. Select the user you want to add to sudo, and click Properties. Under the User privileges tab, check the box that says "Executing system administration tasks" .

A better long term approach is to use GNOME-PPP, an excellent GUI (Graphic User Interface) tool which puts an connection icon in the panel so you easily check on the connection and terminate it - I see no real reason why one should not use it from the start.

GNOME-PPP

Chronologically this is the point at which I discovered and switched to 'front end' to wvdial called GNOME-PPP but this should really be at the top as it is the way forward for most users. GNOME PPP is installed via Add/Remove and seems to be an excellent way to get a simple dial up connection which also puts a icon in the bar which can be used to monitor and stop the connection - first class for a normal simple modem set up which detected my serial modem automatically and connected me up. I do not see a way built in to handle multiple connection but it is so simple to change that may be not worth worrying so having done my initial trials with wvdial with the Vodafone Connect Card I have now switched to GNOME PPP.

Installing Gnome-PPP: Gnome-PPP is not installed automatically so install it with Add/Remove programs and searching for gnome-ppp.

User privileges: On some versions of Ubuntu you do not automatically have all the permissions to run Gnome-PPP as the first (default) user created even with though you should have all the normal administration privileges - you need to be both in the dip and the dialout groups to run it. This apparent mistake was done for additional security but is a nuisance If this is the case the log file which is available from the Log button will have a line complaining about permissions for the executable file /usr/sbin/pppd. The easiest way is to do System -> Administration -> Users and Groups then unlock with your password and highlight your user name and click Properties and on the User Privileges Tab tick Connect to Internet using a Modem. If this does not work or you prefer to use a terminal then the following two commands will add YOURUSERNAME to the dip and dialup groups

sudo adduser YOURUSERNAME dip
sudo adduser YOURUSERNAME dialout

You can check which groups you are in (and get some other useful information by:

id YOURUSERNAME

Normal Configuration: The main configuration is done by a GUI interface and is all fairly obvious if you are making a simple land-line Dial-Up connection. The default modem initialisation strings and other settings will almost certainly work. so only the initial screen needs the username, password and telephone number need setting for a land-line modem.  The only thing missing with GNOME-PPP is help. It uses its own configuration file $HOME/.wvdial.conf which it is reported can have additions despite the warnings not to modify by hand. 

If, however, you want to make a GPRS connection via most mobile phones you need to set up the APN via an initialisation string option on the second tab of the Setup Screen, and then use a special code in place of the phone number. Using Vodafone UK PrePay on the T610 as an example the Initialisation string 3 is set to:

Init 3 AT+CGDCONT=1,"IP","pp.vodafone.co.uk"

and the phone code used is

*99***1#

For other providers replace the pp.vodafone.co.uk with their APN (eg Vodafone UK Contract is internet, Guernsey Wave is pepper and Vodafone NZ is www.vodafone.net.nz)

In general the username and passwords are not checked but for most phones something needs to be present - check with your provider. I use web and web with vodafone. There are more details on my Global Communications and Computing page

Changes required in /etc/ppp/options

Some other options need to be set for most mobile network connections to prevent them timing out. These are not possible via the GUI interface and are taken from the the 'master' pppd set up file which is /etc/ppp/options. We can look at the current options which are set by:

sudo egrep -v '#|^ *$' /etc/ppp/options

It is desirable to make a backup before editing the /etc/ppp/options file so to make a copy and open a terminal:

sudo cp /etc/ppp/options /etc/ppp/options_bak
sudo gedit /etc/ppp/options

I have made three changes to the file. The first two are essential for most GPRS connections and disable the sending and checking of the echo response sent to check the connection is alive - the echo is not implemented by most mobile service providers and the default result is that 4 echo response requests are sent at 30 second intervals and after the 4th failure to receive a response the connection is broken. If you are disconnected after 2 minutes that is the cause. The new values of 0 inhibit the sending and checking:

lcp-echo-failure 0
lcp-echo-interval 0

The third change is essential for the Vodafone Connect Card but does not impact other connections significantly so I do it routinely so I do not forget. It involves disabling negotiation of Van Jacobson style IP header compression by un-commenting

-vj

I have also tried modifying the /ect/ppp/options file to add an extra delay before connection as I was sometimes not getting a correct DNS delivered using Gnome-PP with the Vodafone but it seems to have little benefit but you can try.

connect-delay 5000

Note the information in the /etc/ppp/options file does not recommend that changes are made in this file. The documentation ( man pppd ) says that if there is a file ~/.ppprc it is used for user default options which could be used instead of modifying /etc/ppp/options - I have not tried that method as the simple way works for me and gives me a working system. I have the attitude that "if its not broken don't fix it"

Permissions: I have once had a problem after installing and uninstalling the vodafone connect software with gnome-ppp complaining about permissions which did not seem to be cured by the setting your user permissions as above - it seemed that the other software had modified something and gnome-ppp or more precisely the pppd daemon needed to be run as root. This was cured by setting the setuid attribute with:

sudo chmod u+s /usr/sbin/pppd

When an executable file has been given the setuid attribute, normal users on the system can execute this file and gain the privileges of the user who owns the file.

Firefox 3 and PPP connections - fixing Firefox starting in Offline Mode

The writers of Firefox 3 have tried to be clever and automatically start Firefoz in the Offline Mode if there is no internet connection present. They have done that in Linux by checking if Network Manager has an open connection which is fine for the Ethernet and Wireless Network connections that are looked after by Network Manager. Unfortunately Network Manager does not know about Point to Point Protocol connections - that is dial up and other mobile connections using the PPP daemon. The result is that if you make a connection using Gnome-PPP Firefox swithes into Offline Mode when it starts and it has to be switched back using File -> Work Offline. This is OK for the occasional connection but gets very tedious if you have mobile broadband in use all the time or have a old telephone dial up via a modem. This has been discussed at length and there is now an option to inhibit the feature but only deep in the configuration. The way to access it is by entering about:config in the address bar and then enter. This will bring up a warning screen about dragons being present so you have to take care - click Ok and then enter toolkit.networkmanager.disable into the filter which will reduce you down to a view of the required option which will be currently set to false. Right click -> Toggle - this will change the value to true and inhibit the check in network manager. Exit Firefox and when you next use it on a PPP connection it will no longer start up offline. For more details see The Mozillazine Article toolkit.networkmanager.disable

Bluetooth connection to a Modem in a Mobile Telephone 

We are now getting to the more interesting stage - connecting up through a mobile phone (or PDA with mobile access). Before we can get to the use of a mobile phone we have to consider how to connect it. I prefer Bluetooth as it does not involve cables and has much greater range than Infra Red allowing a phone to be placed in a good signal area (in theory up to 10 metres away) so I will only discuss Bluetooth at this point. 

I plugged in my existing Bluetooth Dongle, a D-Link DBT-120 and it was detected by the system immediately. 

The following procedures to set up to use Bluetooth were those I used initially and are based on http://www.russel.org.uk/gprsInternet.html and modified for my phone which is a Sony Ericsson T610 . 

• Plug in the Bluetooth Dongle and turn on the phone with bluetooth enabled and set to enable discover.
• Run ‘hcitool scan’ on the laptop in a terminal window to ensure the Bluetooth system is up and running and to check Bluetooth is enables on the phone.This should give you the Bluetooth address of the phone – the address is of the form xx:xx:xx:xx:xx:xx where each xx is a hexadecimal digit pair.
• The standard Ubuntu set up has the  PIN in /etc/bluetooth/pin which should be set to the PIN you wish to use.  
• Bond the phone to the laptop by having Bluetooth up and running on the laptop then doing a search and bond from the phone giving the correct PIN. This can be done latter but is a good check that the set up is working.
• Run ‘sdptool browse <phone-bluetooth-address>’ to find out which channels are being used for each of the services. The important entry for using the phone as modem is the one labeled‘Service Name: Dial-up Networking’. For me that is Channel 1.
• Set up an RFCOMM device. Use /dev/rfcomm0 as the device to connect to the phone as a modem by setting up /etc/bluetooth/rfcomm.conf to ensure /dev/rfcomm0 connects to Channel 1 as below. It is a good idea to save a copy of the original file just in case

 sudo /etc/init.d/bluez-utils restart 

For completeness I will note here that I have found that you can set up multiple rfcomm channels in the file to cover multiple devices. I have also found a more useful command to provide the information to set up the RFCOMM file:

The above can be used instead of  sdptool browse <phone-bluetooth-address> when several bluetooth phones are in range. The XDA section below has examples of its output with several Bluetooth devices connected.

Since writing the above I had to get another Bluetooth Dongle, this time a class 2 v1.2 device from Belkin - a F8T009 based on a Broadcom chip. This was detected immediately and was usable with no addition setting other than pairing from the phone. I ordered the Dongle from Amazon for £9.99 and noted that in the existing customers comments somebody had noted that it worked well with Linux 2.6 kernel. I have also discovered another useful command lsusb which list all USB devices plugged in which is a valuable check that devices have been discovered.

I installed Gnome PPP and set up a number to dial, username, and password and changed to /dev/rfcomm0 and immediately got an internet connection on a simple dial-up.

The GPRS connection may need a change in other PPP files as written about before but at least the basics worked very quickly.

GSM Dial-up to an ISP via Bluetooth Phone Modem

Having set up Bluetooth I  swapped the device in the Network Manager (System -> Administration -> Networking) from /dev/ttyS4 (the Xircom Modem) to /dev/rfcomm0 for the Bluetooth phone modem. I could then make a standard dial-up connection through the mobile phone.

The modem seemed very slow and I discovered the modem initialisation string is in /etc/chatscripts/ppp0 which I edited using gedit invoked from the command line:

Better modem initialisation strings will appear latter and use of GNOME-PPP as a dialer follow so really this section is only to complete my 'audit trail'. 

Other Programs and files used in PPP connections

This is following a chronological path and I also want to keep an audit of how I progressed so I should note that my background searches for information have indicated that other files  which used are used by Networking include /etc/ppp/peers/ppp0 and in some cases /etc/ppp/pap-secrets which contains a list of users and passwords.  At this point I tried making connections using the low level commands pon and poff.  pppconfig helps sets you set up connections for pon and poff . The commands and option scripts set up and use the PPP Daemon (pppd). The connections made using pon and poff appeared to work but did not seem to allow Internet access - something I now suspect was to do with setting the default route. I have not used them again.

Other programs I found and used are at a higher level namely wvdial and GNOME PPP which is a GUI front end to wvdial - these are covered below when I come to GPRS connections of various sorts and are a much better way to proceed. wvdial is very useful for testing from the command line and allows one to 'see' more of what is happening if there are problems. Once one is up and running one GNOME-PPP should be all one needs.

Getting Information - manuals and help

I have mentioned several commands and configuration files. There is a very useful command man which provides  'manuals' for many commands and options files eg 

Monitoring Connections and Devices - System Logs

There is a command dmesg which shows all the main activities from startup and the the end is useful for seeing what happens when you plug in a [new]  device to see the channels which are allocated. 

dmesg gives a lot of output so it is worth learning about the grep command and piping the output through it.

dmesg | grep "Connect"

will just give you the lines including Connect - see below for how I use this to get a record of connection times etc.

You can monitor some of what is going on by System -> Administration -> System Monitor

Improvements in procedures for modem connection, disconnection and monitoring

The use of a System  Administration program for making connections lacks elegance and you need to keep the Networking window open so you can close the connection.  The Modem Monitor does tell you when you are connected and disconnected but little more. I felt this was an area where my set up fell far short of Windows. 

The first improvement I found was that you can define an extra set of connections in Network Manager (in the box at the top) which mean that it is possible to at least switch between Landline, Bluetooth, WiFi and Ethernet pre-defined configurations which went a long way to removing my concerns especially if the Network Monitor is opened in a separate workspace. This technique seems to work well and I can switch between connections at a click if the default connection box is correctly set in each.  You do still need to change the connection which is being monitored by right clicking on the Modem Monitor icon -> Properties if you change from a Wifi/Ethernet to Modem Connection. The WiFi (ath0) shows signal strength alongside the modem monitor icon.

This may be all that some users need - the only downside is that the Network Manager needs a superuser password for access and you should also be aware not to open multiple copies of Networking - it has fallen over and needed a reboot when several had been opened!

GPRS access through a Mobile Telephone

It requires two files which I needed to set up manually as the Bluetooth modem is not covered by the auto-detection facility in wvdial.

The files as in my set-up were:

This worked fine and connected up the mobile - the only initial problem was that I did not realise that the only way to stop the connection was by a Cntl C !

The script above copes with a problem I initially had of the connections were being terminated after a couple of minutes - inspection of the log using

tail -f /var/log/syslog

showed that I was being disconnected because there were no echo responses - much googling using the error string as a search term came up with a [well] known but less well warned about problem with many phones/mobile service providers not implementing a lcp-echo. This call for a lcp-echo is done by default every 30 seconds and was included as a way to discover a disconnected telephone connection and after a default of 4 such failures the PPP link was terminated. Setting the echo time and number of checks to 0 inhibits sending lcp-echo requests and responding to lack of replies. See above script.

All the various options can be found using the manual command for the PPP Daemon pppd

Vodafone Mobile Connect Card

• Vodafone 3G (UMTS) Howto - Recommended by the manufacturers
• Using a Vodafone GPRS Connection as a Modem
• Vodafone Mobile Connect 3G/GPRS in Ubuntu Linux
• Linux and the Vodafone Mobile Connect 3G/GPRS Data card
• Linux, Bluetooth and GPRS - A Practical Guide
• Using Option (Vodafone) 3G PCMCIA card in Ubuntu PowerPC - Ubuntu Forums
• Installing Vodafone 3G PCMCIA cards on Fedora Core GNU/Linux 

The first stages went well and the Vodafone Connect Card was detected when I plugged it in. 

All the pages above refer to use of a communication terminal program called minicomm which is not available in Ubuntu Dapper Drake - Add/Remove the Serial Port Terminal instead. You can override the preset selections to use ttyUSB0. After plugging in the card and monitoring with  dmesg I used the terminal to send a few AT commands and monitored/interrogated various settings OK. The card needs to be turned off with:

sudo cardctl eject

before taking it out if the machine is on otherwise there is a risk of damage (as with any card). This command is due for replacement and I will report when I find its replacement. I latter set up a script run from the desktop which ejects the card or warns to turn the machine off instead if you do not have access to the superuser password.

After studying the references above and other information, I started using wvdial with the following files when using the Vodafone Connect card, T610 and XDA:

They make use of  various options (not to be confused with the card which is made by Option Wireless Technology and enable me to test the T610 and XDA as well as the  Vodafone Connect Card with minimum editing. I am not sure I understand the value of all the possible optional sections and only used wvdial option internet in most cases. The pin option is required if your SIM has a pin set (my did not at the time) - however note that you can only enter the pin once - further times causes an error which these scripts do not handle. The easy way is to put the SIM in a mobile you understand and turn the pin off - fine for a Prepay SIM but a risk for a contract SIM. If you want to do it the hard way in a terminal I understand from Linux and the Vodafone Mobile Connect 3G/GPRS Datacard but have not tested the following. Send AT+CLCK="SC",0,"XXXX" (replacing xxxx with your own PIN) to turn off the PIN question. This way, your UMTS card will start searching for a network immediately. To turn the PIN question on again, type AT+CLCK="SC",1,"XXXX" and to check if it's on or off, type AT+CLCK="SC",2if it says "+CLCK: 0" the PIN is off, if it says "+CLCK: 1" the PIN is on.

When using the main scripts above with wvdial and the Vodafone Connect Card I found I got a connection but did not get an IP address or DNS servers issued so I paused to ruminate. I have found from searches that the problem is not mine alone (see Linux - HOWTO - Page 11 . The default IP address provided is not a problem but the DNS (Domain Name Server) addresses of 10.11.12.13. and 10.11.12.14 seemingly coming from Vodafone are no use and their is no domain name resolution. The answer is to use fixed DNS servers and rather than use the Vodafone UK ones I have been using ones from OpenNic

Pings show that the primary one is very fast although it is in Germany.

These can be set by using the Network Manager - System -> Administration -> Networking but they will not remain if you switch to an Ethernet or WiFi connection where new addresses are obtained so further work is needed. They are stored in /etc/resolv.conf which gets overwritten which gets overwritten.

 /etc/wvdial.conf also needs to be modified to have a line in the Default setting when using the Vodafone Connect Card of

I have updated the files above with the new settings. At least this has got me up and running with the Vodafone Connect Card so I do not need to move SIMs round. 

There is also a very useful command mentioned in PHARscape - Vodafone 3G (UMTS) Howto that changes the way the V3G card behaves that I plan to try and use so I can prefer connections with the slower but more reliable GPRS:

Ejection Script for PCMCIA Cards

Code to Set Pin and Connection type

I have  created a wvdial configuration file just to set the pin which can be run as sudo wvdial --config  /etc/wvdial-pin.conf and may need a Cntr C to exit

Vodafone Mobile Connect Broadband package with Huawei E220 USB Modem.

Vodafone now offer a package with a USB broadband stick preloaded with 1 Gbyte of UK data for £49 reduced to £39 for Xmas. Top ups are £15 for 1 Gbyte. The stick I received even has a micro SD reader built. It is branded as a Vodafone K3565 which is detected as a Huawei E220. It handles GPRS, 3g and the latest HSDPA services providing 3.6 Mbaud rates. The MSI Wind has no PCMCIA slot so this is perfect for round the UK and I think I have persuaded Vodafone to unlock it for roaming with different Vodafone SIM cards. The service includes reception of texts but not sending.

I had been reading Linux Format which had an article on Mobile broadband which pointed me towards a control application writen by the Vodafone Group - it is not officially supported but is an Open Source development which can be used for many USB 3G sticks and service providers. It is succintly named the vodafone-mobile-connect-card-driver-for-linux which everyone appreviates to vmc. The vmc installation details are at https://forge.betavine.net/projects/vodafonemobile and www.betavine.net/bvportal/web/linux_drivers and there are also a series of support forums. It took me a little while to work out exactly how to install the software and there were a few things that I wish I had realised in advance which I will cover:

VMC Dependencies: A number of extra packages have to be installed to satisfy dependencies before running the vmc installer. I did realise about these and installed all the following packages - I used System - Administration -> Synaptic Package Manager. Note that you may have to enable some extra repositories. The other way used in their install scripts is:

sudo apt-get install python-dbus python-twisted python-serial python-glade2 python-pysqlite2 wvdial
sudo apt-get install python-notify python-gnome2 python-gnome2-extras python-serial hal python-tz
sudo apt-get install python-setuptoolsbluez-gnome python-gnome2-desktop

Installing: The package is installed by downloading the Auto Install file , copying to your home directory, changing the permissions to make it executable and then sudo ./vmcfilename.net . I took a while to realise the obvious and change the permissions to make the file executable. I installed the latest version 2.0 beta3 - they seem to update every couple of months so you may well find a final version 2.0. I had satisfied all the dependencies so I do not know what happens if you have missed any. Part way through it asks for you to give a list of users who need to run it - I latter found that the list should contain root to avoid the following problem.

VMC Permissions - when I had installed the Vodaphone the VMC software and ran it the first time it complained about the permissions for the opt/vmc/etc/ppp folder which I set to 777 as the easy way by gksudo nautilus to change their properties as root. I actually set all the subdirectories at the same time which caused further complaints about pap-secrets and chap-secrets which I had to set back to 660 after which there program ran happily. They all belong to group vmc.

I found latter in a conference posting that during the install when it asks for users who have permission to run the device one should include root and then the permissions problem disappears. You can uninstall vmc to repeat the install by (both untested):

sudo /opt/vmc/usr/bin/uninstall-vodafone-mobile-connect

Multiple instances of the device: The only problem I have which is independent of software used is that modem is detected when you plug it in but it ends up with multiple installations of the USB devices the first time it is plugged in after the machine is booted up. If you unplug and plug in again it is perfect - you can do it many times without problems. This seems to be to do with the fact that it changes mode - initially it looks like a CD so the drivers can be automatically installed under Windows then switches to a modem mode.

You can check what you have after the first time you plug in by;

lsusb

which should show the Modem as well as other USB devices

pcurtis@triton-ubuntu:~$ lsusb
Bus 004 Device 006: ID 12d1:1003 Huawei Technologies Co., Ltd. E220 HSDPA Modem
Bus 004 Device 002: ID 0c0b:2bcf Dura Micro, Inc. (Acomdata)
Bus 004 Device 001: ID 0000:0000
Bus 003 Device 001: ID 0000:0000
Bus 002 Device 001: ID 0000:0000
Bus 001 Device 001: ID 0000:0000
and then you can check the actual devices which have been installed by

ls -al /dev/ttyU*

you should only have two or three devices /dev/ttyUSB0 /dev/ttyUSB1 and possibly /dev/ttyUSB2 as it also has a built in card reader for a micro SDC card. If you have more it will not work. Unplug and plug in again, wait 30 seconds and check again. At one time I had 12 devices listed! You may want to write an alias for ls -al /dev/ttyU* or put it in a launcher on the desktop as one tends to use it quite often

pcurtis@triton-ubuntu:~$ ls -al /dev/ttyU*
crw-rw---- 1 root dialout 188, 0 2008-12-21 19:00 /dev/ttyUSB0
crw-rw---- 1 root dialout 188, 1 2008-12-21 19:00 /dev/ttyUSB1
When you set up Gnome-ppp the only new information needed for vodafone PAYG Broadband is:

Username: web
Password: web
APN: pp.internet

The APN is unusual as vodafone usually use pp.vodafone.co.uk for PAYG and internet for contract but I got it from the default under Windows and only that seemed to work when I tested in a normal phone as well as in the dongle. I just added an init string to set up the default APN (CID 1) and it also worked immediately. I use the following init strings:

Init2 = ATQ0 V1 E1 S0=0 &C1 &D2 +FCLASS=0
Init3 = AT+CGDCONT=1,"IP","pp.internet"

I am not sure that the init2 is needed - it is the standard set up for a fax modem as used through a mobile phone .

The 'number to dial' is set to be *99# or *99***1# and I use stupid mode to get a quick connection.

I have looked at the codes sent by vmc and tried the codes there to set GPRS prefered as an init code. It does not seem very effective but for completeness the codes I have extracted from the vmc source are:

For GPRSONLY set
Init4 =AT^SYSCFG=13,1,3FFFFFFF,2,4
For 3GONLY set
Init4 =AT^SYSCFG=14,2,3FFFFFFF,2,4
For GPRSPREF set
Init4 =AT^SYSCFG=2,1,3FFFFFFF,2,4
For 3GPREF set
Init4 =AT^SYSCFG=2,2,3FFFFFFF,2,4

I tried the small program provided by vodafone called usb_modeswitch to switch between CD boot mode and Modem mode but it did not work even after various settings in the confg file - recognise the device but does nothing. I am still into having to check if the devices are correct with ls -al /dev/ttyU* if it does not fire up correctly and pluging in again if still a problem.

I have not found a way to check ones credit under linux as the linux vmc software does not support - if the software is improved to do so I will install it. Currently credit checking and Topup via a vouchure is available under the Windows VMC or by telephone and credit card however 1 Gbyte should last a long time!

New Zealand Telecom T-Stick (Sierra 597 Wireless Compass USB Broadband Modem)

I have been accessing a NZ Telecom T-Stick under Ubuntu Hardy Heron 8.04 which I got working in an hour or so using the software built in and that I had already installed for other Dial-up connections. I found sufficient information on the web at Nathan Cooks Blog Article on the Telecom T-Stick under OpenSuse to find the username and password (username: mobile@jamamobile password: telecom) and the pseudo dialling code for a CDMA connection (#777) which gave me the confidence to proceed.

I first plugged the t-Stick into a Windows machine so I was sure it worked and was enabled - I am not sure it is essential but recommend it as a sensible proceedure - no usernames or passwords were required and the Windows software looked very easy to use. The drivers self install as the T-Stick appears as a autostarting CD drive when first plugged in - a trick which is very good under Windows but can cause problems under Linux as I had found with the Vodafone equivalent.

Returning to Ubuntu 8.04.1 I first checked if there were drivers and their version in the system for the Sierra modem by:

modinfo sierra

which told be I had driver version v.1.2.5b available and I did not have to do any further work.

I pluged in the T-Stick and in a terminal typed

dmesg

which had the following at the end of the output:

...........
[ 1018.489995] usb 1-1: USB disconnect, address 3
[ 1018.491730] sierra3 ttyUSB0: Sierra USB modem (3 port) converter now disconnected from ttyUSB0
[ 1018.492344] sierra3 ttyUSB1: Sierra USB modem (3 port) converter now disconnected from ttyUSB1
[ 1018.493098] sierra3 ttyUSB2: Sierra USB modem (3 port) converter now disconnected from ttyUSB2
[ 1018.493180] sierra 1-1:1.0: device disconnected
[ 1022.738202] wlan0: no IPv6 routers present
[ 2886.821961] usb 1-1: new full speed USB device using uhci_hcd and address 4
[ 2886.896942] usb 1-1: configuration #1 chosen from 1 choice
[ 2886.920784] usb-storage: device ignored
[ 2886.925656] sierra: probe of 1-1:1.0 failed with error -5
[ 2887.123273] usb 1-1: USB disconnect, address 4
[ 2888.405129] usb 1-1: new full speed USB device using uhci_hcd and address 5
[ 2888.566272] usb 1-1: configuration #1 chosen from 1 choice
[ 2888.569185] sierra 1-1:1.0: Sierra USB modem (3 port) converter detected
[ 2888.577345] usb 1-1: Sierra USB modem (3 port) converter now attached to ttyUSB0
[ 2888.577514] usb 1-1: Sierra USB modem (3 port) converter now attached to ttyUSB1
[ 2888.577649] usb 1-1: Sierra USB modem (3 port) converter now attached to ttyUSB2
[ 2888.609553] scsi6 : SCSI emulation for USB Mass Storage devices
[ 2888.668915] usb-storage: device found at 5
[ 2888.668925] usb-storage: waiting for device to settle before scanning
[ 2890.561028] usb-storage: device scan complete
[ 2890.563971] scsi 6:0:0:0: Direct-Access Sierra Wireless Storage 2.31 PQ: 0 ANSI: 2
[ 2890.571046] sd 6:0:0:0: [sdc] Attached SCSI removable disk
[ 2890.571150] sd 6:0:0:0: Attached scsi generic sg2 type 0
This showed that the T-Stick had been detected and installed with three USB modem connectors - the one used for normal connections is ttyUSB0. There is also detection of the built in usb storage for booting and the MicroSD card slot - you will see some extra output when a MicroSD card is plugged in and it will appear on the desktop just like any other USB memory dongle

You can get further information to confirm that the modem has been installed correctly by using additional terminal commands to list the USB devices by lsusb and ls -al /dev/ttyU* as shown below. They were explained above for the Vodafone USB Broadband dongle.

pcurtis@wind-ubuntu:~$ lsusb
Bus 005 Device 003: ID 0bda:0158 Realtek Semiconductor Corp.
Bus 005 Device 001: ID 0000:0000
Bus 004 Device 001: ID 0000:0000
Bus 003 Device 001: ID 0000:0000
Bus 002 Device 008: ID 05c7:0113 Qtronix Corp.
Bus 002 Device 001: ID 0000:0000
Bus 001 Device 009: ID 1199:0023 Sierra Wireless, Inc.
Bus 001 Device 001: ID 0000:0000

pcurtis@wind-ubuntu:~$ ls -al /dev/ttyU*
crw-rw---- 1 root dialout 188, 0 2009-02-27 17:49 /dev/ttyUSB0
crw-rw---- 1 root dialout 188, 1 2009-02-27 17:49 /dev/ttyUSB1
crw-rw---- 1 root dialout 188, 2 2009-02-27 17:49 /dev/ttyUSB2
pcurtis@wind-ubuntu:~$

I then tried to connect using GNOME-PPP with the username, password and a phone number of #777 having set the device to /dev/ttyUSB0 . I found that I needed to remove all by existing connection strings just leaving the inbuilt ATZ to reset the modem. I also found I needed to use root to set the username and password in the file /etc/ppp/pap-secrets by adding a the line:

mobile@jamamobile * telecom

to the end of the /etc/ppp/pap-secrets file - I think this was only because I do not have the permissions correctly set up for the access to the file from GNOME-PPP so it may not be needed in your case - I did not investigate further as I have a philosophy of "If it's not broke don't fix it". The file can be edited by using:

sudo gedit /etc/ppp/pap-secrets

At this point I could make a connection using GNOME-PPP and had fast efficient access. The connection time was also very fast, especially with the GNOME-PPP 'stupid mode' option set.

It does not seem to have the problems of my Vodafone equivalent in the UK where it has to be plugged in when the machine is booted from cold - in this case there seems to be no problem in pluging in and removing with the machine on and after the machine has been suspended.

Before comparing the two major competing systems in New Zealand it is worth discussing the very different systems used:

New Zealand Telecom does not use the usual GPRS data coding for its data communication network and phones do not include a SIM card and the 'personalisation' (Phone number etc.,) information is hard wired into the phones/data cards. They use CDMA (Code Division Multiple Access) which is a wideband spread spectrum technique allowing many active users to share the same channel. Each user is assigned a unique digital code, which differentiates the individual conversations on the same channel. In contrast GPRS uses a time division multiple access (TDMA). In both cases further techniques are used to provide the very high data rates used in Mobile Broadband, in conventional systems they are 3G, UMTS etc., culminating in High Speed Downlink Packet Access (HSDPA), sometimes known as 3.5G. Currently, HSDPA can in some implementations provide downlink transfer speeds of up to 21 Mbit/s and Vodafone in the UK provides theoretical speeds of up to 7 Mbytes using the USB dongle described in an earlier section. In New Zealand I have been accessing Vodafone via a mobile phone and even with a rare UMTS connection it has been painfully slow in many locations and on one occasion it took nearly two hours to upload a litle over 2 mbytes of data from Pauline's OU teaching

Telecom uses Evolution Data Optimized ( EVDO) network access which uses multiplexing techniques including Code division multiple access (CDMA) as well as Time division multiple access (TDMA) to maximize both individual user's throughput and the overall system throughput. There are a number of different versions/speeds available and the type/speed of network detected is shown by the colour of the blinking LED on the T-Stick –the slower speed (Orange flashing) supports connections with data rates up to 153 Kbps. The higher speed (Blue flashing) standard Rev. A supports connections with data rates up to 3.1 Mbps (downlink from the network) and 1.8 Mbps (uplink to the network). The lights are continuous when a connection has been made.

The Mobile Broadband speed received in practice will vary based on the number of users on the network, coverage and environmental conditions, applications used, network traffic, and file type. Telecom claim that on average their Mobile Broadband cell sites enable download speeds of 800Kbps and upload speeds of 300Kbps when using a Mobile Broadband Rev A data device such as the T-Stick.

I have had reliable connections everywhere I have tried to use the T-Stick and they have provided good data rates close to the limits claimed by Telecom and in some places it has proved faster and more reliable than using broadband via Wifi. For example I checked the speed on a couple of Linux updates of 2 and 7 mbytes and the download rates never fell below 100 Kbytes/sec with the maximum 230 kbytes/sec (average about 1.5 Mbps) early morning on Waiheke Island near Auckland with a EVDO Rev A (blue light) connection. This exceeds what I get in practice in the UK on a hard wired (copper) broadband connection at most times of day and much better than I have seen on Vodafone in New Zealand on the few occasions I have had a UMTS connection via my phone. In the UK I have seen similar speeds on my Vodafone Mobile Broadband USB stick which has a 3.6 Mbps capability but I did not make extensive quantitative measurements in the few days I had it before leaving for NZ.

During connections with the T-Stick at the lower data rate (orange flashing light on modem) the downlink speed is still a very acceptable 17kbytes/sec in the Auckland area which is faster than I have had with the 2G connections with Vodafone which are all that is currently available over most of the Vodafone coverage areas. In very weak signal areas the rates seemed to fall dramatically with a Telecom connection through the T-Stick and on Great Barrier Island whilst sailing I found it beneficial to bring the laptop up on deck.

Conexant Soft Modems

The laptop also has a soft modem and I initially put off installing it as it seemed that relatively easy driver support was likely in the near future. I also had a Xircom PCMCIA modem and Ethernet card which I plugged in - it was recognised immediately and I had a telephone connect working a few minutes later so their was no urgency. 

The easy way to get a driver is through  linmodems.org  but  they either need a licence fee or are limited in speed. I first tied the procedures at Howto drivers Conexant softmodem - Ubuntu Forums to get a free driver loaded and nearly lost all as I ended  up with an un-bootable kernel and hung machine. Fortunately I had an older kernel left to boot into and I managed to recover eventually so take great care and back up anything valuable before departing from the approved routes. 

I then followed the correct procedures by loading a test program for the modem. A good starting place is DialupModemHowto - Community Ubuntu Documentation which sends one to linmodems.org  where one can get a free program to identify and help you install soft modems. This showed mine to be an Conexant modem packaged by ATI  as I expected. It directed me to download a package (specific to the kernel) to install the modem and the installation was fairly trivial - three lines in a terminal and a couple of questions to answer.

The drivers are restricted to 14 Kbaud unless you pay a $19.99 licence fee which covers one year of support and upgrades and they say will not be kernel or distribution dependent. At present I do not need a faster modem as I have the Xircom anyway. I really tested it all out for the sake of others and it all went together very easily. I believe the a further download and install will be required if the kernel is ever upgraded. If you have a machine with no connectivity you are often in a catch 22 situation for upgrades to set up connectivity. In this case it seems to be possible to download the two files on another machine and transfer them to get one up and running. With a dual boot machine you can, of course, get them under Windows and reboot. The modem software aliases /dev/modem to  /dev/ttySHSF0 and GNOME-PPP found the modem and worked immediately.

If you get a kernel update ie from 2.6.15.27 to 2.6.15.28 you will need to download a new .deb package from linmodems and install it in the new kernel.

One still needs to update the sound drivers each time under Ubuntu 8.04 Hardy Heron

O2 XDA Exec (HTC Universal)

Returning to the chronological order you will recall that I wanted in particular to be able to use the XDA Exec as a modem via Bluetooth in Linux as I had failed in Windows. It has also proved challenging with Linux and it took me 5 or 6 weeks before I found a way to get it working - if you follow the procedures here it should take a similar number of minutes! 

First pair the XDA and the laptop from the XDA Start -> Settings -> Bluetooth -> Tick make discoverable -> Devices -> New Partnership

The first problem was that I had difficulties in obtaining the channel used for Bluetooth as the command ‘sdptool browse <phone-bluetooth-address>’ gave no response. I did a man sdptool and sdptool -h and discovered that their were many more options and used an alternative option to search for all DUN connections which worked. You can use the following commands with the XDA:

hcitool scan
sdptool search DUN

In fact the second is actually all you need - see below for example with both my  T610 Phone and O2 XDA Exec on:

I found, by trial and error that the T610 and XDA also responded to HF  - Hands Free gateway (6 - 2) and HS - Voice Gateway (1 - 3) and OPUSH (10 - 3) The full list includes according to the help file includes: DID SP DUN LAN FAX OPUSH FTP HS HF SAP NAP GN PANU HID CIP CTP A2SRC A2SNK AVRCT AVRTG SR1 SYNCML ACTIVESYNC HOTSYNC PALMOS NOKID PCSUITE

Now I  have access to several Bluetooth modems I thought I would experiment to see if I could set up a rfcomm1 as well as rfcomm0 as a Bluetooth device and found that one can add extra devices to the configuration file as below:

# RFCOMM configuration file /etc/bluetooth/rfcomm.conf. 
    #  
    rfcomm0 { 
    bind yes; 
    # Bluetooth address of T610 
    device 00:0E:07:2C:E8:82; 
    # 
    # RFCOMM channel for the T610 
    channel    1; 
    # Description of the connection 
    comment "T610 Dial-up Networking via Bluetooth"; 
    } 
    # 
    rfcomm1 { 
    bind yes; 
    # Bluetooth address of XDA 
    device 00:12:37:C6:81:31; 
    # 
    # RFCOMM channel for the XDA 
    channel    4; 
    # Description of the connection 
    comment "XDA Dial-up Networking via  Bluetooth"; 
    }  

Remember when testing that whenever you change a configuration file such as the above which you are going to use  you must reboot the machine or restart the relevant sub-system, in this case by:

 sudo /etc/init.d/bluez-utils restart 

The extra device appears in /dev so it can be accessed via the Network Manager, wvdial and the GNOME-PPP Dialer.

Firstly one should note that the XDA Exec is supposed to need the Wmodem program to be running and to be set to Bluetooth before it can be used as a modem. It certainly needs it to act as a USB modem with Windows. I found this was not required for Bluetooth - the only difference I could find was that one got a display of  'lights'. We now have a Palm Treo 750v running Windows mobile and have added a third device and it does not need the Wmodem program running wither.

I however still had one major problem - I could make a standard GSM dial up connection  via the XDA but not a GPRS connection using the initialisation string followed by a dialing string of *99***1#  - a mechanism that works perfectly with the T610. I  tried using both wvdial and GNOME-PPP and the same occurred with both - good GPRS connections via the T610 but only a GSM dial-up connection possible via the XDA.  I recall that I never made a satisfactory GPRS connection under Windows via Bluetooth and the XDA either and gave up.

I was at a loss until after many weeks I read in a script for a Mac accessing a Windows Mobile machine that Windows for Pocket PCs does not allow GPRS configuration changes through the modem interface - I therefore tried taking out the setting-up Init string and used a dialing code of  *99# which activates a connection using default parameters already in the XDA. It all then seemed to work fine. One obviously needs a working connection set up on XDA before using it as a modem and I note that the Options screen in WModem has a tick box to select a preset GPRS connection which was also set up in my case.

I initially tested it using a Prepay SIM then changed to a Contract SIM. I made a new connection for the Contract SIM in Start -> Settings -> Connections -> Connections -> Add a new modem connection in the usual way and then used it briefly so the new default route is set up. I also went into Start -> Programs -> Wireless Modem -> Tools -> Options and gave the new APN an left the box ticked - I do not know if that was necessary. I made no changes in  GNOME-PPP which then connected through the changed SIM.

Connection Times and Data Volumes

Having got everything working I decided it is useful to know how much ones sessions have cost. All the data is available in the system logs if you are prepared to plough through them.  The current on is in /var/log/syslog and older versions are /var/log/syslog.0 - even older versions are in compressed format. The first of the following searches for Times and Data flows in the current and previous log which usually cover at least a couple of days. The second does the same for the previous 6 zipped logs. Both must be copied as a single line into the terminal or set up an alias. Between them you should have a good idea of the bills you have run up over a week or more!

I leave the reader the exercise of creating a script and launcher to join them up and perhaps send it all to a file rather than scan it on the screen! An alternative is to put both commands onto a single line separated by a semicolon - again this must be copied and pasted as a single line into the terminal to work.

Wifi Connections and drivers

Wifi cards in laptops are much better supported than modems but even so there may be a need to install drivers. The problem is that the drivers for a WiFi card are complex and few manufacturers support Linux. There are however a limited number of chips which are used by a large number of manufacturers and some of the most popular now have built in support in the kernel. Even then proprietary considerations mean that in some cases such as the Broadcom 43xx chips used by many manufacturers including Belkin and Dell need the proprietary firmware installed. Those that have no native Linux support can often be installed by a technique of 'wrapping' part of the Windows drivers in code to integrate them into Linux - the ndiswrapper mechanism. The basic windows .inf and .sys files can be obtained in many ways including from your Windows system or the CDs provided.

The code below shows that it need not be complex and covers the Belkin Card using the Broadcom 4306 chip. It got the WiFi up but had problems in integration into the system so one could not easily start a useful connection through it. This is based on a thread  Broadcom (4306) - Ubuntu Forums  and the thread covers the two possible methods. All these mechanisms which use Windows drivers or place drivers into the kernel have a downside in that they need to be repeated if you upgrade to a new kernel - you do have the choice at boot-up which kernel you chose so you can use the old one until you are sorted. In some simple cases you may even be able to experiment using the LiveCD. The ndiswrapper actions are shown to indicate that it is not rocket science. First the tools are installed (that can be from Add/Remove Advanced). Then you find the windows drivers and put them in a directory ready for the next time, make the modules, install them and check and add them into the drivers loaded during booting up. In this case there may be an incomplete driver already present which needs to be removed from the sequence loaded at boot-up which is covered in the thread.

The alternative using bcm43xx-fwcutter to load the extra firmware follows noting that the kernel version may need to be changed

Configuration can be done using the System -> Administration -> Network or possibly wifi-radar on Dapper Drake but the Network Manager Applet takes care of it all in Hardy Heron

Email on the Move

Mail System Considerations - IMAP versus POP

This is a new section covering the advantages of using an IMAP mail system whilst one is mobile over the more conventional POP mail.

POP Mail: Most people understand how POP (Post Office Protocol) mail works, the incoming mail is delivered and held on a server at your ISP and, in the simplest case, you download it to your machine on demand (usually called a Send/Receive activity in your email package) and it is deleted from the server. This was fine when a user had only one machine in one place. Nowadays many people access email from home, their office, from a mobile and perhaps a PDA and Phone. This leaves ones incoming mail fragmented in many places. Most email packages therefore allow you to collect your email whilst leaving a copy on the server and many allow one to just download the headers or a restricted amount of data in each email. This is a much better way of working as the download and deletion from the server is done on one machine. It does not help with outgoing mail which has to be copied to another account or some other method to allow an audit trail. If one uses different email packages which can not interchange saved emails it all becomes a bit of a nightmare.

IMAP: There is an alternative to POP mail called IMAP which stands for Internet Message Access Protocol. In this case the email is stored on the server and not downloaded automatically and one can create addition folders on the server so one can have a complete filing system on the server and available whilst one is online, on corporate systems there may be shared folders accessible by many users. Email stored on an IMAP server can be accessed and manipulated from a desktop computer at home, a workstation at the office, and a notebook computer while traveling as well as ones phone/PDA, without the need to transfer messages or files back and forth between these computers. In the simplest case data is only transferred as required - when you select your inbox or a remote folder the headers are transfer - when you select an email the message body is transferred and only when you open an attachment is the attachment transferred You can of course copy anything to a local folder on your machine to work on it and when you finally send a reply (whilst online) it is saved on the server and accessible from any other machine. IMAP is at its best when you are on a corporate LAN or a Broadband always on internet connection - a GPRS/3G connection charged on data transfers is acceptable. There is more comprehensive analysis at The IMAP Connection -- Comparing Two Approaches to Remote Mailbox Access: IMAP vs. POP.

Implementation: Many ISPs including Freezone offer the choice of a mix of POP or IMAP mailboxes and in some cases such as the Horde IMP system used by Freezone they are identical other than the access protocols and port number used. Most IMAP mailboxes are also accessible via a webmail interface in an internet cafe or on any friends machine. Even if you do not want to change completely without extensive trials it is worth setting up a single IMAP mailbox when you are traveling so you can transfer mail from your 'mobile' machines via an IMAP mailbox to you home machines filing system via the IMAP mailbox without worries about the incompatible local mailbox and folder formats. You can also tidy up whilst traveling whenever you get a fast WiFi data link in an internet cafe. I do not feel comfortable solely depending on a remote server to store my emails long term but for a few months between archives it seems a very sensible way to proceed. With a GPRS/3G connection where one pays for data transfers rather than time online, it seems a very economical way to operate and most email packages allow one to download selected local copies for offline working as well as remote copies. So far I have set up one IMAP box for archive and transfer purposes whilst continuing to use POP mailboxes set up to only download copies to my machines for my main accounts whilst traveling but I suspect that will change shortly when I have developed data flow efficient procedures.

Some cautions: Both POP and IMAP protocols are define by RFCs but the implementation by email packages may not be rigorous when it comes to some of the more advanced features. Note that the folders you create in an IMAP environment may not be where you would expect - the RFCs imply they should be under the IMAP Inbox folder and that only shared folders are at the same level. The display may be different between Webmail and different MAPI email systems. It is possible that POP implementations of features such as leaving emails on the server for a fixed time differ between email packages and may be implemented locally or on the server so if you have different settings or ways those features are implemented between packages you may have a problem. The same may go for subscriptions to folders (whether they are visible on the client automatically) and I have noticed that if I unsubscribe for a high level folder with a subfolder still visible on Thunderbird then Outlook produces some inconsistent error messages. I suggest keeping it simple initially if you are using different email clients and machines and experiment at each step if you decide on a fancy way to work. A good way to start investigation is to see if the same mailbox can be accessed by POP, IMAP and Webmail without problems.

There is a useful free Windows program called IMAPSize which does far more than its name implies and has associated utilities which can be very useful in transferring email between Outlook Evolution and Thunderbird - I have found that it will run well enough to be useful under Wine 1.0 and Ubuntu Hardy Heron 8.04

A Comparison of Outlook 2003, Evolution and Thunderbird on the Move

Synchronising Windows Mobile machines with a Dual Boot Desktop and Laptop.

I have been looking into ways of working with our Windows Mobile PDAs and our dual boot machines with Linux and Windows XP. Vista is similar but the Windows programs have different names. The standard way of synchronising a Windows Mobile based PDA is using a program called Active Sync which links between Microsoft Outlook 2000, 2003 or 2007 with a Bluetooth of USB cable connection. There are various limitations including that PDA can only Synchronise with two PCs although the PCs can link with many PDAs and can not separate address books so that entries for a home address book migrate to a work machine and vice-versa.

However Microsoft Outlook is arguably the best overall integrated email, contact, calendar and task manager application sofor a long time we kept using it for home machines and for archiving whilst using Thunderbird for email whilst mobile. We have kept emails in step on all our machines by the simple expedient of leaving them on the server so each machine can download them. We periodically clear older emails from the server at a time we know that all the machines have downloaded them using a utility such as IMAP size which also runs under Wine. It is also possible to set the machines up to leave emails for a limited time such as 120 days, ie the longest period we ever expect to be away.

Contacts are kept in step between our PDAs, a Palm 750 and an O2 Executive and the main machines running Outlook 2003 by periodic synchronisation with ActivSync - the PDAS serve as the transfer mechanism between the laptop and desktops. That does not help with Thunderbird as the import process loses some important fields such as contacts.

We now want to reduce our dependence on any Microsoft software so we are looking for solutions where we can use Thunderbird on all our dual booted machines under Windows as well as Linux yet still be able to use the PDAs. The first step was to share the Thunderbird profile data between Windows and Ubuntu Linux so emails etc could be collected and address books updated whichever was booted up. This has been covered above and has been implemented on two dual boot machines so far without any immediate problems although we have not tried using any Extensions so far. This now leaves the Synchronisation of Windows Mobile devices with Thunderbird under Linux (or Windows) whic has proved a considerable problem to many people. Several pieces of software exist to replace ActiveSync but seem to give problems in practice judging by reading the many entries on bulletin boards etc. In looking into all of this I discovered a syncronisation package between Thunderbird and a Windows Mobile device call FinchSync which I have been experimenting with and which others seem to make work reliably but I gave up on after a long frustrating period as it could not handle address books over about 400 without contaminating them.

BirdieSync to Synchronise Windows Thunderbird and PDAs

I had been finally forced to give up on Finchsync as it just does not work reliably and has no support left. In order to sync my XDA and Treo PDAs I have resorted to using a Windows based program which actually costs money (19Euro), it however does seem to work and I suppose it is too much to be able to expect to avoid use of Windows altogether when one is using a Windows based PDA as Bill really hates to make life easy. Birdiesync extends Microsoft ActiveSync so that it will also synchronise the Windows Mobile to Thunderbird and Sunbird with the usual limitations that each mobile can only synchronise to 2 other other machines. There is one additional feature in that Thunderbird and Sunbird have to be running so that they are automatically started up when you plug in the mobile on the USB link. Another feature with Windows Mobile 5 is that the synchronisation is carried out every 5 minutes regardless of what you specify.

So what is the procedure for synchronising - I have dual boot system but normally I am using Ubuntu 8.04 Hardy Heron and Thunderbird 2 but when I want to synchronise the PDA I reboot into Windows and plug in the PDA by USB. Thunderbird and Sunbird are fired up and synchronisation takes place - it takes a while as Birdiesync also seems to check and remove any duplicate entries which is good if you make mistakes and end up with double entries. I can then reboot back to Ubuntu and because I have a shared profile my Thunderbird address books and calendars are now all up to date and synchronised. For an occassional activity this is not too bad.

One very good thing about Birdie sync is that is also deals properly with some of the fields that do not exist in Thunderbird at present although there must be some fields reserved for future expansion. In my case I made extensive use of categories and they are implemented in Windows Mobile. These get synchronised although you can not see them which is not the case when you import from Outlook when you lose them.

SyncKolab for Synchronising Thunderbird Address Books, Calendars and Tasks via an IMAP Server

Kolab is a Open Source Groupware Solution for Emails, Appointments, Contacts and more. It supports mixed client environments because of an open storage format. Kolab was created for the needs of the German Federal Office for Information Security (BSI) by contract in 2002-2004 and is used there in a heterogenous environment. Kolab won the Linux New Media Award: Category 'Best Groupware Server' in November 2005. In other words it is a professional set-up. We are interested in only a very small part, namely SyncKolab which is the client which runs as an extension in Thunderbird. This can also be run using any standard IMAP mailbox to synchronise Thunderbird Addresses, Calender Items and Tasks. Each item is converted into a standard Vcard or iCal format and stored on the IMAP server as though they were an email. When you synchronise only the headers need to be downloaded for checking for changes whilst the local copies are checked fully. Any changes are then 'mailed' to the server or the local copy updated. There is no limit to the number of clients that can be synchronised.

You need to have the Lightning extension version .9 installed which may be a problem if you have loaded the modified version .8 for Ubuntu - I had to uninstall it v .8 and also use a new profile. The SyncKolab extension is very small and very easy to set up once you have made some folders on the IMAP mailbox. The configuration tool (Tools -> SyncKolab Options) takes you through a very obvious wizard and ends up in is a single screen with every possible parameter identified on your system and available to select via drop down boxes (IMAP mailbox, folders on the selected mailbox, address book and calender). It can take a few minutes with an address book of 900 items like I have to download and check all the headers but that is a small price to have everything synchronised. I have customised the toolbar to have a Synchronisation Icon.

If you have to completely uninstall it to, for example, use one of the nightly builds the proceedure is:

Odd characters: I initially found that I had 16 entries in an address book of 870 which always showed up as changed. I eventually tracked it down to certain characters and strings in the address book which had orinally been converted from Outlook 2003 via a PDA! Tabs were the first problem and did not seem to be consistently handled in the conversion to and from vCards. The other problem was that some of the address stings had been concatonated by the previous conversions onto a single entry and some combinations of 'terminators' in particular | and ; gave problems. Once I found this out it was easy to check the notes for tabs and the address field for a concatenated entry on the problem entries and fix them. I am not sure if this is fundamental or a flaw in the particular implementation of vCard import/export - I suspect the latter. I believe all the problem had come from using cut and paste to fill in address cards in Outlook 2003 so now they are fixed I seen no future problems although I will be carefull pasting into the notes field.

Data usage during a synchronisation: I was worried that there would be a significant cost in data but it seems to be small enough to be OK for mobile use. My address book has 850 entries and there are a small number of calendar entries - the total data flow th first time after connection when the complete set of headers has to be downloaded is about 50 kbytes and subsequent synchronisations reduce this to 10 kbytes, to this has to be added any cards which need to be updated but unless you have a lot of notes that will be under 1 kbytes for each. You might not want to do it every day whilst abroad but it is fine on a mobile in the UK.

The MoreFunctionsForAddressBook Extension for Thunderbird

I found a really useful extension for Thunderbird which enables one to see and use several of the hidden fields which are not yet implemented and I have installed it on all my machines. It is called MoreFunctionsForAddressBook and seems almost too good to be true. It adds in many of the functions which are missing in the Thunderbird address book as well as allowing display and use of some hidden fields such as Categories and Birthdays. You can also select and copy addresses and notes from the 'preview' and do searches which include extra fields such as categories and notes. It also extends the actions to handle exporting, sending and importing Vcards. I have done an extensive search looking for the catch but have not found any problems on the internet so far, if you search for problems you just get examples of the problems it solves! I still think that Outlook 2003 is the best overall email, contact management and calendaring package against which the opposition must be judged but Thunderbird with the Lightening and MoreFunctionsForAddressBook Extensions is now coming very very close before you take into account the costs, security and general Windows issues.

Unison - Data Synchronisation between two machines or backup external hard drives

Backing up is very important even when mobile and often the easiest way has been to make copies onto the small USB external hard drives . But what does one do when one has multiple copies of important folders and different files are being added or edited two machine. This is a particular problem when one uses a laptop or netbook away from home for a few months and returns especially when only subsets of files may have been taken with one or the two machines were not in sync when one started.

Linux has a very powerful tool available to synchronise folders, and all their subfolders, between drives on the same machine and across a local network using a secure transport called SSH (Safe 'S Hell). At its simplest you can use a Graphical User Interface (GUI) to syncchronise two folders which can be on any of your local drives, a USB hard drive or on a networked machine which also has Unison and SSH installed. Versions are even available for Windows machines but one should make sure that the Unison versions numbers are compatible. You just enter or browse for the two folders if local and it will give you a list of differences and recommended actions which you can review and it is a single keystroke to change any you do not agree with. It uses a very efficient mechanism to transfer/update files which minimises the data flows (rsync). The initial Syncronisation can be slow but after it has made its lists it is very quick even over a slow network between machines because it is running on both machines and transfering minimum data. For more complex synchronisation with multiple folders and perhaps exclusions you set up a very simple configuration file for each Synchronisation Profile and then select and run it from the GUI as often as you like. It is easier to do than describe - the file to synchronise my four important folders My Documents, My Web Site, Web Sites, and My Pictures is only 10 lines long and contains under 200 characters yet synchronises 25,000 files! The review list is intellegent and if you have made a folder full of sub folders of pictures whilst you are away it only shows the top folder level which has to transfered which is a good job as we often come back with several thousand new pictures!

Both Unison and SSH are available in Ubuntu but need to be installed using System -> Administration -> Synaptic Package Manager and search for and load the following two packages:

unison-gtk ssh

The first is the GUI version of Unison which also installs the terminal version if you need it. The second is a meta package which install the SSH server and Client, blacklists and various library routines. Unison is then accessible from the Applications Menu and can be tried out immediately however there is one caution - it is initially configured so that the date of any transfered file is the current date ie the file creation date is not preserved although that can easily be set in the configuration files - the same goes for preserving the user and group.

Permissions: Unison checks for matches in the permissions so one has to make sure that USB drives are mounted with suitable and matching permissions - by default external drives are mounted with 0700 whilst automounted drives are usually set to 777 or 775 in fstab. The permissions for mounting via Places or automount can be set using gconf-editor -> system -> storage -> default_options and set umask=000 for the drive type which will probably be vfat. Double click down till you get to umask=077 and change it to 000 or 002 to match drives mounted by fstab. Even so one still seems to get some miss-matchs which have to be Resolved. I have set up a number of profiles for back-up and synchronisation with my WD Passport portable 120 Gbyte USB drive to keep Pauline's Laptop and netbook synchronised as well as a Profile to Synchronise my Netbook and Desktop over the local network.

Enabling Hostname Resolution under Ubuntu: Ubuntu does not seem to be set up to carry out hostname resolution by default and use of fixed IP addresses is dangerous when they are dynamically allocated by a router. I eventually found out use the machine's name on the network for the SSH link by enabling WINS resolution by:

Install winbind using System -> Administration -> Synaptic Package Manager and search for winbind - it is often installed with samba so you may find you already have it.

then in a terminal:

sudo gedit /etc/nsswitch.conf

and change this line:

hosts: files dns

to:

hosts: files wins dns

Note - in some cases there may be more on the line - if that is the case put it immediately after files.

Now ssh <hostname> will log you into <hostname> and you can use hostnames in Unison

Some sample Profiles for Unison:

The profiles live in /home/username/.unison

#Example Profile to synchronise from desktop triton-ubuntu to netbook vortex-ubuntu
# with username pcurtis on vortex-ubuntu
#
# Roots for the synchronisation
root = /media/DATA
root = ssh://pcurtis@vortex-ubuntu//media/DATA
#
# Paths to synchronise
path = My Backups
path = My Web Site
path = My Documents
path = Web Sites
path = My Pictures/2009
#
# Options - only times is essential
#
# When times is set to true, file modification times (but not directory modtimes) are propagated.
times = true
#
# When fastcheck is set to true, Unison will use the modification time and length of a
# file as a ‘pseudo inode number’ when scanning replicas for updates, instead of reading
# the full contents of every file. Faster for Windows file systems.
fastcheck = true
#
# When owner is set to true, the owner attributes of the files are synchronized.
owner = true
#
# When group is set to true, the group attributes of the files are synchronized.
group = true
#
# The integer value of this preference is a mask indicating which permission bits should be synchronized other than set-uid.
perms = 0o1777

The above is a fairly comprehensive profile file to act as a framework - if the hostname is a problem then use the absolute address such as 192.168.1.4

The Bottom Line

I now have everything I need for connections to the Internet on the move using Ubuntu Linux, in fact I am better off than with Windows as I now have a connection via Bluetooth and the O2 XDA Exec. I can use all my SIM cards in the T610 phone (GPRS not 3G) via Bluetooth, use the Vodafone Connect Card or Broadband USB dongle with the Vodafone SIMS offering GPRS/3G/3.5G and use all the SIMs in my XDA which provides 3G. I can also use my Xircom modem for Land-line connections (and only need to pay to speed up the internal modem) and off course I have WiFi and Ethernet for Hotspots and Broadband at home. All the connections can be made with minor changes in the  GNOME-PPP using a GUI interface. There were many dead ends followed but for those following me only need to spend a few minutes in low level terminal activities to obtain some information and then make small changes in two or three files - why did it take me so long?

Copyright © Peter and Pauline Curtis Content revised: 15th May, 2009