Sharing, Networking, Backup, Synchronisation and Encryption under Ubuntu Linux

This page has been extended from my original page covering standard backing up and now covers the prerequisites of Mounting Drives, File Sharing between Computers running both Linux and Windows, secure access to remote machines and Synchronisation. It also covers other techniques in the armory for preservation of data, namely separation of data and system and encryption

The ability to Backup the system has always been considered an essential to any system. The amount of data involved has increased dramatically from the days when data was preserved on floppy disks and our internal drives were 850 Mbytes (which was adequate for a Windows 95 system with Office Office 95 - now a minimum size Windows XP or Ubuntu Linux system is about 6 Gbytes and even a 5 year old laptop has a 40 Gbyte drive . Our photographs occupy 40 Gbytes and Audio 15 Gbytes - videos are not practical to back up and occupy some 750 Gbytes.

Backing up takes many forms now that the norm is for machine to be networked so this article covers many forms of achieving redundancy and preserving data. It covers shared drives between different operating systems on dual booted computer, networking of both Linux and Windows machines, techniques to separate system and data areas, conventional backing up to internal and external drives and most important in this time of mobility - synchronisation both between machines and to external hard drives.

File Systems and Sharing

Preliminaries - Background on the Linux File System

I am not going to extol the virtues of Linux, in particular the Ubuntu Distribution here, but I need to explain a little about the ways that file systems and disks differ in Windows and Linux before talking about backing up. In Windows, Physical Disk drives and the 'virtual' Partitions they are divided up into show up in the top level of the File System as Drives with names such as C: , a floppy disk is almost always A: and the system drive is C: . This sort of division does not appear at all in Linux where you have a single file system starting from root ( / ) which is where the operating system is located booted. Any additional disk drives which are present or are 'mounted' at a latter time such as a USB disk drive will by default in most distributions appear in the file tree in /media and the naming will depend on several things but expect to entries such as /media/cdrom , /media/floppy and /media/disk. If you create a special partition for a different part of the filesystem such as /home where all the users live then it can be 'mounted' at /home. In theory you could mount partitions for all sorts of other parts of the file system. If, for example, you add a new disk and choose to mount a partition to just contain your home directory it is only the addition of a single line in a file although you need to exactly copy the old contents to the new partition first - I will cover that in detail latter in this article.

There is a nearly perfect separation of the kernel, installed programs and users in Linux. The users each have a folder within the home folder which has all the configuration of the programs they use which is specific to them - it is set up the first time they run a program. A users folder only contains a few tens of kbytes until programs are used and all the program settings are hidden (hidden files and folders start with a . (dot). There are usually a number of folders generated by default including /Desktop /Documents /Music /Pictures /Videos /Templates and /Public which are used by program as their defaults. This makes backing up very easy - an exact copy of the home folder allows the system to be restored after a complete reload of the operating system and programs. Note the word exact as the 'copy' has to preserve Symbolic Links and the Permissions of all the files - permissions are key to the security of Linux so special archiving utilities are best employed.

Networks

Permanently Mounting a shared drive in Ubuntu 8.04 and 9.04 (Advanced)

If we are going to use a shared drive for data then we must ensure that it is permanently mounted. The mounting points in all major flavours of Linux are defined in a file system table which is in /etc/fstab and and the convention is that the mount points are in /media. We therefore need to set modify /etc/fstab to set up to mount points in /media and we must also create the folders for them using sudo to make them owned by root and set the permissions so they are accessible to all users.

This allows me to use the graphical file browser to create the folders and set permissions by standard navigation and right click menus for create folder and Properties -> Permisions. It is best to make these folders with the same names as those assigned by mounting from 'Places' which is derived from the partition label if it is set – see below. Do not continue to use this Root File Browser after setting up the shared folder as running anything as Root has many dangers of accidental damage although the aware reader will realise that the terminal can be avoided in the next stage by also opening fstab from within the Root File Browser - but do take care!

It is desirable to back up /etc/fstab and then make the changes using the editor gedit. This is done in a terminal by:

The procedure for modifying /etc/fstab in Hardy Heron and Jaunty is the same as it was for Dapper except that it is now recommended to use the drives UUID rather than the device's location ie append lines to /etc/fstab looking like:

# /dev/sda5
UUID=49D3-6030 /media/DATA vfat iocharset=utf8,umask=000 0 0
# /dev/sda2
UUID=F6EC309DEC305A57 /media/WINDOWS ntfs nls=utf8,umask=0222 0 0

Note – The first string is for a FAT32 partitions and the second for a NTFS partitions. The first provide a read/write mounts ( umask=000 for read/write for owner, group and everyone else) and the second a read only mount (umask=0222). If you want to only allow others read access then use umask=002

pcurtis@vortex-ubuntu:~$ blkid
/dev/loop0: TYPE="squashfs"
/dev/sda1: LABEL="WINRE" UUID="80AE-9D55" TYPE="vfat"
/dev/sda2: UUID="F6EC309DEC305A57" LABEL="WINDOWS" TYPE="ntfs"
/dev/sda5: LABEL="DATA" UUID="49D3-6030" TYPE="vfat"
/dev/sda6: UUID="8993fcc3-6a5a-4f94-9e6a-c77f5091c1ad" TYPE="swap"
/dev/sda7: UUID="83ac79eb-c78f-42a8-bbde-60ef0145a289" TYPE="ext3"
/dev/sda8: UUID="e31b581a-85fa-4681-bf11-aebb9a9656cd" TYPE="ext3"
pcurtis@vortex-ubuntu:~$

After modifying /etc/fstab and rebooting the three Windows Drives are mounted and appear on the desktop in addition to being in 'Places' - they can not be unmounted without root privileges which is just what we want.

There is one 'feature' of this way of mounting which seems to be totally universal and that is that only root (the owner) can set the time codes - this means that any files or directories that are copied by a user have the time of the copy as their date stamp. What seems to happen is this:

A solution for a single user machine is to find out your user id and mount the partition with option uid=user-id, then all the files on that partition belong to you - even the newly created ones. This way when you copy you keep the original file date. This is important if you have a file synchronisation program such as Unison which checks file creation and modification dates.

# /dev/sda5
UUID=706B-4EE3 /media/DATA vfat iocharset=utf8,uid=yourusername,umask=000 0 0

In the case of multiple user machines you should not mount at boot time and instead mount the drives from Places.

The uid can also be specified numerically and the first user created has user id 1000.

Labelling/Relabelling a FAT 32 file-system (partition)

New partitions which you create when installing Ubuntu will not have a label and will show up as a size when mounted on the desktop. The easy way is to do it under Windows by right clicking on the disk in My Computer -> Properties and entering a label. Labelling a Windows type filesystem in Linux is a black art and it took a lot of searching to find the way below which I have have used on principle rather than do it in Windows! The article referenced below also covers labelling file-systems in other formats and is very useful for USB sticks so they have a meaningful name and mount point.

Install the tools required

sudo apt-get install mtools
Find out device corresponding to the partition:

sudo blkid
Unmount device which we have found is /dev/sdb7 :

sudo umount /dev/sdb7
Check existing label:

sudo mlabel -i <device> -s ::
Change label to SHARED

sudo mlabel -i /dev/sdb7 ::SHARED
and that's it

Accessing Files on Windows Machines over the Windows Network

I originally considered this would be an important step but the order of difficulty was not what I expected - I could access files on Windows machines over the Network immediately using Places -> Network Servers although the creation date is only displayed correctly for files on NTFS partitions. I have not found a workaround yet or any reference to it despite extensive web searches. You will usually be asked for a password the first time.

Accessing Files on your Linux machine from Windows machines (and other Linux machines on your local network).

All these settings need a restart to become active. To save a full restart you can just restart Samba by typing on a terminal:

You can now use e Network Browser on another machine to access the machine you have just set up by Places -> Network - you will be asked for the Username and Password you have just set up and you can specify how long the password remains active.

For information: There are other ways of file sharing between Linux machines but samba is the popular choice as it also covers Windows machines.

Printing over the Network

This was again much easier than I expected. Use System -> Administration -> Printing -> New printer -> Network Printer:Windows Printer which takes one on to identify the printer on the network and has a selection procedure which has a huge range of printers with drivers available including the members of the Epson Stylus series - I have a C66.

Moving an Ubuntu Home Folder to a dedicated Partition (Expert level)

This is based on Psychocats - Create a separate home partition in Ubuntu with quite a few modifications to get it to work for me using Ubuntu Hardy Heron. As an aside the original article is one of an excellent series which are well worth a look. The proceedures here are not very well tested as I have only done it for one machine and one should make backups of everything of value but the techniques are used in many places. Both the partitioning and the "copying" and mounting have risks but the proceedures here are designed to minimise the risk and provide some escape routes if it does not work. It is however best to think ahead and provide the separation of system and data when the initial installation is carried out.

The first stage is to make a new partition using Gparted from an Ubuntu LiveCD. This will usually involve shrinking an existing partition, probably the root partition with the home folder you are moving. It is always safest to only reduce/increase the size of a partition leaving the start position the same. In the following I have used as an example a new ext3 partition /dev/sdb8 - you need to substitute what it turns out to be for you from Gparted. I have covered Partitioning elsewhere so I will not do more here than suggest you make sure you have a sensible division leaving space for both the main system and the /home directory and enough working space so you can keep the original /home for a while until everything is debugged.

Assuming you have made a new partition you now need to go back into your normal system. If you are using Hardy heron you can mount the new partition using Places -> Removable Disk (check the mount point is /media/disk or modify the following suitably) then use cpio which is an archiving program to 'copy' your home directory to it by the following commands in a terminal:

I initially thought doing this from a LiveCD would be more sensible as it would avoid problems with files being in use or changing but it does not work as the permissions end up being incorrect as the original root folder one mounts and uses for the "copy" does not have the original permissions when using the live CD. I found that out the hard way as it was not clear in the original article and I had to start again several times! It takes a while to realise that Linux is very different from Windoze and one can work on a live system without problems of files being locked and/or in use. The complexity and use of cpio instead of cp is to maintain permissions and the various links and comes from a variety of sources. It is worth doing a man cpio and a man find to understand the programs and parameters but I will try to explain some of basics of it step by step:

sudo find . -depth -print0 - this provides a list of filenames. I am not sure if the sudo is desirable or essential, it is not needed for a single user system. The - printo parameter specifies "null terminated strings" which will work even if some of the files have spaces, newlines, or other dubious characters in them. The -depth parameter specifies processing each directory’s contents before the directory itself.

The results are written into a pipe by the | and the program reading them must be capable of using this list. The cpio archiving command has this feature.

| cpio -p0vd /media/disk - here's the tricky part. This uses the "passthrough" mode of cpio, an archiving program that normally copies files "in" or "out" but it can do "both" using this "passthrough" mode. The -p sets the "passthrough" mode for cpio which then expects a list of filenames on its standard input (which we are providing from the 'find' command). It then copies the corresponding file "in" from the path specified (as part of the input line) and "out" to the the path specified as the final one of cpio's arguments (/media/disk in this case).

The rest of the switches on this cpio command are: -0 - expect the input records (lines) to be null terminated, -v provides verbose output, and -d - make leading directories as needed. -u forces overwriting of any existing files but it is best to make sure that everything is deleted before starting.

We sould now have an exact "copy" of the /home folder on the new partition with the correct permissions (at least for the username you used when carrying out the cpio copy). It is prudent to mount the partition and have a look to see if everything is reasonable. It should be [still] be visible in Places -> Removable Media for you to check the permissions and ownership is correct for each user.

Now we need to mount this in the correct place at turn on. Automatic mounting is carried out by backing up then modifying /etc/fstab by:

# /dev/sdb8
UUID=3b50dbce-28c8-46fc-bc63-f89bb06c54e5 /home ext3 nodev,nosuid 0 2

At the next start-up the modified fstab will mount the new home folder from the disk partition over the top of the existing one which is then invisible. A return to the old set up is as simple as restoring the original fstab or commenting out the new final line at which point the old /home is once more visible. At a point in the future when one is completely happy the old contents can be deleted to save space using the LiveCD.

An alternative is to rename /home to /home-backup using the LiveCD and create a new /home ready for the mounting with the appropriate permissions before changing fstab as in the original article. This is more complex and risky than mounting over the top but the 'backup' /home directory can be deleted without having to use the LiveCD.

The proceedure above was what finally worked for me with a single user setup, it is possible that permissions may need to be set up for other users. If the permissions are wrong, as happens if you do the cpio 'copy' using the LiveCD you get messages when you try to log in such as:

Users $Home/.dmrc file is being ignored. This prevents the default session and language from being saved. File should be owned by user and have 644 permissions.
Users $Home directory must be owned by user and not writable by others.

If you can get past this message, perhaps by changing permissions from the liveCD the following brute force procedure (which is a refinement of what I did in one of my early attempts) to set the groups and permissions can be tried on the other users:

sudo chown -R eachusername:eachusername /home/eachusername
sudo chmod -R 755 /home/eachusername
sudo chmod 644 home/eachusername/.dmrc

This needs to be done before each username is logged into the first time otherwise many desktop settings will be lost at the point the warning message appears - this means that the cpio 'copy' will need to be repeated and the permissions set for each additional user as above before logging in the first time.

The best way I believe now is to create a new user just before making the change and logging into that user to carry it out. The overhead in size of a the new user's directory in /home is only about 28K and it makes sure that there is very little going on and no important files are missed or changed during the copy - I worry about programs with Daemons running such as Picasa and Skype. The new user can be removed after the change has taken place. The new user is easy to creat by System -> Administration -> Users and Groups. The new user needs a name, to have the Profile of an Administrator set in the box and the passwords input, nothing else matters and can be empty or default. The new user will automatically have the ability to Sudo if set up with the profile of an Administrator.

When you delete the user using the GUI you do not delete the /home directory or another few files - to also remove the home directory it is easiest to instead use the command line and the deluser utility:

Alternatively all files owned by the user, including those in the user's home directory may be removed as follows:

One possible improvement which I have not tried would be to use rsync instead of cpio to do the intelligent copy by

you do not need to be in the home directory. rsync is an advanced archiving and copying tool which can work over ssh (Secure Shell) to other machines as just a simple intellegent copy. It is also useful for regular 'backups' as it will only copy files which have changed so it can be run on a regular basis to syncronise two machines although unison (which uses rsync) is a much better way to go.

In summary: there are many advantages in having ones home directory on a separate partition but overall this change is not a proceedure to be carried out unless you are prepared to experiment a little. It is much better to get it right and create one when installing the system.

Backing up.

There are a number of backup utilities available for Ubuntu Linux but use Simple Backup which is a simple backup solution intended for desktop use created within Google Summer of Code 2005 for Ubuntu with the mentoring of the Ubuntu team. It is a tiny download and seems very comprehensive in the ways you can chose to include and exclude files based on location, size, type etc. It can run automatically and will do periodic full backups with incremental backups of changed files. By default it saves the backup as a compressed file in /var/backup and the file seems very small at under 400 Mbytes with default settings. You can restore individual files etc and in theory they can be restored to a different location although that feature has not worked for me. You can also access the various levels of archive folders and use archive manager to extract directly from the .tar archives in the folders.

It is all very easy with a good GUI interface and is found in Systems ->Administration -> Simple Backup. Simple Backup is downloaded by Add/Remove and search for Simple Backup or Synaptic package manager and search for sbackup.

The default settings create a daily incremental backup and a weekly full backup. Although the files are quite small you need to have a policy of a monthly tidy up otherwise you will end up with a full disk - it has happened to me twice

There is also a very powerful command line archiving tool round which many of the GUI tools are based which should work on most Linux Distributions. In many circumstances it is best to access this directly to backup your system. The resulting files can also be accessed (or created) by the archive manager accessed by right clicking on a .tgz .tar.gz or .tar.bz2 file. To show the power the following commands will back up your entire file system but can easily be adapted to just back up your home folder by starting there instead of in the root directory:

sudo tar cvpzf /media/FREEAGENT/mybackup.tgz / --exclude=/mybackup.tgz --exclude=/proc --exclude=/lost+found --exclude=/media --exclude=/sys --exclude=/mnt

This is a single line if you cut and paste. The options cvpzf are: create archive, verbose mode, retain permissions, gzip archive and file output. Then follows the name of the file to be created, mybackup.tgz which should use the date for easy reference - in this case we are using a USB hard drive called FREEAGENT for the backup. Next is the directory to back up, in this case / the root so everything will be stored. (Another common possibility would be /home or /home/USERNAME). Next are the objects which need to be excluded - the most important of these is our back up file or it would be recursive! It also excludes directories which are recreated dynamically.

The backup process is slow (15 mins plus) and the file over a gbyte for the simplest system. After it is complete the file should be moved to a safe location, preferably a DVD or external device. If you want to do a higher compression method the command "tar cvpjf mybackup.tar.bz2" can be used in place of "tar cvpzf mybackup.tgz". This will use bzip2 to do the compressing - j option. This method will take longer but gives a smaller file.

You can access parts of the archive using the GUI Archive Manager by right clicking on the .tgz file - again slow on such a large archive. A full system backup is most useful if you can effectively restore your entire system or data. This can be again done by a few commands but do not try this for fun as it will overwrite the entire Ubuntu file system, thus restoring the older image that we took. I have not yet had problems so large I have had to try this! The following assumes the backup image is still in the root directory or has been replaced there from DVD:

The restoration uses the options - extract, verbose, retain permissions, from file and gzip. This will take a while because all your files will be overwritten with the versions from the image you previously backed up. The "-C /" ensures that the directory is Changed so the file is restore to the root directory /.

If this has been done onto an empty drive from a LiveCD one needs to replace any excluded directories before rebooting the system.

If the old system is still present note that it only overwrites files, it does not deleted files from the old version which are no longer needed.

This is one of the simplest and fastest methods for backing up. It may take some customizing and tweaking for your purposes but it is powerful, versatile, and free.

It is very useful if you want to preserve your home directory to allow you to upgrade or 'clone' machines and the archive and restore are then:

sudo tar cvpzf /media/FREEAGENT/mybackup.tgz /home/USERNAME --exclude=/mybackup.tgz --exclude=/proc --exclude=/lost+found --exclude=/media --exclude=/sys --exclude=/mnt

Synchronisation between machines and hard drives - Unison (Advanced)

I have put this as an advanced activity as a number of features of the normal set up of the way removable and fixed drives are mounted need to be understood and configured to avoid unexpected results. See below under Permissions and Automounted Drives (Removable Drives) and Drives Mounted Permanently at startup using fstab.

Backing up is very important and often the easiest way has been to make copies onto external hard drives and onto other machines. But what does one do when one has multiple copies of important folders and different files have been added or edited on each machine or external hard drive copy. This is a particular problem when one uses a laptop or netbook away from home for a few months and returns especially when only subsets of files may have been taken with one or the two machines were not in sync when one started.

Unison

Linux has a very powerful tool available called Unison to synchronise folders, and all their subfolders, between drives on the same machine and across a local network using a secure transport called SSH (Safe 'S Hell). At its simplest you can use a Graphical User Interface (GUI) to synchronise two folders which can be on any of your local drives, a USB external hard drive or on a networked machine which also has Unison and SSH installed. Versions are even available for Windows machines but one should make sure that the Unison versions numbers are compatible.

You just enter or browse for the two folders if local and it will give you a list of differences and recommended actions which you can review and it is a single keystroke to change any you do not agree with. Unison uses a very efficient mechanism to transfer/update files which minimises the data flows based on a utility called rsync. The initial Synchronisation can be slow but after it has made its lists it is quite quick even over a slow network between machines because it is running on both machines and transferring minimum data - it is actually slower synchronising to another hard drive on the same machine.

For more complex synchronisation with multiple folders and perhaps exclusions you set up a very simple configuration file for each Synchronisation Profile and then select and run it from the GUI as often as you like. It is easier to do than describe - a file to synchronise my four important folders My Documents, My Web Site, Web Sites, and My Pictures is only 10 lines long and contains under 200 characters yet synchronises 25,000 files! The review list is intelligent and if you have made a folder full of sub folders of pictures whilst you are away it only shows the top folder level which has to transferred which is a good job as we often come back with several thousand new pictures!

Both Unison and SSH are available in Ubuntu but need to be installed using System -> Administration -> Synaptic Package Manager and search for and load the following three packages:

The first is the GUI version of Unison which also installs the terminal version if you need it. The second is a meta package which install the SSH server and Client, blacklists and various library routines. The third is required to allow ssh to use host names rather than just absolute IP addresses which will also requires some extra configuration described below - you may find with some versions of Ubuntu that winbind is already installed but it will still need the configuration.

Unison is then accessible from the Applications Menu and can be tried out immediately. There is one caution - it is initially configured so that the date of any transferred file is the current date ie the file creation date is not preserved although that can easily be set in the configuration files - the same goes for preserving the user and group.

Permissions for External USB Drives and Drives Mounted via 'Places': The file systems on External USB Drives are almost always Windows type file systems such as FAT32 which do not support the definitions of owners and groups and their associated permissions. The same applies to such file systems on internal drives which are mounted through Places or permanently mounted when the machine is started up. The Owner, Group and Permissions are therefore set up each time when the drives are mounted. Unison checks for matches in the permissions so one has to make sure that USB drives are mounted with suitable and matching permissions - by default they are mounted with 0700 whilst drives permanently mounted in the File System Table at startup usually have the permissions set to 777 or 775 in /etc/fstab. See above for information on permanently mounting drives and /etc/fstab

The permissions for internal drives mounted via Places or automounted when external drives are plugged in can and should be set using the Gnome Desktop Configuration Editor to match what is set up for permanently mounted drives in /etc/fstab. The Configuration Editor is norminally installed but does not show under Applications -> System -> Configuration Editor (because it is powerful and open to abuse by the unwary) . You can set up the system to display by a Right click on the Applications drop down -> Edit Menus and find and tick the box for it to be displayed. It is a useful program so it is worth making it easily available but if you are in a hurry it can also be run from a terminal by:

Once you have the configuration editor running expand the parameters on the right through system -> storage -> default_options and set umask=000 for the drive type which will probably be vfat. Double click down till you get to the existing umask=077 and change it to 000 or 002 to match drives mounted by fstab. Note umask is a mask and you set the inverse of the permissions you want which are 777 or 775 !

Even so Unison may seems to get some miss-matchs on the first synchronisation which have to be Resolved. I have set up a number of profiles for back-up and synchronisation with my WD Passport portable 120 Gbyte USB drive which is formated to FAT32 (vfat) to keep Pauline's Toshiba Laptop and MSI Wind netbook synchronised and a Profile to Synchronise my MSI Wind Netbook and homebuilt Desktop over the local network.

Ownership of Drives Mounted permanently using fstab: There is one 'feature' of mounting using the File System Table in the usual way is that the owner is root and only the owner (root) can set the time codes - this means that any files or directories that are copied by a user have the time of the copy as their date stamp which can cause problems with Unison when Synchronising. What seems to happen is this:

# /dev/sda5
UUID=706B-4EE3 /media/DATA vfat iocharset=utf8,uid=yourusername,umask=000 0 0

In the case of multiple user machines you should not mount at boot time and instead mount the drives from Places.

The uid can also be specified numerically and the first user created has user id 1000.

So far we have looked at the set needed up for Unison to Synchronise files on the same machine. We will now look at synchronising between machines using ssh. When you set up the two 'root' directories for synchronisation you get four options when you come to the second one - we have used the local option but if you want to synchronise between machines you select the SSH option. You then fill in the hostname which can be an absolute IP address or a hostname (but see below on the need to set this up). You should then specify the username you will use to log into the other machine ie the samba sharing username as set up for windows sharing, leave the port blank. You will also need to know the folder on the other machine as you can not browse for it. When you come to synchronise you will have to give the password corresponding to that username.

I have always checked out that ssh has been correctly set up on both machines before trying to use Unison. In its simplest form ssh allows on to log using a terminal on a remote machine. Both machines must have ssh installed (and the ssh daemons running which is the default after you have installed it). For reasons we will come to (and sort out) the set up of Ubuntu only allows one to use and absolute IP address so you need to know the address on your network of the other machine. This can be found within a mass of other information by opening a terminal on the machine whose IP address you need and typing

which will give a lot of information on all your connections including the active network - the IP address you are looking for will follow an inet addr: . There will be two addresses, one will be for the local loopback (127.0.0.1) and the other is the external address which is what we want.

if you want to reduce the amount of information to just the lines containg inet addr then you can pipe the output through a program called grep - I leave it as a task for the reader to find out more about pipes and grep once you have tried it and can see the power of the command line!

It is now time to try ssh and I assume that the machine we want to access has an IP address 192.168.1.2

You will get some warnings that it can not authenticate the connection which is not surprising as it is the first time and will ask for confirmation and you have type yes rather than y. It will then tell you it has saved the authentication information for the future and you will get a request for a password which is the start of your log in on the other machine. After providing the password you will get a few more lines of information and be back to a normal terminal prompt but note that it is now showing the address of the other machine. You can enter some simple commands such as a directory list (ls) if you want. When you have tired of this type exit to return to your own machine.

Enabling Hostname Resolution under Ubuntu: Using absolute IP addresses is not very satisfactory in the long term as they are allocated dynamically from your router or other server on most modern networks. This means that they may change so you could end up synchronising with the wronge machine. In some ways the consistency and ssecurity checks run by ssh to prevent hacking are even more of a problem when the IP address changes and I have sometimes had to clear out the file containing that information (~/.ssh/known-hosts) . It is therefore much better to use the hostname to access the machine. Unfortunately the default set up in Ubuntu does not support hostname resolution without an additional utility being loaded and a change made in a configuration file to enable what is called WINS (Windows Internet Name Service) resolution by:

Installing winbind using System -> Administration -> Synaptic Package Manager and search for winbind - it is often installed with samba so you may already have it but it is best to check.

Note - in some cases there may be more on the line - if that is the case put it immediately after files.

Now ssh <hostname> will log you into <hostname> and you can use hostnames in Unison

Problems with some networks giving slow response especially after wins modification for ssh: There is a problem with some ISPs who use DNS redirection for advertising purposes which is accentuated when the above modification is made. It is a known bug see https://bugs.launchpad.net/hundredpapercuts/+bug/389909 and the effect is that the ability to click on Places > Network and browse the local network has essentially been unusable. It takes over a minute and a half to just display the gui and then it fails to display the network properly, if at all. Network places that are displayed often return error messages if clicked on. Sometimes running the Network browse gui several times in a row returns normal function, sometimes it does not. The fix is very simple. Edit

In the file you need to change the resolve order and remove the leading semicolon so that bcast comes before host

# What naming service and in what order should we use to resolve host names
# to IP addresses
; name resolve order = lmhosts host wins bcast

# What naming service and in what order should we use to resolve host names
# to IP addresses
name resolve order = lmhosts bcast host wins

Some sample Profiles for Unison:

# Profile to synchronise from desktop triton-ubuntu to netbook vortex-ubuntu
# with username pcurtis on vortex-ubuntu
# Note: if the hostname is a problem then you can also use an absolute address
# such as 192.168.1.4 on vortex-ubuntu
#
# Roots for the synchronisation
root = /media/DATA
root = ssh://pcurtis@vortex-ubuntu//media/DATA
#
# Paths to synchronise
path = My Backups
path = My Web Site
path = My Documents
path = Web Sites
#
# Some typical regexps specifying names and paths to ignore
ignore = Name temp.*
ignore = Name *~
ignore = Name .*~
ignore = Name *.tmp
#
# Some typical Options - only times is essential
#
# When fastcheck is set to true, Unison will use the modification time and length of a
# file as a ‘pseudo inode number’ when scanning replicas for updates, instead of reading
# the full contents of every file. Faster for Windows file systems.
fastcheck = true
#
# When times is set to true, file modification times (but not directory modtimes) are propagated.
times = true
#
# When owner is set to true, the owner attributes of the files are synchronized.
owner = true
#
# When group is set to true, the group attributes of the files are synchronized.
group = true
#
# The integer value of this preference is a mask indicating which permission bits should be synchronized other than set-uid.
perms = 0o1777

The above is a fairly comprehensive profile file to act as a framework and the various sections are explained in the comments.

Secure access to a Remote File System using SSH and SSHFS (Advanced)

SSHFS (SSH File System) uses SSH to mounting a remote filesystem on a local machine. SSHFS uses FUSE (Filesystem in Userspace) which is included in Ubuntu and authenticates connections so only those who are authorised can mount them. SSH connections are encrypted, no one can see your files as they are transferred over the network and once mounted the FUSE filesystem is owned by the user that created it and other users on the local machine can not access them.

Firstly we must install sshfs using the Synaptic Package Manager by System -> Administration -> Synaptic Pacakage Manager -> Search for sshfs, Rigtht Click and Mark for Installation -> Apply.

Secondly add yourself to the fuse group by System -> Administration -> Users and Groups -> select user -> Properties -> User Privileges, then tick the following option:

Once you have added yourself to the fuse group, you need to either log out and log back in again or reboot for the change to take effect.

There is a risk that your ssh session will automatically log out if it is idle. To keep the connection active add the following parameter to ~/.ssh/config or to /etc/ssh/ssh_config on the client.

Using a Terminal

A local mount point must exist and have permissions set for you to use it before you connect. If you have it in your home directory you can use the file manager to create the new directory which will have the correct permissions.

Assuming that you have ssh installed and the server running on the remote machine you now just need to run the sshfs command to mount the remote directory. In a terminal type:

The idmap=user option ensures that files owned by the remote user are owned by the local user. If you don't use idmap=user, files in the mounted directory might appear to be owned by a different user.

fusermount -u ~/localfolder
This method seems simple and reliable and you should test its use.

Using ‘Connect to Server’ in the Nautilus file browser with the SSH option

An Alternative way to mount a secure remote file system is to open the Nautilus file browser then File -> Connect to Server and select SSH from the drop down menu. Fill in the server name and other parameters as required. This way will give you a mount on the desktop and you can unmount by a right click -> Unmount Volume. You still need SSH on both machines and the fix to the Hostname resolution problem using winbind covered above. I am not sure if you also need the keep alive signals as I already had the fix in place.

If this fails with a timeout error try deleting or renaming ~/.ssh/known_hosts - you will get a message saying that it is a new host you are connecting to and everything will then work. It seems to be caused by the IP address changing for the remote hostname and you not getting the warning message.

Using Places menu

This is very similar to the above but is accessed by Places -> Connect to Server and filling in as above. Nautilus is then opened. It seems less reliable than accessing the connection Window from in Nautilus under one of my machine with Hardy Heron whilst it works fine under Jaunty.

Conclusion

I discovered this way of working much later than Networking using samba. It has many advantages once the basics have been set up as it is a much more secure solution especially over Wifi as snooping is impossible with an encrypted connection. However you still need to have a strong password to prevent a remote login. I plan to bring it up in this document once I have more experience.

Encryption

OpenPGP

We require encryption to protect any sensitive information whilst we are travelling. Linux has OpenPGP which offers a superset of the usual PGP standard but defaults to be compatible with the encryption levels available in PGP 8 which we are using. It is accessed by the gpg program which operates in terminal mode.

There is also a GUI interface called Seahorse which can be installed by Add/Remove which handles the creation and management of keys much easier than using gpg directly. It is also supposed to add facilities into the text editor and file browser. The right click encryption of a file and encryption/decryption in the text editor work fine as does the GUI program to create and manage keys. In theory double clicking on a PGP file ought to bring up the screens to open it but there seems to be a problem in Seahorse 8.1 or Ubuntu Dapper Drake which prevent that. After a bit of searching and playing about I realised that it worked if the .gpg extension was used so it was a simple job to add the same program as an option for opening .pgp files namely seahorse --decrypt using the right click menu on a .pgp file -> Open with other application -> Use a Custom Command and filling in the box with seahorse --decrypt .

Evolution has built in encryption and signing for emails using keys created in terminal mode or managed by Seahorse. Full details of how to set it up and use it are in the Evolution help files.

Secure deletion of data - shred

The other feature which is required for looking after data securely is a way to erase files without traces. It is no good being able to encrypt a file if you can not delete the original or working copies. Linux has a built in command shred which does a multiple pass write of data selected to make a read based on residual information at the edges of the magnetic tracks almost impossible before the file is deleted. This is not foolproof for all file systems and programs as temporary copies made be made and modern file systems do not always write data in the same place however on an ext2 or ext3 system with the default settings in Ubuntu Linux it is acceptable. Do a man shred to find out more.

Scripting Example - Secure Delete

There is no GUI interface for shred so I wrote a simple script. This took a few days to get up the learning curve of programming in bash and how the system was set up which will pay off handsomely in the future. A good place to start is LinuxCommand.org: Learning the shell. The important piece of information is that the addition of a path to a /bin directory is set in ubuntu linux in .bashrc not .bash_profile as is described in some places. Also note that files starting with a . are hidden - use View -> Hidden Files in the File browser to find them. The lines I added were:

# Additions to the standard ~/.bashrc file to set up path to
# /bin directory in home folder
PATH=$PATH:~/bin

I then had a folder in which to put script files which could be accessed from any directory. My first script to shred a file follows - if you want to follow it in detail remember that man any_command gives a summary of what it does and its options:

#!/bin/bash
#
# script to shred a file ~/bin/Shredfile
# echo "This will Shred - overwrite many times - and"
echo "delete a file in a secure manner"
echo "The filename should include the path"
echo "from your home directory"
echo "eg /Desktop/filetoshred.doc or /Safe/filetoshred)"
echo -n "Enter Filename to Shred ? "
read filename
temp=/home/pcurtis$filename
if [ -f ~/$filename ]
then
echo "File $temp will be deleted"
else
echo "File $temp does not exist"
echo "Hit Enter to exit ? "
read
exit
fi
echo "About to Shred" $temp "!!!"
echo -n "This is irrevocable - y to continue or n to Abort ? "
read t1
if [ "$t1" = "y" ]
then
shred -n 50 -u -z -v $temp
else
echo -n "Aborted- hit any key to exit ? "
read
exit
fi
echo -n "File shredded - hit any key to exit ? "
read
exit

The reads at the end of each part are necessary to prevent the Terminal Window closing before you have seen what happens.

The last step is to create a launcher on the desktop which can also be dragged onto the bars. Right click anywhere on the desktop -> Create Launcher Fill in a name; browse to the ~/bin directory and script name; tick run in terminal; add an icon if required and that is it. The Launcher can also be dragged onto the panel.

It all sounds very simple but it took me a while to get scripting together the first time despite having done some programming in my time.

Truecrypt

It does very much what PGPDisk does - it creates a Virtual Disk with the contents encrypted into a single file or onto a disk partition or removable media such as a USB stick. In fact it does it rather better than PGPDisk in many ways and in the Windows version it has all the same automatic demount etc options. The encryption is all on the fly so you have a file, you mount it as a disk and from then on it is used just like a real disk and everything is decrypted and re-encrypted invisibly in real time. The virtual Drive is unmounted automatically at close down and one should have closed all the open documents using the Virtual Drive by that point just like when you shut down normally. The advantage is that you never have the files copied onto a real disk so there are no shadows or temporary files left behind and one does not have to do a secure delete. I have loaded it onto two of my Windows systems.

Truecrypt obviously installs deep into the operating system in order to encrypt decrypt invisibly on the fly. This has meant in the past that it was specific to a Linux Kernel and had to be recompiled/installed every time a Kernel was updated. Fortunately TrueCrypt has now reached version 6 which can be downloaded as a .deb file for both 32 and 64 bit versions of Ubuntu – make sure you get the correct version. The .deb is packed into a compressed file .tar.gz just double click truecrypt-6.x-ubuntu-x86.tar.gz to open the archive and drag the folder to the desktop, double click it then click 'run in Terminal' to run the installer script.

The new version of Truecrypt is a great improvement and has a GUI interface almost identical to that in Windows. It can be run by just typeing truecrypt in a terminal or from a launcher. It opens virtual disks which are placed on the desktop. Making new volumes (encrypted containers) is now trivial – just use the wizard. Old files can be opened but it is best to create a new volume and copy the contents across. This is now a very refined product under Linux.

The only feature I have found is that one has to have administrative privileges to mount ones volumes. This means that one is asked for ones administrative password on occasions as well as the volume password. There is a way round this by providing additional 'rights' specific to just this activity to a user (or group) by additions to the sudoer file. There is information on the Sudoers file and editing it at:

https://help.ubuntu.com/community/Sudoers

Because sudo is such a powerful program you must take care not to put anything formatted incorrectly in the file. To prevent any incorrect formatting getting into the file you should edit it using the command visudo run as root or by using sudo ( sudo visudo ). In Ubuntu 8.04 the default editor for visudo has changed to vi, a terminal editor, which is an incomprehensible at least to those used to Windows. To change this behaviour we can edit the users setup file:

Add the lines to use gedit as the standard GUI editor and nano as a terminal editor:

# Addition to standard file to select visual and terminal editors
[ $DISPLAY ] && \
export VISUAL="gedit" || \
export EDITOR="nano"

Now you can launch visudo in a terminal with the standard editor gedit by:

sudo -E visudo

There are now two ways to proceed, if you have a lot of users then it is worth creating a group for truecrypt, changing truecrypt to be a member of that group and adding all the users that need truecrypt to that group. You then use sudoer to giving group members the 'rights' to use it without a password. See:

If you only have one or two users then it is easier to give them individual rights by adding a line(s) to the configuration file by launching visudo in a terminal into the standard editor gedit by:

sudo -E visudo

USERNAME ALL = (root) NOPASSWD:/usr/bin/truecrypt
#or
%truecrypt ALL = (root) NOPASSWD:/usr/bin/truecrypt

Save and exit - if there are errors there will be a message and a request what to do in the terminal.

I have used it both the simple way and by creating a group called truecrypt. In the case of the Wind where I have two users I created a a new group truecrypt using System -> Administration -> Users and Groups and Truecrypt now runs under all both users without continual prompts for passwords. There is a third Brute Force method which I have tried which works for all users without a truecrypt group needed but this is much less secure - append the following string

Contents of this Page

Introduction