Home Uniquely NZ Travel Howto Pauline Small Firms Search
Mint and Ubuntu Linux on the Take

Contents of this page

Introduction

 The main page covering my experiences in making the transition from Microsoft Windows to Ubuntu Linux - Fun with Ubuntu Linux quickly become excessively long.  This was my second specialised page covering how to handle all the pictures we take in the Digital Camera era using Ubuntu Linux. The page was first extended to include my experiences learning how to turn the output of my Digital Video Camera which uses DVC tapes into edited DVDs with a navigation menu. I finally decided that it was an appropriate home for my experiences with Digital TV which overlaps in many respects with the Digital Video section especially in the media player area.

One of my main reasons for making the transition to Ubuntu was the huge, expensive and uncontrollable flow of data whilst on the move when using Microsoft Windows XP. I got so tired of the continual updates to the Windows XP system and the associated Virus checkers, Firewalls and Malware detectors. Linux was attractive because it needs no virus checkers or separate firewalls which need updating and no unstoppable or essential automatic system updates. The subject of my first specialised page was therefore Ubuntu Linux on the Move which covered all aspects of communications when away from home. Perhaps the second most important activity after communications for inveterate travelers like ourselves is handling all the photographs and video we take with digital cameras, viewing them and ultimately putting them into a form that can be used on our web site. These days we also have the ability to watch Digital TV using USB DVB-T sticks, record programs and play videos so that is also covered. Retaining the fun approach I have called it Ubuntu Linux on the Take. This has evolved and I am in the processing of pruning much old material and taking into account the use of Linux Mint instead of Ubuntu. The old version covering Ubuntu and up to Mint 17.3 is still available at Ubuntu Linux on the Move - Legacy Activities

 

Digital Photography

Digital Photography support has become a major use for computers over the last 4 or so years and increasing so for laptops for use whilst on holiday to download and backup the cards in the camera. This provides a major logistic and storage problem - we have taken some 20,000 pictures in the last 4 years of which less than 25% are printed and even less added to our web site. Many more are used as reference - we take many pictures of, for example information boards for background information so they still have to be easily found.

Firstly it is worth looking at the important differences between graphic images from a digital camera and other images. The photographic images are mostly stored as JPEG (Joint Photographic Experts Group) compressed images, the same format used for most web images and most other photo type images. The difference is that there are now extensions to the format which allow additional information on the camera settings when the picture was taken such. Some are well known and defined such as EXIF, some are more flexible in the metadata they can contain. Early graphics packages ignore the metadata or strip it out and when you modify and save the graphics file it will have been lost permanently. Newer packages give the choice of striping it out or retaining it unchanged - some allow limited changes or new metadata data to be added.

Most Digital Cameras come with their own basic software for Windows PCs and sometimes Apple computers - none come with software for Linux however most of the functionality is built into Ubuntu Linux although some may need to be installed from the CD or repositories. In fact the Ubuntu software is often superior and makes it easy to use cameras from several vendors and integrate the pictures.

Before we start on the details there is one basic point which needs making: Whatever software and hardware you use it is essential that you set the time in your camera accurately and reset it as one changes time zones. If you have several cameras you need to be relatively accurate to a few seconds to interleaf pictures correctly - that needs some care and both cameras side by side. If you have not got it right some editors allow you to adjust the time code in the EXIF data but that is very time consuming.

Requirements for supporting Digital Photography

First we should look at what is required.

Support available in Ubuntu Linux for Digital Cameras

Now we have laid out our requirements we can look at what is available in Ubuntu Linux.

Hardware

Firstly almost all cameras are recognised as a USB mass storage device if plugged into a Ubuntu Linux computer. The System -> Preferences menu has a Cameras tab which allows you to set up for automatic importing using your selected program (gThumb by default, F-Spot by preference or Pix under Mint 18). I prefer to plug my camera card in directly and it is automatically recognised , mounted and the import program started.

Programs - needs updating Mint 18 has Pix based on gThumb

F-Spot

F-Spot no longer seems to be maintained (since 2010) and is no longer in the repositories.  

Shotwell

This program was the default photo-manager for Ubuntu 10.10 and can be installed on earlier versions although it is best to use the PPA as the version in the standard repositories is already out-of-date and the program is developing fast. If you used F-Spot it can import your current tags etc. Like F-Spot it creates a folder called Photos in your home folder,(or you can specify a folder of your choice in Preferences - the Folder structure has sub-folders for year, month and day and adds them to its catalog. It does not download pictures in the catalog a second time.

Shotwell groups pictures taken together in time into 'Events' which start off named with the date but can be changed to anything you like and Events can also be merged. You can also add multiple tags and ratings (1-5 Star and reject) so the sorting is fairly flexible.

When we come to editing, Shotwell has a number of built in basic sizing and correction functions including cropping and a colour temperature correction. The originals are not changed and the changes can be reverted back to the original. The changes are held in a database rather than files in the folders so they do not get preserved when you move your pictures to a backup device. It does not have the fine rotation function in use a lot in Picasa but you can also use an external editor such as GIMP and the original is preserved with changes made on a copy.

I have not had a lot of time to try it out but it looks a very good starting point and can be used for importing and quick look displays etc. Further thought about backing up and moving the picture 'correction' data with the picture folders is needed as are optional permanent tags in EXIF metadata. Straightening and permanent tags in the EXIF data are under development for release 8.

Tip: I have a large collection of pictures on a separate partition and most are not in the F-Spot database. I have input these by pointing the Shotwell default database to the top level folder by Edit -> Preferences and then importing them using File -> Import - it seemed to recognise what I was doing as it did not offer the option to copy them or link to them (if you have to make a choice use link) or create duplicates and then re-importing from a camera again did not generate duplicates.

gThumb now enhanced as Pix in Mint 18

gThumb was the default photomanager in Mint 16 and does an excellent job of importing so I switched from f-spot which has a habit of crashing.

GIMP (Sophisticated Image Processor) is considered my many to be on a par with Adobe Photoshop in most ways but is Open Source and runs under Linux and Windows. It is always installed in most Linux distributions including Ubuntu. It is an understatement to say that GIMP seems daunting when you first open it but is worth persevering and after a short time it become apparent that it is actually very easy to use and provides a huge number of facilities with instantaneous previews. I will just mention those I find indispensable: Rotation, perspective correction, cropping with and without the aspect ration being maintained and changes in resolution. Colour corrections are always difficult but hue and RGB corrections are available as usual but not the colour temperature correction I find helpful in F-Spot and Picasa. Brightness and contrast are obviously available. I also used to use a gamma correction a lot in PaintShop Pro 4.2 - this allows one to bring out detail in the shadows by a non-linear brightness curve. This and more can be done done just as simply by GIMP using the curves function - it is difficult to describe but a few seconds experimenting makes its power and effectiveness clear.

I find a combination of simple corrections whilst cataloguing in F-Spot or Picasa and then more complex ones when required in GIMP is perfect. In practice my Canon A75 is so good that only a few photos require manipulation in colour so one is only correcting ones own errors in rotation or abuse of perspective and perhaps gamma type adjustments to bring out details in dark areas of a picture. One exception is taking pictures of information boards where one almost inevitably has to angle shots to avoid reflections - the perspective corrections in GIMP make it trivial to drag each corner to the correct place before cropping.

Google Picasa (Complete Suite of programs) I can not leave the handling of digital photographs without mentioning Google's Picasa which I have come to depend on for managing (but not importing) all may pictures. Picasa was written for Windows by Google and also is provided for Linux using a custom version of WINE which Google have supported. I have covered installation in Fun with Ubuntu Linux. It is a self contained Photo Management and Cataloging system which will index a selection of folders that you select using the 'folder manager' . This can be displayed both on a folder basis and a time-line. It has a basic tag like function by 'starring' pictures and a collection of photos can be gathered into a 'tray' from different folders to work on or export. Pictures can also be given Tags (keywords) which implemented as IPTC keywords and can be used with a Search function. The keywords are permanently attached so they follow any directory changes, backups etc

A very useful feature is 'albums', an indexing process where collections of pictures can be gathered together without making actual duplicates. The ability to create albums is very powerful. For example lets say we want to create a folder of pictures to print and a folder of pictures which will end up on the web. First we go through all the pictures in the time frame and put a Star on those to Print. Now those pictures are also in the Special System Album called Starred Photos. If we look at that we can select those in the correct time frame (you may have other starred pictures) by a click at the start and shift-click at end then at the bottom middle click Add To -> New Album and call it MyPicturesToPrint1. We now have all those pictures in an album and if we display that we can clear all the existing Stars and then Star those for the Web folder, select them and Add To another New Album called MyPicturesForTheWeb1. Those folders can eventually be exported (which makes a copy) to a folder within Windows to use outside of Picasa.

All the changes and the 'Star' are held in a file in each directory of pictures so the changes go with the pictures when they are backed up or moved to another machine. One can use a removable drive and the albums database stays intact when the drive is unplugged and the pictures return when it is plugged back in. The albums database is however machine specific so I use Tags to add IPTC tags for print, web and sometimes other designations for all the files I print or add to the web. The albums can then be usually be re-created by a search for the tags print and web and then selecting the time frame.

A big advantage of Picasa is that it has some very effective automatic procedures for enhancing pictures which can have almost unbelievable results - for example I took a picture in Hong Kong in mist/pollution which had all the detail washed out and a colour tint; Picasa's one click enhancement made it look as if it had been taken on a clear day. It has the usual rotation, crop and red eye functions as well as colour enhancement and a number of various filters. It has a very effective colour temperature adjustment which I used to correct all the pictures of posters taken indoors in natural light which slightly fooled the cameras white balance. The modified pictures can be exported to a folder which can be put on a CD or used by other programs. The originals are not modified and each change can be progressively undone. The only shortfall is that it has no perspective correction but a click on the folder symbol opens the directory containing the picture so a sophisticated editor such as GIMP can be accessed easily and Picasa automatically finds the changed or additional file.

On the subject of tricks you can get a full screen image by hovering the mouse over the photo and press CTRL then also ALT - it is full screen until you release a key - however slideshow does the same and has more features available.

A shortfall in Picasa is in importing pictures from a camera card where there is no ability to import all the pictures in a folder and sub-folders that I have found which means that they have to be imported folder by folder (approx 100 at a time) and each import has to be given a name (this may have been changed in the latest version). I have not tried direct from a camera. Picasa is however good in that it ensures that one does not import duplicates. The photos are imported into a folder structure in the home directory.

Google have announced that they do not intend to update beyond version 3.0 under Linux which is a shame as recent surveys show it as the most popular such package used under Linux despite not being 'native'. I have however found a site with information on how to do it yourself using Wine and the Windows version rather than the Google customised version which hides its use of Wine from you. See http://www.webupd8.org/2010/04/how-to-install-picasa-36-in-ubuntu.html

Wine: Wine Is Not an Emulator nor is it a graphics program however Wine does enable one to run many useful Windows Programs in Linux including graphics programs. It is under continuous development and the number of applications it can handle has increased dramatically including a number of the most demanding games. Because it is under continuous development one needs the latest version and that has obvious risks. I have used Wine to run two well behaved Windows graphics applications; Paintshop Pro 4.2 and Irfanview 3.95. I got the information from Ubuntu Hacks (See the book list below) and their website at Wine HQ. To loosely quote "Wine is an Open Source implementation of the Microsoft Windows API on top of the X windows system and Unix. Wine is a compatibility layer for running Windows programs and does not require Microsoft Windows, as it is a completely free alternative implementation of the Windows API consisting of 100% non-Microsoft code"

I successfully loaded two graphics programs namely Paintshop Pro 4.14 and Irfanview 3.95 which even gave me an Icon to run it on the Ubuntu Desktop.

There is more about Wine under Ubuntu 8.04 Hardy Heron in Fun with Ubuntu Linux - Wine

IrfanView (Graphics Browser, Editor, Slideshow Manager and Batch Converter) is another Freeware program written for Windows which I find indispensable. Fortunately it runs perfectly under Ubuntu Linux using WINE. I covered installing Wine and using it to install and run programs in Fun with Ubuntu Linux-Wine. IrfanView preserves and to some extent allows you to manipulate EXIF and IPTC data. It allows you to Browse graphic images, Create Slide Shows and save their contents, convert from one format to another and carry out powerful Batch commands with renaming. You can save most of the configurations for batch commands and load lists of Files. You can also carry out most of the functions one needs on photos including cropping, resizing, rotation, contrast brightness, gamma, compression, red eye and colour balance adjustments (only colour temperature and perspective are missing). You have a preview as all these adjustments are done from one control panel before finalising them. It was my preferred program for slideshows until I started to use Picasa.

Irfanview also allows you to join up pictures taken to give panoramic views, usually a feature of the Cameras software - I have not tried it yet.

Irfanview can be downloaded at www.irfanview.com and there are plentiful help files although most of it use is very intuitive. I am currently using version 3.95 under Windows XP and in Ubuntu running Wine 0.9.29. The latest version of IrfanView is 4.20 which is Windows Vista compatible and runs under Windows XP but the installer does nothing under Wine 0.9.29. 

Utilities and Techniques I have developed.

I have my system set up with a Partition accessible from both Windows and Ubuntu on our lap top. This has a folder called My Photographs into which I used to download my photographs from the two Canon Cameras using Canon's Zoombrowser software. I wanted to use F-Spot which is generally much more powerful but still keep the photographs on the shared drive. F-Spot has no problem cataloguing the existing photographs but its built in download under the version in the Ubuntu Dapper Drake distribution went to a folder Photos in my home folder. Furthermore when I download from the camera card it makes copies if I inadvertently download the same pictures twice or more times which I tend to do with big cards - one likes to back-up and have a look before taking 500 or more pictures. I have therefore written a single line terminal command to do an intelligent copy from ~/Photos to /media/hda6/My Pictures. It only copies the original, not the multiple copies and only overwrites if the file is a newer version. Having run the copy one should delete the photos in Photos using F-Spot and only then re-import them from My Pictures on the shared drive - this avoids having two copies in the database.

The command uses one of the most powerful of Linux's built in commands find which executes cp on all the matches.

cd ~/Photos; find -not -name *-[123456789].jpg -name *.jpg -exec cp --parents -u {} ~/Photos2 \;

This is how it works: First one changes into the Photos folder with a cd ~/Photos; so that some of the clever copy options work correctly later. Then find searches for all JPEG Images (-name *.jpg) except those ending in -1.jpg, -2.jpg, -3.jpg, -4.jpg etc (-not -name *-[12345].jpg ) which are the multiple copies. The find command has some very powerful options including -exec command \; which runs command on all the matches - if the string is needed one uses {} to represent it. My one liner executes the cp command with two options --parents means that the path to the files is included and the -u option to prevent overwriting unless the file being copied is more recent. For this demonstration I am copying into Photos2 in my home directory whilst in real life I am copying to My Pictures in my shared partition ie "/media/hda6/My Pictures"

This can be run from a launcher on the desktop but as it is a complex command it needs to be put in a BASH script file which by convention lives in the ~/bin directory. I called mine PhotoCopy and the file contents is:

#!/bin/bash
cd ~/Photos
find -not -name *-[123456789].jpg -name *.jpg -exec cp --parents -u {} ~/Photos2  \;

This needs to have the permissions set to execute for Owner as well as read and write (Right click -> Properties -> Permissions tab) 

Then right click anywhere on the Desktop -> Create Launcher - Give it a name such as PhotoMove, browse to ~/bin and highlight the file. You can click on Icon to select an Icon and OK and you are finished.

The new version of F-Spot which is available under Ubuntu Jaunty 8.04 allows one to select the download location but still imports duplicates so the technique above is still useful or could be modified to automatically delete duplicates rather than move the originals. I understand that version 5.0 available under Jaunty Jackalope 9.04 handles duplicates internally but does not always handle them as I would expect in folders in non standard locations.

The code to delete duplicates in folders below the folder containing my pictures from 2010 with extension .jpg or .JPG:

cd "/media/DATA/My Pictures/2010"; find -name *-[123456789].jpg -name *-[123456789].JPG -exec rm {} \;

Jhead - a very useful program for utilising the Exif data in Jpeg files

This is a command line driven program for manipulating the non-image parts of JPEG files with Exif data that most digital cameras produce. There is full information on the options on the Jhead web site but in summary:

jhead can extract from the EXIF data in a jpeg file the: jhead can modify jpeg files with EXIF information to:

I have highlighted what I believe are the most useful functions in Red.

A common problem when you have several cameras and want to combine the pictures and then sort in date order in a display (gallery) program can be solved several ways by Jhead. A single short command line will set the file time and date to match the time the picture was taken that is stored in the Exif data and slightly more complex commands can change the file names to include information from the date stamp.

I needed to set the file date stamp for my complete set of 'print files' to allow them to be sensibly displayed in the QuickPic program on the Android based box I have bought - this was a set of 120 folders containing about 24000 pictures - a daunting task which was done by running the following in the top level folder (which had the actual folders containg the print files one level below with all the files ending in .JPG)

jhead -ft */*.JPG

What was even more impressive was that it only took a couple of minutes to run. NOTE: Make sure you are in the folder otherwise you will change the date stamp on every .JPG file with Exif information in the whole system! Also check that you do not also have files ending in .jpg as it is case sensitive and may need to be repeated.

You can also adjust the internal exif date stamp to compensate for mistakes in reseting the time when moving time zones or between cameras on a folder basis before doing the above, however I recommend backing up the data well before experimenting with any of the very powerful Jhead functions! I used the following to correct because one camera had not been reset for daylight saving - the images had a different image sequence so I could just correct the pictures from a single camera in the folder(s).

jhead -ta+1:00 IMG_4*.JPG

An introduction to Video Editing under Ubuntu

Video has been one of the outstanding areas that I have not investigated up till now. I believed that there was little choice but to carry it out under Windows using my Pinnacle Studio Version 10 but I have been experimenting and it seems that there are now a number of tools which may add up to an adequate suite for capturing, editing and authoring DVDs from my Digital Video Camera which uses the small DVC tapes and a DV firewire connection. At present there is only one program which carries out all the stages but even Pinnacle Studio gives them almost complete separation in practice and only recently has it integrated the DVD authoring at all. There is a lot to be said for keeping the stages separate.

Firstly I think one needs to specify what one needs to be able to do to edit a Video.

Minimum Video Editing Requirements

The cutting edge - Repositories, building software and dependency nightmares.

Once you get involved in video processing you are entering into the land of geeks. Little of the software is stable and the libraries it builds on are changing by the day so the geeks build everything themselves from the floor up and there is a lot to build. One program alone I looked at installing had 40 dependencies on other programs and library routines and many of them would have their own dependencies. This is why Ubuntu, and Debian which it is based on, have a package management system which means that anything you try to install automatically installs the required dependences or will not install at all. The price one pays is that everything will lag by 6 months to year which is normally not a problem but in the case of Video then it may mean you are working harder. It is possible to add additional repositories which contain packages but it is possible to get some conflicts if they diverge in the updates, sometimes the actual versions are important both backwards and forwards. It is also possible to install individual packages via the package manager .deb files. In both cases a consequence is that an upgrade between versions ie Hardy Heron to Intrepid Ibex will probably fail unless the extra packages are removed first.

I have tried to chose programs which are supported within the Ubuntu package manager but that is not sufficient and initially I had to use .deb files from Getdeb.com and also added an unofficial Ubuntu Repository at Akirad ( http://akiradproject.net/repository) - now there are better alternatives. I did not want to break my main systems so everything I have tried has been on a special Wubi install of Intrepid Ibex on my HP DX2250 with AMD Athlon dual core 64 bit processor. The Wubi install was for AMD 64 bit and I have found that some programs are not supported yet for 64 bit. The one early attempt at trying to install a major program from scratch lead to a long period of attempting to satisfy dependences and eventually giving up and having to reload from scratch. My end system may not be optimal but I have something working to build on.

The Ubuntu Launchpad Personal Package Archive (PPA) Repositories

The Launchpad Personal Package Archive (PPA) service is a the way Ubuntu developers build and publish packages (Repositories) of their code, documentation, artwork, themes and other additions to Ubuntu . This is only part of Launchpad which allows users to report bugs, contribute code, submit translations and generally collaborate in an efficient and transparent fashion. PPAs enable developers to publish ready-to-install packages of their software directly to users through the mechanism of Repositories. Launchpad PPA repositories exist for a large number of programs including:

Video Capture

The first stage of producing a video is the capture of the video footage from the camera as a video stream or by transfer of files. A video stream needs to be broken up into individual scenes based on the time code and the file names need to be unique and such that they can easily identified. The best way is to include the date and time of the start of the clip in the filename - one can also include a reference to the labeling on the name, location etc.

Setting up Permissions and Groups for Firewire capture

Access by Firewire is made very difficult under Ubuntu because there are security issues - a loopback Firewire connection can allow an ordinary user root access if they are expert in hacking. This is not an issue for a home machine but is an issue in a business.

The first action you need to carry out is to add yourself to the video group which has the necessary permissions for accessing video streams. You can add yourself to group video by: System -> Administration -> Users and Groups -> Click to make changes and enter your password -> Manage Groups -> Select video and click properties -> Tick your username box -> OK -> Close -> Close.

You now need to set the permissions for device which is capturing the firewire (IEE1394) video stream which under Ubuntu Lucid Lynx is /dev/raw1394. This section is not required for Natty onwards as there is a new firewire stack and the device is mounted as /dev/fw1. with correct permissions for video devices.

First you have to know that this device is only loaded when a firewire device such as a video camera is plugged.

There are now two ways to proceed, the first is simple and quick, especially if you are happy using a terminal. You plug in and turn on your video device such as a camcorder, wait a few seconds for the driver to be loaded and then set the permissions by typing in a terminal:

sudo chmod 666 /dev/raw1394

You will be asked for your administrator password for security. This will be good until you reboot for video capture

The second way provides a permanent fix. For those using Ubuntu or any of its derivatives from 9.04 onwards you can modify the udev rules. First some background information; udev is the device manager for the Linux 2.6 kernel series and it dynamically mounts and unmounts devices such as USB drives and firewire devices as and when they are plugged in and removed. This is a rule based system that matches against exported values of the event generated by the kernel when devices are plugged in and removed and the properties of the device. The rules may possibly lead to naming and creating a device node and also running programs to set-up and configure the device. It is the mechanism that opens a file browser on the desktop when a usb drive is plugged in. We can use a simple rule to set up the group of the /dev/raw1394 device when it is plugged in:
  1. Find the udev rules folder - /lib/udev/rules.d in in recent versions . It may be in /etc/udev/rules.d in older versions prior to 9.04.
  2. If it is the standard place you can now open the file 50-udev-default.rules for editing using gedit by:

    sudo gedit /lib/udev/rules.d50-udev-default.rules
  3. Scroll down until you reach the firewire section and add this line under the existing entries:

    KERNEL=="raw1394", NAME="raw1394", GROUP="video"
  4. Reboot to set up new udev rules.

You should now be able to use the firewire device for control of and video capture from the programs following.

Kino and dvgrab

Kino is a well developed program which can be used as a simple editor in its own right but is most useful as an easy way to capture video from a camcorder and to split it up. It is available via Add/Remove.

The usual way to operate is to use Kino to control the video camera to set it to wherever you want to start and then capture video. It has three main capture formats, Raw DV and two formats where the audio is split out, format 2 is preferred if you plan to use Windoz at any time. The best way to split up the video is to make each video clip into a separate file and to add the time code to the name. This is also the recommended way for use by the Open Movie Editor. The only problem I had with Kino was that it had to be run from the command line as with Sudo (root privileges) to control the video camera. I overcame this by adding myself as user to the 'disk' group and also changing my authorisation to include additional privileges by System -> Administration -> Users and Groups. Note that disk does not appear as a group there for security reasons so a terminal command has to be used. The command line to do this is:

sudo usermod -aG disk $USER

The authorisations screen after changes looks like this:

Screen Shot Screen Shot

dvgrab

dvgrab is the command line program that Kino uses to do the control of the video camera and acquisition of the data from the video stream and is written and maintained by the Kino project. It can, and is also used by other programs for video capture such as Kdenlive. I prefer to do my video capture using Kino as it has the best GUI interface as you would expect as they were both written as an entity. The Kino interface gives excellent and reliable control of the video camera, good diagnostics including information on any dropped frames and easy setting up of the file naming and locations for the capture. You can view the video in parallel if you have a fast machine as well as viewing thumbnails of the captured video.

I have not used dvgrab directly from the command line but it could be the most reliable way on a very slow machine or for high resolution video. You can see all the options by looking at the manual file and some programs such as Kdenlive allow you to add options when they integrate dvgrab into their suite.

man dvgrab

Video editing

OpenMovieEditor

This is now very out of date as I have not used it for many years

This is the work of a single person and shows a lot of promise although I was not using the most up to date version as it was loaded via Add/Remove. It has a reputation for instability although not for losing data - I had few problems compared to Pinnacle Studio. It is slightly unusual in its interfaces and it took me a while to, for example, find out how to reload a saved project (They are all in a drop down list top right) and the way buttons work, for example the button to the right of the project name allows one to change the name - I never found out how to delete a project! There is an excellent tutorial at http://propirate.net/repos/openme-developers/doc/tutorial.html in which I found is an almost exact mimic of my requirements list!

Each scene has to be dragged individually to the timeline where it clicks to the end of the earlier project. You can not do a multiple selection (that I can find). You can drag either end to make cuts but then the scenes have to be rejoined. This means it is best to work on each clip as you bring it in - different from what I tend to do but probably a better way.

When you have finished the project needs rendering and here there is a real problem as there is only one preset which is for the Apple Quicktime format although other containers are possible there seems to be nothing which is very compatible and acceptable to turning into a DVD. This means that we will have to do a transcoding.

It is best to split the video into a number of sections of 15 - 20 minutes and then combine them as chapters at the DVD authoring stage. It makes editing quicker and less time is spent when changes are made as each stage runs at real time or slower in the case of transcoding.

Kdenlive as an alternative to OpenMovieEditor

Since I first wrote the video editing section Kdenlive (KDE Non-Linear Video Editor) has undergone a lot of development and now needs to be considered very seriously as an alternative to Open Movie Editor, especially as it may not need all of the additional stages, programs and transcoding required by Open Movie Editor.

I have loaded it onto the machine described above which already had a lot of extra programs and libraries installed and nothing seemed to be missing from it. The only downside is that it is designed to run on a KDE desktop, ie Kubuntu so the interface does not look as familiar, however I am already running other KDE programs such as the K3b CD/DVD burner so it should not be a problem - it will just bring in the parts of the KDE system that it needs. Kdenlive is available in Ubuntu/Mint but the versions are sufficiently out of date and the development cycle so fast that it is very desirable to add the Knedlive Launchpad PPA Repository so it can be loaded by the Synaptic Package manager and kept updated automatically.

Kdenlive's main features are:

The versions provided in Ubuntu are inevitably out of date as progress has been so fast and version updates are now on a bi-monthly basis so you really need to load their repository which also has some of the essential packages such as FFmpeg which need to be kept up to date.

Initial Evaluation:

I have tried out most of the essential editing functions and have input some video as a test.

I have now used it quite extensively and see no reason to look further.

DVD Production

The Video Editing stage may be the most important and challenging part of producing a Movie but turning it into a good DVD is equally important if one wants a professional looking result which is easy to show.

Standard DVDs are limited to about 2 hours of high quality video but can be longer if a little quality is sacrificed. This is longer than the attention span of most people watching home made videos and longer than justified by most subjects. A DVD is normally broken down into a number of separate Titles which can be selected from a menu at the start. Each Title can be broken down into Chapters and DVD players almost always allow one to move forwards and backwards by a Chapter (and Title) at a time. Chapters can also be reached from a menu and additional sub-menus can be used below or chained from the main root menu which is displayed when the DVD is first played.

What does this all mean for editing and producing our movies. I find it best to edit the movie in sections which each correspond to a separate Title. I try to limit each chapter to about 20 minutes in length and that corresponds to 4 Gbytes of 'raw' DV format or 750 Mbytes of MPEG2 which is the underlying video format on a DVD. There are advantages in keeping the MPEG2 file sizes to under a Gbyte because there are some limitations in file sizes on some systems although I have never had any problems with larger ones.

It is very important to limit the number of format changes, which are usually referred to as Transcoding. The best way is to make a single transcoding by doing the editing in the native format in which the unedited video exists which is DV (Digital Video) from a camcorder using Compact Digital Tapes (DVC) which last for an hour. Camcorders with hard disks and DVDs will most likely use an MPEGx format and Digital cameras use a variety of formats and will normally have a lower resolution than a DV recorder which is 720x576 - you may wonder how this matches to a 4:3 but this is because the pixels are defined to be rectangular with a 16:15 ratioso the result needs to be expanded to 768x576. You need to remember this when adding still pictures and make sure if the editor converts for you from 4:3 or if you need to convert yourself.

Returning to the main point I capture in one of the DV formats which all leave the video unchanged and only differ in how they handle audio. I set up the video editor to use the same format internally so no transcoding is required until the editing is completed. At this point the files are unchanged and there is a lot of information on all the effects and transitions. The production of the output is called Rendering and obviously there are changes wherever there are transitions or effects applied - this is the time to carry out the transcoding to the format used on a DVD so no further changes will be needed during Authoring.

This also gives me a very well defined interface between the video editing and DVD authoring steps. I can produce files with MPEG2 Video and AC3 audio streams combined into .vob files from Kdenlive each of which corresponds to a single Title on the final DVD which can be used without any need for further transcoding in the DVD authoring process. If the files are under a Gbyte they will not even need to be split up can be passed directly into the DVD image. This only leaves menu creation and addition of chapters to the DVD authoring tool. In an ideal world the information on the chapter positions and headings would be added in the editing operation and passed from editing to the authoring stage. This integrated approach is used in Pinnacle Studio but also implies you need to work with a whole DVDs worth of video simultaneously which is very slow and inefficient and I feel the split approach is quicker and more reliable.

Looking at the two Editors I have used. Open Movie Editor produces files which need transcoding to .vob files and has no built in DVD authoring. Kdenlive can render directly into .vob files suitable for an Authoring tool to pass directly onto a DVD and has a simple DVD authoring tool integrated into it in as much as it can be run automatically on the output from rendering as well as being used as a stand alone program. It is actually a front end to a command line program called dvdauthor which has been the stand tool for many years and offers almost anything you can conceive of using and a few other options as well. The GUI front end offered in the Kdenlive suite is fairly simple but convenient. However to get the best results there are other more flexible GUI programs based on dvdauthor and I prefer the one called DvdStyler for quick and professional results but Qdvdauthor is also very popular and may offer some extra facilities.

DVD Authoring - DVD Styler

I tried several programs to combine the output files from Open Movie Editor (or kdenlive) and this one suits me although it is not in any of the repositories at present. It has a very simple interface but enable one to make very complex menus if one desires - it is in a different class to the facilities built into Pinnacle Studio. It is actually a very good GUI front end to the almost universal dvdauthor command line program which assembles multiple mpeg program streams into a suitable DVD filesystem. I had difficulty making the other common alternative Qdvdauthor work reliable, or at all, whilst DVD Styler did exactly what I wanted to make simple DVDs in a very intuitive and user friendly manner. There are Ubuntu Packages on the Sourceforge DVDStyler site and a full manual on the DVDStyler site and recent copies of Ubuntu have a version accessible from Add/Remove but not always the most recent one. It is therefore advisable to update if you are using Ubuntu versions prior to Lucid Lynx

The latest DVDStyler for Jaunty is now available from a repository which can be added by:

System -> Administration -> Synaptic Package Manager -> System Menu > Software Sources > Third-Party Software; click add and paste this line in:

deb http://ppa.launchpad.net/dvdstyler-maintainers/ppa/ubuntu jaunty main

Repeat with the following line

deb-src http://ppa.launchpad.net/dvdstyler-maintainers/ppa/ubuntu jaunty main
close Software Sources and click reload. Note change jaunty to kalmic or your current ubuntu version as required.

After adding the repository, you also need to add the key for the repository to your system's list of trusted keys by typing the following command line in the terminal:

sudo apt-key adv --recv-keys --keyserver keyserver.ubuntu.com 0x4212adf59724ec2d

Now click the reload button of synaptic, search for and install the dvdstyler package or type the following line in a terminal:

sudo apt-get update && sudo apt-get install dvdstyler

Screen Shot Screen Shot Screen Shot Screen Shot

I have added a couple of screen dumps to show the interfaces and how easy and flexible it is to use

DVD Burning - Brasero or K3b

DVD Styler can burn the final DVD itself but there are advantages in splitting the stages in case you have a problem and do not wish to wait another hour for DVD Styler to redo a complex job. Brasero is the standard built in DVD burner but I have been using K3b (which can be installed by Add/Remove) for a while and it is very flexible and powerful so I tend to use it most of the time - it does a reliable burn and a good DVD and CD validation but note it was designed for a KDE desktop so has a somewhat unfamiliar style of interface when you first meet it.

DVD playback - Xine, Gxine, Mplayer and VLC

Xine is used internally by DVD Styler and at one time was the most reliable for DVD playback so it is best to install it via System -> Administration -> Synaptic Package Manager and search for and install ui-xine gxine and mplayer. The Gnome front end to Xine is called Gxine which seems to be better for playing DVDs with menus.

VLC how has the high ground as a player and has the greatest number of options. It is also available for mobile devices. Mint 18 has xplayer, a fork of Totem player as its standard video player (just Videos in the menus) - more basic than VLC

For playing commercial encrypted DVDs with any of the above one needs to also install libdvdcss2 libdvdread3 libdvdnav4

Summary of packages to install using the Synaptic Package Manager for video processing and DVD playback

Tricks of Video Editing under Ubuntu

Transcribing old VHS videos to DVD

I, like many people have a lot of old VHS videos which are rapidly becoming unreadable. I still have two VHS video recorders, one of which is a JVC HR-DVS1 which handles the miniature DV tapes for camcorders as well as VHS and SVHS tapes and can automatically dub between them and can also output a Digital Video stream on a firewire connection whilst 'dubbing' from VHS to DV tape - better still you do not even have to be recording to get the stream. This means that I can very easily save my VHS to hard drive in DV format in exactly the same way as from a DV tape and use the same techniques to edit (hopefully very little more than trimming) and convert them to a convenient format and/or save on DVD, either singly or with several grouped on the DVD. The overall process involves several steps.

  1. Convert the VHS tapes to AV files in a DV format - note this gives circa 14 Gbytes per hour of VHS video!
  2. Join the AV files together and tidy them up at start and end.
  3. Output the material from the VHS tapes as AV files in a format and quality for storing and burning onto a DVDs.
  4. Author the DVDs

1. Capture the DV stream from the VHS tape to firewire converter. (Kino)

The HR-DVS1 produces an output on the firewire output only when the VHS tape is running so it is easiest to use a Capture program with a GUI rather than the command line utility dvgrab (which almost every program uses internally). I have described setting up and using Kino above so I will not go into a lot of detail other than to note any differences in configuration which are used. The configuration used is set by Edit -> Preferences and I show below the two tabs where I have changed settings.

Screen Shot Screen Shot

The important setting in this case is the max file size as every hour of DV format video from the VHS tape takes 14 Gbytes. It is therefore essential to split this up into a series of files, preferably under 1 Gbyte for compatibility with all other programs. I have chosen 800 Mbytes which is just under 4 minutes of video. Normally the Auto Split Files option keeps the file size down as it is unusual for any video clip from a camcorder to be over 4 minutes. The Timestamp is in this case the start of the recording as there is no timestamp in the the data as there is from a camera. DV AVI Type 2 is the most generally accepted format as it has a separated out audio stream. Raw DV is also acceptable to Kdenlive and I tend to use it as the files are a tiny bit smaller. The settings above have also been chosen to be compatible with using Kino/Kdenlive with a camcorder so they do not need to be changed.

This is also the screen where you set up the default location for your captures. The convention is to have the captured files names starting with capture and the timecode is added on to give a unique filename. You can also add into the name a designation for the tape you are capturing to make life easy when you come to find them to use in the Video editor see above.

Screenshot of Kino Display Preferences Screenshot of Kino Display Preferences

This tab shows the set up for the monitor display for Kino. This may vary depending on the speed of your machine and if you want to do anything else whilst capturing video. Firstly there are three methods of interfacing to the display. GDK should work on any machine but is slow. XVideo uses more of your hardware in the video card but may not work on all machines - use it if it works. You should definitely tick the 'Drop video frames as needed' unless you want to find out how fast your system actually is! If you want to do other things and/or are using GDK then disable preview during capture to minimise the chance of dropped frames. There is a display during capture of dropped frames - if you drop any frames turn off the preview and do nothing else and also disable any screen savers.

Having set everything set up Kino to Capture (right hand tabs) just rewind the VHS and start it playing. Once it had got to the program (or close to the start) then click Capture and that's it until the end when it is best to stop the capture rather than let the tape get to the end. You can tidy up the start and end points at the next step using the editor before you save it to the final file or DVD. CHECK for any dropped frames before you leave the Capture screen.

Editing the Captured Video and transcoding it for storage or adding to a DVD. (Kdenlive)

The editing should be minimal compared to producing a movie from a camcorder and will normally just consist of trimming the start and end, removing any adverts and perhaps adding an extra title so I will not go into any great detail. I have normally been working on several VHS conversions at one time so I have tended to put them all into a single 'project' and alternated the track for each set of captures comprising the whole VHS. I have added a Guide with the name of the VHS at the start of each so I can easily render them separately.

Rendering is the only area which differs slightly from working on a movie. A VHS tape is normally much longer than a movie you produce - showing a two hour home movie is a good way to lose friends and a lot of wine. Commercial and those you have recorded from TV programs also tend to be quantised in half hour lengths. This means we need to do a bit of extra optimisation if we are going to efficiently fill up DVDs as the default quality level set in Kdenive for a VOB (mpeg2) file for use on DVDs gives a maximum length of DVD of 115 minutes. This uses a data rate of 5000k baud which is a good choice for high quality output from a source such as a DV tape and camcorder. In our case the input is of a lower quality and we need to squeeze the data rate to 4500k with little visible difference to get 126 minutes or to 3000k (186 mins) for an adequate quality from a three hour tape. It is worth trying some experiments to see the differences and I recommend doing a recording at 1000k so you can easily see all the artifacts and look out for them at you chosen rates. The size is not directly scaled with video data rate as there is a fixed overhead equivalent to about 110k from the audio.

So how do we set this up in Kdenlive? Let us first look at the Rendering Window:

screenshot from Kdenlive screenshot from Kdenlive

Firstly the settings are for a DVD in PAL, the Output File is defined and I am rendering a part of the project defined by Guides added to the timeline. Looking at the choices under PAL one sees the 4 preset choices that the system has provided and one would normally use one of those - the 2 pass options take twice as long to render but give better optimised use of your bandwidth. The date rate used for all of these is 5000k baud which gives an excellent quality but only allows a total time of 115 minutes on a DVD. I have added 4 more options of my own which are the ones with a heart beside them. I found finally that double clicking on a system preset opens a window as below where you can create your own options. I have shown two of mine below and have highlighted the changes I made and put a description in the Profile name. Do not make changes you do not understand as getting the optimum parameters for libavcodec seems to be a black art!

screenshot from Kdenlive screenshot from Kdenlive Screen Shot Screen Shot

You should now have a mpeg2 file with a .vob extension which can be used directly by a DVD authoring program without needing any further transcoding to change the format or data compression and it can also be played on your computer by most players such as Xine, Gxine and Movie Player. They will be much shorter than the original capture at a tad under 2 Gbytes/hour instead of over 12 Gbytes/hour. If you have created ones for 3 hour DVDS then you may also want to archive a higher quality version before deleting the DV files and bining the VHS tapes. I also recommend scanning or photographing the front and back of the VHS case.

NOTE: During subsequent inspection of the videos I find that the output has not changed the size from 690 x 552 when using Kdenlive versions 0.9x - this seems to be corrected in versions 16.0.8 where the editing of profiles is no longer required.

There is a simple Authoring facility built into Kdenlive but I normally use a more sophisticated program DVDStyler so I can use buttons rather than text.

Turning it into a DVD - DVDStyler

Again I have covered the use of DVDStyler quite fully above so all I need to talk about is a few tricks to make an easy life for DVDs from VHS tapes. Firstly you do not need a complex menu structure - just entries to each VHS or each program if they held several should be sufficient. You may want to make your menus enter at the start of the program rather than the preliminaries, copyright notices etc. I set up so each Title program corresponds to a program and may set up so each is in a separate file or use Guides to mark the start of each.

I normally do not find a need to set up specific chapters but it is useful if they are set up at say 6 minute intervals so one can quickly move forward and back when playing. This can be done on the Configuration -> Settings window - see below.

screenshot from DVD Styler screenshot from DVD Styler

Now when a file is added it has chapters spaced at 6 minute intervals automatically set up. If you need to edit them to match a button you want to add for a specific time then you can double click the file (Title) to get the screen below:

screenshot from DVD Styler screenshot from DVD Styler

You can now edit chapter times. Note also the tick box for 'do not remultiplex/transcode' this is why we needed to get the settings correct in Kdenlive - it saves a lot of time and keeps the quality unchanged. This is also where you decide where to go at the end of each chapter - here it goes back to the top level menu but you could chose to jump to the next Title or to a chapter within it. This is also the window where you can add additional files to a chapter.

The final setting up is in DVD -> Properties where the Video Quality ought to be Auto to avoid transcoding or set to the quality you want if you have to change the quality and transcode because you got it wrong earlier or have other files to add which are not in a suitable format.

screenshot from DVD Styler screenshot from DVD Styler

I usually just generate an image of the DVD first and then burn it using k3b or the CD/DVD burner of you choice. I give the file the same name as earlier. See below:

screenshot from DVD Styler screenshot from DVD Styler

Now all that is left is to burn the DVD by right clicking on the .iso and choosing your favourite burner.

Digital TV: Digital Video Broadcast - Terrestrial (DVB-T)

I expected this to be difficult under Linux but it has proved very easy if you have a Digital Video Broadcast - Terrestrial (DVB-T) USB stick receiver which is supported by the kernel and most are. I bought an Artec T14BR which was a bargain at 7dayshop.com and installed Kaffeine and it all worked within minutes 'out of the box' - the longest period was scanning for channels. There is an Electronic Program Guide (EPG) in Kaffeine which shows every program on your channel, with details, for 8 days and allows single click adding to your recording list. If you are watching live TV in Kaffeine a single click starts or stops recording. Most commands have keyboard shortcuts (hot keys) making it very easy to use

DVB-T Receivers - Examples

I have selected the two USB sticks I own as examples. The Artec T14BR was bought after checking the list of Supported DVB-T USB Devices on the LinuxTV Wiki . The list is comprehensive - I counted 191 entries for supported hardware with brief notes. The Pinnacle PCTV DVB-T stick was bought many years ago and proved more of a challenge but did work.

Artec T14BR USB Mini Digital TV Receiver Receiver (DVB-T)

This is the stick I am currently using and I bought it from 7DayShop.com for under £15. It came with a very effective flat antenna and remote control which I have not bothered to get working. As far as TV goes it is plug and play with all the drivers required built into the kernel - you do not even need to extract the firmware which needs to be loaded every time it is turned on. It is accessible from all standard TV software - I have tried it with Kaffeine and Me-TV. Some details from the Artec information are:

Artec - T14BR USB 2.0 Driver Details

The stick is supported in the kernel used by Lucid and I have also used in under Jaunty. The information following is largely for reference if you are just going to use it for TV on your PC in standard programs such as Kaffeine or Me TV where no special action is required beyond doing a channel scan.

This is the output from lsusb:

pcurtis@triton-ubuntu:~$ lsusb
Bus 004 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
Bus 003 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
Bus 002 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
Bus 001 Device 004: ID 05d8:810f Ultima Electronics Corp.
Bus 001 Device 002: ID 1058:1003 Western Digital Technologies, Inc.
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub

This shows it has been identified with usb id 05d8:810f and as a product of the Ultima Electronics Corp (Artec)

The following are terminal outputs from the end of a dmesg:

pcurtis@triton-ubuntu:~$ dmesg
......
[37790.684173] usb 1-1: new high speed USB device using ehci_hcd and address 6
[37790.817295] usb 1-1: configuration #1 chosen from 1 choice
[37791.089076] dib0700: loaded with support for 13 different device-types
[37791.089508] dvb-usb: found a 'Artec T14BR DVB-T' in cold state, will try to load a firmware
[37791.089524] usb 1-1: firmware: requesting dvb-usb-dib0700-1.20.fw
[37791.143344] dvb-usb: downloading firmware from file 'dvb-usb-dib0700-1.20.fw'
[37791.351908] dib0700: firmware started successfully.
[37791.852075] dvb-usb: found a 'Artec T14BR DVB-T' in warm state.
[37791.852304] dvb-usb: will pass the complete MPEG2 transport stream to the software demuxer.
[37791.852976] DVB: registering new adapter (Artec T14BR DVB-T)
[37792.068654] DVB: registering adapter 0 frontend 0 (DiBcom 7000PC)...
[37792.287302] DiB0070: successfully identified
[37792.287488] input: IR-receiver inside an USB DVB receiver as /devices/pci0000:00/0000:00:1d.7/usb1/1-1/input/input12
[37792.287650] dvb-usb: schedule remote query interval to 50 msecs.
[37792.287662] dvb-usb: Artec T14BR DVB-T successfully initialised and connected.
[37792.287877] usbcore: registered new interface driver dvb_usb_dib0700

This shows that the device has been correctly identified as needing a dib0700 driver and the actual device detected within the driver as an Artec T14BR DVB-T, one which is supported. It shows that the device needs firmware which is already available (unusual - you usually have to lift it from a windoz driver) and it has been loaded. It also shows that the remote control has been identified and will be polled every 50 msecs. I have had a few tries to get the remote working without success (see below) with the aim of using the lirc software to convert to inputs for other programs without success although it may be possible.

One hint I found was that there is an onboard amplifier which can be turned on by:

gksudo gedit /etc/modprobe.d/dvb-options.conf
Add: options dvb-usb-dib0700 force_lna_activation=1

If dvb-options.conf does not exist it will create it and it will automatically be used. I have noticed no difference so the latest driver/firmware may already do so.

Remote Control

The remote control should be recognised by the standard Ubuntu kernel. Check it by running:

cat /proc/bus/input/devices

Look for an appropriate entry:

cat /proc/bus/input/devices
...........
I: Bus=0003 Vendor=05d8 Product=810f Version=0100
N: Name="IR-receiver inside an USB DVB receiver"
P: Phys=usb-0000:00:10.3-2/ir0
S: Sysfs=/devices/pci0000:00/0000:00:10.3/usb1/1-2/input/input5
U: Uniq=
H: Handlers=kbd event5
B: EV=3
B: KEY=14afc336 284284d 0 0 0 4 80058000 2190 40000801 9e96c0 0 900200 ffd

The important bit is the event5 line. This shows us which input device the remote is using.

We can install a utility to test it by the IR remote

sudo apt-get install evtest

which gives information on the codes which are programmed in allows testing by pressing buttons on your remote:

sudo evtest /dev/input/event5

This gave me:

pcurtis@triton-ubuntu:~$ sudo evtest /dev/input/event5
Input driver version is 1.0.0
Input device ID: bus 0x3 vendor 0x5d8 product 0x810f version 0x100
Input device name: "IR-receiver inside an USB DVB receiver"
Supported events:
Event type 0 (Sync)
Event type 1 (Key)
Event code 0 (Reserved)
Event code 2 (1)
Event code 3 (2)
Event code 4 (3)
Event code 5 (4)
Event code 6 (5)
Event code 7 (6)
Event code 8 (7)
Event code 9 (8)
Event code 10 (9)
Event code 11 (0)
Event code 41 (Grave)
Event code 52 (Dot)
Event code 55 (KPAsterisk)
Event code 102 (Home)
Event code 103 (Up)
Event code 105 (Left)
Event code 106 (Right)
Event code 108 (Down)
Event code 111 (Delete)
Event code 113 (Mute)
Event code 114 (VolumeDown)
Event code 115 (VolumeUp)
Event code 116 (Power)
Event code 119 (Pause)
Event code 128 (Stop)
Event code 139 (Menu)
Event code 158 (Back)
Event code 164 (PlayPause)
Event code 167 (Record)
Event code 168 (Rewind)
Event code 173 (Refresh)
Event code 207 (Play)
Event code 208 (Fast Forward)
Event code 210 (Print)
Event code 223 (Cancel)
Event code 226 (Media)
Event code 352 (Ok)
Event code 354 (Goto)
Event code 355 (Clear)
Event code 358 (Info)
Event code 363 (Channel)
Event code 365 (EPG)
Event code 370 (Subtitle)
Event code 375 (Screen)
Event code 377 (TV)
Event code 385 (Radio)
Event code 386 (Tuner)
Event code 388 (Text)
Event code 389 (DVD)
Event code 392 (Audio)
Event code 393 (Video)
Event code 398 (Red)
Event code 399 (Green)
Event code 400 (Yellow)
Event code 401 (Blue)
Event code 402 (ChannelUp)
Event code 403 (ChannelDown)
Event code 405 (Last)
Event code 407 (Next)
Event code 410 (Shuffle)
Event code 412 (Previous)
Testing ... (interrupt to exit)

However the commands from the remote were not recognised. I tested the program using the event for the mouse and could see all the buttons so I concluded that the remote for the TV card did not work and gave up at that point. I even tried with an earlier version of the firmware which was a fraction better in that a couple of key strokes were detected but it was very random. I suspect some additional option is required which I have not found. If it had been working I would have then needed to install lirc and configure it to convert and pass the commands to other programs - there are very full instructions at http://parker1.co.uk/mythtv_ubuntu2.php

If you want to turn the remote off then you can use an option for the driver by adding

options dvb_usb disable_rc_polling=1

to the configuration file we created earlier in /etc/modprobe.d or creating a new one by:

gksudo gedit /etc/modprobe.d/dvb-options.conf
Add: options dvb_usb disable_rc_polling=1

So in conclusion - the driver for providing the TV data streams etc works 'out of the box' but there is little chance of getting the remote to work.

Pinnacle PCTV DVB-T USB Stick for TV reception

My Pinnacle PCTV DVB-T USB Stick was bought to go with the Toshiba Laptop to give TV on our narrowboat before we first got involved with Linux so it must be over 4 years old. The software from Pinnacle was always flaky under Windows. I never really thought there would be any possibility under Linux so had made no serious effort to get it working until a few weeks ago. I did get it working for a while but it then failed and did not work under Windows or Linux. By this time I already had an Artec T14BR which is known to work with Linux on order.

TV Display and Recording Software

Kaffeine

Kaffeine is a general purpose media player oriented towards supporting TV playing and recording. It has a very clear Electronic Program Guide which allows you to schedule any program to be recorded in the future and provides a guard band of -5 and +10 minutes by default. It is easy to search for channels and chose the ones you want. It will use any DVB-T USB stick which is loaded by the kernel or custom drivers. Kaffeine is installed by the Synaptic Package Manager and searching for kaffeine - it is not in the Software Centre as it is optimised for the KDE desktop but I prefer it to the simpler Me-TV. You should also install dvb-apps using the Synaptic Package Manager or you will not have a list of local transmitters available for you to scan. There are 100 Digital Transmitters in the UK. Kaffeine needs to have some configuration information before you can scan for channels and watch TV - you have to:

You need to install dvb-apps using the Synaptic Package Manager if you do not have a list of transmitters as noted above.

Kaffeine is reluctant to do another automatic scan if you change region often like we do on the boat. I have found you need to:

If you do not carry out the procedure you can only update channels corresponding to the transmitter you are using. There may be several transmitters in you list and you have to tune to each one.

Mostly you can control Kaffeine by keys as well as the drop down menus and control icons under the channel list. F is important as it toggles in and out of full screen. G gives the Onscreen Guide which also allows you to schedule recording with a guard band either end.

The best way to find the nearest transmitter and its direction from an OS map Reference is Wolfbane's UK digital TV reception predictor . The OS map reference can be obtained from a https://www.freemaptools.com/find-british-national-grid-reference-from-map.htm or directly from an app on your phone such as Grid Reference Free OS

You can also use a program called w-scan which can be installed by the Synaptic Package Manager to create a scan list suitable for Kaffeine - enough details to get one going are at http://hftom.free.fr/phpBB2/viewtopic.php?t=178

Me-TV

A simple TV player with the ability to record programs which is well integrated into gnome. I run it as well as using kaffeine. The earlier versions had a signal strength meter but that seems to have been dropped.

I have added the development ppa repository for me-tv by:

sudo add-apt-repository ppa:me-tv-development/ppa && sudo apt-get update

This will enable me to update to the latest version (currently 1.2.4) when other updates take place.

Note: The developer has changed recently and the last time I tried to install me-tv the repository had not been updated and did not cover the latest versions of Ubuntu and Mint

MythTV

MythTV turns your computer into a complex general purpose media centre and can be installed from a LiveCD or added onto an existing Ubuntu installation. A MythTV system consists of backends, for recording and storing TV, and frontends, for watching the TV. These can be on multiple computers or combined onto a single PC. You can use several DVB-T sticks and satellite receivers. It records TV from a tuner card onto the hard disk of a PC. It uses TV listings data to schedule recordings allowing it to automatically record whole series. I am not planning to use it but a lot off useful information is available from users including the definitive MythTVGuide from Garry Parker This includes a description of how to get a remote control working which could be modified to control kaffeine instead of MythTV.

Sharing Audio, Video and Pictures over a network including using DLNA/UPnP

Introduction and the Humax HRD Fox T2 500 Gb

We have bought a fancy High Definition Video Recorder which offers many additional features including the ability to access video, photo and sound files over a network from a DLNA (Digital Living Network Alliance) server via cable or Wifi.

The Humax is very choosy about the file formats it will play - it will play many formats but the number where the 'controls' to allow stepping forwards and back and pausing are strictly limited and undocumented. The good news is that all my 'captures' from my old Video8 tapes (360 Gbytes) are in one of the few formats that plays. The bad news is that all the files ready for authoring into DVDs are .vob files which will not play but I have a way of transcoding them which does not degrade the video stream which is copied unchanged into a new container with a converted audio stream - the cost is 80 Gbytes extra storage. Standard jpeg files can be put into a slide show which adds another 67 Gbytes. Audio is less important for the Humax at present but tests show that audio .mp3 files are handled perfectly and there is a sophisticated indexing allowing selections to be made.

The MediaTomb Server

I am using MediaTomb as a server for this data. MediaTomb is an open source Universal Plug and Play (UPnP) MediaServer complient with the required sections of the V 1.0 specification. A UPnP server allows one to stream digital media, usually from a local hard disk on the server, through ones home network for listening to/watching on a variety of UPnP compatible devices. Humax is advertised as being DLNA (Digital Living Network Alliance) complient MediaRenderer (player). In practice these 'standards' are very similar so the Humax HRD Fox T2, being a DLNA complient player, ought to be able to browse, request and play audio, video and photos over a network from a UPnP server such as MediaTomb.

MediaTomb is somewhat differently to most programs - it can either be started either as a daemon running in the background when the machine is booted up or as a program started from a normal launcher or in a terminal. In both cases it is then almost 'invisible' - there is not even an icon in the status panel to show it is running. Whlst running, access is purely through the network and control is through a web interface connected to the IP address and port on which it is serving the media. Selection of the media for the database can therefore be from any machine on the network. In a similar manner 'Players' browse over the network the media which is available in the database and select what to play. In practice MediaTomb can serve far more media formats than the Humax can play. The initial configuration is almost entirely through a single options file after which you point your browser at the relevant Address and Port (usually port 49152) where you can view your local filesystem and chose which files and folder structures from which you will make selected classes of media available using a tree view of the database and the file system. The local database can updated on a regular timed basis or whenever a file or folder is added, removed or changed. The initial or timed scanning takes a significant time.

The MediaTomb for Ubuntu is set up slightly differently to the standard version and is configured to run as a Daemon which is started during the init.d sequence and runs as user mediatomb. This default gave some problems to me because I have my media files on a removable drive and:

I could not access files on the drive because of permissions when it was a NTFS volume as it was mounted with read permissions only for the current user. I could get round that by formating and mounting as a ext3 drive and setting the permissions to give read access to Others which allowed it to work in Daemon mode.

The daemon was started before the drive was mounted so it could not be found initially and the database was cleared. This needed the daemon startup to be removed from the the init.d sequence and the program started manually from a terminal or launcher. This also meant that the server was running as the same user as the one who had mounted the drives so now all drives and contents are accessible

Preventing Ubuntu 11.04 version of MediaTomb being run automatically as a daemon.

The version supplied with Ubuntu has been set to run as a Daemon and is started during the boot process automatically once it has been installed and run the first time. This may not be what you want so the first stage is to prevent the Daemon being started. The script which starts the mediatomb Daemon during the boot process is in /etc/init.d in early versions of Ubuntu. A quick and dirty way to stop it being run in that case was to change it from being executable by changing the permissions:

sudo chmod 0644 /etc/init.d/mediatomb

Onlatter versions of Ubuntu Mediatomb uses the Upstart method of starting the mediatomb daemon service and a different waf inhibiting startup is required - see http://askubuntu.com/questions/19320/recommended-way-to-enable-disable-services . I edited the Service file by

sudo gedit /etc/init/mediatomb.conf

and commented out the start on line as below

# Commented out following line to prevent automatic start of mediatomb daemon
#
start on (local-filesystems and net-device-up IFACE!=lo)
stop on runlevel [!2345]
respawn

There are other methods involving a single command possible but I could understand what I was doing in the method above and it worked!

 

The Mediatomb Web Interface

The Mediatomb Web Interface is accessed through your normal browser. You can provide the address in several ways:

  1. 192.168.2.234:49152
  2. http://computername:49152
  3. file:///.mediatomb/mediatomb.html

The first way using a standard IP address and port will change depending on the address allocated to your computer and will change everytime your router is turnered on and off. The second way uses you computer name depends on the software being enabled on your machine to resolve it. The third uses a file which is kept up-to-date by mediatomb itself so is ideal for the machine running Mediatomb.

You have two views options for the left panel - filesystem and database. You choose the folders to include in the database in filesystem view using the icons at the top of the right panel which produces a form to fill in - choose the Inotify Basic Recursive options. Then check what you have in Database view - it may take a while to complete the scan and fill it all in. I have my photo and video files on one USB drive and the audios on another.

MediaTomb Configuration files

The MediaTomb runs in slightly different ways and has the configuration files in a different place depending on whether it is run as a Daemon during system bootup or has been run by a user.

It is now time to make some changes in the configuration file which is ~/.mediatomb/config.xml when run by a user.

gedit ~/.mediatomb/config.xml

All the important changes are at the top of the file and are in red below:

<?xml version="1.0" encoding="UTF-8"?>
<config version="2" xmlns="http://mediatomb.cc/config/2" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://mediatomb.cc/config/2 http://mediatomb.cc/config/2.xsd">
<!--
Read /usr/share/doc/mediatomb-common/README.gz section 6 for more
information on creating and using config.xml configration files.
-->
<server>
<ui enabled="yes" show-tooltips="yes">
<accounts enabled="no" session-timeout="30">
<account user="mediatomb" password="mediatomb"/>
</accounts>
</ui>
<name>Matrix</name>
<port>49152</port>
<udn>uuid:b4669fad-3f8e-4d30-94a7-fec9bb4395e9</udn>
<home>/home/pcurtis/.mediatomb</home>
<webroot>/usr/share/mediatomb/web</webroot>
<storage>
<sqlite3 enabled="yes">
<database-file>mediatomb.db</database-file>
</sqlite3>
<mysql enabled="no">
<host>localhost</host>
<username>mediatomb</username>
<database>mediatomb</database>
</mysql>
</storage>
.........

  1. The first change is to enable the user interface via the web - it was not enabled on my version by default.
  2. The second change is optional and just changes the name of the server which is seen from the clients.
  3. The third line is an addition and fixes the port used to be 49152 rather than the first available starting at 49152 - this also prevent multiple copies on increasing ports. This will need to be different if you have servers on several machines

The choice of database is set to sqlite3 by default - this is the best option but note you need to have sqlite3 installed so check using the Synaptic Package Manager and install the sqlite3 package if required.

Using media on a USB drive in MediaTomb.

USB drives are normally mounted late in the booting process and after the user has logged in. They are mounted by the file manager - Nautilus in Ubuntu and Thunar in Xubuntu and the permissions only allow access by the user who was first logged in. This gives two problems when a Daemon is started during boot process - firstly the drive is not mounted by the time Mediatomb has been started and the database is first accessed and is then cleared. Secondly Mediatomb is run in the mediatomb group and drives mounted this way do not have the permissions for the mediatomb daemon to access them.

Auto-mounting a USB hard drive during system boot.The first approach I tried was to make an entry in the filesystem table for the USB drive in the same way that a fixed drive can have an entry. I have covered this for fixed drives in Ubuntu All Together - Sharing, Networking, Backup, Synchronisation and Encryption under Permanently Mounting a shared file-system - use of fstab (Advanced). The trick in this case is to make sure the mounting uses the UUID (Universal Unique IDentifier) which every disk has in the file system table /etc/fstab.

This worked well for me on Ubuntu 10.04 and 11.04 and I could get an early mount with the correct permissions for MediaTomb to run at startup in daemon mode however there are some serious disadvantages. One Must have the USB drive plugged in or the boot-up needs intervention to continue when the drive is not found and even if you then plug it in it will not mount through nautilus (only checked in U10.04) although it is still possible to mount it through the mount command in a terminal. I also discovered to my surprise that fstab does not like mount point folder names which have spaces even if you quote them in fstab (again only checked in U10.04). I have used 'defaults' and auto detection for the drive so my filesystem table (/etc/fstab) now looks like:

# /etc/fstab: static file system information.
#
# Use 'blkid -o value -s UUID' to print the universally unique identifier
# for a device; this may be used with UUID= as a more robust way to name
# devices that works even if disks are added and removed. See fstab(5).
#
# <file system> <mount point> <type> <options> <dump> <pass>
proc /proc proc nodev,noexec,nosuid 0 0
/dev/sdb6 / ext4 errors=remount-ro 0 1
# /home was on /dev/sdb7 during installation
UUID=a411101c-f5c6-4409-a3a1-5a66de372782 /home ext3 defaults 0 2
# swap was on /dev/sdb5 during installation
UUID=a7c16746-1424-4bf5-980e-1efdc4500454 none swap sw 0 0
/dev/fd0 /media/floppy0 auto rw,user,noauto,exec,utf8 0 0
# /dev/sda5: LABEL="DATA" UUID="47859B672A5D9CD8" TYPE="ntfs"
UUID=47859B672A5D9CD8 /media/DATA ntfs nls=utf8,uid=pcurtis,gid=pcurtis,umask=0000 0 0
#
# USB Drive /dev/sdf1: LABEL="DE_MEDIA" UUID="4690477f-effa-4d60-a6f1-243cfdc7e065" TYPE="ext3"
UUID=4690477f-effa-4d60-a6f1-243cfdc7e065 /media/DE_MEDIA auto defaults 0 0

I have added information about this proceedure for USB drives into Permanently Mounting a shared file-system - use of fstab (Advanced) .

Starting Mediatomb with a delay after login is complete

The solution I have adopted is to start Mediatomb by using a script which I add to the users Startup Applications using System Applications -> Startup Applications -> Add and filling in the boxes.

The reason to use a script rather than just entering the program name is so I can add a long delay before the program is started to ensure the drive is mounted. I keep all my scripts in ~/bin and have added that folder to the PATH Environment Variable so it can be called just by name but if you do not want to modify ~/.bashrc by adding the liine PATH=$PATH:~/bin to the end you can specify the full path in the Startup Application details.

My additions to the end of ~/.bashrc so I can run scripts in ~/bin are:

.........
# enable programmable completion features (you don't need to enable
# this, if it's already enabled in /etc/bash.bashrc and /etc/profile
# sources /etc/bash.bashrc).
if [ -f /etc/bash_completion ] && ! shopt -oq posix; then
. /etc/bash_completion
fi
# Additions to add path to ~/bin for local scripts
PATH=$PATH:~/bin

The script itself is very simple and the following can be copied into ~/bin/delayed_mediatomb using gedit.

gedit ~/bin/delayed_mediatomb

And paste in the following

#!/bin/bash
# Script to copy start mediatomb as a daemon after a 40 second delay.
# Lives in ~/bin/delayed_mediatomb where it must be given execute permissions
# and ~/bin should have been be added to to #PATH in ~/.bashrc
# by adding the line
# PATH=$PATH:~/bin
#
sleep 40
mediatomb -d

The file needs to be made executable

chmod 777 ~/bin/delayed_mediatomb

Now when I start the machine mediatomb is started and all my settings are preserved as the USB drives which contain the media files are alredy mounted.

More about Media Streams/Files

Media files are very large so that it is impossible or impracticle to transfer them in their entirety between machines before playing them. The information is therefore requested as required and streamed from the server to the player. Depending on the server/player and type of file/stream it may, in order of complexity, be possible to pause, step forwards and backwards or fast forwards and back. Again, as a generalisation, it is easier to control a local file than one provided from a server. Most video media contains a number of streams, the audio, the video and sometimes other streams with, for example, subtitles or alternative audios. Almost all media is heavily compressed and there are many alternative formats for the audio and video streams in the same way that pictures may be in jpeg, png gif, etc formats. Within, say a video format, there are a huge number of options - some obvious like the size 720x578 or 1920x1080, the frame rate and whether the stream is interlaced. Others are more complex such as average, maximum and minimum bandwidths when variable bandwidth encoding is used and some which a normal person has great difficulty in understanding at all. If you want to get an idea of the complexity of the modern standards have a look at http://en.wikipedia.org/wiki/H264 for the latest video format H264. Again there are different audio formats and parameters for the streams, Dolby, Stereo, MP3, OGG etc with variable and fixed bandwidths. The audio and the video streams are not, of course, independent and need to be played in synchronisation so they are in a container - what you normally see is just a container such as AVI, MPG, TS, VOB. Startup Applications

Converting between Audio and Video Formats

There are many options for file convertors, some are gui based and are often part of a video player such as VLC but the best way is often to call the underlying conversion programs which usually means use of a terminal. perhaps the best of all is ffmpeg which many other programs use. This is what I have been using to convert VOB (.vob) files (containers) to MPG files (containers with extension .mpg) and whlst changing the format of the audio stream but leaving the video untouched as any changes seemed to lead to a stuttering video playback.

ffmpeg -i "/media/WD400EXT3/VOB Files for Authoring/The Thorn Birds Volume 4.vob" -f mpeg -vcodec copy -acodec mp2 -ab 384k -ar 48000 "/media/WD400EXT3/VHS Copies for Humax/The Thorn Birds Volume 4.mpg"
This is actually a very simple example! The input parameters for the file are automatically detected from the input file specified by -i filename and normally the container or output format is specified by the extension of the output file, in this case .mpg. In this case the output'container' is also forced by the -f mpeg parameter. The video codec is set by -vcodec copy to a straight copy and the audio codec by -acodec mp2 The audio sample rate is set by -ar 48000 and the badwidth by ab 384k .

The important bit of the output follows:

....................
Input #0, mpeg, from '/media/WD400EXT3/VOB Files for Authoring/The Thorn Birds Volume 4.vob':
Duration: 02:21:05.53, start: 0.500000, bitrate: 3273 kb/s
Stream #0.0[0x1e0]: Video: mpeg2video, yuv420p, 720x576 [PAR 16:15 DAR 4:3], 6000 kb/s, 25 fps, 25 tbr, 90k tbn, 50 tbc
Stream #0.1[0x80]: Audio: ac3, 48000 Hz, stereo, s16, 192 kb/s
Output #0, mpeg, to '/media/WD400EXT3/VHS Copies for Humax/The Thorn Birds Volume 4.mpg':
Metadata:
encoder : Lavf52.64.2
Stream #0.0: Video: mpeg2video, yuv420p, 720x576 [PAR 16:15 DAR 4:3], q=2-31, 6000 kb/s, 90k tbn, 25 tbc
Stream #0.1: Audio: mp2, 48000 Hz, stereo, s16, 384 kb/s
Stream mapping:
Stream #0.0 -> #0.0
Stream #0.1 -> #0.1
Press [q] to stop encoding
...................................

Transcoding in MediaTomb

MediaTomb can transcode on the fly but I have not yet made it work satisfactorily. The relevant sections of the set up file follows:

....
<mappings>
<extension-mimetype ignore-unknown="no">
<map from="vob" to="video/mpeg"/>
<map from="mp3" to="audio/mpeg"/>
....
....
<transcoding enabled="no">
<mimetype-profile-mappings>
<transcode mimetype="video/mpeg" using="ffmpegcopy"/>
<transcode mimetype="video/x-flv" using="vlcmpeg"/>
<transcode mimetype="application/ogg" using="vlcmpeg"/>
<transcode mimetype="application/ogg" using="oggflac2raw"/>
<transcode mimetype="audio/x-flac" using="oggflac2raw"/>
</mimetype-profile-mappings>
<profiles>
<profile name="oggflac2raw" enabled="no" type="external">
<mimetype>audio/L16</mimetype>
<accept-url>no</accept-url>
<first-resource>yes</first-resource>
<accept-ogg-theora>no</accept-ogg-theora>
<agent command="ogg123" arguments="-d raw -o byteorder:big -f %out %in"/>
<buffer size="1048576" chunk-size="131072" fill-size="262144"/>
</profile>
<profile name="vlcmpeg" enabled="no" type="external">
<mimetype>video/mpeg</mimetype>
<accept-url>yes</accept-url>
<first-resource>yes</first-resource>
<accept-ogg-theora>yes</accept-ogg-theora>
<agent command="vlc" arguments="-I dummy %in --sout #transcode{venc=ffmpeg,vcodec=mp2v,vb=4096,fps=25,aenc=ffmpeg,acodec=mpga,ab=192,samplerate=44100,channels=2}:standard{access=file,mux=ps,dst=%out} vlc:quit"/>
<buffer size="14400000" chunk-size="512000" fill-size="120000"/>
</profile>
<profile name="ffmpegcopy" enabled="no" type="external">
<mimetype>video/mpeg</mimetype>
<accept-url>no</accept-url>
<first-resource>yes</first-resource>
<hide-original-resource>yes</hide-original-resource>
<agent command="ffmpeg" arguments=" -i %in -f mpeg -acodec mp2 -ab 384k -ar 48000 -y -vcodec copy %out"/>
<buffer size="576000000" chunk-size="1024000" fill-size="240000"/>

</profile>
</profiles>
</transcoding>

The transcoding can be seen to start when mediatomb is run in a terminal and a mpeg or vob file accessed but fails shortly into ffmpeg - the problem may be in ffmpeg rather than MediaTomb

djmount - a UPnP Client for Linux which mounts as a FUSE File system

I have had difficulty in identifying a client for UPnP for Linux. The best seems to be a very old utility called djmount which mounts as a Linux filesystem (using FUSE (File System in User Space)) the media content of compatible UPnP AV devices. The Audio and Video content on the network is automatically discovered, and can be browsed as a standard directory tree.It is available through the Ubuntu Software Center or Synaptic and needs the FUSE utilities to be installed. Mine seemed to be installed by default on 10.04 and 11.04 but check. You need to run in a terminal and to specify a mount point which needs to be set up with permissions to access for all users. I created a new folder /media/UPNP_APV and set up the permissions. This is best done in a terminal.

sudo mkdir /media/UPNP_APV
sudo chmod 777 /media/UPNP_APV

I could then mount the UPnP server by

djmount /media/UPNP_APV

The server should now be visible in /media/UPNP_AVP and as we are in a terminal it is worth checking:

xxxx@xxxx:~$ ls -l /media/UPNP_APV
total 1
-r--r--r-- 1 root root 7 2000-01-01 11:00 devices
dr-xr-xr-x 7 root root 512 2000-01-01 11:00 Triton
xxxx@xxxx:~$:~$ ls -l /media/UPNP_APV/Triton
total 2
dr-xr-xr-x 9 root root 512 2000-01-01 11:00 Audio
dr-xr-xr-x 4 root root 512 2000-01-01 11:00 PC Directory
dr-xr-xr-x 7 root root 512 2000-01-01 11:00 Photos
dr-xr-xr-x 5 root root 512 2000-01-01 11:00 Video
xxxx@xxxx:~$

We can see the server called Triton is mounted and the standard folders from MediaTomb are below it and have sensible permissions.

It should be possible to add the program to the autostart list but it will need a delay as for mediatomb to ensure the Wifi network is up and running - I use 35 seconds as 20 seconds occasionally failed.

Extra Material and work in progress August - September 2014

The following is some current development work which has yet to be fully polished and integrated.

Video Capture Device - EasyCAP USBTV007 version by Fushical

I have been rationising my old computers and want to get rid of my old AMD desktop but it is tthe only machine left which could take an old analog AV input from VHS and Video8 tapes using a Pinnacle Card under Windows XP so I wanted to find a suitable input device capable of working under Linux before consigning it to the loft or bin.

There are a number of USB devices available cheaply (mine was £6.29) and branded EasyCAP which contain a number of different chips. Few of the sellers give any indications which. All have some support under Linux and the sources of information include:

LinuxTV has a good reference article on the various versions of EasyCAP, how to identify them, and what drivers are available under Linux

TV viewing and recording solutions for Linux is a blog with a lot of useful suggestions, although written for the STK1160 device based EasyCAP much is applicable to capturing from all the devices - especially the page on the mtvcgui program.

The USBTV007 version is arguably the least desirable as the drivers currently available do not support sound capture although all that means in practice is that one needs an extra cable from the video source to the sound input on the computer. You also need a very up to date Kernel to get out the latest usbtv module to provide of the box support - the 3.13.0-24 kernel in Ubuntu 14.04 Trusty and Mint 17 Quiana is fine and supports PAL video unlike earlier versions but not audio. There is also a module under development which you can build which adds audio which I have not tried.

I have done most of the testing so far using VLC which is a very good player but also has transcoding and capture built in. There are a few 'tricks' to using it for capture

MiniDV Video - Kino and Handbrake

Using Kino to Capture the DV Video

I usually capture video from the MiniDV Video Recorders using Kino as DV AV Type 2 with Auto Split Files and Add Time Stamp set. See earlier description with screen dumps. The video files in all the DV formats are huge (12 Gbytes/Hour) so I have been looking at converting to use a MP4 container with an efficient H.264 video stream and AAC audio which reduces it down to circa 2.5 Gbytes/Hour.

Using Handbrake to Convert the DV AV Video to H.264 for efficient storage

This can be done directly by FFMpeg as mentioned earlier but it is difficult to get optimum parameters, some of which may need to be optimised for the form of video, especially if one wants to deinterlace at the same time. It is capable of converting a folder full of DV captures at one time - Kino is set up to put each recording into a separate file with a timestamp in my default set up so this is essential as an hour tape often contains over 200 scenes.

Unfortunately the latest versions of Handbrake in the Ubuntu repositories do not handle MP4 because Ubuntu do not want to distribute software with MP4 support because it is patent-encumbered. Installing the "restricted-extras" package solves the problem for many proprietary formats, but this alone is not sufficient for Handbrake and furthermore HandBrake does not support external encoders so it is not possible to add an encoder to HandBrake at runtime. All the encoder libraries are built in and not dynamically linked/loaded at runtime and Ubuntu has unhelpfully recompiled its version of Handbrake for the repositories without the MP4 support. The last time Handbrake was available from a PPA rather than the Ubuntu Repositories was for Ubuntu 13.04 raring but the appropriate stebbins handbrake-releases PPA (from the author) has been updated to a reasonably recent version of Handbrake. So how do we fudge things to install from a PPA for an older version of Handbrake?

Fudges to install a Linux version of Handbrake which will convert to MP4.

and you will have a version of Handbrake with MP4 support.

Converting a folder with Handbrake.

The commands are not immediately obvious and one uses Source to select a Folder then Queue -> Add All Queue then Queue to check and start.

Note: The title is the same for all the 'clips' in a folder and by default is the filename of the first clip - use the Tags tab to set the title to the folder name instead. This needs to be done for every folder.

Deinterlacing

Deinterlacing is required to remove the comb artifacts from interlaced video when there is any motion in the scene. Deinterlacing however reduces the quality so it's not good to run a deinterlacer all the time and it can really slow down an encode. Often content is mixed, cutting back and forth between interlaced and progressive video. This is important on my JVC video camera which automatically switches to progressive when taking still shots on the video tape. Even in interlaced content, filtering is only necessary when either something in the scene or the camera moves, producing those combing artifacts.

Handbrake has a clever alternative called decomb. The decomb filter looks at each pixel of each frame of a video. It then only deinterlaces frames that show visible amounts of combing. This means you can just run the decomb filter all the time and it'll take care of everything.

The Deinterlacing is set up from Picture Settings -> Filters and Decomb and Default should be used unless you know that the video was recorded entirely with progressive video.

Other Changed Settings

I have been using the Normal preset but have tried increasing the x264 preset slider to Medium under the Video tab which increases the encoding time considerably but gives a very compact result. The 5 seond long progressive 'stills' compress down to as little as 250 kbytes !

Saving Settings

You can save your modified settings as a new preset using the buttons on the bottom right.

Changing from a .m4v to .mp4 extension for all video clips in a folder.

By default Handbrake adds a .m4v extension to MP4 containers. That is the only difference and it is perfectly acceptable to just change the extension of the files. The following command line string will change all files in the folder where it is run and starting with DV with a .m4v extension to .mp4. The DV start matches my MiniDV capture files and acts as a protection in case one calls it from the wrong folder!

for i in DV*.m4v;do eval mv \"$i\" \"$(basename "$i" m4v)mp4\";done

Concatenating all video clips in a folder - Install of ffmpeg is required

FFmpeg has a utility to concatenate video files fast and losslessly but Ubuntu has switched to a branch from ffmpeg called avconv which does not include this. However from Mint 18 onwards ffmpeg is back in the repositories and only needs to be installed using the Software Manager or the Synaptic poackage manager

For completness I have left the instructions for earlier versions of Mint. I initially reinstalled ffmpeg from a the jon-severinsson/ffmpeg PPA which may cause problems I am yet to discover as not only avconv and ffmpeg overlap but also a number of libraries were updated which avconv uses. To quote Jon Severinsson:

This is FFmpeg, from ffmpeg.org. Recent Debian and Ubuntu packages feature Libav (from libav.org), a prominent FFmpeg fork, instead. You should be safe upgrading from Libav to FFmpeg, as the FFmpeg developers regularly pull from the Libav git tree, and thus have all Libav features, as well as several of their own.

However, both Libav and FFmpeg periodically break backward compatibility in order to more easily provide new features, so applications built against one version will not always work with the next:

FFmpeg 0.7 is backward compatible with FFmpeg 0.6 and 0.5, as well as Libav 0.6 and 0.5; FFmpeg 0.10 is backward compatible with FFmpeg 0.8 and 0.9, as well as Libav 0.7 and 0.8; FFmpeg 1.2 is backwards compatible with FFmpeg 1.1, as well as Libav 9; FFmpeg 2.1 is backwards compatible with FFmpeg 2.0; FFmpeg 2.2 is backwards compatible with Libav 10.

For trusty (14.04) this PPA provides FFmpeg 1.2 (My note: and updates libav from 6.9 to 7.12).

To install (at your own risk) using a terminal:

sudo apt-add-repository ppa:jon-severinsson/ffmpeg
sudo apt-get update
sudo apt-get install ffmpeg

File concatenation using FFmpeg

Installing ffmpeg allowed me to test out the concatenation. The two stages are to create a file containing the list of files to concatenate and then use that. If the files all have the same containers and streams this is fast and lossless. See https://trac.ffmpeg.org/wiki/How%20to%20concatenate%20(join,%20merge)%20media%20files

printf "file '%s'\n" ./*.m4v > DVlist.txt ; ffmpeg -f concat -i DVlist.txt -c copy AllClips.mp4

Note that I have used an intermediate file rather than piping as I found it gave problems and also it is useful to have the file list at the end. Also note that the output file has a different extension for the same container to avoid duplicating if the command is run a second time.

Using Kdenlive

Having converted most of my MiniDV tales first to DV Clips using Kino and then to H.264 Clips using Handbrake, the next stage was to put them into the Kdenlive Video editor.

Input to Kdenlive

This did not go well initially and I could not put them into the Project Tree (List of the clips for use) in bulk and then I had further troubles adding them in bulk to the time line and other Kdenlive crashes. There turned out to be two problems.

Rendering the output file.

The standard output profile for H.264 had defaults of 6000 kbaud for video and 384 kb for audio and furthermore the audio sample rate was 48 kbaud. This differs whilst the input from the MiniDV camera was 32 kbaud with a 160 kb bandwidth. I therefore created a custom render profile - the audio and video bandwidths are easily changed but the audio sample rate meant that I had to edit the control string as well as the the values in the boxes. I also took the opportunity to add extra 'allowed options' for the audio and video rates to experiment with the ideal settings. See the edit window below:

Screen Shot Screen Shot

Note the additions to Video Bitrates of 1500 and 3000 and the addition to the Audio Bitrate of 160. A cut and past of the string in the Parameters box with the sole change to the audio sample rate to 32000 shown in red follows:

f=mp4 hq=1 acodec=aac ab=%audiobitrate+'k' ar=32000 pix_fmt=yuv420p vcodec=libx264 minrate=0 vb=%bitrate+'k' g=250 bf=3 b_strategy=1 subcmp=2 cmp=2 coder=1 flags=+loop flags2=dct8x8 qmax=51 subq=7 qmin=10 qcomp=0.6 qdiff=4 trellis=1 aspect=%dar pass=%passes

I initially chose this default video bitrate of 3000 to approximately match the input files but I have been experimenting down to 1000 (with 2 pass rendering) with no degradation I can see on the laptop screen but the check will be when one watches a variety of scenes on a large 'TV' screen via my Android media box - I plan a series of comparative tests. A video rate of 1000 gives under 550 Mbytes per hour and 1500 circa 750 Mbytes and hour which seems a good compromise at present for the initial viewing tests. The Chillblast Defiant laptop is fast enough to use a two pass render which took 20 mins per hour of video a video rate of 1500 when kdenlive is allowed to use the whole 8 threads for encoding.

Installing the Latest Kdenlive

The Versions in Mint 18 are recent enough currently to not merit adding the PPA.

The latest kdenlive can be obtained from the https://launchpad.net/~sunab/+archive/ubuntu/kdenlive-release PPA. One change in kdenlive 0.9.10 from the current repository (version 0.9.6 in Trusty) is a rationisation of the render profiles to match those in mlt (the framework used internally). I do not yet know how the performance varies yet. The extra profiles I set up for the Project and for Rendering seem to survive the update on my Desktop but I have not updated yet on the Chillblast Laptop.

Status of VHS, Video8 and MiniDV Projects

VHS Videos

This has been completed. The size has been typically cropped to 590x552 to remove artifacts which, with a pixel aspect ratio of 16x15, retains the aspect ratio at 4x3 to a high degree of accuracy and the video was de-interlaced so it is progressive at ~25 frames/sec and the conversion to H264 has also led to a conversion of the audio to AAC at ~160 kb/sec stereo. In an ideal world these should have been transcoded to a standard 720x576 etc with lower bandwidth but they should play on most machines without any problems. The files are stored as MPEG-4 (.m4v) files in sub-folders of My Video/ VHS Videos with H264 video and AAC Audio. Not all are on my machines at any time with the masters being on a 2 Tbyte USB drive mounting as SA_D .

Video8

All the video8 tapes from (1990 to 1999) were transfered to MPEG2 format files using Pinnacle Studio 10 running under Windows XP. They are mostly one file per 90 minute tape - 118 MPEG files totalling 386 Gbytes and 135 Hours of video. These are stored on my DATA drive in sub-folders of My Video/Video8 Capture/ when being edited but not all are on my machines at any time with the masters being on a 2 Tbyte USB drive mounting as SA_D . The total on SA_D occupies 390 GBytes.

They have then been converted using Handbrake to .m4v files which are considerably smaller and will be used for input to the editor. These are stored on my DATA drive in sub-folders of My Video/Video8 Capture/Video8 Videos at between 1 and 1.4 Gbytes per hour. There are 117 videos which occupy 184 GBytes total.

MiniDV

Total 101 Tapes. All have been captured using kino (wrapping dvgrab) in DV AV type 2 files, one file per scene with time and date code in the filename. The files have however been limited to 800 mbytes so there are cases where there are a series of 4 minute blank clips at the end because all the tapes were 'striped' (recorded with black before use to get round time code problems on the JVC camera and some single frame or other very short files also possibly from a known camera problem. The tapes were read on either a JVC HRDVS1 or the original JVC camera. Some tapes have 'noise' during the first few minutes and cleaning heads or changing camera does not seem to affect the capture appreciably but there is little which can not be used in editing. In general the DV capture folders have not been changed. The files take up approximately 14 Gbytes per full one hour tape and most have not been retained, certainly not in one place.

All the folders of captures have then been transfered to H.264 (main profile) and deinterlaced using Handbrake with aac audio 32 kb sample rate and 160 kb bandwidth. The sizes vary as there are many 'stills' which consist of 5 seconds of progressive ouput which typically compresses down to only 250 kbyes whilst a similar video would be ten times larger. These are stored on my DATA drive in sub-folders of My Video/MiniDV Videos at between 1 and 1.4 Gbytes per hour with the whole 105 files taking up 152 GBytes.

All have been concatenated into single files for easy playing up till when they are edited.

Sony Video Recorder Image Qualities

Sony Models with the PAL color system typically have the following settings:

High definition image quality (HD):

MP4:

Standard definition image quality (STD):

In practice I have only used the MP4 setting and the files have all been added to My Video an the standard folder structure of YYYY/MM/DD I use for My Pictures but not merged in with the pictures.

Conclusions

Digital Photography: Ubuntu Linux has all the facilities to handle photographs that I had under Windows, in fact the main photo and graphics editors I use (Paintshop Pro, IrfanView, Picasa and GIMP) all run both under Linux and Windows. Under Windows I used the Canon Zoombrowser program provided with the camera to 'download' pictures from the camera/card whilst under Linux there is a powerful program called F-Spot for downloading and photograph management. The problem is then one of deciding which programs to use - initially I used F-Spot to download, carry out minor corrections and cataloguing and selection for printing and web use followed by InfanView in batch mode to reduce the resolution, increase compression and rename them for web use. This is the best approach for purists who wish to avoid use of any proprietary software which is not completely Open Source and free of commercial considerations. I have now compromised and use F-Spot for Downloading and then use Picasa for normal trimming/corrections and selection using the albums feature - for major corrections I use GIMP before final tuning in Picasa. I then export the 'print album' from Picasa to a folder on a CD for Printing and the 'web album' to a folder I save for use by InfanView in batch mode to reduce the resolution, increase compression and rename them for web use.

Video Camera Support: Less refined and integrated than packages such as Pinnacle Studio under Windows but useable. I initially took three DVC tapes through to DVDs with Titles, Chapters and Menus on a reasonable powerful machine using Open Movie Editor. I am now using Kino for Video Capture, Kdenlive for editing and rendering and DVDStyler for DVD authoring and have completed a number of projects. Handbreak provides easy file conversions and deinterlacing and VLC is used for playback and some capture and conversions from my USB EasyCap capture device. I have also uploaded to YouTube.

Digital TV: The Linux programs are equivalent to those provided with Windows. My Artec T14BR installed immediately and using Kaffeine you can be watching TV in 15 minutes or less if you know which transmitter you are using and do not have to do a full automatic scan for channels. The only shortfall is that it seems to be a nightmare to make any remote controllers work.

Before You Leave

I would be very pleased if visitors could spare a little time to give me some feedback - it is the only way I know who has visited, if it is useful and how I should develop it's content and the techniques used. I would be delighted if you could send comments or just let us know you have visited by Sending a quick Message to me.

Link to W3C HTML5 Validator Copyright © Peter & Pauline Curtis
Fonts revised: 28th April, 2021