Tuesday, January 16, 2018

What to do after installing VirtualBox

This post is part 2 of the 3-part series on creating a virtual machine using VirtualBox hosted on a Debian machine. Part 1 focuses on the installation of VirtualBox and the guest OS (FreeBSD). Part 2 addresses things that you should do after installing VirtualBox. The upcoming part 3 deals specifically with accessing USB flash drives from the guest OS.

Install extension pack

The base VirtualBox package is missing some important functionalities: support for virtual USB 2.0 and 3.0 devices, host webcam passthrough, Intel PXE boot, and disk image encryption. To obtain those functionalities, you need to download and install the VirtualBox extension pack.
Note that the extension pack operates under a much stricter license than the GPLv2 of the base VirtualBox package. Specifically, the VirtualBox Extension Pack Personal Use and Evaluation License (PUEL) 'is a free license for personal, educational or evaluation use'. For commercial use, you do have to pay a fee to Oracle.
The procedure to install the extension pack is as follows:
  1. Download the VirtualBox extension pack to your host machine.

    The extension pack is available from the VirtualBox website. The package is universal for all host and guest OSes. However, you should use the same version for both the base and extension pack. In other words, when you upgrade your base, you should also upgrade the extension pack.
  2. Run VirtualBox Manager.
  1. Add extension pack.

    From the File menu, click Preferences. Then, select Extensions in the side panel.

Click the + icon to specify the location of the extension pack file (Oracle_VM_VirtualBox_Extension_Pack-5.2.4-119785.vbox-extpack).


At the end, click Install. Before it begins installation, you will be prompted to accept the software license, and to enter the administrative password.

Bridge host & guest networks

By default, the virtual machine is NATed (not bridged) to your host network. Consequently, the FreeBSD guest is on a different subnet than the host. For example, my FreeBSD guest has a 10.0.0.2 IP address while my Linux host has a 192.168.1.49 IP address. Practically, it means that I cannot ssh to the guest OS from anywhere in my host network. To make that possible, I need to bridge the host network and the guest. The procedure is below (the VM does not have to be powered off for you to change networking).
  1. Open the VirtualBox Manager, and click Settings.
  2. Click Network in the side menu.
  1. Select Bridged Adapter in the Attached to dropdown menu.

Enable copy & paste between host & guest

I administer the FreeBSD guest extensively from its virtual system console running on the Linux host. By default, the clipboard is not shared between the host and the guest systems. That means I often need to manually re-enter commands on the guest OS rather than copy and paste from the host. To share the system clipboard (and save yourself plenty of laborious typing), follow the steps below:
  1. Open the VirtualBox Manager, and click Settings.
  2. Select General, and click the Advanced tab.
  1. Select bidirectional for both Shared Clipboard and Drag and Drop.
Note that, to share the clipboard, the FreeBSD-specific guest additions package (emulators/virtualbox-ose-additions) must also be installed. The instruction for that is already detailed in the Post-installation section of part 1 of this series.

Related posts

Tuesday, January 2, 2018

axi-cache: a new search tool for Debian packages


Debian has no shortage of tools when it comes to searching for packages. Revered oldtimers include apt-cache, apt, and apt-file. axi-cache is the new kid on the block. This post explains what is novel about axi-cache and how to use it.

First, axi-cache needs to be installed and initialized as follows:

# apt-get install apt-xapian-index
# update-apt-xapian-index
The index /var/lib/apt-xapian-index is up to date

The basic axi-cache search is syntactically very similar with the aforementioned search commands:

$ axi-cache search browser
930 results found.
Results 1-20:
100% chromium-driver - web browser - WebDriver support
99% chromium - web browser
99% ruby-browser - browser detection for Ruby
97% libwwwbrowser-perl - Platform independent means to start a WWW browser
96% libhtml-display-perl - module for displaying HTML locally in a browser
96% gcu-plugin - GNOME chemistry utils (browser plugin)
96% python-zope.browser - Shared Zope Toolkit browser components
96% python3-zope.browser - Shared Zope Toolkit browser components
96% chromedriver - web browser - WebDriver support transitional package
96% mythbrowser - Small web browser module for MythTV
96% chromium-widevine - web browser - widevine content decryption support
96% swfdec-mozilla - dummy package for transition to browser-plugin-gnash
95% python-livereload - automatic browser refresher
95% mozilla-plugin-gnash - dummy package for renaming to browser-plugin-gnash
95% python3-livereload - automatic browser refresher (Python 3)
95% qupzilla - lightweight web browser based on libqtwebkit
95% python-livereload-doc - automatic browser refresher (documentation)
95% firefox-esr - Mozilla Firefox web browser - Extended Support Release (ESR)
95% ipig - integrating PSMs into genome browser visualisations
94% epiphany-browser-data - Data files for the GNOME web browser
More terms: refresher livereload safer stable refresh browsing webdriver
More tags: field::religion culture::dutch uitoolkit::gtk field::chemistry use::browsing web::browser interface::x11
`axi-cache more' will give more results

A similar search using apt-cache returns 962 results with the top 20 results being:

389-admin - 389 Directory Administration Server
libds-admin-serv0 - Libraries for the 389 Directory Administration Server
xul-ext-adblock-plus - advertisement blocking extension for web browsers
ajaxterm - Web based terminal written in Python
alevt - X11 Teletext/Videotext browser
alice - Web browser (WebKit or Gecko) based IRC client
xul-ext-all-in-one-sidebar - sidebar extension for Firefox
node-almond - minimal AMD API implementation for use in optimized browser builds
pilot - Simple file browser from Alpine, a text-based email client
ams - Realtime modular synthesizer for ALSA
amule-gnome-support - ed2k links handling support for GNOME web browsers
libjs-angularjs - lets you write client-side web applications as if you had a smarter browser
libjs-animate.css - cross-browser library of CSS animations
libapache2-mod-upload-progress - upload progress support for the Apache web server
apachedex - Compute APDEX from Apache-style logs
xfonts-kapl - APL fonts for A+ development
artemis - genome browser and annotation tool
libjs-asciimathml - Library to render high quality mathematical formulas in a browser
aspectj - aspect-oriented extension for Java - tools
auctex - integrated document editing environment for TeX etc.

Note that, unlike apt-cache, axi-cache returns, by default, only the top 20 hits. You can see the entire result set by specifying the --all option (e.g., axi-cache --all search browser). Alternatively, you can page through the results by running the following command after the initial search.

$ axi-cache more

In general, axi-cache returns more relevant results than apt-cache. The latter implements a rudimentary grep-like search by matching regular-expression text patterns against the package name and description of a package. Unless you have some idea of the package's name, an apt-cache search often returns many irrelevant results, as indicated by the above example. In contrast, axi-cache can rank the search results by relevance with the help of the Apt Xapian Index(axi). This index is a database of package meta-data which includes much more than just a package's name and description. To examine what is indexed, run the following command:

$ axi-cache info
...<snipped>...
Plugin status:
aliases enabled, up to date (430 days, 15:47:18.356149 older than index)
app-install disabled
apttags enabled, needs indexing (6 days, 23:59:10.729973 newer than index)
cataloged_time enabled, needs indexing (6 days, 23:59:10.729973 newer than index)
debtags disabled
descriptions enabled, needs indexing (6 days, 23:59:10.729973 newer than index)
relations enabled, needs indexing (6 days, 23:59:10.729973 newer than index)
sections enabled, needs indexing (6 days, 23:59:10.729973 newer than index)
sizes enabled, needs indexing (6 days, 23:59:10.729973 newer than index)
template enabled, up to date
translated-desc enabled, needs indexing (1 day, 10:56:58.643851 newer than index)
...<snipped>...

As an aside, you can update the Apt Xapian Index by executing the following command as root:

# update-apt-xapian-index

The most distinctive feature of the index is its use of tags (apttags). These tags categorize a package by predefined facets such as role, protocol, suite, culture, use, works-with, etc.

Suppose you are multi-lingual and you are looking for packages that install, say, Chinese fonts. You can quickly identify the packages you need using the following command.

$ axi-cache search x11::font and culture::chinese and role::data
13 results found.
Results 1-13:
100% fonts-arphic-bkai00mp - "AR PL KaitiM Big5" Chinese TrueType font by Arphic Technology
100% fonts-arphic-bsmi00lp - "AR PL Mingti2L Big5" Chinese TrueType font by Arphic Technology
100% fonts-arphic-gbsn00lp - "AR PL SungtiL GB" Chinese TrueType font by Arphic Technology
100% fonts-arphic-gkai00mp - "AR PL KaitiM GB" Chinese TrueType font by Arphic Technology
100% fonts-cwtex-fs - TrueType Font from cwTeX - FangSong
100% fonts-cwtex-heib - TrueType Font from cwTeX - HeiBold
100% fonts-cwtex-kai - TrueType Font from cwTeX - Kai
100% fonts-cwtex-ming - TrueType Font from cwTeX - Ming
100% fonts-cwtex-yen - TrueType Font from cwTeX - Yen
100% ttf-wqy-zenhei - transitional dummy package
100% xfonts-intl-chinese - international fonts for X - Chinese
100% xfonts-intl-chinese-big - international fonts for X - large Chinese
100% xfonts-unifont - PCF (bitmap) version of GNU Unifont
More terms: chinese fonts truetype cwtex font koanughi cwttf
More tags: made-of::font culture::taiwanese role::dummy role::app-data culture::greek culture::korean culture::russian

Note that axi-cache supports logical operations such as AND, OR, and NOT.

The following table compiles a non-exhaustive list of common facets and example values. For a complete list, please click here. To make a tag, join the facet and the value using 2 colons, e.g., role::program.

Facet Values
admin filesystem, forensics, monitoring, power-management, virtualization
culture chinese, latvian, russian
field arts, astronomy, finance, mathematics, medicine, statistics
game arcade, board, card, fps, mud, puzzle, rpg, sport, toys, typing
implemented-in c, php
interface 3d, commandline, graphical, shell, x11
network client, configuration, server, service, vpn
office finance, groupware, presentation, spreadsheet
protocol ip, ipv6, smtp, webdav
role app-data, data, debug-symbols, devel-lib, documentation, kernel, metapackage, plugin, program, shared-lib, source
security antivirus, authentication, cryptography, log-analyzer
suite bsd, debian, eclipse, emacs, gnome, gnu, kde, mozilla, mysql, openoffice, openstack, postgresql, xfce, xmms2
system cloud, embedded, laptop, mobile, server, virtual
uitoolkit gtk, motif, ncurses, qt, sdl, tk, xlib
use analysing, browsing, calculating, chatting, checking, compressing, configuring, converting, downloading, driver, editing, entertaining, filtering, gameplaying, learning, login, measuring, monitor, organizing, playing, printing, routing, scanning, searching, simulating, storing, synchronizing, transmission, typesetting, viewing
works-with archive, audio, calendar, db, file, font, image, logfile, mail, network-traffic, spreadsheet, text, unicode, video
x11 font, screensaver, theme, window-manager

In closing, axi-cache is a good search tool for Debian packages because it usually gives you more relevant results. One caveat is warranted, however. Not all Debian packages have tags defined, which is particularly true for packages that are downloaded from third-party non-standard repositories.

Thursday, December 21, 2017

Using VirtualBox to build FreeBSD VM on Debian

Many virtualization software (aka hypervisors) can run on the host operating system of Linux. Notable open-source software include VirtualBox, KVM, and Xen. As part 1 of a 3-part series, this post outlines how to 1) install VirtualBox on Debian 9 (Stretch), and 2) build a FreeBSD virtual machine. The upcoming part 2 and 3 cover additional post-installation tasks, and how to access USB flash drives on the guest OS.


About VirtualBox


In 2010, Oracle acquired Sun Microsystems and, as a result, took over the ownership of several open-source technologies including VirtualBox and mySQL. Subsequently, VirtualBox was rebranded as Oracle VM VirtualBox, but the software has remained free and open-source.


VirtualBox, like KVM, is a type-2 hypervisor, meaning that it runs as a process on the host OS (Linux). In contrast, XEN is a type-1 hypervisor which runs directly on top of the bare-metal hardware. Type-1 hypervisors have an inherent advantage over type-2 regarding performance. Yet, type-2 hypervisors, such as VirtualBox, benefit from the continual advancement of their underlying OS.


Hypervisors typically support the same multitude of guest operating systems including Linux, BSD, Solaris, Mac OS X, and Windows. But, hypervisors vary in what host OSes they support. For instance, KVM only runs on Linux. VirtualBox, on the other hand, runs on multiple host platforms such as Linux, Solaris, macOS, and Windows.


Prepare for installation


Perform the following steps prior to installing VirtualBox:

1. Enable virtualization technology in BIOS.

2. Download FreeBSD ISO.


Enable virtualization in BIOS


In 2005, chip manufacturers introduced the instruction set extensions Intel VT and AMD-V to support hardware virtualization. Before that, all virtualization was done with software. VirtualBox supports both software and hardware virtualization. If hardware virtualization support is not available on a machine, VirtualBox will switch to software virtualization. Note that without hardware virtualization, VirtualBox can only create 32-bit virtual machines. In other words, a 64-bit virtual machine is only possible if hardware support is available.


To determine if your CPU supports hardware virtualization, run the following command:


$ egrep '^flags.*(vmx|svm)' /proc/cpuinfo


The above command searches the CPU flags register for either 'vmx' (Intel VT) or 'svm' (AMD-V). If the result is non-nil, your machine does have hardware-assisted virtualization.


Even if your machine supports hardware virtualization, it does not necessarily mean that it is currently enabled. To confirm that it is enabled, reboot the machine and enter BIOS. Once inside the BIOS, look for an option to turn on virtualization technology.


Download FreeBSD ISO


VirtualBox is a full-virtualization hypervisor(as opposed to para-virtualization), meaning that the guest OS, FreeBSD in this case, can be used as is, without modification. Having said that,
you are responsible for downloading the guest OS yourself.


You can download the latest installer image (release 11.1) from the FreeBSD website. You have the options of downloading regular Installer Images, Virtual Machine Images, and SD Card Images. For this tutorial, use regular Installer Images. (VirtualBox does not support all Virtual Machine image formats.) Make sure that you select the right hardware architecture (amd64 vs i386).


Within the architecture, download the compressed disk image file with the disc1.iso.xz suffix and the corresponding SHA512 checksum file. The .xz file is about 40% smaller in size than the uncompressed version and will save you download time.


After you have downloaded the 2 files, decompress the disk image file and do a checksum using the respective commands below.


$ unxz FreeBSD-11.1-RELEASE-amd64-disc1.iso.xz
$ sha512sum -c CHECKSUM.SHA512-FreeBSD-11.1-RELEASE-amd64
...
FreeBSD-11.1-RELEASE-amd64-disc1.iso: OK
...


Install VirtualBox


The following commands need to executed with root privilege.


  1. Add the source repository for VirtualBox.


    # echo deb http://download.virtualbox.org/virtualbox/debian stretch contrib > /etc/apt/sources.list.d/virtualbox.list
  2. Import the Oracle GPG public key.


    # wget -q https://www.virtualbox.org/download/oraclevbox2016.asc -O- | apt-key add -
    OK
  3. Install VirtualBox.


    # apt update
    # apt install virtualbox-5.2

Create FreeBSD VM


After installing VirtualBox, run it for the first time by clicking Oracle VM VirtualBox in your menu system. For the Xfce desktop, the option is under the System submenu. Then, click New to create a virtual machine.



Next, provide a host name, and specify the Type (BSD), and Version (FreeBSD 64-bit or 32-bit).



The remaining steps configure the memory size, create a virtual hard disk and specify its file format, its pathname and size, and whether storage is dynamic or fixed-sized. At each step, reasonable defaults are given, and can be accepted unless your requirements dictate otherwise. Click Create at the end to create the virtual machine.


The virtual machine is in the powered-off state after creation. Next, power up the machine by clicking Start.



At this point, the VM has no operating system. Powering it on prompts you to insert a start-up disk. Click the folder icon to specify the file path for the disk image file downloaded earlier (FreeBSD-11.1-RELEASE-amd64-disc1.iso).



Continue by selecting Start. The FreeBSD boot-up screen appears. Hit Enter to boot.



Select Install to continue installation.



The FreeBSD installer now takes over, and prompts you for information it needs to configure the system.


At the end, the installer will prompt you to reboot into the installed system. At this point, you should reboot, but first you must unmount the disk image file (otherwise, you will boot into the CD image again). The reboot procedure I recommend is as follows. First, close the VM window by clicking the X in the upper right-hand corner.



In the ensuing dialog box, select Power off the machine.



To unmount the CD disk image, click the Machine Tools down-arrow, and select Details.



In the Storage section, right click the Optical Drive besides the IDE Secondary Master label, and select Remove disk from optical drive.



Now that the CD image is unmounted, you can click Start in the upper left-hand corner to boot the VM.


Post-installation


VirtualBox offers additional functionalities via guest-specific addition packages. Guest additions packages enable the sharing of folders and the clipboard between host and guest. Follow the procedure below to install VirtualBox guest additions for FreeBSD.


  1. Login to FreeBSD as root.

  2. Install the VirtualBox guest addition packages.


    # pkg install emulators/virtualbox-ose-additions

  3. Append the following 2 lines to the startup configuration file /etc/rc.conf.


    vboxguestenable="YES"

    vboxservice
    enable="YES"

  4. Add each X11 user (say peter) to the wheel group.


    # pw groupmod wheel -m peter

  5. Reboot the VM to have the changes take effect.


Summary & conclusion


VirtualBox is easy to install. With VirtualBox, Creating a FreeBSD virtual machine on a Linux host machine is straightforward. If you are looking for a open-source multi-platform hypervisor, VirtualBox should be high on your priority list.

Related posts

Wednesday, July 27, 2016

ExpressVPN protects your on-line anonymity and privacy

This post reviews ExpressVPN, a hosted Virtual Private Network (VPN) service. A hosted VPN service is a paid subscription service. With a VPN, all your Internet communication is encrypted and passed through a secure proxy (the VPN server) before continuing to the intended destination. To the rest of the world, the Internet traffic appears to come from the VPN server, not your home computer.


Why VPN?

Subscription to a VPN service provides many benefits. For a brief video introduction, please click here.


Anonymity and privacy



When you connect to the Internet, you are exposing yourself to the world of hackers and government spy agencies who want to track your on-line activities, and steal your private information.

Many people have misplaced their trust in the Internet Service Providers (ISP) to protect their on-line anonymity and privacy. It is generally well known that ISPs do log your Internet activities. They are obliged to hand over the data if they are subpoenaed by government authorities.

The tech savvy may run Tor, a popular security tool, on their home computers. But, running Tor by itself is not good enough. While the data is well protected, your ISP does know that you are using Tor. The mere usage of Tor may arouse suspicion and attract extra unwanted attention from the authorities.

By using VPN, all your Internet data and activities (including Tor usages) are protected, even from your ISP. The key is that the VPN vendor must not track your VPN traffic. This "no data logging" policy is written into ExpressVPN's terms of service agreement.


Breaking censorship


Even if you live in a democratic country, you may be subject to regional Internet restrictions. For example, I cannot watch NBC on-line because I live in Canada and they restrict access to American viewers only. Likewise, American viewers cannot stream hockey games from the Canadian CBC website.

With a VPN, you can break censorship by opening a VPN connection to a server located in the target country. For example, to watch the American NBC, I connect my computer to a VPN server located in the USA. In this way, my request to watch NBC is granted because it appears to come from an American location.

For better security and service to you, VPN vendors must deploy servers in as many cities and countries as possible. ExpressVPN's servers are located in 136 cities across 87 countries. This should cover the location requirement for most people.


VPN service features


Pricing & payment options

ExpressVPN's pricing is not the cheapest in the industry. However, it does offer a flexible term: you can sign up for 1 month, or you save money by committing for 6 or 12 months. All plans come with a generous 30-day money-back guarantee, and 7x24 live chat support.

I like the payment options that ExpressVPN offers. In addition to the major credit cards, ExpressVPN also accepts PayPal and Bitcoin. If you pay with a credit card, your identity is associated with your VPN account. On the other hand, you can buy bitcoins anonymously. If you pay with bitcoins which you purchased anonymously, you remain anonymous even to the VPN vendor.

Linux support

Many VPN vendors claim support for Linux. Windows users can download a VPN client which automatically configures your VPN connection. In contrast, the degree of Linux support is often in the form of instructions on how to manually set up a VPN connection.

ExpressVPN's Linux support is exceptional. You can download the VPN client for major Linux distributions such as Debian, Ubuntu, Fedora, and CentOS. You use the client for all your day-to-day VPN operations, such as connecting and disconnecting from the VPN, listing available servers, and reporting the connection status.



Installing ExpressVPN client


After you sign up for the service, download the VPN client according to the instructions in the official welcome email. You need to specify the Linux distribution before the download can proceed. For Debian or Ubuntu, select Ubuntu (there is no Debian option per se). Similarly, for Fedora or Centos, select Fedora.

It is also a good idea to download the VPN client's
signature file. For instructions on how to use the signature file to verify the client download, click here.

After you successfully download the client (expressvpn_1.1.0_amd64.deb), installing it is as easy as running the following command:


$ sudo dpkg -i expressvpn_1.1.0_amd64.deb

Next, you need to activate the VPN service. Note that you only need to do it once. You will be prompted to enter the activation code as provided to you in the welcome email.


$ expressvpn activate

You can download and install the VPN client on as many devices as you wish. But, you can only
have a maximum of 3 simultaneous VPN connections.

The next section explains how you will use the ExpressVPN client.

Useful commands

To connect to the VPN, run this command:


$ expressvpn connect
Connecting to Smart Location...
Connecting to Canada - Montreal - 2... 100.0%
Connected.

Note that I did not specify which VPN server to connect to. When you connect for the very first time and you do not specify the server, it will default to a recommended server, the 'smart' server, e.g., Montreal. In subsequent connections, it will default to the previous server.

What if I don't want to connect to the Montreal server? I live in Vancouver which is about 4,000 kilometres (or 2,485 miles) away from Montreal. So, I want to connect to a nearer server.

To switch servers, follow the steps below.

  1. List the available servers using the following command.
  2. $ expressvpn list    
    ALIAS COUNTRY            LOCATION          RECOMMENDED     
    ----- ---------------    ----------------- -----------    
    smart Smart Location     Canada-Montreal-2  Y
    ...
    usny  United States (US) USA - New York     Y
    ...
    usse                     USA - Seattle      Y
    ...    
    
    Of all the servers, it turns out that Seattle is closest to Vancouver. Later, I will use the alias "usse" from column 1 as a short form for the Seattle server.
  3. Disconnect from the current server.
  4. $ expressvpn disconnect 
    
  5. Connect to the target server.
  6. $ expressvpn connect usse   
    Connecting to USA - Seattle...  100.0%   
    Connected.
    

To verify the status of the VPN connection, run this command:


$ expressvpn status
Connected to USA - Seattle


Performance test



A VPN service encrypts and reroutes your Internet traffic through the VPN server. Because of this indirection, it adds some level of overhead to the VPN speed performance.

To measure the performance overhead of ExpressVPN, I run the following tests.

  1. Baseline (No VPN)

    I run 3 tests without VPN, using speedtest.net. Each test measures the download and upload speeds. Results from the 3 tests are averaged and recorded in the row labeled 'No VPN' in the table below.

  2. VPN connection to the nearest server (Seattle)

    3 more tests were run with VPN connection to the Seattle server. Note that the download and upload speeds take a 21% and 18% hit respectively when you compare the results with tests performed without VPN. A drop in speed is unavoidable because of the inherent VPN performance overhead. This level of performance degradation is often acceptable to most users, and can be viewed as the cost of protecting your on-line privacy and anonymity.

  3. VPN connection to the smart server (Montreal)

    The Montreal server is located 4,000 kilometres (or 2,485 miles) away from Vancouver. In contrast, the Seattle server is only 200 kilometres (or 124 miles) away. In light of the greater distance, it is not surprising that the Montreal speed tests took a bigger hit than the Seattle tests.

VPN status Ave download speed (Mbps) Download speed penalty (%) Ave upload speed (Mbps) Upload speed penalty (%)
No VPN 26.73 N/A 6.69 N/A
Connected to USA - Seattle 21.10 21 5.49 18
Connected to Canada - Montreal 18.80 30 5.40 19

Summary & conclusion



Pros


  • Excellent Linux command-line interface

  • 30-day money back guarantee

  • 7x24 customer support via live chat or email

  • PayPal, Bitcoin and many more payment method options



Cons


  • Restricted number of devices for simultaneous connections






There are many VPN solutions in the market. But, if you are looking for Linux support, you should definitely give ExpressVPN a try. Linux power users will enjoy the use of the command-line VPN client.


Disclaimer

Linuxcommando was provided a free ExpressVPN subscription for this review.

Tuesday, January 26, 2016

Running bash commands in parallel

Introduction


A modern server is typically multi-core, perhaps even multi-CPU. That is plenty of computing power to unleash on a given job. However, unless you run a job in parallel, you are not maximizing the use of all that power.


Below are some typical everyday operations we can speed up using parallel computing:

  1. Backup files from multiple source directories to a removable disk.
  2. Resize image files in a directory.
  3. Compress files in a directory.

To execute a job in parallel, you can use any of the following commands:

  • ppss
  • pexec
  • GNU parallel

This post focuses on the GNU parallel command.

Installation of GNU parallel

To install GNU parallel on a Debian/Ubuntu system, run the following command:

$ sudo apt-get install parallel

General Usage

The GNU parallel program provides many options which you can specify to customize its behavior. Interested readers can read its man page to learn more about their usage. In this post, I will narrow the execution of GNU parallel to the following scenario.

My objective is to run a shell command in parallel, but on the same multi-core machine. The command can take multiple options, but only 1 is variable. Specifically, you run concurrent instances of the command by providing a different value for that one variable option. The different values are fed, one per line, to GNU parallel via the standard input.

The rest of this post shows how GNU parallel can backup multiple source directories by running rsync in parallel.

Parallel backup

The following command backs up 2 directories in parallel: /home/peter and /data.

$ echo -e '/home/peter\n/data' | parallel -j-2 -k --eta rsync -R -av {} /media/myBKUP

Standard input

The echo command assembles the 2 source directory locations, separated by a newline character (\n), and pipes it to GNU parallel.

How many jobs?

By default, GNU parallel deploys 1 job per core. You can override the default usint the -j option.

-j specifies the maximum number of parallel jobs that GNU parallel can deploy. The maximum number can be specified in 1 of several ways:

  • -j followed by a number

    -j2 means that up to 2 jobs can run in parallel.

  • -j+ followed by a number

    -j+2 means that the maximum number of jobs is the number of cores plus 2.

  • -j- followed by a number

    -j-2 means that the maximum number of jobs is the number of cores minus 2.

If you don't know how many cores the machine has, run the command below:

$ parallel --number-of-cores
8

Keeping output order

Each job may output lines to the standard output. When multiple jobs are run in parallel, the default behavior is that a job's output is displayed as soon as the job finishes. You may find this confusing because the output order may be different from the input order. The -k option keeps the output sequence the same as the input sequence.

Showing progress

The --eta option reports progress while GNU parallel executes, including the estimated remaining time (in seconds).

Input place-holder

GNU parallel substitutes the {} parameter with the next line in the standard input.

Each input line is a directory location, e.g., /home/peter. Instead of the full location, you can specify other parameters in order to extract a portion thereof - e.g., the directory name(/home) and the basename (peter). Please refer to the man page for details.

Summary

GNU parallel is a tool that Linux administrators should add to their repertoire. Running a job in parallel can only improve one's efficiency. If you are already familiar with xargs, you will find the syntax familiar. Even if you are new to the command, there is a wealth of on-line help on the GNU parallel website.