Tutorial: DSH - Sebastien Varrette, PhD.

DSH (Distributed / Dancer’s Shell) is a wrapper around SSH that allows to run commands over multiple machines. There are many similar wrappers (i.e. meant for executing the same command on multiple hosts over ssh in parallel):

DSH – Distributed / Dancer’s Shell, described in this post
PDSH – archive
ClusterShell

Resources:

Techmint Tutorial on DSH

Table of Content

Installation Notes
- Bash auto-completion
DSH Configuration
Basic DSH Usage
DSH Group Layout

Installation Notes

OS	Installation command*
Debian	`apt-get install dsh pdsh`
CentOS/RedHat	`yum install dsh pdsh`
Mac OS**	`brew install dsh pdsh`

*: pdsh should be installed too as it brings the dshbak command which format output from [p]dsh commands (see below)

**: Assumes you have install HomeBrew, a package installer for Mac OS.

Bash auto-completion

Note that to enable bash completions, you will need to install the corresponding package, and load it within your bashrc – see this old tutorial for instance.

You can find the Bash completion file for dsh here. You can install it as follows:

$> mkdir .bash.d    # Eventually
$> git clone https://gist.github.com/920433.git ~/.bash.d/dsh
$> sudo ln -s ~/.bash.d/dsh/dsh  /etc/bash_completion.d/dsh

Nothing found for zsh so far…

DSH Configuration

The configuration of DSH is done within the ~/.dsh/ directory.

Main configuration file: ~/.dsh/dsh.conf
DSH groups definitions directory: ~/.dsh/group/

Example of main configuration file ~/.dsh/dsh.conf

##############################################################################################################
# Time-stamp: <Sat 2016-01-30 22:47 svarrette>
#  ____  ____  _   _           ____  _     _        _ _           _           _   ____  _          _ _
# |  _ \/ ___|| | | |         |  _ \(_)___| |_ _ __(_) |__  _   _| |_ ___  __| | / ___|| |__   ___| | |
# | | | \___ \| |_| |  _____  | | | | / __| __| '__| | '_ \| | | | __/ _ \/ _` | \___ \| '_ \ / _ \ | |
# | |_| |___) |  _  | |_____| | |_| | \__ \ |_| |  | | |_) | |_| | ||  __/ (_| |  ___) | | | |  __/ | |
# |____/|____/|_| |_|         |____/|_|___/\__|_|  |_|_.__/ \__,_|\__\___|\__,_| |____/|_| |_|\___|_|_|
#
# Source : http://www.netfort.gr.jp/~dancer/software/dsh.html.en
# Tutorial: http://www.tecmint.com/using-dsh-distributed-shell-to-run-linux-commands-across-multiple-machines/
#
# Configuration file for dsh (Distributed / Dancer's Shell).
# `man dsh.conf` for details
############################################################

verbose = 0

remoteshell      = ssh
showmachinenames = 1

# Specify 1 to make the shell wait for each individual invocation.  See -c and -w option for dsh(1)
waitshell        = 0  # whether to wait for execution

# Specify the number of parallel connection to create at the same time.
#forklimit=8

remoteshellopt   = -q

You can create as many meaningful groups as you wish with under ~/.dsh/group/ A given group <groupname> would be thus defined as ~/.dsh/group/<groupname> and simply lists the target servers (one per line) by hostnames (as defined within your SSH configuration).

Then, assuming you wish to run the command uptime on all the hosts listed under the group <groupname> (i.e. <hostname1> and <hostname2>), you can run:

$> dsh -g <groupname> -- uptime
# Equivalent of:
# for h in $(grep -v \# ~/.dsh/group/<groupname>); do ssh $h uptime; done

Basic DSH Usage

   dsh [-c | -w] { -a | -g <group> | -m <hostname> } <command> | dshbak -c

using -a, you run the command on all nodes listed in machines.list
using -g <group>, you restrict the commands to the group of hosts <group>
using -m <hostname>, you run the command only on hostname
using -c, you run the commands in parallel (default)
using -w, you run the commands in sequential

DSH Group Layout

Of course you can organize as many groups as you wish to reflect your working conditions. Here as some hints on how I manage these groups:

First of all, I maintain manually a set of groups per topics, organized as follows per site <site>:

.
├── Makefile     # Automatic generation of missing, super groups
├── home         # All home/NAS servers/workstations
├── ...
├── <site>.kvm.yum   # CentOS/RHEL physical server, KVM hosts for site <site>
├── <site>.kvm.apt   # Debian/ubuntu physical server, KVM hosts for site <site>
├── <site>.nokvm.yum # CentOS/RHEL physical server in site <site>, not meant for virtualization
├── <site>.nokvm.apt # Debian/Ubuntu physical server in site <site>, not meant for virtualization
├── <site>.<host>.vms.yum  # CentOS/RHEL VMs on host <host> belonging to site <site>
├── <site>.<host>.vms.apt  # Debian/Ubuntu VMs on host <host> belonging to site <site
├── ...
└── puppet.master    # Special group for puppet masters

Now the Makefile is meant to generate automatically the groups that derives from these atomic groups. For instance:

<site>.<host>.vms holds all VMs on the host <host> of site <site>, and is the concatenation of <site>.<host>.vms.yum and <site>.<host>.vms.apt
<site>.vms holds all VMs on the site <site> and is the concatenation of all <site>.*.vms
<site>.kvm holds all KVM hosts on the site <site> and is the concatenation of <site>.kvm.yum and <site>.kvm.apt
<site>.nokvm holds all non-KVM hosts (not meant for virtualization servers) on the site <site> and is the concatenation of <site>.nokvm.yum and <site>.nokvm.apt
<site>.servers holds all physical servers on the site <site> and is the concatenation of <site>.nokvm and <site>.kvm
<site>.all holds all servers for the site <site> and is thus the concatenation of <site>.nokvm, <site>.kvm and <site>.vms
vms holds all VMs, and is the concatenation of all *.vms
kvm holds all KVM hosts, and is the concatenation of all *.kvm
nokvm holds all physical/non-KVM hosts, and is the concatenation of all *.nokvm
all contains all hosts, and is thus the concatenation of nokvm, kvm and vms

The Makefile used to generate these files is rather complex, but here is the skeleton structure you might wish to reuse and adapt to your needs

#############################################################################################
# ~/.dsh/group/Makefile - Configuration file for GNU make (http://www.gnu.org/software/make/)
# Automatic generation of sub-groups for DSH
##############################################################################################
SHELL = /bin/bash
# List below your sites names
SITES =  ul home
MACHINES_ALL = ul.all
SRC =
TARGETS = $(MACHINE_ALL)

### Site <site> -- /!\ ADAPT '<site>' and '<SITE>' accordingly
<SITE>_ALL     = <site>.all
<SITE>_SERVERS = <site>.servers
<SITE>_NOKVM   = <site>.nokvm
<SITE>_KVM     = <site>.kvm
<SITE>_VMs     = <site>.vms
<SITE>_YUM     = <site>.yum
<SITE>_APT     = <site>.yum
# corresponding sources -- /!\ TO BE ADAPTED
SRC_<SITE>_ALL     = $()
SRC_<SITE>_SERVERS = $(<SITE>_KVM) $(<SITE>_NOKVM)
SRC_<SITE>_NOKVM   = $(wildcard <site>.nokvm.*)
SRC_<SITE>_KVM     = $(wildcard <site>.kvm.*)
SRC_<SITE>_VMs     = $(wildcard <site>.*.vms)
SRC_<SITE>_YUM     = $(wildcard <site>.*.yum)
SRC_<SITE>_APT     = $(wildcard <site>.*.apt)

<SITE> = $(<SITE>_ALL) $(<SITE>_SERVERS) $(<SITE>_KVM) $(<SITE>_NOKVM) $(<SITE>_VMs) $(<SITE>_YUM) $(<SITE>_APT)
SRC     += $(SRC_<SITE>_NOKVM) $(SRC_<SITE>_KVM) $(SRC_<SITE>_VMs) $(SRC_<SITE>_YUM) $(SRC_<SITE>_APT)
TARGETS += $(<SITE>)

### [...] (repeat for all sites)


###############################################################

# General Make Macro to generate a given group file
define CREATE_GROUP

$(1): $(2)
	@echo "==> Generating $$@ from '$(2)'"
	@cat $(2) | grep -v '^#' > $$@

endef
define CLEAN_GROUP

$(1)-clean: $(2)
	rm -f $$?
endef


##################################
all: $(MACHINES_ALL) $(TARGETS)

######### Site '<site>' (to be repeated for each of them)
info-<site>:
	@echo "<SITE>         = $(<SITE>)"
	@echo "<SITE>_ALL     = $(<SITE>_ALL)"
	@echo "<SITE>_SERVERS = $(<SITE>_SERVERS)"
	@echo "<SITE>_NOKVM   = $(<SITE>_NOKVM)"
	@echo "<SITE>_KVM     = $(<SITE>_KVM)"
	@echo "<SITE>_VMs     = $(<SITE>_VMs)"
	@echo "<SITE>_YUM     = $(<SITE>_YUM)"
	@echo "<SITE>_APT     = $(<SITE>_APT)"
	@echo "==================================="
	@echo "SRC_<SITE>         = $(SRC_<SITE>)"
	@echo "SRC_<SITE>_ALL     = $(SRC_<SITE>_ALL)"
	@echo "SRC_<SITE>_SERVERS = $(SRC_<SITE>_SERVERS)"
	@echo "SRC_<SITE>_NOKVM   = $(SRC_<SITE>_NOKVM)"
	@echo "SRC_<SITE>_KVM     = $(SRC_<SITE>_KVM)"
	@echo "SRC_<SITE>_VMs     = $(SRC_<SITE>_VMs)"
	@echo "SRC_<SITE>_YUM     = $(SRC_<SITE>_YUM)"
	@echo "SRC_<SITE>_APT     = $(SRC_<SITE>_APT)"

<site>: $(<SITE>)
<site>-clean:
	rm -f $(<SITE>)
$(eval $(call CREATE_GROUP,$(<SITE>_ALL),      $(SRC_<SITE>_ALL)))
$(eval $(call CREATE_GROUP,$(<SITE>_SERVERS),  $(SRC_<SITE>_SERVERS)))
$(eval $(call CREATE_GROUP,$(<SITE>_NOKVM),    $(SRC_<SITE>_NOKVM)))
$(eval $(call CREATE_GROUP,$(<SITE>_KVM),      $(SRC_<SITE>_KVM)))
$(eval $(call CREATE_GROUP,$(<SITE>_VMs),      $(SRC_<SITE>_VMs)))
$(eval $(call CREATE_GROUP,$(<SITE>_YUM),      $(SRC_<SITE>_YUM)))
$(eval $(call CREATE_GROUP,$(<SITE>_APT),      $(SRC_<SITE>_APT)))

That’s all folks ;-)

Next step is to configure your Git hook to automatically invoke the appropriate make command when a commit involves changes on the source files. But that will be covered ion another post.