Come out of your shell and learn Bash

One thing I noticed whenever I'd kick off a new cohort at We Got Coders, was that despite all my efforts, often students would arrive without knowing much about how to use a shell. The shinier the laptop, the greater the reluctance to let go of the GUI and get their hands on the keyboard.

I'd put together a cheat sheet. Yet I can't think of many other technologies more likely to sap the enthusiasm of a new coder than bash (maybe SQL?). It would make a boring first day: let's learn how to make a directory and then remove it again! Especially when ultimately the goal would be to learn web development, or get a job as a programmer; why go over these basic command line commands?

There are some concepts that are assumed when you learn programming. Take the filesystem. It's not obvious to new starters what a path is, how a relative path works or how to navigate around the concept of a tree. It's not hard - but it's enough to deter newcomers who will wrongly believe they are up against 'clever' people, and it's enough to distract from an actual high-level concept we might cover in class.

Trouble is, the longer we'd leave it, the more of a stumbling block the shell would become. That first couple of weeks, where I'm bombarding them with new concepts about Ruby and Object-orientated programming, they'd be wrestling with their operating systems, using a text editor to rename files and create directories, or the finder to delete folders and lose their work. Everything was slower, and all of that extra effort was taking away from learning to code.

While there is a lot to learn about bash programming, unless you're planning on being a Debian package maintainer, I found over the last fifteen years that I'd stick to a relatively small subset of bash commands, and the odd bit of scripting on top. I never listened to those proponents of other shells, and how zsh would change my life and so on. I felt like I had learnt just enough, and that enabled me to be much better at my job than if I never bothered to learn at all.

In the end, many of us learn shell programming as a subtask of something else, a necessary obstacle to be overcome. Once we've learnt Ruby, JavaScript, Go or whatever we're focussing on, we're not going to spend much time on our bash scripting. But for automating repetitive tasks, converting files, creating initial project directory structures, navigating the filesystem and finding where we left our code, knowing the basics around shell is a vital skill for any serious programmer.

I decided to put together a short lesson for beginners, that covers some basic bash concepts. I tried to focus on that subset of knowledge that I'd come to rely on over the years, rather than regurgitating a bash bible. I wanted to make a essential guide - something that would underpin any online course, bootcamp or training program. The content is free to use and download - and you can request my help if you get stuck on any of it.

To give you a flavour, here is one part of the exercises that we'll work on during the course.

Generating a HTML project skeleton in Bash

One of the nice things about Bash is that it doesn't require any other dependencies to run, it usually comes with a UNIX machine. Therefore it's a good choice for code that creates a new project, as we can run it straight away without any further setup.

Suppose we wanted to achieve a directory structure ready for web development. We might want to keep separate folders for our different assets, and a starter HTML file. To begin with, I'll aim for something like this:

site/
├── css
├── images
├── index.html
└── javascripts

Using the commands we had before, we could easily achieve this using mkdir and touch. You can try any of these snippets by just dropping them in your shell, or you might want to create a script (For this example I'll assume you've created a new_html_project script and made it executable).

mkdir -p site/images
mkdir -p site/javascripts
mkdir -p site/css
touch site/index.html

But it is a bit repetitive. Once our file list gets longer, we might want to avoid writing the mkdir command many times. This is where the for loop comes in. Any code inside the loop gets repeated, so that the code can be run many times with different inputs. Let's roll these up into a loop and see if it saves any repetition:

for folder in "site/css" "site/images" "site/javascripts"
do
  mkdir -p $folder
done

Here we're using a bash variable (the bit with the dollar sign before it) to store the values of "site/css", "site/images", "site/javascripts" consecutively, and using value that as the argument to mkdir.

As for the index page, we need to use a different approach. You might use touch to create a blank file, or even echo to add some content using redirection:

echo "<h1>My amazing webpage</h1>" > index.html`

Since we generally need a bit of content, we could consider using a here document to build a larger string, and using that as the input to cat, before redirecting the output into the index file. A here document uses an arbitrary delimiter (here HTML) to denote the start (<<HTML) and end (HTML on it's own line) of the string.

cat > site/index.html <<HTML
<!DOCTYPE HTML>
<html>
  <head>
    <title>My Amazing HTML document</title>
  </head>
  <body>
    <h1>My Amazing HTML document</h1>
  </body>
</html>
HTML

So far we've got a script that looks like this:

#!/bin/bash
for folder in "site/css" "site/images" "site/javascripts"
do
  mkdir -p $folder
done

cat > site/index.html <<HTML
<!DOCTYPE HTML>
<html>
  <head>
    <title>My Amazing HTML document</title>
  </head>
  <body>
    <h1>My Amazing HTML document</h1>
  </body>
</html>
HTML

So far we've hit on a nice coincidence, that the commands we've chosen work even on subsequent uses - that is, even if we ran the script twice. However, we probably don't want to keep running the command to set the index.html, otherwise there is a chance that we'd lose our work if we re-ran the script. Fortunately, this is a good use case for the if statement, to check whether the file already exists before creating it. Let's update our script to use if statements, using the -f flag to determine whether a file exists.

#!/bin/bash
for folder in "site/css" "site/images" "site/javascripts"
do
  mkdir -p $folder
done

if [ ! -f site/index.html ]
then cat > site/index.html <<HTML
<!DOCTYPE HTML>
<html>
  <head>
    <title>My Amazing HTML document</title>
  </head>
  <body>
    <h1>My Amazing HTML document</h1>
  </body>
</html>
HTML
fi

Finally, we're assuming that we always want to create a site folder. The script would be considerably more useful if we could parameterise it, to allow us to name to subfolder in question. Bash scripts do accept arguments, which are space delimited on the command line. Each argument is delimited by the dollar sign, and a number index of which argument in the list you want to use. If you try typing this into your shell, you'll see that $1 becomes hello, and $2 becomes there:

echo $1 $2 hello there

The arguments are zero-indexed, which means there is also an argument $0, which represents the program you called, in this case echo.

We might therefore use this to make our site folder more dynamic:

#!/bin/bash
for folder in "$1/css" "$1/images" "$1/javascripts"
do
  mkdir -p $folder
done

if [ ! -f $1/index.html ]
then cat > $1/index.html <<HTML
<!DOCTYPE HTML>
<html>
  <head>
    <title>My Amazing HTML document</title>
  </head>
  <body>
    <h1>My Amazing HTML document</h1>
  </body>
</html>
HTML
fi

We can now generate multiple projects using ./new_html_project my_amazing_webpage, replacing the argument for each folder. But there's a bug! If we try to call this program, but forgot to pass in an argument, the value of $1 will be blank, leaving a shell script that notionally looks like this:

#!/bin/bash
for folder in "/css" "/images" "/javascripts"
do
  mkdir -p $folder
done

if [ ! -f /index.html ]
then cat > /index.html <<HTML
<!DOCTYPE HTML>
<html>
  <head>
    <title>My Amazing HTML document</title>
  </head>
  <body>
    <h1>My Amazing HTML document</h1>
  </body>
</html>
HTML
fi

You may have spotted that without our dynamic site folder name, we're now referencing paths that start with /, which means the root of the entire filesystem. This folder is protected from ordinary users (partly for this reason!), so unless you had root privileges, you will probably see permission errors.

To fix this, we can use the $# operator to provide the number of arguments given, and display a message if we got it wrong. Given that the program name counts as an argument, we are looking for exactly two as the result.

#!/bin/bash
if [ ! $# == 2 ]; then
  echo "Usage: $0 <folder_name>"
  exit
fi

for folder in "$1/css" "$1/images" "$1/javascripts"
do
  mkdir -p $folder
done

if [ ! -f $1/index.html ]
then cat > $1/index.html <<HTML
<!DOCTYPE HTML>
<html>
  <head>
    <title>My Amazing HTML document</title>
  </head>

  <body>
    <h1>My Amazing HTML document</h1>
  </body>
</html>
HTML
fi

I reckon this example contains enough concepts for further study in other languages. In the course we look at some more examples of basic bash commands, and I leave you an exercise to build a directory structure to support a bootcamp syllabus.