Introducing the Shell

Overview

Teaching: 5 min
Exercises: 0 min

Questions

• What is a command shell and why would I use one?

Objectives

• Explain how the shell relates to the keyboard, the screen, the operating system, and users’ programs.

• Explain when and why command-line interfaces should be used instead of graphical interfaces.

Background

Humans and computers commonly interact in many different ways, such as through a keyboard and mouse, touch screen interfaces, or using speech recognition systems. The most widely used way to interact with personal computers is called a graphical user interface (GUI). With a GUI, we give instructions by clicking a mouse and using menu-driven interactions.

While the visual aid of a GUI makes it intuitive to learn, this way of delivering instructions to a computer scales very poorly. Imagine the following task: for a literature search, you have to copy the third line of one thousand text files in one thousand different directories and paste it into a single file. Using a GUI, you would not only be clicking at your desk for several hours, but you could potentially also commit an error in the process of completing this repetitive task. This is where we take advantage of the Unix shell. The Unix shell is both a command-line interface (CLI) and a scripting language, allowing such repetitive tasks to be done automatically and fast. With the proper commands, the shell can repeat tasks with or without some modification as many times as we want. Using the shell, the task in the literature example can be accomplished in seconds.

The Shell

The shell is a program where users can type commands. With the shell, it’s possible to invoke complicated programs like climate modeling software or simple commands that create an empty directory with only one line of code. The most popular Unix shell is Bash (the Bourne Again SHell — so-called because it’s derived from a shell written by Stephen Bourne). Bash is the default shell on most modern implementations of Unix and in most packages that provide Unix-like tools for Windows.

Using the shell will take some effort and some time to learn. While a GUI presents you with choices to select, CLI choices are not automatically presented to you, so you must learn a few commands like new vocabulary in a language you’re studying. However, unlike a spoken language, a small number of “words” (i.e. commands) gets you a long way, and we’ll cover those essential few today.

The grammar of a shell allows you to combine existing tools into powerful pipelines and handle large volumes of data automatically. Sequences of commands can be written into a script, improving the reproducibility of workflows.

In addition, the command line is often the easiest way to interact with remote machines and supercomputers. Familiarity with the shell is near essential to run a variety of specialized tools and resources including high-performance computing systems. As clusters and cloud computing systems become more popular for scientific data crunching, being able to interact with the shell is becoming a necessary skill. We can build on the command-line skills covered here to tackle a wide range of scientific questions and computational challenges.

Let’s get started.

When the shell is first opened, you are presented with a prompt, indicating that the shell is waiting for input.

$

The shell typically uses $ as the prompt, but may use a different symbol. In the examples for this lesson, we’ll show the prompt as $ . Most importantly: when typing commands, either from these lessons or from other sources, do not type the prompt, only the commands that follow it. Also note that after you type a command, you have to press the Enter key to execute it.

Changing your prompt

Your prompt may be super long (containing your username, computer name, etc.). To change your prompt type in:

$ export PS1= ">"

This will last as long as the length of your shell session.

The prompt is followed by a text cursor, a character that indicates the position where your typing will appear. The cursor is usually a flashing or solid block, but it can also be an underscore or a pipe. You may have seen it in a text editor program, for example.

So let’s try our first command, ls which is short for listing. This command will list the contents of the current directory:

$ ls

Desktop     Downloads   Movies      Pictures
Documents   Library     Music       Public

Command not found

If the shell can’t find a program whose name is the command you typed, it will print an error message such as:

$ ks

ks: command not found

This might happen if the command was mis-typed or if the program corresponding to that command is not installed.

Exercise Pipeline: A Typical Problem

Throughout this workshop lesson, we will make progress on developing a programmatic pipeline on the files in “ex-files”. In the folder exercise files, there are 1520 text files. The goal of the pipeline is to run all 1520 files through an imaginary program called goostats.sh.

If we run goostats.sh by hand using a GUI, we’ll have to select and open a file 1520 times. If goostats.sh takes 30 seconds to run each file, the whole process will take more than 12 hours. With the shell, we can instead assign the computer this mundane task.

The next few lessons will explore the ways we can achieve this. More specifically, how we can use a command shell to run the goostats.sh program, using loops to automate the repetitive steps of entering file names, so that the computer can work without consistent supervision.

Once we has put a processing pipeline together, we could use it again.

In order to achieve this task, we needs to know how to:

• navigate to a file/directory
• create a file/directory
• check the length of a file
• chain commands together
• retrieve a set of files
• iterate over files
• run a shell script containing our pipeline

Key Points

• A shell is a program whose primary purpose is to read commands and run other programs.

• This lesson uses Bash, the default shell in many implementations of Unix.

• Programs can be run in Bash by entering commands at the command-line prompt.

• The shell’s main advantages are its high action-to-keystroke ratio, its support for automating repetitive tasks, and its capacity to access networked machines.

• The shell’s main disadvantages are its primarily textual nature and how cryptic its commands and operation can be.