Chapter 1 Introduction

Motivating Questions

  • What is a command shell and why would I use one?

Objectives

  • Explain how the shell relates to the keyboard, the screen, the operating system, and users’ programs.
  • Explain when and why command-line interfaces should be used instead of graphical interfaces.
  • Explain the steps in the shell’s read-run-print cycle.
  • Identify the actual command, flags, and filenames in a command-line call.
  • Demonstrate the use of tab completion and explain its advantages.

Key Points

  • Most commands take flags (options) which begin with a -.
  • A shell is a program whose primary purpose is to read commands and run other programs.
  • The shell’s main advantages are its high action-to-keystroke ratio, its support for automating repetitive tasks, and its capacity to access networked machines.
  • The shell’s main disadvantages are its primarily textual nature and how cryptic its commands and operation can be.

The Unix shell has been around longer than most of its users have been alive. It has survived so long because it’s a power tool that allows people to do complex things with just a few keystrokes. More importantly, it helps them combine existing programs in new ways and automate repetitive tasks so they aren’t typing the same things over and over again. Use of the shell is fundamental to using a wide range of other powerful tools and computing resources (including “high-performance computing” supercomputers). These lessons will start you on a path towards using these resources effectively.

1.1 Prerequisites

This lesson guides you through the basics of file systems and the shell. This lessons assumes you know how to store files on a computer, and recognize the word “file” and either “directory” or “folder” (two common words for the same thing). Also, this lesson assumes you know how to unzip a compressed file, install programs on your computer, and perform simple computer administation tasks.

1.2 Setup

You need to download some files to follow this lesson:

  1. Download data-shell.zip and move the file to your Desktop.
  2. Unzip/extract the file. You should end up with a new folder called data-shell on your Desktop.
  3. Open a terminal and type cd, then press the Enter key. That last step will make sure you start with your home folder as your working directory.

In the lesson, you will find out how to access the data in this folder.

1.3 Where to type commands: How to open a new shell

The shell is a program that enables us to send commands to the computer and receive output. It is also referred to as the terminal or command line. Some computers include a default Unix Shell program.

The steps below describe some methods for identifying and opening a Unix Shell program if you already have one installed. There are also options for identifying and downloading a Unix Shell program, a Linux/UNIX emulator, or a program to access a Unix Shell on a server.

If none of the options below address your circumstances, try an online search for: Unix shell [your computer model] [your operating system].

1.3.1 Linux

The default Unix Shell for Linux operating systems is usually Bash. On most versions of Linux, it is accessible by running the (Gnome)Terminal or (KDE) Konsole or xterm, which can be found via the applications menu or the search bar. If your machine is set up to use something other than bash, you can run it by opening a terminal and typing bash.

1.3.2 Mac OS

For a Mac computer, the default Unix Shell is Bash, and it is available via the Terminal Utilities program within your Applications folder.

To open Terminal, try one or both of the following:

  • Go to your Applications folder. Within Applications, open the Utilities folder. Locate Terminal in the Utilities folder and open it.
  • Use the Mac ‘Spotlight’ computer search function. Search for: Terminal and press Return.

Reference:

1.3.3 Windows

Computers with Windows operating systems do not automatically have a Unix Shell program installed. In this lesson, we encourage you to use an emulator included in Git for Windows, which gives you access to both Bash shell commands and Git.

Other solutions are available for running Bash commands on Windows. There is now a Bash shell command-line tool available for Windows 10. Additionally, you can run Bash commands on a remote computer or server that already has a Unix Shell, from your Windows machine. This can usually be done through a Secure Shell (SSH) client. One such client available for free for Windows computers is PuTTY. See the reference below for information on installing and using PuTTY, using the Windows 10 command-line tool, or installing and using a Unix/Linux emulator.

References:

1.4 Background

At a high level, computers do four things:

  • Run programs,
  • Store data,
  • Communicate with each other, and
  • Interact with us.

They can do the last of these in many different ways, including through a keyboard and mouse, touch screen interfaces, or using speech recognition systems. While touch and voice interfaces are becoming more commonplace, most interaction is still done using traditional screens, mice, touchpads, and keyboards.

We are all familiar with graphical user interfaces (GUI): windows, icons, and pointers. They are easy to learn and fantastic for tasks no more complicated than clicking on icons and buttons. But this simplicity relies on having programs that can do only a limited and prescribed number of tasks.

If you wish to do complex, custom, and purpose-specific tasks it helps to have a richer means of expressing your instructions to the computer. This mechanism doesn’t need to be complicated or difficult–just a vocabulary of commands and a simple grammar for using them.

This is what the shell provides–a simple language and a command-line interface to use it through.

The heart of a command-line interface is a read-evaluate-print loop (REPL). When you type a command and press Return the shell reads your command, evaluates (or “executes”) it, prints the output of your command, loops back and waits for you to enter another command.

1.5 The Shell

A shell is a program like any other program on your computer. What’s special about a shell is that its job is to run other programs rather than to perform tasks itself. The most popular Unix shell is Bash, the Bourne Again SHell (so-called because it’s derived from a shell written by Stephen Bourne). Bash is the default shell on most modern implementations of Unix and in most packages that provide Unix-like tools for Windows.

1.5.1 What Does the Shell Look Like?

A typical shell window looks something like:

bash-3.2$ 
bash-3.2$ ls -F / 
Applications/         System/
Library/              Users/
Network/              Volumes/
bash-3.2$

The first line shows only a prompt, indicating that the shell is waiting for input. Your shell may use different text for the prompt. Most importantly: when typing commands, either from these lessons or from other sources, do not type the prompt, only the commands that follow it.

The part that you type, ls -F / in the second line of the example, typically has the following structure: a command, some flags (also called options or switches) and an argument. Flags start with a single dash (-) or two dashes (--), and change the behaviour of a command. Arguments tell the command what to operate on (e.g. files and directories). Sometimes flags and arguments are referred to as parameters. A command can be called with more than one flag and more than one argument, but a command doesn’t always require an argument or a flag.

In the second line of the example above, our command is ls, with a flag -F and an argument /. Each part is separated by spaces: if you omit the space between ls and -F the shell will look for a command called ls-F, which doesn’t exist. Also, capitalization matters: LS is different than ls.

Next we see the output that our command produced. In this case it is a listing of files and folders in a location called /–we’ll cover what all these mean later. Those using macOS might recognize the output in this example.

Finally, the shell again prints the prompt and waits for you to type the next command.

In the examples for this lesson, we’ll show the prompt as $. You can make your prompt look the same by executing the command PS1='$ '. But you can also leave your prompt as it is–often the prompt includes useful information about who and where you are.

Open a shell window and try executing ls -F / for yourself (don’t forget that spaces and capitalization are important!). You can change the prompt too, if you like.

1.6 How Does the Shell Know What ls and the Flags Mean?

Every command is a program stored somewhere on the computer, and the shell keeps a list of places to search for commands (the list is in a variable called PATH, but those are concepts we’ll meet later and are not too important at the moment). Recall that commands, flags, and arguments are separated by spaces.

So let’s look at the REPL (read-evaluate-print loop) in more detail. Notice that the “evaluate” step is made of two parts:

  1. Read what was typed (ls -F / in our example). The shell uses the spaces to split the line into the command, flags, and arguments
  2. Evaluate:
    1. Find a program called ls
    2. Execute the program, passing it the flags and arguments (-F and /) to interpret as the program sees fit
  3. Print the output produced by the program, and then print the prompt and wait for you to enter another command.

If the shell can’t find a program whose name is the command you typed, it will print an error message like:

$ ls-F
-bash: ls-F: command not found

Usually this means that you have mis-typed the command–in this case we omitted the space between ls and -F.

1.7 Is Learning the Command Line Difficult?

Using the command line interface is a different model of interacting with the computer than using a GUI, and understanding that model will take some effort and time to internalize. A GUI presents you with choices and you select one. With a command line interface (CLI) the choices are combinations of commands and parameters more like words in a language than buttons on a screen. However, unlike icons and buttons on your screen, the available commands and parameters may not be immediately visible to you, so you must learn a few basic commands to get you started, like learning some basic vocabulary in a new foreign language. But even a small number of commands will take you a long way, and we’ll cover those essential few commands in this guide.

1.8 Flexibility and Automation

The grammar of a shell allows you to combine existing tools into powerful pipelines and handle large volumes of data automatically. Sequences of commands can be written into a script, improving the reproducibility of workflows and allowing you to repeat them easily. In addition, the command line is often the easiest way to interact with remote machines and supercomputers. Familiarity with the shell is often required to run a variety of specialized tools and resources including high-performance computing systems. As clusters and cloud computing systems become more popular for scientific data crunching, being able to interact with the shell is becoming a necessary skill. We can build on the command-line skills covered here to tackle a wide range of scientific questions and computational challenges.