Unix/Linux, Shell, and Git

Introduction to Operating Systems¶

An operating system (OS) is the software layer that connects the computer hardware to users and applications (and AI agents now). Instead of writing instructions that directly manipulate processors, memory chips, or disk drives, we interact with the OS, which manages these resources for us.

The Structure of an Operating System¶

An OS typically consists of three main parts:

• Kernel: the core component. It directly manages hardware (CPU, memory, devices) and enforces rules for resource sharing.

• System Programs and Applications: provide services built on top of the kernel, such as file utilities, compilers, or networking tools.

• Shell and User Interface: the layer through which users interact with the OS. This can be:

  • Command-line shells (e.g., bash, zsh), where users type commands, or

  • Graphical interfaces (e.g., desktops, windows, icons).

In this lab, we will focus on the shell, because computational astrophysicists often work on large remote systems (HPC clusters and Cloud) where the command line is the most efficient and sometimes the only available interface.

Common Features of Operating Systems¶

Despite differences, most operating systems share these responsibilities:

• Process Management: starting, stopping, and scheduling programs.

• Memory Management: allocating, tracking, and protecting system memory.

• File Systems: organizing data into files and directories.

• Device Management: controlling access to hardware like disks and network cards.

• Security and Access Control: permissions, authentication, and isolation.

• User Interfaces: shells or graphical environments for interaction.

Unix¶

Unix, developed at Bell Labs in the 1960s-70s by Ken Thompson and Dennis Ritchie, set the standard for many OS design principles:

• A multi-user, multi-tasking architecture.

• A hierarchical file system.

• “Everything is a file” (even devices).

• Small, composable programs connected via pipes.

Linux¶

Linux is a Unix-like operating system (technically only the kernel) created by Linus Torvalds in 1991. Unlike traditional Unix systems, it was built independently. Its open-source license (GPLv2) lets anyone to study, modify, and redistribute the code.

Here is the original humble email that changed the world:

Hello everybody out there using minix -

I'm doing a (free) operating system (just a hobby, won't be big and
professional like gnu) for 386(486) AT clones.  This has been brewing
since april, and is starting to get ready.  I'd like any feedback on
things people like/dislike in minix, as my OS resembles it somewhat
(same physical layout of the file-system (due to practical reasons)
among other things).

I've currently ported bash(1.08) and gcc(1.40), and things seem to work.
This implies that I'll get something practical within a few months, and
I'd like to know what features most people would want.  Any suggestions
are welcome, but I won't promise I'll implement them :-)

              Linus (torvalds@kruuna.helsinki.fi)

PS.  Yes - it's free of any minix code, and it has a multi-threaded fs.
It is NOT protable (uses 386 task switching etc), and it probably never
will support anything other than AT-harddisks, as that's all I have :-(.

Unix Philosophy¶

The power of Unix and Linux comes not just from its technical features, but from a design philosophy. Some of the guiding principles are:

• Do one thing well. Each program should have a single, focused purpose. To solve a new problem, build a new tool rather than overcomplicating an old one.

• Build programs to work together. The output of one program should serve as the input to another. This encourages simple text-based interfaces and avoids unnecessary formatting.

• Prototype early and refine. Software should be tested quickly, with the freedom to discard clumsy parts and rebuild better versions.

• Rely on tools, not manual effort. Create reusable tools to simplify tasks, even if they are only needed temporarily.

Another core idea is that “everything is a file”. As a result, devices, processes, and data can all be accessed through a unified file interface.

Because of these simple yet powerful design choices, Unix and Linux are extremely flexible and extensible.

Unix evolved into a broad family of operating systems, including the BSDs (FreeBSD, OpenBSD, NetBSD), Solaris, and eventually NeXTSTEP, which became macOS (Mac OS X). Linux, meanwhile, has grown into an ecosystem with countless distributions.

Today, Linux has surpassed both traditional Unix and Windows in many domains and become the #1 OS for the internet and scientific computing:

• Runs directly on bare-metal servers in data centers and on virtual machines in the cloud.

• Powers the fastest supercomputers in the world.

• Serves as the backbone of scientific computing, HPC, machine learning, and AI.

• Provides the kernel for Android smartphones, used by billions of people worldwide.

Shells¶

The terms “shell” and “terminal” are often used interchangeably today, but they actually refer to different parts of the system:

• Terminal (or terminal emulator): A text-based interface that lets you interact with the operating system. On modern computers this is usually a software application (e.g., Terminal on macOS, GNOME Terminal on Linux).

• Shell: A program that runs inside the terminal. It interprets the commands you type, sends them to the operating system, and prints the results back. Examples include sh, bash, and zsh.

Basic Unix/Linux Commands¶

Here are some commands every Unix/Linux user should know.

Navigation¶

Basic File Management¶

Viewing Files¶

Wildcards, Globbing, and Brace Expansion¶

The shell can automatically expand patterns into lists of files or strings, saving you from typing them out manually.

Many of these commands deal with the file system, which makes the point that in Unix/Linux, “everything is a file”. Hence, regular files, directories, devices, and even some processes are all accessed using the same interface.

Combining Programs¶

Unix programs are designed to work together. The shell provides simple mechanisms to connect these small tools into powerful workflows.

Redirection and Piping Operators¶

Filters¶

Some of the most useful programs to use with pipe are “filters”. They take input from stdin, transform them according to some rules, and then output the results to stdout. Here are some filters that I use frequently.

Shell Scripting¶

Shells allow you to automate repetitive tasks by writing scripts. A shell script is simply a text file containing a series of commands. Here is an example of a simple Bash script:

#!/bin/bash
echo "Hello, World!"

To run the script, save it to a file (e.g., hello.sh), make it executable (chmod a+x hello.sh), and then execute it (./hello.sh).

For almost all Unix/Linux systems, bash are installed by default, and sh is just a symbolic link to bash. On Mac, because of license comptability, the default shell is zsh; and sh is a minimal “POSIX-compliant command interpreter”.

Variables and String Manipulation¶

Control structures¶

The shell is not only an interface for running commands, but also a scripting language. The most common control structures are for conditions and loops.

Shortcuts for Interactive Terminal¶

Here are some tips and tricks to enhance your terminal usage:

• Use Tab for auto-completion of commands and filenames.

• Use Ctrl+R to search through your command history.

• Use Ctrl+C to cancel the current command.

• Use Ctrl+L to clear the terminal screen.

• Use !! to repeat the last command.

• Use !<command> to repeat the last occurrence of a specific command. Example: !ls repeats the last ls command.

File Permissions and Ownership¶

Managing file permissions and ownership is crucial for system security and proper access control on Unix/Linux. Here are some commands related to file permissions and ownership:

Viewing Running Processes¶

You can view and manage running processes using the following commands:

Getting Help in Bash¶

When working in the shell, you often want to learn more about a command or explore advanced features. Common ways to get help include:

Further resources:

• Bash Official Documentation

• Bash Source Code

• Advanced Bash Scripting Guide

Text Editors¶

To work effectively on Unix/Linux systems, you need a text editor to create and modify files such as code, configuration files, or scripts within a terminal. Three most common editors you will encounter are nano, vim, and emacs.

• nano: Simple and Beginner-Friendly

  • Command: nano FILE

  • Easy to learn: commands are listed at the bottom of the screen.

  • Use Ctrl+O to save, Ctrl+X to exit.

  • Great for quick edits or when you are just starting out.

• vim: powerful but Minimal

  • Command: vim FILE

  • Modal editor:

    • Normal mode: default, used for navigation, editing commands.

    • Insert mode: typing text, entered by pressing i.

    • Visual Mode: allows for selecting blocks of text, lines, or rectangular blocks, enter by v, V, or Ctrl-v.

    • Command mode: colon commands, e.g., :w to save, :q to quit.

  • Famous learning curve

  • Almost always comes with Linux

• emacs: Extensible and Feature-Rich

  • Command: emacs -nw FILE

  • Full-featured editor that is also an environment.

  • Key commands: Ctrl+X Ctrl+S to save, Ctrl+X Ctrl+C to quit.

  • Highly customizable with its own programming language (Emacs Lisp).

Which one should you use?

• Start with nano if you are brand new.

• Learn enough vim basics to be productive, since it is installed almost everywhere (including supercomputers).

• Explore emacs if you like a fully integrated, extensible environment.

Remote Login and ssh¶

You may wonder why we spend so much time on the command line when laptops and desktops offer shiny graphical interfaces.

The reason is that a large fraction of the world’s computing power, especially in scientific computing, supercomputing, and cloud services, is still accessed primarily through the command line.

Many of these machines don’t even have a screen or keyboard connected to them! Instead, they are designed to be managed and used remotely. To interact with them, you must log in to the computer from another machine, usually over the network using command-line tools.

This is the standard way scientists, engineers, and developers work with shared computing resources such as high-performance computing (HPC) clusters, university research servers, and cloud-based systems.

HPC is a shared environment that has some steep learning curves. We will go through how to load software modules, compile/install your own packages, job submission, etc, in a later lab.

Version Control and Git¶

As projects grow, keeping track of changes becomes difficult:

• Which version of the code worked last week?

• What exactly changed between two drafts of a paper?

• How do we collaborate without overwriting each other’s work?

Version control systems (VCS) solve these problems by recording changes to files over time. They allow you to:

• Roll back to previous versions.

• Compare changes between versions.

• Work in parallel with others without losing work.

Git is the most widely used version control system today. It was created also by Linus Torvalds and has become the backbone of modern software and research collaboration.

We will use these slides to learn the basic of Git.

GitHub also supports many interesting features including GitHub Action. However, we will go through it in a later lab.