Recent searches
Search options
Administered by:
This week, I'm again starting my #SysAdmin class where we focus on a lot of practical aspects commonly missing in Computer Science curricula. I'm running the same class as an internal study group at work as "Internet Operations".
The syllabus and all course info is available here:
https://stevens.netmeister.org/615/
All video lectures are public and available for free on YouTube:
https://www.youtube.com/@cs615asa/videos
Follow along! I'll be posting weekly links in this thread throughout the semester.
System Administration
Week 1, Introduction
We cover all the meta info for the course, how System Administration fits into a Computer Science curriculum, our syllabus, systems used, grading policy etc. (There will be Venn diagrams...)
System Administration
Week 1, The Job of a System Administrator
In this video, we try to show what things SysAdmins may encounter in their day to day routine, ranging from power tools to actual and symbolic duct tape and WD40. With this, we try to sort out just how the job may be defined. It's not that easy, as it turns out...
System Administration
Week 1, Core Principles
In this video, we present a few core principles that will guide us throughout the semester, including our three pillars: Scalability, Security, and Simplicity. We'll also get to know a few basic "laws", well known by any System Administrator. If you're wondering what all this has to do with Legos, please tune in...
System Administration
Week 1, #Unix History
Exhibiting one of the cardinal virtues of System Administrators — laziness — we're reusing a video from my #programming class to give a whirlwind history tour: Bell Labs, Berkeley & the Unix Wars, USL v. BSDi, the birth of #NetBSD & #FreeBSD, the development of GNU & its adoption of #Linux, all leading to Unix on your fridge, car, and mobile phone.
Last semester's #apue thread:
https://mstdn.social/@jschauma/113087999769442250
System Administration
Week 1, AWS Aliases
In this video, we demonstrate the use of shell aliases and functions to save ourselves some typing whenever we run #AWS #EC2 commands.
To ensure we get a full dual-stack IPv4 / #IPv6 environment, we're also following this guide:
https://www.netmeister.org/blog/ec2-ipv6.html
The functions shown in the video make use of the subnet and security group described in the blog post and are available here:
https://github.com/jschauma/cloud-functions/blob/main/awsfuncs
System Administration
Week 2, Storage Models and Disks
This week is all about filesystems and storage. We start with a discussion of conceptual storage models, such as Direct Attached Storage (DAS), Network Attached Storage (NAS), Storage Area Networks (SANs), and Cloud Storage.
System Administration
Week 2, Devices and Interfaces
In this video, we'll talk a bit about storage devices and their interfaces, including SCSI, ATA, SSD, Fibre Channel, and hinting at storage configurations like JBOD and RAID.
System Administration
Week 2, Storage Virtualization
In this video, we cover the concept of storage virtualization -- combining individual disks into larger storage pools and utilizing resources from such a pool. This includes a discussion of RAID and some of the different supported levels as well as Logical Volume Management (by example on #Debian #Linux). We also illustrate some of these properties by example of ZFS (on #OmniOS).
System Administration
Week 2, Physical Disk Structure
In this video, we'll take a quick look at what an old-fashioned spinning-magnetic-platters hard disk drive actually looks like. This will help us understand the underlying addressing schemes such as CHS and LBA as well as what physical aspects affect hard disk performance as well as partitioning requirements.
System Administration
Week 2, Partitions
In this video, we'll talk about how to divide a single disk -- physical or virtual -- and how the partitions relate to the physical structure of the disk. We show examples partitioning disks on #NetBSD, #OmniOS, and #Linux using the disklabel, fdisk, and format tools.
System Administration
Week 3, The Boot Process & the MBR
In this video, we discuss the boot process on a high level as well as take a fairly detailed look at the MBR. We'll create a suitable #NetBSD BIOS partition by hand, utilizing the dd(1) command because using fdisk(8) would be just too easy. In the process, we learn a fair bit about the structure of the boot sector.
(I really should do an update of this with UEFI/GPT...)
System Administration
Week 3, Filesystems
In this video, we pretend to be a filesystem, trying to store all our cat photos in a reasonable manner on a raw disk by manually writing bytes to the raw block device. We also show how the tar(1) utility creates output that very much resembles a filesystem format. (There's a reason for that...)
System Administration
Week 3, The Unix Filesystem
In this video, we look at the traditional #UNIX Filesystem and how that is structured. We use the newfs(8) tool to create such a filesystem (well, an FFS, anyway), and the dumpfs(8) tool to inspect it.
For a slightly deeper look at the UNIX Filesystem, take a look at this video from my "Advanced #Programming in the UNIX Environment" lecture series:
System Administration
Week 3, Files go hier(7)
In this video, we're wrapping up our discussion of file systems and partitions with a look at file types (e.g., directories 
, fifos ↴↱, character devices 
, block devices 
, sockets 
, hard links 
, symbolic links →) as well as the file systems mounted by default. We close with a look at the filesystem hierarchy as defined in the hier(7) manual page.
System Administration
Week 3, Resizing file systems
Closing out our week of discussing the Dark Arts[1] of file systems, we show how to resize an existing file system on #NetBSD using the resize_ffs(8) tool. We first increase the size of a 512MB partition to 1GB, then shrink it down to 256MB:
For comparison, we then repeat the same exercise on #Debian #Linux, using resize2fs(8):
System Administration
Week 3, Additional Reading
GUID Partition Table (GPT)
https://uefi.org/specs/UEFI/2.10/05_GUID_Partition_Table_Format.html
The Security War in File Systems: An Empirical Study from A Vulnerability-centric Perspective
https://dl.acm.org/doi/10.1145/3606020
And to dive a bit deeper into #ext4 and FFS.
Understanding ext4 Disk Layout:
https://blogs.oracle.com/linux/post/understanding-ext4-disk-layout-part-1
ext4 Data Structures and Algorithms:
https://www.kernel.org/doc/html/latest/filesystems/ext4/
A Brief History of the BSD Fast Filesystem:
https://freebsdfoundation.org/wp-content/uploads/2016/04/A-Brief-History-of-the-BSD-Fast-Filesystem.pdf
System Administration
Week 4, Types of Software
With this video, we begin our topic of "software": what types of software there are, how they fit together, how to install software, and how to manage dependencies. We try to draw a terrible analogy to - what else - cars, and quickly realize that the distinctions between firmware, operating system, system software, add-on software are difficult to make.
System Administration
Week 4, OS Installation
In this video, we run through the manual installation of #NetBSD onto a virtual machine to illustrate the details of the process from partitioning and mounting the disk, extracting the sets, installing the bootloader, creating device nodes to updating /etc/rc.conf.
We also discuss planning of the OS installation by looking at data classification into shareable/non-shareable and static/variable data.
System Administration
Week 4, Package Management
In this video, we continue our discussion of the difference between the operating system and so-called "add-on software". We conclude that in order to install and maintain all such software, we want to use a package manager, and illustrate common features by example of the 'dpkg', 'rpm', and #NetBSD's #pkgsrc tools.
System Administration
Week 4, Package Manager Pitfalls
In this video, we discuss some of the problems with package managers, native language-specific packaging solutions, and the implications of their use on dependency resolution, package integrity, and trust. We revisit the "left-pad" (2016) and "dependency confusion" (2021) incidents to illustrate some of these problems.
System Administration
Week 4, Multiuser Fundamentals
We don't have a video for this segment, but here are the slides discussing group dynamics, foundations of trust (and #ZeroTrust), like Least Privilege, headless users or service accounts, group permissions, many examples of different forms of multi-factor authentication #mfa, and raising privileges (a topic we'll come back to again later in the semester).
System Administration
Week 5, Networking I: Layers
In this video, we begin our longer discussion on the topic of "networking". We're using tcpdump(8) to capture a single TCP SYN packet and start looking at the MAC and IP information, teasing out each individual byte. And yes, we will mention those OSI layers...
System Administration
Week 5, Networking I: IPv4 Basics & CIDR subnetting
In this video, we cover the basics of the 32-bit IPv4 address and how we organize networks using Classless Inter-Domain Routing or CIDR subnetting. (Don't worry, #IPv6 comes in our next video.)
System Administration
Week 5, Networking I: #IPv6 Basics
In this video, we get familiar with our Big Hero IPv6, looking at the structure of the IPv6 header and IPv6 address representations.
System Administration
Week 5, Networking I: IP Allocation & IPv4 Exhaustion
Mommy, where do IP addresses come from? We discuss how IANA allocates IP addresses to the Regional Internet Registries and try to illustrate just how large the #IPv6 address space is.
System Administration
Week 5, Networking I: The Physical Internet
In this video, we look at the physical structure of the internet, with a focus on submarine internet communications cables. Jumping from the bottom of the OSI stack all the way to Layer 9 ("political"), we then discuss how different countries use their political power to enforce internet blocks on their citizens, leading us to warrantless wiretapping in AT&T's room 641A.
System Administration
Week 5, Networking I: A Network of Networks
In this video, we look at how independent networks connect to one another, how Autonomous Systems numbers allow us to identify network operators, and how peering between independent ASs works.
System Administration
Week 6, Networking II: A simple request
In this video, we trace a simple HTTP request made via telnet to find out just how exactly our application knows how to connect to the remote server. In the process we learn about the ktrace(1) utility (strace(1) on #Linux), as well as the nsswitch.conf(5), hosts(5), and resolve.conf(5) configuration files.
System Administration
Week 6, Networking II: A simple request II
In this video, we pick up our tcpdump output collected in the previous video and begin to analyze the packets we find in there. We observe our AWS instance being scanned by some random host on the internet, see the ARP requests and replies, our DNS query, and the TCP packets for our HTTP request, yielding an illustration how a simple request causes traffic across multiple layers.
System Administration
Week 6, Networking II: ARP and NDP
In this video, we illustrate the functionality of the Address Resolution Protocol (ARP) and it's #IPv6 equivalent, the Neighbor Discovery Protocol (NDP).
System Administration
Week 6, Networking II: ICMP
In this video, we demonstrate the use of the Internet Control Message Protocol or ICMP (and its #IPV6 equivalent ICMP6) by tracing and analyzing ping(1) and traceroute(1) invocations.
System Administration
Week 7, The Domain Name System, Part I
In this video, we are beginning our discussion of the #DNS, the Domain Name System. We go back to the early days of the internet when copying /etc/hosts from system to system was the way to resolve hosts, and we cover the structure of the domain name space and the creation of the top-level domains.
System Administration
Week 7, The Domain Name System, Part II
In this video, we dissect #DNS lookups performed on our EC2 instance, then discuss just how a caching resolver performs the lookup before we turn our instance into such a resolver.
System Administration
Week 7, The Domain Name System, Part III
In this video, we try to wrap up our discussion of the Domain Name System by addressing the nature of the root nameservers, looking at various different resource record types, observing reverse lookups, and thinking about how we can have assurance of authenticity and integrity of the #DNS results returned to us via #DNSSEC.
System Administration
Week 7, HTTP and CDNs
After discussing the DNS, we now move on to #HTTP and HTTPS. While we don't have videos for these sections, hopefully the lecture slides can help you get an idea of what we're covering there. We review the basic HTTP protocol, peek at #QUIC and H3, and talk about load balancing and content delivery networks:
System Administration
Week 8, The Simple Mail Transfer Protocol
In this video, we begin our discussion of E-Mail by looking at the components of the larger mail system (the Mail User Agent, Mail Transfer Agent, Mail Delivery Agent, Access Agent); we observe the packets involved in a simple #SMTP exchange and track an email from one system to the other, both through the logs and on the wire, before we then learn to speak SMTP via telnet(1).
ah, a simpler time. when you didn't need a MUA to send mail because you could just telnet to the SMTP port and input your message from the terminal. :)
@paul_ipv6 @jschauma I still routinely do that as a means testing mail servers. So much fun!