Networking for System Administrators: IT Mastery, #5

Start Reading

Full Table of Contents

About the Author

Copyright Information

More Tech Books from Michael W Lucas

Absolute BSD

Absolute OpenBSD (1st and 2nd edition)

Cisco Routers for the Desperate (1st and 2nd edition)

PGP and GPG

Absolute FreeBSD

Network Flow Analysis

the IT Mastery Series

SSH Mastery

DNSSEC Mastery

Sudo Mastery

FreeBSD Mastery: Storage Essentials

Networking for Systems Administrators

Tarsnap Mastery (coming Feb 2014)

Acknowledgements

The people who most deserve thanks for this book are the folks who struggled through me learning networking as I stood between them and what they wanted to accomplish. Every one of you brought me some horrible issue that educated me even as you ranted and cried and begged for me to fix the problem. I learn slowly, and you suffered for it. Thank you. Fortunately, suffering builds character, so you got something out of it and I don’t have to feel too bad.

This book had a crew of excellent technical reviewers. Some of them have an understanding of networking that crushes mine. Others knew nothing about networking, but were able to tell me when I confused them. Both are invaluable. They are, in alphabetical order: Alexiei Bottino, Donald Cooley, Fred Crowson, Michael Dexter, Dominik Douville-Bélanger, Edwin Groothuis, Josh Grosse, Bryan Irvine, Chris Josephes, Frank Moore, Kurt Mosiejczuk, Scott Murphy, Chris Parr, Martin Pugh, Mike O’Connor, A.J. Reese, Amanda Robinson, Jim Salter, Justin Sherrill, Carsten Strotmann, Grant Taylor, and Giovanni Torres. You all had excellent advice and lots of really good recommendations. Those recommendations would have made this book four times longer, but they really were excellent.

This book was made possible through hardware purchased from iX Systems (http://www.ixsystems.com). Well, not exactly possible. More like a heck of a lot easier than trying to keep a maddening mishmash of recycled debris booting without bursting into great fountains of toxic flame.

And a special thanks to the people who’ve thrown a few bucks into my Tilted Windmill Press tip jar. You folks make writing for a living a lot more realistic.

For Liz

Chapter 0: The Problem

Dear systems administrators: the firewall people don’t want to talk to you, either.

It’s nothing personal. We all share the goal of delivering service to users, but once you break that goal down into meaningful parts our teams completely diverge. Our tools differ. Our equipment differs. We even think differently. Sysadmins care about bytes, network administrators measure everything in bits. Network equipment might be built on computer hardware, but it’s very specialized hardware that doesn’t have any of the tools that systems administrators take for granted. Servers have network interfaces, but not nearly enough of them to do anything interesting.

Neither one understands how the other can possibly perform their job without the basic tools their platform offers.

Both roles require a high degree of specialization, especially in modern enterprises. The firewall administrator doesn’t have time to dig into the specifics of the latest version of whatever operating system you’re using. You don’t have time to figure out why the newest version of the big firewall is mangling your carefully-crafted HTTP headers.

This is all complicated stuff. While I spend more time in systems administration than network engineering, I’ve filled both roles in the last twenty years. Each time I switch from one hat to the other I spend a few weeks catching up with the latest annoyances.

The end result? The network folks blame the servers. The server people blame the network. Often the blame gets personal. It’s the sysadmin’s fault! If the firewall crew knew what they were doing, this wouldn’t happen! Meanwhile the helpdesk folks—correctly—blame everyone for not making customers stop whining. I’ve been in organizations where the only thing that prevented open warfare between IT teams was a shortage of sharp stabby objects.

Even in the best environments, differing expertise and priorities make both jobs more difficult than they have to be. Many organizations avoid this warfare by applying trouble tickets, workflow, and meetings. Lots and lots of brain-numbing meetings.

It doesn’t have to be this way.

A systems administrator can’t learn the ins and outs of each version of networking gear any more than a network administrator can learn the ins and out of the latest generation of your operating system. Neither one of you has the time to keep up with this constantly changing information on top of your own area of expertise.

A network administrator can—and should—learn the basics of how a server operates. Every network administrator should understand the basics of user access control and privileges, processes, services and daemons, basic installation and removal of software, and so on. But this information varies wildly between operating system platforms. Sometimes it’s a language difference—Unix-like operating systems have daemons, while Microsoft systems have services. Sometimes even closely related operating systems have very different ways to handle similar tasks, such as the myriad ways of installing software on various Unix-like operating systems, or even on one operating system!¹ The network administrator might learn the basics of the operating systems in your organization, but this knowledge won’t carry forward to her next assignment.

Systems administrators can learn the basics of networking, however. And this knowledge will serve you no matter what organization you work with or what sort of network gear your organization uses. You don’t need to know how to configure a router or a firewall or any other network device—they’re all ephemeral anyway.

But basic TCP/IP knowledge endures. While people add new protocols all the time, these are incremental changes and easily mastered. It’s much easier to teach a systems administrator the basics of networking than it is to teach a network administrator the basics of systems administration, and that knowledge will last your entire career.

Understanding the network saves you time. You won’t wonder if a network change has been made—you’ll check it yourself. You won’t call to see if a problem is inside your network—you’ll look and find out. You’ll quickly determine if problems exist on your systems, on your network, or outside your network.

Most network administrators quickly learn which systems administrators understand basic networking and which don’t. When I’m a network administrator I’m happy to work with the sysadmins that don’t ask me if I’ve opened that firewall port or if there’s a problem between here and our office in Farawayistan. Being asked When will the link to Farawayistan be fixed? might be harder to answer, but it does save a loop in the conversation.

If I’m a network administrator with a whole stack of issues to resolve, but I know that you speak from evidence when you say This traffic isn’t reaching my server, I’ll address your problem before everyone else’s. There’s a really good chance that I can fix your problem quickly because you provide me with actual information. If my phone is ringing like mad and everything seems to have collapsed, resolving your problem might solve problems for a whole bunch of other people.

Make yourself the most valued member of your systems administration team. Take a couple hours to read this book, learn a little networking, and become a bridge to other critical IT groups.

We’ll start by discussing network principles, and then go into detail on how to view or use those principles on multiple operating systems. This book covers Windows, Linux (CentOS and Debian), and BSD platforms, but the principles and tools run on just about any modern networked operating system, including portable devices like phones and tablets.

Who Should Read This Book?

Every sysadmin, database admin, web admin, developer, and computing professional should understand the basic principles of networking. This book grounds you in modern TCP/IP without demanding a month’s dedicated study. Understanding the network will empower you to identify the real source of problems, solve your own problems more quickly, and make better requests of your team members.

This book is also for network administrators who need to educate others in their team about the essentials of networking. After a few years, a network administrator’s understanding of TCP/IP turns into an interconnected morass of window scaling and sequence numbers and malformed packets. Someone asks what a port is, and moments later you’re explaining SYN floods and the questioner has learned the vital lesson of "never ask the network administrator anything." (This trait isn’t exclusive to network administrators—it’s endemic in the IT industry. Ask a database administrator to explain databases sometime.²) That stuff is all vital to a network administrator’s job, but the average user doesn’t need to understand it. You can use this book to explain only what the average sysadmin absolutely must know about TCP/IP.

Server versus Network Device

By server I mean a computer, running an operating system, whose main task is providing services to other servers or users, rather than supporting the network. A sysadmin is someone responsible for managing such devices.

Some network administrators build routers, firewalls, proxy servers, intrusion detection devices, and more out of carefully selected server hardware. When this book says server, I’m explicitly excluding such custom-built devices. Elsewhere I say that a server should never do X, but if you’ve built a device whose purpose is doing exactly that, it’s an exception.

When I mention a router or proxy server or any number of other network devices, I mean a device that fills that role. It doesn’t matter if it’s a black box solution or something built out of commodity hardware. The network administrator is the person who manages that equipment.

A Note to Network Administrators

Some readers are network administrators, wondering how the heck I’m going to teach networking in a few pages. Let me answer your questions before you ask them.

I don’t dive deep into network protocols. My explanations might not be totally accurate for all situations and all environments. Every protocol has its edges, and I’m not trying to cover them all. The goal of this book is not to make sysadmins networking professionals, but to equip them with the skills they need to take better care of themselves and disturb you less frequently.

I skip a lot of old knowledge. The Ethernet chapter covers switches and not hubs. You won’t encounter a hub by accident these days. They are specialized devices only deployed in specific cases.

I don’t cover many traditional networking topics, because they’re not absolutely essential. SNMP, or the Simple Network Management Protocol, is one example. I could fill a book this size discussing SNMP. I do discuss how ICMP is built on top of IP, when the specification wedges it in this misshapen role between the network and transport layers. But someone who’s unclear on TCP versus UDP doesn’t need to go into SNMP right now, or Netflow, or VLAN propagation, or any of the innumerable protocols used to manage and diagnose networks. Understanding SNMP won’t change someone’s relationship with the network team the way understanding TCP/IP will.

Always remember that I’m talking to non-network administrators. I’m not going to tell sysadmins that they can, say, use a /112 IPv6 subnet, because not everybody’s equipment can do that. I play the heavy here by spelling out the rules: you get to swoop in and tell your people that yes, your network can break certain rules because you are so totally amazing. I’m also consciously and deliberately glossing over some protocol details, so you can explain them and make your sysadmins think you’re terribly smart. You can thank me later.

Network Tools

If you look around you’ll find innumerable server-side tools for analyzing the network. Many of these tools work only on very specific operating systems or have limited utility. I will cover tools that work across both Microsoft and Unix platforms, and have been widely ported to less common operating systems like VMS.

Yes, some operating systems have their own network analysis tools, but that knowledge isn’t portable throughout your career. If a piece of software is Windows-only, or Oracle-only, or Debian-only, I won’t cover it. Once you understand the cross-platform tools, you can quickly apply that knowledge to platform-specific tools.

The Windows network debugging tools are all designed for the command line. I recommend you not use the traditional Windows command window CMD.EXE. Try Microsoft’s PowerShell or Cygwin’s mintty. Any version of either one should suffice, although newer versions are probably less buggy and more secure. Neither one will sink its fangs into you, and after about three minutes of acclimation you’ll be much happier. I normally use Cygwin and mintty, but the software behaves the same under PowerShell.

You can combine all of these tools with your usual system utilities like tail(1), Wordpad, or whatever your preferred platform offers.

Here’s the main tools I cover.

ifconfig, route and ipconfig

A host’s network configuration includes its IP addresses and gateway. On Unix, use the route and ifconfig commands to view the system’s network configuration. Windows systems put all of these in the ifconfig command. While different operating systems have different versions of these commands, you can sort out the information you need from any of them.

Microsoft has added networking functions to PowerShell. While I do recommend learning and understanding PowerShell, recommending you start with PowerShell would be like shoving you in the deep end of the ocean and telling you to swim back to Switzerland. If networking becomes a big part of your work, get comfortable with PowerShell’s networking commands.

Similarly, some Linux variants have started to move