Originally Published: Friday, 25 February 2000 Author: Rob Bos
Published to: featured_articles/Featured Articles Page: 1/1 - [Printable]

More Stupid Linux Tricks

Let's face it: Unix isn't a single, large product -- it's a fantastically modular system. It's so modular, in fact, that every component can in theory be replaced! So much so, that there is no single part of the Unix architecture that can, strictly speaking, be called Unix.

   Page 1 of 1  

Surprisingly, I got quite a bit of response from my last piece, "Stupid Linux Tricks." That article was something I'd dashed off at the last second because I hadn't thought of anything provocative or rant-worthy to write about. Still, it seems to have hit an interesting chord with quite a few people. The idea that the best way to convert a person is to demonstrate some of the incredibly cool, useful, or just plain well-designed stuff about Linux. So, since I'm out of ideas this week too, I'll do something similar -- but instead concentrating on some of the smaller tools and what they can do.

Let's face it: Unix isn't a single, large product -- it's a fantastically modular system. It's so modular, in fact, that every component can in theory be replaced! So much so, that there is no single part of the Unix architecture that can, strictly speaking, be called Unix. The Linux kernel is just an implementation of an API, and an abstraction layer for hardware. That's not Unix. Neither are any of the potentially thousands of programs that make up Solaris, Linux, BSD, or any of the other ones. Any individual component is trivial, and is a part of a larger, cohesive whole.

Within that collective framework of tools, the user learns how to take advantage of them, one by one, learning how they interact and lock together, and their purposes and use. Often, a tool is used for purposes very unlike its intent. Almost frighteningly so, in fact. Very impressively so. Showing someone what these tools, few of which work as well as eye candy, can do to make your life easier is how you "clinch the sale" so to speak. So what if Linux can do all these cool things to impress people? Now you have to show them what it can do to make their lives easier in some small way.

dd. Yes, dd. Take or generate a stream of bytes, and do something with it. With this nifty little utility, you can create disk images from scratch and format them. You can create disk images from existing devices (and then mount those images, mount -o loop filename.img!), write disk images to devices, send output from one device to another, or any number of interesting things. Say I want to create an image of a driver floppy for later, just in case I need it, and stick it in an archive somewhere:

dd if=/dev/fd0 of=driverfloppy.img

(if and of being "inputfile" and outputfile" respectively)

Or say I want to create a file exactly one gigabyte in size, composed of nothing but null bytes.

dd if=/dev/zero of=zerofile bs=1024 count=1000000

(bs being "block size" and count being the number of blocks to write)

Note that this gigabyte-of-zeroes compresses quite nicely. Just under a hundred bytes, with a little work. Now say you wanted to make a backup of your hard drive as a disk image, compress it, and write the raw data to a file:

dd if=/dev/hda | gzip > hda.img.gz

(Note that if you put hda.img.gz on the same disk that you're reading from, you likely won't get very good compression. Note that there will be a lot of "free" space on the drive consisting mostly of deleted files. If you do a dd if=/dev/zero > blah.img, you'll fill up all the free space on your drive with null bytes, enabling far higher compression -- in theory.)

And so on. This tiny little 25-kilobyte utility, designed to do exactly one thing in life -- take bytes from somewhere and do something with it -- is conceivably one of the most nifty things about Unix. Now, we take this utility and add a little something -- say, wget, or curl.

Both wget and curl have exactly one job in life: to parse a URL, grab the file referenced to it, and save it to disk. It could be a zip file, Web site, FTP site, or whatever. Nothing more. For instance:

wget http://www.linux.com will return the file that that URL references. But that's not all it can do.

wget -r http://www.linux.com will recursively follow every link at Linux.com referenced within its Web pages and download them, creating directories as it goes along. It will not download anything outside www.linux.com, but it will grab everything at that site.

wget can do way more than that, of course. But combine it with dd, and what do we have? A way to store a disk image on a remote server and get it with minimal hassle. Now, given the very nifty little single-floppy Linux distributions that have been kicking around, suppose you have a lab full of NT machines that you've imaged at some point, perhaps using multicasting or something similar. One of them breaks for some indefinable reason, so you need to reimage the bugger. Setting up a crossover cable would be a pain (although it's doable), but there's an easier way. With this method, you can put together a boot disk, create a disk image and store it virtually anywhere (ftp, http, or even an ssh server, or a Netware server, NFS server, Appleshare volume), and use wget or dd to image the machine directly! wget -O - ftp://blah/blah.img > /dev/hda is a relatively trivial thing to do, when you consider the alternative. In some situations, this can be a very useful thing to do.

wget can recursively grab an entire ftp site. Hell, wget could in theory mirror the entire Internet with a relatively simple command and lots of disk space, by recursively following links on Yahoo to a hundred levels. You'd miss a lot of content (databases, CGI scripts and the like) but you'd certainly get a lot of it.

Now is that just blindingly cool or what? You could put wget in a loop to grab content from a CGI-generated script (for example, for i in 1 2 3 4 5; do wget http://server/blah.pl?option=$i;done), or to submit information to a Web site (wget -U"Mozilla/4.6 [en] (X11; I; Linux 2.2.13 i686; Nav)" -O/dev/null http://www.thehungersite.com/cgi-bin/donate.pl 1> /dev/null 2> /dev/null -- this makes wget pretend to be Netscape, putting the output to /dev/null, sending the rest of the output to /dev/null, and loads thehungersite's auto-donation clickthrough), or whatever. The uses for this deceptively simple command go far, far beyond simply grabbing a file, just like every other Unix command.

Now, I'll digress a bit into more user-interface niceties. "alias ls='ls --colour'" is one that you can put into your shell startup scripts (.cshrc, .bashrc, /etc/profile, whatever) to colourise "ls" output. This makes it much easier to pick out files and directories.. since directories are highlighted in blue, executables in green, and so on, you spend much less time flipping through mundane text. Why this isn't enabled by default in every Linux distribution is a mystery to me, but there must be a good reason for it.

There are hundreds of these little tips and tricks. Check out the s database at Linux.com TuneUp for more of them, sorted by category. Each of them cover either novel uses of old commands, interesting interactions between them, or interesting ways of configuration.

In any case, most salesmen will tell you that in order to sell something, you have to get the target's attention, and get them interested -- then show them how the product will fit into their lives and make their lives easier. There are many ways to do this, and they're different for each person. You just have to find it.

Of course, it's easy to forget that Linux isn't for everybody. The power, freedom, and flexibility that Linux and Unix give people is accessible to anyone who puts in the effort and wants to learn it For those people, Linux will reap great rewards. For the people who don't want to dive into the guts of their system, Linux will only be another vaguely different magical interface to a computer. While those people will be a vital element of protecting Linux from monopoly power and as such must and will be welcomed, they're going to miss out on a lot of things. So what should be the general reaction? Let people learn no more about their computer than they want or need to learn. This is what Linux is working toward. The guts of the system are still there for anyone to see, but they'll work as well whether you understand them or not. This is where I could make a snide comment about other operating system forcing you to dig into their guts to fix problems inherent in their design, but I won't. Linux's maturation process will make this ideal possible. Right now, Linux takes expertise -- no doubt about it -- but it's a good, solid foundation that we can build on. Worry about making something work first, then make it easy to use.

Rob Bos is on the computer lab staff and a student and Simon Fraser University. He listens to way too much music in languages that he does not understand.

   Page 1 of 1