Shiny and New: Emacs 22

I finally upgraded to Emacs 22 a few weeks ago, and now I'm wishing I'd braved it sooner. Technically it's not released yet; I'm working from a build of a cvs snapshot from a month or so ago. But the Emacs dev team works pretty hard to make sure it has problem-free builds on a whole slew of platforms, so just following their instructions has a pretty good chance of working for you.

It's worth the effort. Truly. Reading through its NEWS file, there's just tons and tons of new functionality. It's going to take me some time, maybe a few weekends, just to absorb it all.

Personally, though, I think there are two features that by themselves justify the entire effort of upgrading: the Unicode and UTF-8 support, and the enhanced replace-regexp command.

International At Last

It's been a very long wait for Unicode and UTF-8 support, and now that I have it, I could never go back. There isn't much to say about it, except that it works. Seamlessly. It used to be hard to get international characters into and out of Emacs, because it had its own custom way of dealing with them. Now it's a snap.

In fact — here, I'll show ya. If you type C-h h, it brings up the HELLO file, which contains greetings in a variety of languages. Here's some Chinese: 中文,普通话,汉语. Here's some Korean: 안녕하세요, 안녕하십니까. Here's some Russian: Здравствуйте!

I'm not doing anything special; I'm just copying the strings out of the HELLO buffer and into my html buffer, and saving the file. I added the content-type header line in this HTML file, and all the characters just show up effortlessly in Firefox.

If you can't see them in your browser, well... Firefox is free. Or it might be a font problem on your system. As far as I'm concerned, any problems you may have in viewing them is no longer the fault of my Emacs session, which makes me Happy. Speaking as a developer who needs to internationalize every program I write, I can't begin to tell you how useful it's been to have seamless editing of utf-8 encoded files for the past month.

Right there, that feature alone is worth the upgrade.

But wait, there's more... Even though Emacs 22 has a bunch of noteworthy and exciting new features, blah blah blah, I'm going to blithely ignore them all today and focus with single-minded zeal on just one feature. It has a teeny tiny entry in the NEWS file; it's barely mentioned, really. I'm sure it was a thousand times less work than the UTF-8 support, but even so, it might well be strong enough on its own to justify the (moderate) pain of upgrading from Emacs 21.

Take a look at my examples and see if you agree!

Replacement Super-Powers

Emacs 22 sports an amazing new editing feature that's had me drooling in anticipation since I first heard about it, maybe six or eight months ago. As you can well imagine, that's a lot of drool.

And what might the feature be, you ask? Well, they've enhanced M-x {query-}replace-regxp to accept lisp expressions to be evaluated in the replacement string.

That might not seem like a big deal, so let's run through some examples, from simple to very fancy.

Example: Changing Case in Replacement Strings

Have you ever wanted to change the case of certain letters in the replacement string for M-x replace-regexp? It used to be a real pain; you either had to write a Lisp function or fix them all by hand. Now it's trivial.

As a simple demonstration, let's say you have a list of names that you need capitalized, like so:

bob
sue
ralph
alice
jimmy
preston
billy joe jim bob

It's a contrived example, since Emacs already has M-x capitalize-region. Or you could use C-u 10 M-x capitalize-word. But let's try it with the new replace-regexp evaluation feature to see how it works.

It's just like a normal M-x {query-}replace-regexp, but you'll prefix any lisp expressions in the replacement string with the sequence `\,' (i.e., a backslash and a comma). In this case, we match the whole word, and invoke the Emacs-Lisp function `capitalize' to capitalize the word we just matched:

M-x replace-regexp
  Replace regexp:  \(\w+\)
  Replace regexp with:  \,(capitalize \1)

and we wind up with each word capitalized, just like we wanted:

Bob
Sue
Ralph
Alice
Jimmy
Preston
Billy Joe Jim Bob

Unlike the capitalize-{word/region} commands, which have hardwired behavior, the {query-}replace-regexp commands give us tremendous flexibility. For instance, we could have capitalized the last letters of the names instead, by splitting each word into two regexps, with the second regexp matching just the last character. Then it's a simple matter to reconstruct the word with the last letter capitalized:

M-x replace-regexp
  Replace regexp:  \(\w+\)\(\w\)
  Replace regexp with:  \1\,(capitalize \2)

..to get the reverse capitalization we wanted:

boB
suE
ralpH
alicE
jimmY
prestoN
billY joE jiM boB

For a somewhat more realistic example, let's say you've defined some "getter" functions in a Java class, like so:

  public Relative father() { return this.father; }
  public Relative mother() { return this.mother; }
  public Relative sister() { return this.sister; }
  public Relative brother() { return this.brother; }
  public Relative auntie() { return this.auntie; }
  public Relative uncle() { return this.uncle; }
  ...

and your code reviewer wants you prepend the word "get" to each of them.

Well, this is a classic refactoring situation ("rename method"), but you're going to have to invoke the refactoring manually for each method, and you may have hundreds of them.

If these methods have been around a while, and they're being referenced by many external callers, then you're safest using a refactoring tool. You may even consider writing a one-off refactoring script, perhaps in Jython or Mozilla Rhino, that makes programmatic use of either your IDE's refactoring APIs or a lower-level tool such as ANTLR or JavaCC.

Whew! That's going to be a lot of work, no matter how you slice it. And if you've published your APIs externally, then you're screwed; all you can do is @deprecate the old names and hope people stop using them someday.

But that's why you get your code reviews done early, right? In many real-world situations, you're performing a rename-method on a new class that has no external callers yet. And in those situations, Emacs 22 will get the job done far faster than a refactoring IDE can.

In this case, we'd just do a straightforward replacement with capitalization, similar to the one in our last example, like so:

M-x replace-regexp
  Replace regexp: \(public Relative \)\(\w\)\(\w+\)
  Replace regexp with: \1get\,(capitalize \2)\3

Et voilà:

<-- (p.s.: C-x 8 ` a gives you that neat 'à' character.)

  public Relative getFather() { return this.father; }
  public Relative getMother() { return this.mother; }
  public Relative getSister() { return this.sister; }
  public Relative getBrother() { return this.brother; }
  public Relative getAuntie() { return this.auntie; }
  public Relative getUncle() { return this.uncle; }
  ...

Even if you do most of your coding in the comfy confines of a visual IDE, it can be awfully handy to keep Emacs around for your fine-grained text surgery.

Now we can move on to some more interesting examples, so you can feel you got your money's worth out of today's blog entry. But first...

A Note About Emacs Regexps

Emacs regular expression syntax is very old, predating Perl 5's fancier regex syntax by almost a decade. Perl's regexp enhancements are the defacto standard, supported in virtually all major programming languages. Unfortunately, nobody has ever seen fit to retrofit poor Emacs with an alternate Perl-compatible regexp syntax, so Emacs regexps are now nonstandard and a bit awkward. Here are a few of the noteworthy differences:

• You have to escape the (, ), {, }, and | metacharacters. That is, they're not metacharacters by default — without the backslash, they match themselves.


• There's no '\d' shortcut for the [0-9] character class.


• There are no lookahead or lookbehind assertions.


• There are no direct equivalents for Perl's {n}?, {n,}?, {n,m}?, /i, /m, /s, /x, \G, or (?# ...) constructs.

But Emacs still supports some of the constructs you've come to expect:

• You can specify exactly N repetitions with `\{N\}', and between N and M repetitions (inclusive) with `\{N,M\}'. E.g. [0-9]\{3\}-[0-9]\{4\} matches a 7-digit phone-number in the format xxx-xxxx.


• You can have backreferences to previously matched groups in the regular expression. E.g. \(re\).*\1 matches words with "re" appearing at least twice, like carefree and première.


• You can specify "shy" groups that don't record the match with \(?: ... \).

There are also some Emacs-specific enhancements, such as matchers for entries in the mode-specific syntax tables. The Info pages have more details on Emacs regular expressions.

If you plan to be more than a casual Emacs user, you should study the regexp syntax carefully, because there are many useful commands in Emacs that operate on regular expression matches. The better you know the Emacs-specific syntax, the more productive you'll be.

I suppose before we move on to the next example, I should preemptively answer one of the most frequently asked questions about Emacs regexps.

Q: How do I embed a newline in a regexp I'm typing into the minibuffer?

A: You use the key sequence C-q C-j. The C-q invokes the Emacs `quoted-insert' command, which basically says "insert the next character literally, without invoking any commands with it." C-j (i.e., control-j) is how a newline character is represented in Emacs.

C-q is a useful general-purpose Emacs command. Whenever you want to insert a character (in the minibuffer or a regular buffer), and it's just refusing to go in, C-q <char> will almost always do the trick.

Example: Numbering Lists

With our new replacement-with-evaluation feature, it becomes straightforward to create numbered lists. Emacs 22 has introduced a new backreferencing metacharacter, `\#', which counts the number of replacements we've done so far in the current command. So even without using any Lisp, we already have one way to make numbered lists.

Let's see... we'll need a short list of words as an example. How about all the words in /usr/share/dict/words that don't end in [a-z]? Easy enough to find out. We M-x find-file /usr/share/dict/words (only if you're on a Unix system, of course, and the location varies), and then M-x list-match [^a-z]$. Ah, perfect — our *Occur* buffer shows 32 matches:

   1987:Bogotá
   5243:Fabergé
   9772:Mallarmé
  12044:Paraná
  12499:Poincaré
  16956:abbé
  19923:appliqué
  20932:attaché
  23704:blasé
  26223:café
  26511:canapé
  29314:cliché
  31431:consommé
  38981:décolleté
  42995:fiancé
  43623:flambé
  44996:frappé
  48317:habitué
  58328:macramé
  58898:manqué
  62514:naiveté
  65243:outré
  66710:passé
  71609:protégé
  73675:recherché
  76387:risqué
  76847:roué
  77811:sauté
  82455:soufflé
  89055:touché
  96268:émigré
  96274:études

They're prefixed by their line number, but we can make that disappear during the replacement. Let's turn them into a numbered list. First copy the matches into a new, writable buffer, then M-C-< to go to the top of the list, and then:

M-x replace-regexp
  Replace regexp: \(.+:\)
  Replace regexp with \#. 

Boom!

0. Bogotá
1. Fabergé
2. Mallarmé
3. Paraná
4. Poincaré
5. abbé
6. appliqué
7. attaché
8. blasé
9. café
10. canapé
11. cliché
12. consommé
13. décolleté
14. fiancé
15. flambé
16. frappé
17. habitué
18. macramé
19. manqué
20. naiveté
21. outré
22. passé
23. protégé
24. recherché
25. risqué
26. roué
27. sauté
28. soufflé
29. touché
30. émigré
31. études

Oooh, but only Computer Science students like lists numbered from zero. So let's M-x undo (using C-/ of course — everyone uses C-x C-u, but that's way too many keystrokes for something as common as Undo!) and use a tiny bit of Lisp to start the numbering at 1.

It so happens that Emacs-Lisp defines a function called 1+, which increments a number. So we can just wrap the `\#' in our replacement string with that function, like so:

\,(1+ \#).

<-- (There's a trailing space after the ".")

The result is just what we wanted:

1. Bogotá
2. Fabergé
3. Mallarmé
4. Paraná
5. Poincaré
6. abbé
7. appliqué
8. attaché
9. blasé
...

The lisp 1+ function operates on numbers, not strings, so you might have expected it to barf with a wrong-type-argument error. Our example works because the \# metacharacter returns a count of matches so far, which is a number, not a string value. In our next example, we'll have to do the conversion ourselves.

Example: Re-numbering Lists

We can use Lisp-code snippet similar to our previous one to renumber an existing list. Let's say we want to insert a word in a numbered list, like so:

1. Bogotá
2. Fabergé
3. Flambé     <-- (We inserted this one.  Nice word, eh?)
3. Mallarmé
4. Paraná
5. Poincaré
6. abbé
7. appliqué
8. attaché
9. blasé
...

Easy to fix in Emacs22: place the cursor just after the word we inserted, and M-x replace-regexp `^\([0-9]+\)' with `\,(1+ (string-to-int \1))'. The result:

1. Bogotá
2. Fabergé
3. Flambé
4. Mallarmé
5. Paraná
6. Poincaré
7. abbé
8. appliqué
9. attaché
10. blasé
...

This time we used a numbered backreference (\1), which always returns a string. Don't be fooled by the fact that we appear to be matching a number: the regexp `^\([0-9]+\)' matches a string containing numeric digits, and we have to do a type conversion (using the Emacs-Lisp builtin function string-to-int) if we want to increment it.

I hope by now you're beginning to suspect that knowing a little Emacs-Lisp can help you immensely with your editing tasks. Believe it! (Not surprisingly, knowing a lot of Emacs-Lisp helps even more.)

Example: Alphabetically Numbered Lists

Let's say we have a list of 26 or fewer items, and we want to "number" it with A, B, and C rather than 1, 2, and 3.

Well, shoot. The list we've been using has 32 items. I'd prefer a list of 26 (or so) for this example.

Let's see... we can write a wee Lisp function to group the words in /usr/share/dict/words by their ending letters a-z:

(cl-prettyprint
 (save-excursion
   (set-buffer "words")
   (loop for c from ?a to ?z
         collect (let ((i 0)
                       (tail (string c)))
                   (beginning-of-buffer)
                   (while (re-search-forward (concat tail "$") nil t)
                     (incf i))
                   (cons tail i)))))

We just evaluate this snippet in our *scratch* buffer (by typing C-j after the last paren). It crunches the words buffer and produces:

(("a" . 1625)
 ("b" . 167)
 ("c" . 772)
 ("d" . 8331)
 ("e" . 7190)
 ("f" . 191)
 ("g" . 7401)
 ("h" . 991)
 ("i" . 457)
 ("j" . 4)
 ("k" . 784)
 ("l" . 2041)
 ("m" . 920)
 ("n" . 4347)
 ("o" . 718)
 ("p" . 450)
 ("q" . 5)
 ("r" . 4279)
 ("s" . 44857)
 ("t" . 4454)
 ("u" . 140)
 ("v" . 46)
 ("w" . 247)
 ("x" . 182)
 ("y" . 5519)
 ("z" . 124))

Hmmm... nothing really promising. We only get 9 words if we combine the ones ending in "q" and "j". A minor tweak to our search function will show us words ending with a doubled letter:

(cl-prettyprint
 (save-excursion
   (set-buffer "words")
   (loop for c from ?a to ?z
         collect (let ((i 0)
                       (tail (concat (string c) (string c))))
                   (beginning-of-buffer)
                   (while (re-search-forward (concat tail "$") nil t)
                     (incf i))
                   (cons tail i)))))

Evaluating it gives us:

(("aa" . 2)
 ("bb" . 3)
 ("cc" . 1)
 ("dd" . 5)
 ("ee" . 136)
 ("ff" . 75)
 ("gg" . 4)
 ("hh" . 0)
 ("ii" . 4)
 ("jj" . 0)
 ("kk" . 0)
 ("ll" . 291)
 ("mm" . 2)
 ("nn" . 38)
 ("oo" . 30)
 ("pp" . 4)
 ("qq" . 0)
 ("rr" . 13)
 ("ss" . 1276)
 ("tt" . 58)
 ("uu" . 1)
 ("vv" . 0)
 ("ww" . 0)
 ("xx" . 0)
 ("yy" . 0)
 ("zz" . 9))

Looks like we have more options with this list. Maybe if we just take the ones with a count of 5 or less... looks like 26 of them. Perfect!

We could write a little more code to extract the words matching our criteria, but it's clearly going to be fastest to eyeball it. So we call M-x list-matching-lines with the regexp \(aa\|bb\|cc\|dd\|gg\|ii\|mm\|pp\|uu\)$, and we get our list of 26 words:

   2145:Bragg
   3436:Cobb
   5662:Fromm
   6284:Gregg
   6317:Grimm
   6675:Hawaii
   8025:Judd
   8290:Kellogg
   8411:Kidd
   8509:Knapp
   8645:Krupp
   8689:Kwanzaa
   8804:Lapp
  12549:Pompeii
  15425:Todd
  16257:Webb
  16641:Yacc
  17702:add
  21427:baa
  39145:ebb
  39372:egg
  46305:genii
  62411:muumuu
  64283:odd
  72801:radii
  78139:schlepp

And what a fine bunch of words they are. Just try doing that exercise in Java or C++ sometime.

In any case, now we have a list for our example, and we want to number it alphabetically. So we need to replace all the cruft up through each ':' with a counter converted to an alphabet character.

As we saw in our little function that produced this word list, Emacs uses `?c' syntax to represent characters, and internally they're just ints. So we just add the character `?a' to our `\#' counter this time, to loop through the characters 'a' to 'z':

M-x replace-regexp
  Replace regexp:  ^\(.+:\)
  Replace regexp with:  \,(+ ?a \#)) 

And, ladies and gentlemen... behold!

97) Bragg
98) Cobb
99) Fromm
100) Gregg
101) Grimm
102) Hawaii
103) Judd
104) Kellogg
105) Kidd
106) Knapp
107) Krupp
108) Kwanzaa
109) Lapp
110) Pompeii
111) Todd
112) Webb
113) Yacc
114) add
115) baa
116) ebb
117) egg
118) genii
119) muumuu
120) odd
121) radii
122) schlepp

D'oh!!!! I forgot to convert the counter back to a character. Haha. Oops.

After a quick C-/ to undo the operation, we can just change the replacement regexp to `\,(string (+ ?a \#))) ', and we finally have our alphabetically-enumerated word list:

a) Bragg
b) Cobb
c) Fromm
d) Gregg
e) Grimm
f) Hawaii
g) Judd
h) Kellogg
i) Kidd
j) Knapp
k) Krupp
l) Kwanzaa
m) Lapp
n) Pompeii
o) Todd
p) Webb
q) Yacc
r) add
s) baa
t) ebb
u) egg
v) genii
w) muumuu
x) odd
y) radii
z) schlepp

Or we could get capital letters by using `\,(upcase (string (+ ?a \#)))) ' instead.

Note that M-x replace-regexp has its own command history list, so you can just use up-arrow to fetch old regexps you've entered, and tweak them in place. Easier than re-entering them from scratch every time.

Some Even Snazzier Examples

So far we've used this amazing little new feature to generate (and renumber) various lists, and to change the capitalization of the replacement text on the fly (in two different ways). Both very practical and useful transformations.

In our next example, we'll assume you're working on a Java-based Web application, because your company is too lame to let you use Ruby on Rails. Hypothetically speaking, of course.

Suppose you have some JSP files containing references to various static images, e.g. <img src="images/foo_bar.gif">, and you decide you want to change them to calls into Java code to fetch the image URLs as the page is composed. So "images/foo_bar.gif" needs to change to (say) <%= StaticImageManager.FOO_BAR_GIF.getUrl()%>.

Well, clearly no fancy-pants refactoring IDE on the planet is going to be able to help you with this. If you're an Eclipse or IntelliJ or Visual Studio user, get ready for some carpal tunnel while you manually change every instance.

However, if you've followed the examples so far, you know it's trivial in Emacs 22:

M-x replace-regexp
  Replace regexp:  "images/\([a-z_]+\)\.\(gif\|jpg\)"
  Replace regexp with: <%= StaticImageManager.\,(upcase (concat \1 "_" \2)).getUrl() %>

and they're all fixed in the blink of an eye.

But you knew that by now. This example wasn't more complex than the others, just a little longer.

It starts to get even more interesting if you permit side effects in your lisp expressions. That is to say, persistent changes to the world, whether it's Emacs variables, your buffer configuration, or even your filesystem. You have to be a bit more careful, but you can use the new replace-regexp eval feature as a powerful interactive scripting engine.

Our last example will be opening files. Often you'll find yourself looking for files using the Unix `find' command in a shell. But what if you want to open the files it turned up?

Again, it's a slightly contrived example, because it's already possible to use Unix shell commands and the "emacsclient" program to instruct Emacs to open the files you find. But it should suffice to show you what we mean by "side-effecting replacements".

A simple example should work. Let's go to our installed emacs lisp directory. Mine's /usr/share/emacs/22.0.50/lisp, which I found by looking at my `load-path' variable in my *scratch* buffer. There are various subdirectories, including textmodes/, progmodes/, and others.

To have Emacs open (say) all the elisp files beginning with the letter `x', we M-x shell, cd to /usr/share/emacs/22.0.50/lisp, and use the `find' command:

/usr/share/emacs/22.0.50/lisp>find . -name "x*.el"
./progmodes/xscheme.el
./term/x-win.el
./term/xterm.el
./obsolete/x-apollo.el
./obsolete/x-menu.el
./x-dnd.el
./xml.el
./xt-mouse.el

If you select the lines naming the 8 files above, then M-x replace-regexp will operate just in the selected region. Opening the selected files is then one easy command:

M-x replace-regexp
  Replace regexp:  .+
  Replace regexp with:  \,(find-file-noselect \&)

The files are silently opened in the background when you execute this "replacement" command. We could alternately have used `find-file' to watch them opened noisily in the foreground, but when opening lots of files I personally prefer to open them in the background. (This also makes them appear at the bottom of your buffer-list.)

Note that we used a new metacharacter here, `\&', which grabs the entire string that matched. That means we were able to omit the grouping parens. We also relied on the fact that Emacs regexps are, by default, anchored to the beginning and end of the line, so `.+' matches exactly one line, not counting the newline character. Pretty convenient!

After the replacement, the lines in your *shell* buffer are replaced with the return values of the calls to `find-file{-noselect}', which in this case is just the name of the file. But we don't really care, since it's a shell buffer; we were doing it purely for the side effect and not for the replacement.

Armed with your new-found knowledge, replace-regexp and query-replace-regexp should become some of the most powerful tools in your editing toolchest. The more experience you have with Emacs regular expressions, and with Emacs Lisp, the more bang for your buck you'll get out of this enhancement.

Example: Heading-Tag Promotion/Demotion

Oh, OK, fine. One laaaaaast example, because I just ran into it as I was putting in my final edits. You know how most browsers like to render <h1> tags in 10-foot tall letters? So we all start with <h2> or even <h3> tags and work down from there? (Those of us too lazy to muck with CSS overly much, that is. Which is most of us.)

Well, I started my blog entry today with <h1> tags, and decided to bump them all up a number (and thus down in size.)

I used to do this operation with N successive replacements, with N ranging from 2 to 5. You replace the <h5>'s with <h6>, then the <h4>'s with <h5>, and so on. No more of that hooey for me! I can renumber them all with a single replacement. You guessed it:

M-x replace-regexp
  Replace regexp:  <\(/?\)h\([0-9]\)>
  Replace regexp with:  <\1h\,(1+ (string-to-int \2))>

Zoom, zoom, zoom! All fixed in one swell foop. That's just awesome.

There's also a new query-replace-regexp-eval function, but it's not all that different from what we've talked about here, so you can read about it when you upgrade.

You are going to upgrade now, right? Well, it's your choice, it's your time. Gotta spend time to save time, as the old saying (almost) goes.

But I think it was worth it.

How do I learn this Emacs-Lisp doohickey, anyway?

It's really not too hard. Honest. Especially since I'm going to tell you some things that will make it much easier on you, because Emacs Lisp is pretty different from other languages you're used to. If you keep these points firmly in mind, learning it will be a snap, really. They're all things I wish someone had told me when I started learning elisp.

It'll Always be Useful

First, recognize that Emacs Lisp isn't going anywhere. Emacs is not going to magically become programmable in Python or Ruby or JavaScript or Perl overnight. (Or, God save us, Java or C++ or C# — all fine languages, to be sure, but they all suck at scripting.)

And judging from the last decade's pace of innovation in editing and coding environments, it will be many years before any editor begins to approach Emacs in the things Emacs does well. [The one noteworthy exception is VIM, which is also very powerful by all accounts, though I have no experience with it. If you have already developed a preference for vi over emacs, then you may experience greater happiness pursuing expertise with VIM. Psh.]

There do, in fact, exist packages that make it possible to write Emacs extensions in Python (PyMacs), Ruby (El4r), and Perl (EPL). But they're far from seamless: they're hard to install and they're hard to learn. They will only appeal to you (maybe) if you're a truly die-hard programmer in one of those languages, and you already know a fair amount of emacs-lisp, because they're closely tied to the elisp programming model. You will still need to know about buffers, overlays, markers, plists, symbols, and all the other Emacs-Lisp abstractions. And you'll have to deal with the sometimes complex mapping between language X and Emacs-Lisp.

Plus, if you want to share your extensions with your friends, they'll have to go through the install process as well. I just wouldn't go there, not if you're trying to learn how to get better with Emacs. If you're an expert in both languages, then sure. The package authors could use your help.

In the meantime, Emacs isn't going anywhere, and Emacs-Lisp isn't going anywhere, not for several decades at least, so it will benefit you to learn them deeply. It will never be obsolete knowledge. You might as well start learning it now, and reap the benefits now.

It's Kinda Like XML

You can think of Emacs Lisp as being very much like XML. There are some differences that will become apparent as you use it, but thinking of it as XML will help a lot.

In most programming languages, it's good style to avoid deeply indented code. With XML, indentation depth is entirely a function of your data domain; you don't generally think about restructuring it to avoid indentation. (Think of XHTML, for instance, in which the nesting can become arbitrarily deep.) With XML, you're building a tree structure, and it's easy to see it that way. Your XML processing tools help you manage navigating your way around complex documents.

With Lisp, you're also building an explicit tree structure. That means it's going to be indented very differently from your C/Java/Perl/Python code. The indentation is less something you decide, and more something that's decided for you based on the approach you take to the problem.

The perpetual indentation used to drive me nuts, but I've come to appreciate its advantages. I won't go into them here, but given what you know about XML, I'm sure you can imagine a few benefits without too much effort.

It Gets Better With Practice

If you do it enough, eventually you'll enjoy programming in Emacs-Lisp, no matter how much you hate it initially. You'll probably never love it, and you'll pine for a more powerful Lisp dialect, and for features from other languages you know. But like any other programming language, it becomes way more fun as you go from beginner to expert.

Programmers really hate new syntax. Most people find it harder to learn new syntax than to learn new Design Patterns or APIs or frameworks. So you'll initially dislike Emacs-Lisp's syntax; it's virtually guaranteed. Fortunately, it doesn't really have much in the way of syntax; almost everything follows the exact same s-expression form. So you should get past the syntax pretty quickly, and in a few weeks you'll start liking it just fine.

It's Oddly "Zippy"

Emacs-Lisp uses a radically different programming model from other languages. There's a strong (nearly 1:1) correspondence between what you can do in the editor and what you can do in the language. Writing elisp code is very much like scripting your actions in the editor.

For instance, to get the length of the current line (assuming the function doesn't exist), your code will remember where you are, then move the cursor to the beginning of the line, get the buffer position, jump the cursor to the end of the line, get the buffer position there, subtract the two buffer positions, return the cursor to where you were, and then finally return the value. That's a lot of moving around! [You can also use point-at-bol and point-at-eol, but in general, you still zip around a lot.]

As another example, if you wanted to determine whether any lines in the buffer started with a particular regexp, then you'd go look at them! You don't call a function that returns a list of lines in the buffer (though you could write one). You just save your position, go to the beginning of the buffer, and call next-line and looking-at to check each line against the regexp. It's like you've got a little worker-bee version of yourself, doing automatically what you could have done by hand (albeit much more slowly) using editor commands.

Over the years, they've piled up thousands of shortcut functions, and much of the time you wind up programming "normally" by invoking functions on data structures, like in other languages. But it really helps to remember that little worker bee that zips around like Feynman's lonely electron. Your coding will go more smoothly if you keep it in mind.

You can Learn From its Peers

Lastly, it's useful to know that Emacs-Lisp has a lot in common with two other famous Lisp dialects: Common Lisp and Scheme. It's arguably closer to Common Lisp, and in many ways it's inferior to both of them, but learning a little about Common Lisp or Scheme will improve your Emacs-Lisp coding dramatically. And, as it happens, there are far more books published about Common Lisp and Scheme than there are about Emacs Lisp. I'll list a few of my favorites here.

Emacs-Lisp Books

You should start with Richard M. Stallman's book. He wrote Emacs, and he wrote the Gnu Emacs Manual. It's a classic, and probably remains the best book on Emacs to date. It's a good idea to read it just to get an overview of all the things Emacs can do out of the box. Otherwise it'll be hard to know what kinds of Emacs-Lisp programs you can write.

Next, you'll want the all-time classic, Mastering Regular Expressions, by Jeffrey Friedl. Don't leave home without it.

Starting with Emacs 22, the Emacs-Lisp Reference Manual comes bundled with the distribution. I don't know if it's sold in hardcopy anymore, but it's chock-full of critically important information, so you'd do well to read it. (And re-read it periodically. It's a lot of information.)

I do have a couple of books on Emacs Lisp:

An Introduction to Programming in Emacs Lisp (Robert Chassell)
Writing GNU Emacs Extensions (Bob Glickstein)

They're not bad, but I honestly never got much from them. However, your mileage may vary. Go to Amazon, peek through them a bit, and decide for yourself whether they'll be helpful.

Common Lisp Books

There are lots — lots and lots — but these are the ones I personally found most directly relevant to helping me learn Emacs-Lisp:

ANSI Common Lisp (Paul Graham)
On Lisp (Paul Graham) — out of print, but available as a PDF. I printed it out and bound it at FedEx/Kinko's.

I own (and have read) essentially all of the other books on Common Lisp in print today, and the two above got me the furthest towards Emacs-Lisp proficiency. I'm not counting Peter Norvig's AI books, since their focus is AI, not Lisp. They're awesome books, though; I recommend them both highly.

And if you're actually trying to learn Common Lisp to use it (as opposed to applying what you can of it to Emacs), then you'd better get a copy of Peter Siebel's Book. It's essential.

Scheme Books

Again, lots to choose from, and Scheme books tend to be more didactic, so I found they had a bigger impact in terms of ingraining the core ideas of Lisp. Listed in decreasing order of mind-opening wow-ness:

Structure and Interpretation of Computer Programs — a good candidate for the "Best Computer Science Book Ever" award.

The Little Schemer — I worked through every single exercise twice: once in Scheme, once in Emacs-Lisp. Ditto for the sequel, The Seasoned Schemer, and I just started on the brand-new third volume, The Reasoned Schemer.

The Scheme Programming Language — good book, though a bit heavy going, as it's long on concepts and short on explanations. I found it well worth wrestling through, though.

Scheme is a wonderful language, and worth learning in its own right.

Caveats

If Emacs 22 goes on a horrible disk-eating rampage, don't blame me.

As one might expect of alpha software, it's got some bugs and glitches. I've never had anything really scary happen. It hasn't corrupted my data so far, and it has only crashed once or twice (i.e., far less often than the supposedly "stable" releases of XEmacs). But it occasionally does surprising things, like minimizing the entire frame if I try to 'q' (quit) certain read-only windows, or suddenly bringing up a completions buffer when I'm typing along in fundamental mode. They're rare enough not to have bothered me much.

They've also broken backwards-compatibility with several in-house functions and modes, so I've had to do some work to rewrite them. If you rely on proprietary Emacs-Lisp software as part of your job, and you're not proficient with elisp yourself, then you should make sure you have a local guru available before you upgrade to Emacs 22, or some stuff may stop working for you.

This blog entry consists, as usual, of only my very own whimsical opinions. I don't speak for my employer, nor for anyone else's employer, nor for any of the authors cited here, nor for the most excellent development teams working on Emacs, XEmacs, VIM, Eclipse, IntelliJ, Visual Studio, Firefox, and Ruby on Rails. I just speak for me.

If you didn't like this article, please be sure to run over to Reddit and call me stupid there. I'm sure people would hate to miss an opportunity to hear how you've taken such a boldly prominent and decisive stand on some random guy's personal blog.

And if you did like the article — well, go play with Emacs 22!

(Not) Managing Software Developers

Manager Secret Sauce

I've managed software developers at various companies, on and off, for about fifteen years. Doing so I've made or watched just about every mistake in the very big book o' management mistakes. So, like many others before me, I thought I'd offer a few observations and tips.

I'm not trying to be comprehensive here. It's just some thoughts, just enough of them to fit in a blog. And wouldn't you know it, they fit exactly. Lucky us! Also, I don't mean to be controversial here. However, given that nearly everything I write seems to generate at least some controversy, I eagerly await seeing how far I miss my mark.

If today's rant seems boringly obvious to you, then you may very well be a rare breed: a good software engineering manager. I say you may be, because knowing these things isn't the same as practicing them effectively. You may do all my Dos and don't all my Don'ts, yet still find some clever way to be an awful manager that I hadn't thought of. The path is fairly narrow, and there are surprisingly many dimensions along which you can make mistakes.

However, I'll offer you one almost magical tip that can help you smooth over nearly any mistake, a tip that can get you through just about any bad situation. I'll tell you the tip right now, with no fanfare or ado. This hint is the most important one I'll offer you today. It's the secret ingredient to Great Manager Sauce. Unfortunately, it's not easy to learn. You either already understand it, down in your bones, or you have years of head-scratching ahead of you. The tip is just one word: Empathy.

If you have true empathy for your engineers, they can forgive almost anything. Which is good, because you will make mistakes. We all do.

Of course, you'll need to be more than a quivering blob of empathy to be a good manager, so I have a few more tips coming your way. But first, if you're reading these tips with an eye towards practicing them, let's revisit why you're interested in management in the first place.

Absolute Power

The catch-22 of software management is that the ones who want it most are usually the worst at it. Some people, for worse or for worst, want to be managers because it gives them power over their peers. There's nothing good that can come of this arrangement: you should never give power to someone who craves it, for reasons that I hope are obvious.

Unfortunately, many tech companies do exactly that, because they don't know any better. And they exacerbate the problem by setting up a bad feedback loop, in which managers get to make all the decisions and effectively have all the power, or at any rate too much of it. A company may say they value their engineers, but if compensation decisions are all made by managers, guess who gets all the compensation? And then everyone sets a long-term goal of becoming a manager, at which point the company is no longer focused on innovation.

If you're an engineer at a company where becoming a manager is considered a promotion, then you only have three choices: become a manager yourself, or leave, or resign yourself to being a second-class employee. It should be obvious — you can work through the math using three sock puppets — that this is an arrangement that pushes a company inexorably towards mediocrity. The best engineers either leave the company or try their hand at management, often with doubly disastrous consequences: they simultaneously lose the company a great engineer and gain them an awful manager.

Every company is subject to nearly invisible forces set up by their organizational and cultural choices, whether deliberate or accidental. Software companies that prize managers above engineers are guided by their own Invisible Hand to become a henhouse of clucky managers pecking viciously at harried engineers. Sadly, most software companies fall into this trap, because they're borrowing traditional ideas about management from non-tech industries, and they're not bright enough to think through the implications, let alone design a better system.

It takes a long time for a big company to die: so long that it's non-obvious that most of them are in fact dying, or at best treading water. We're a hit-driven industry. A few big successes can make it seem like everyone's doing well. But most of them have only had one hit. Go visit most tech companies, and all you'll find is a fussy henhouse parading around an aging goose that laid one or two golden eggs. All their innovation happened in the first act, and now they're focused on "managing for success." But that kind of managing is just staving off insolvency until a real innovator takes their business away. Any tech company overly focused on (or dependent on) its management is probably a good candidate for short-selling.

Many big-company CEOs still wish for the good old startup days where everyone was motivated by idealism, or greed, or both, and worked as hard as they could without the need for management overhead. However, some sort of formal management heirarchy seems to be a necessary evil. I don't think anyone's figured out how to make a no-management structure work for an org with hundrds or thousands of engineers. I do know one great company that's come really close. I won't tell you their name, but you can, um, Google for them. Unfortunately I have neither time, nor space, nor in all likelihood permission to explain their recipe for success with almost no management. You'll just have to take my word for it: if you take all the managers away, great engineers will still build great things. Maybe even faster.

And even companies brave enough to try flat management still need to find a few good managers. But the management catch-22 I opened this section with makes finding good managers a bit problematic. If you simply line everyone up and ask anyone who wants the job to take one step backwards, you wind up with two rows: a row of bad managers who want the job, and a row of bad managers who don't. It's hard to find good software managers and it's hard to grow them. I'm afraid I have to leave the task of acquiring good managers out of scope for today, and instead focus on how you might go about becoming a better manager if you already have the job, or think you might someday.

So You Want To Be A Manager

If you want to manage badly enough, then you will manage, badly enough. Hence, before you jump in, stop and think about why you want it. Are you tired of engineering, or were you perhaps never very good at it? If so, technical management isn't much of an escape, because your engineers will know, and they won't respect you. Do you want to manage because you want authority? If so, it's a trap: you'll still be on a leash held by the folks above you.

Or maybe you just want to be a little higher in the pecking order, so you can peck downhill? If so, then you're what we call, colloquially speaking, a "pecker".

Think hard about why you want to be a manager. I've worked with a hundred managers with a hundred different motivations, and all of the underlying reasons, including my own, seem suspicious to me now. Especially now that I work for a company that works, and well, with almost no managers or management overhead. Now that I've seen it working, I question the motivations of anyone who wants to manage.

I'm suspicious of all the mother-hen types: they want to nurture their teams, but tend to smother them. And I'm suspicious of the overly-organized types: they want to bring process to chaos, but process stifles invention, and it can be used to disguise incompetence for an entire career. I'm suspicious of empire builders; too often they lower their hiring bar. I've heard or seen a hundred reasons for becoming a manager, and I now view all of them with suspicion, because each reason is a potential psychological problem waiting to manifest itself on a soon-to-be-unhappy engineering team.

I know plenty of good managers, even great ones, and none of them are managing. They're leading, and there's a world of difference. You've heard a hundred clichéd descriptions of leadership, but you probably also know at least one or two people you consider great leaders, so you know intuitively how it can work via their examples. And if you know enough great leaders, you know there are vastly different styles at work.

I won't try to characterize those styles here; it would take us too far afield. But I think the best managers don't want to manage: they want to lead. In fact most leaders probably don't think about it much, at least at first, because they're too busy leading: rushing headlong towards a goal and leading everyone around them in that direction, whether they're on the team or not. Leadership stems from having a clear vision, strong convictions, and enough drive and talent to get your ideas and goals across to a diverse group of people who can help you achieve them. If you have all that, you're close. Then you just need empathy so you don't work everyone to death. If you're a great leader, you can put the whip away; everyone will give you everything they've got.

Put in that light, management no longer seems so glamorous, does it? Ironically, "I want to be a manager" is just about the worst sentiment a would-be manager could possibly express, because the statement has absolutely nothing to do with leadership. A leader doesn't fixate on management, which is after all just a bureaucratic framework that attempts to simulate leadership through process and protocol. Great teams building great things don't worry about process. They just build whatever it is as fast as they can.

The more HR-oriented a tech organization becomes, with manager training and manager forms and manager evaluations and manager this and that, the harder it is for a real leader to get any work done. Often as not, the actual leaders in the organization (at all levels, from individual contributors up through senior VPs) tend to be very slightly unpopular with HR, because they're always bending the rules and not doing things strictly by the book.

The true leaders in an organization are seeing the world through a very different set of eyes: the eyes, almost, of someone reading a story unfolding, except they're the ones writing the story. They can see clear as day how the world should be different in some way, and they're doing whatever it takes to get from here to there. And they're enlisting all the help they can get along the way, because getting others on board with your ideas is one of the best ways to accomplish your goals. They'll align their own goals with yours if they agree with you strongly enough.

Great companies recognize that leadership is orthogonal to management, and that people can be highly influential leaders with or without direct reports. The management heirarchy isn't generally helping the leaders. If you're lucky enough to have truly great leaders in your org, the best thing you can do is get out of their way and let them lead.

Any time I hear someone say "I want to be a manager", I just want to smack them. But maybe it's just me.

Management in the Real World

It's wonderful when we have great leaders and visionaries around to lead the charge, but in practice our lives at tech companies are usually a bit more mundane than that. Also, a visionary who's too fired up (read: product manager) can develop a habit of discounting the laws of physics — laws about time, in particular — and asking you to give birth to a brand-new baby every 2 weeks.

You need the visionaries or you'll have one dull company, but it's also good to have people around who represent rationality. This is where actual managers come in, or at least they should.

Let's put aside what I said about leadership earlier, because great leaders and perfectly tuned teams are fairly rare. A more typical situation is that you're a new manager who has inherited a bunch of miserable engineers maintaining some crufty old legacy code for a product that's losing market share but is still generating enough revenue to fund your group, albeit with no room for new headcount. Sound familiar?

In situations like that, true leadership is still possible and desirable, but there are enough constraints to make it more of a long-term goal. You can't just drop the ball and send your team charging off in some new direction; you have existing products and customers to deal with.

This is where those management tips come in handy. Because if you're a bad manager, you've got serious problems ahead of you. The tech industry is still growing fast enough that good engineers won't stick with you if they're unhappy. Bad engineers might, but then you've got another set of problems.

If you don't know whether you're a bad manager, then you're a bad manager. It's the default state, the start-state, for managers everywhere. So just assume you're bad, and start working to get better at it. It's actually a pretty good assumption, because you are going to make some mistakes, even after you have lots of experience. Even if you're a good manager, you can always get better at it. And let's face it: you might actually be an awful manager. Most managers are average or below average, so statistically it's a pretty good bet that you're one of them. And it's not as if your team or your peers would ever tell you, so how would you really know?

OK, got it. We're bad managers. Let's get better at it.

Sadly, listing all the possible ways you could go wrong as a tech manager would fill fat books. There's no way to give you a reasonable list in a blog entry. So we'll have to stick with meta-tips: that is to say, tips about how to figure out how to be a better manager. Teaching us to fish so we can eat for a lifetime, and all that.

I've given you three pretty good ones so far. The first was empathy. That's all about having the rather deep realization that people are all pretty much like you in many ways, and their feelings matter a lot more than you probably think. I don't know how to teach empathy. Maybe get yourself a dog or cat, preferably not a pit bull, and then learn to love it. Once you realize just how much you and your dog or cat are alike, you'll be getting close. For instance, we all like to eat. You'd probably be surprised at how much time good managers spend thinking about food for the team, at least at companies that don't have catered lunch and dinner every day. I.e., yours.

Like I said, it's a meta-tip. Get in touch with the right side of your brain, all touchy-feely and stuff, and you'll suddenly start having all kinds of great ideas that make you a better manager.

That, and maybe one or two ideas that could get you fired and/or thrown in jail. So one concrete base-level tip would be to pay very close attention to your company's HR policies. They basically boil down to "don't do anything your mom wouldn't do." 'nuff said.

The second meta-tip was not to get into management at all. Be a leader, and try to do so at a non-dysfunctional organization. Then if management is thrust on you, your priorities will be... if not right, then at least better than they'd have been if you'd been dying to be a manager.

And my third meta-tip is to assume you're a bad manager, because you really can't go wrong with that assumption. Look for things you're doing wrong. Look for ways to improve. If you're not looking, you're probably not going to find them.

Management is hard. I'm serious. Just when you think you've got it all figured out, you'll run into some bizarre new situation. Or they'll give you more responsibility, or a harder problem space. Something always comes up. If nothing does come up — if things are all running smoothly and you're hardly having to work at it — then you've got a more insidious problem on your hands: if it all seems easy, then your company is probably sailing smoothly towards insignificance. Because I guarantee you that your competitors are working really, really hard. So in a sense, good management is hard by definition: the challenge is to balance pushing hard with the good of the team. Again: fuel for fat books here. No space to discuss it. But you get the idea.

How To Manage

Tune in next time. Blogs are so episodic; it just drives you nuts, doesn't it? Plus I'll probably disappear for another month, playing some video game.

But at least I'm not doing performance reviews.

Seriously, though, this blog has gone on long enough, so I'll have to leave the non-meta tips for another blog. For now, just remember. Empathy. If you have that, then even without the vision, the rest should follow nicely. You can go far just by being nice.

Oblivion

So I haven't been blogging lately, because Oblivion doesn't have that feature. On the plus side, though, I've been leveling completely out of control because I was stupid enough to have Athletics as one of my primary skills.

If you don't know what I'm talking about, consider yourself lucky. As in, your luck attribute is maxed at 100. Because the game is so damned addictive that when you're not playing, you wind up looking outside and thinking "man, that lighting model is really realistic!" before you remember you're not actually playing at that particular moment. And then you get all bummed, because you're not playing Oblivion.

Oblivion is the latest in a series of very long games called "The Elder Scrolls", from Bethesda. It's a single-player first-person RPG, or "Role-Playing Game" for those of you who've lived under a rock for the past 30 years. RPGs are a genre defined by its immersively intense realism. For instance, when you're hurling fireballs at vampires, you can hardly tell them apart from real fireballs and vampires.

Bethesda called this latest incarnation "Oblivion" because it sounded better than "core dump" or "segmentation fault", but it has the same basic connotations. "Oblivion" is where the game sends your Windows sessions, at least if you're unfortunate enough to be running the game on any computer manufactured before the year 2017.

Crashes? *gasp* -- I bet you'll never guess what programming language it's written in.

Anyhoo, the frequent crashes are tolerable, because the game lets you save anywhere you want, and you can tell when it's getting ready to crash by watching the frame rate, which goes from frames per second to seconds per frame. The game also has a nice autosave feature, so that when a mountain lion rips your throat out, and you're lying on your back trying to subdue the lion by spurting blood all over it, the game reassures you with: "Autosave successful."

For those of you who played Morrowind, you'll find Oblivion to be comfortingly similar. There are a few differences, of course. One is that Oblivion's countryside is beautiful, whereas Morrowind was a hideous island dominated by volcanoes and disease storms and mud flats and rotting undead zombies, much like a C++ users convention.

In fact, walking around in Oblivion is one of the most appealing aspects of the game. You get so caught up admiring the beautiful flowers and trees and birds and ancient ruins and whatnot that you usually fail to notice the lightning bolt from the imp behind you until it fries the back of your head. Fortunately, you can always revert back to that autosave with the lion.

On the other hand, while Oblivion's scenery has grown breathtakingly lovely, the increased realism hasn't been so kind to the NPCs (Non-Player Characters, for you Rock People) in the game. The graphics have improved to the point where you can now see every last throbbing vein and clogged pore in their wrinkled, hung-over faces. And they all have a nasty habit of standing too close to you when they talk, so you also get an insider's view of their nostrils. Why did they have to make everyone so ugly? It's like the soap-opera director said of Moe the Bartender: "I wanted Mary-Ann-on-Gilligan's-Island ugly, not ugly ugly!"

Plus, in order to achieve the realistic facial expressions during conversations, they evidently had to make all the characters look vaguely alike, as if they all had one parent in common. So they're worse than merely ugly: they're ugly relatives. The game's supposed to be an escape from reality, but at times it feels more like an escape to Arkansas.

But it's cool that they have such realistic facial expressions when they talk to you, so I suppose it's worth it. The NPCs' expressions generally change to match what they're saying. And, just like in real life, the characters will often seem to be staring fixedly at a zit in the middle of your forehead, because yes, you guessed it: you're ugly too. That is, unless you spent a long time customizing your character's face before you started playing, as I did. My wife and I spent well over an hour agonizing over every last detail of my (female) character's face, and she turned out pretty cute. But most of the in-game characters are just plain old fugly.

In addition to painfully realistic characters, the game also has extensive voice acting. Every single interaction with every character in the game is voice-acted, which is truly amazing, at least at first. Eventually it becomes a little annoying because there are only a few voice actors, and you quickly start recognizing their voices. "Oh, it's Mr. Rogers again." "Oh, there's Boris from Snatch again."

And for some reason many of the voice actors, particularly the men, apparently think they're sounding dramatic when they randomly and violently modulate their intonation. In practice, however, when a voice ACTOR does this!?, with UPs and doooowns and lows and *sudden* -HIGHS-, it sounds more like he or she has a choke-chain tied around his or her testicles, and he or she keeps tripping on it.

But really... aside from the frequent crashes and frightfully ugly characters and comical voice acting, the game is just awesome. They do all kinds of things I've never seen in a video game before, and you probably haven't either.

For one thing, the monster AI is exceptionally good. Enemies use all sorts of evasion tactics and generally fight almost as well as human opponents. And they no longer stay in their areas: they'll chase your ass right out of the dungeon and all the way into town. I know this because my character is a "light armor" specialist, which means I get frigging pounded (and I mean *hard*) by everything in the game, including sewer rats. I honestly don't think there would be any detectable difference if I took off all my armor and fought with the nude mod applied. So I specialize in fighting while running backwards at top speed, which generally moves us into a new area where I can attract even more monsters.

The cool thing is that when monsters run outside, they'll fight pretty much anything that moves. If an Imperial Guard is riding by, or some random townsperson, or even another monster, chances are pretty good that the monsters chasing you will suddenly be chasing them. I've even stopped my horse to watch an Imperial Guard romp on a troll. I'd have offered to help, but... you know... I had this, um, excuse.

For the record, Trolls in Oblivion are just green apes. They evidently went through the whole Zoo (I've encountered rats, wolves, bears, and mountain lions outdoors so far), and then at the last minute they decided that trolls are cooler than apes, so they turned 'em green and that was that. It's sort of a sad take on Tolkien, if you ask me. Or Norweigian trolls, which are even further removed from their Oblivious counterparts.

Anyway, as you can clearly see, this is one of my it's-Friday hence let's-get-hammered rants, so I didn't really have much of a point, except to say that if you're not playing Oblivion, then I highly, nay strongly recommend that you don't start, or you'll suddenly develop an aversion to Real Life, and who knows how long it'll last. Probably until you're fired, I'm guessing, at which point Real Life will become at least passingly interesting again, although not in any happy way.

Oblivion is... what can you say? It's a great game. A great, big game. A big, crashy game. Yes. Oblivion is a crashy game. It's a case study in why the world shouldn't be using C++. But in spite of that, they did a great job. As is always the case in the game industry, they spent too much time on the graphics and not enough time on the gameplay — I still think I'll ultimately log far more hours in Nethack than any Bethesda game — but it's still very cool, and I'm having fun with it.

It's not their fault, of course: they have to impress critics in a very short time, during the first conference where the game is demoed. So everyone spends their time making the game look realistic. Fortunately, there really is a game behind Oblivion, and it's a fun one: graphics can't carry a game on their own, no matter what the 12-year-olds might claim on the newsgroups.

You'd think single-player games are on the decline, but MMORPGs haven't figured out that the vast majority of their potential market hates subscriptions, and yet would gladly pay their life savings for cheats and hints and ways to differentiate themselves in the mass of unwashed players. Someday they'll get this, maybe, and the massivey-multiplayer game industry will be based off advertisements and micropayments, the way everything else on the internet is these days. Someday. Maybe.

But also don't forget: single-player games have the distinct advantage that they're, well... single-player. They're all about you. Like my friend Brunson says: when you hear an NPC talking about the exploits of some hero, you know it's about you. You don't have to deal with all the assholes you run into in multiplayer games, and believe you me, I don't use the term "asshole" lightly. MMORPGs bring out the worst in 12-year-olds, or perhaps they bring out the 12-year-old in adults. Whatever. The point is that when you play a single-player game it can have a plot, and you don't have to compete for resources with anyone but yourself, and there are definite plusses there.

I wonder if someday someone will figure out how to combine the best of single-player with the best of (massively) multi-player. Who knows.

Anyway, um, if you don't hear from me for a while, you know where I'll be. Working for my employer on stuff I'm being paid for, of course! Jeez!

Only two more working days 'til Monday... I think I'll go clean out some goblin caves. Know what I mean?

Psh. Whatever!

How the hell do I write stuff? It just comes out, like poop, and the result is usually nearly indistinguishable.

Brad Pitt, in an interview, once said something like: "I don't know what the hell I'm doing up there in front of the camera. Really, I have no f---ing clue." He's become quite an actor, hasn't he? Maybe you've decided he'll never be Bogart (he's got a few years to go, though, doesn't he?), but you have to admit he's a lot better than various actors-who-always-play-themselves whose names rhyme with Rom Ruise.

Sorry, didn't mean to imply even indirectly that I'm a good writer; I know better. Don't ask me how I know — I'd have to think of seven words in the right order to explain it. I just know. I suck.

I do read my old entries occasionally, though, and I think: "Gosh, how the hell do I write that stuff? I could never do that! Well, er, I mean now, that is. Obviously I did it before. Just what are you trying to say, anyway? Do you think I'm some sorta dumbass?"

I wonder.

Dave

I learned to write from my brother Dave, who as far as I can tell wasn't a very good writer. He was a good Liver, to coin an utterly awful phrase; he lived, and that's always better than writing. Writing just tries to show you how people lived, and it never comes close, never gives you that rush of being there just when something wonderful is happening.

Dave died, you know. I've often thought that if I ever wrote a book, it'd be about Dave. He just kinda got sick one day. We golfed, we biked, we had long late-night discussions about his philosophy class, we reminisced about things you could never repeat, during the years we lived in different states, stuff happening mostly at parties, teenager sins you committed as a teenager. Fun stuff. Nothing you didn't do yourself, of course, but it's still unprintable.

He died. I've been dead ever since then, you know. I didn't close my eyes for a single night, for four years after he died, without thinking of offing myself, becaused I missed him so much. My brother Mike knows; my brother Kevin knows; my Dad and Mom both know. We were all there. It's like losing a limb, but worse. So much worse. I realized after Dave died that I'd glady have given both arms and both legs to have him back. I would.

Everyone loved Dave. Everyone wanted to be Dave. Dave had wine-tasting parties at his little apartment; he taught us all to golf and to keep aquariums and to mountain bike and to appreciate the fine points of football strategy. He had a soul-beagle, Bentley, who still misses him to this day.

He had a girlfriend, Nancy, who was more of a wife to him than any wife I know. My wife agrees, and she's only met Nancy once.

Dave just got sick one day. It happens. He didn't feel good. He was coughing. He had night sweats. You know, a cold. A fever. Bronchitis. Something nasty. Didn't want to get anyone else sick — I remember he would sit away from us on the couch so he wouldn't give it to anyone else. He was so considerate.

We didn't know it was a tumor. He was only 23; I mean, come on. He was an all-star football player in high school. You don't get tumors when you're 23.

His doctor mis-diagnosed him, twice. It's OK; we forgive him. It's been 8 years, and I think we all forgive him now. I had a friend in the Navy, back when I was in the Navy, who got mis-diagnosed. It happens. His doctor said he had, I dunno, a cold, something lame. He had leukemia. Doctor mis-diagnosed him twice, just like Dave. With Dave, the doc said it was bronchitis, nasty case, definitely needed bedrest.

The Navy friend got heli-lifted off the sub to a medical facility in Oregon. He didn't make it.

Dave didn't get to ride in a helicopter, but he did get an ambulance. On the 3rd visit to his doctor, the doc took his oxygen level and said: "Ooh, time for the ambulance." Dave trusted him. Turns out Dave had lymphoma, and pretty advanced. No reason. No family history. He just had it. Maybe the environment. Maybe too much coffee or nutra-sweet. Nobody knows. And like everything else in Dave's life, his lymphoma was world-class. It ate him up in a way that no other cancer could.

You really don't want to hear about that part. Trust me.

We trusted Dave. He had a sense of humor like you've never, ever seen or heard before. It would take up several chapters in my book. Dave could make people laugh who had obviously not laughed in years; he'd crack a joke, something made up on the spot, context-sensitive and all, and they'd HAW, HAW, *hack* *cough* HAW HAW HAW until we thought they were going to have a stroke. Dave was the only person I knew who could make someone laugh so hard that I thought they were physically uncomfortable.

So it goes.

I'm not going to make my blog about anything at all. You can peg me as a programming-language guy, or a would-be math guy, or an inconsiderate jerk who says bad things about the vehicle with which you earn your living. But that's not what I'm about. Because I appreciate that you're reading this, I really do, but I'm not writing it for you. I don't know you.

I'm writing for Dave. I sure miss him. We all do. Everyone from Geoworks misses him. They made him a big banner, back when we knew he was sick, but we didn't know it was that bad. I mean, I should have known. The first day in the hospital, after his ambulance trip, after he passed out from lack of oxygen at community college from climbing six flights of stairs from a broken elevator, with a tumor the size of your fist growing between his heart and lungs, and the nurses asked him what he'd been doing that day, and he said he passed out trying to go to class, and he said their eyes got all big and round, I should have known.

Because his doctor started crying. I've never seen that before, and I hope to God I never see it again. She was the visiting doctor, the resident, whatever they have at 10pm at Swedish Hospital in Seattle, Washington, the place Dave spent the next year and a half, the rest of his life. She looked at his chart that first day, and said some encouraging words to him, and then as we were walking down the hall, me and my mom and this strange doctor, she was crying. We thought that was kind of weird, because you don't cry in public, especially if you're a doctor, especially hanging around the family of someone you just saw.

Dave made so many jokes that we couldn't even understand them all, there in the hospital. The nurses loved him. One nurse told him: "You were the best! Even Dr. Wasserman says so!" Dave kidded her for the next four months over that one. "Don't refer to me in the past tense!" I saw her blush every time he mumbled it through the morphine. But he knew and she knew and I knew that he was just messing with her, just having fun on his deathbed. Who else can do that? Not me, I don't think. I don't know.

I remember a road trip, one we did in our parents' van way back when, and our little brother Kevin was about seven years old. Dave taught Kevin to say: "Psh! Whatever!" It took him a little while to grok the concept. The idea was that any time someone said anything you disagreed with, or even if you just felt like it, you would reply: "Psh! Whatever!" You could substitute the socially acceptable variant "Tsk! Whatever!" as long as you could produce a suitably sardonic clicking sound with your tongue, a sound to make Zulu heads turn in envious surprise. Dave had mastered the depreciating tongue-click. For mere mortals, the acceptable default was "Psh!"

My stepmom Mindy was less than pleased. "Kevin, don't you listen to them!"

You wouldn't believe the cheering that me, Mike and Dave produced at 7-year-old Kevin's beautifully crafted response: "Psh, whatever!" (Hi, Min!)

Dave imparted me and everyone near him with a sense of humor, by osmosis, although we were all really just a pale shadow. He had his world-class sense of humor until the very very end. A few months before then, I was visiting him in his hospital room, and he told me in thick, steroid-induced tones (after having thrown up his esophagus the night before, which he recounted to me with some surprise as being like spitting up long filets of salmon) that he'd lost bowel control a few days back, because of the chemotherapy, and he'd had an incident "like the one in Trainspotting". I hadn't seen Trainspotting at the time, but I got the picture. I didn't know what to say, so he chimed in, almost unintelligibly:

"Look on the bright side: at least I didn't have to clean it up!"

How could I laugh? How could I not laugh?

I'll say some pretty strange or seemingly mean things in my time, in my blogs, but you have to keep it all in perspective. My brother was tortured to death. I'll spare you the gruesome details, but aside from the miracle of morphine, those folks in medieval torture chambers had nothing on him. His suffering lasted 18 months, during which he basically dissolved, for all intents and purposes, and in the end I think (not really knowing, myself, but guessing) that the worst part was psychological. Facing your own death at 23 years old is pretty scary. Especially when you're melting.

So I probably have a slightly different perspective than you do. To me, it doesn't matter all that much anymore. I just try to make people around me happy, and enjoy myself, until, you know, I have some sort of major Trainspotting incident. Hopefully one that I don't personally have to clean up.

I don't really mean to be mean, though. I hope you realize that.

Stuff

Despite my best intentions, my blood pressure occasionally rises when I blog. Or more precisely, after I blog, because no bowel movement is ever inspected as scrupulously as the articles posted to Reddit. Even when my blog tries to be innocuous, the comments always seem to get to me.

My doctor says I might have bronchitis. What the hell do they teach med school students, anyway? It's as bad as a Computer Science degree.

I wish there were a way to request, respectfully, that certain of my articles not be posted to Reddit, because even though I want people to read them, I want them to read them at the right time. And that's different for everyone. Probabilistically speaking, the right time for most people is not going to be the day after I post them.

Then of course there's Digg, the Reddit for... Digg people, I guess. Diggers. Duggers. Whatever! Lord help you if you get Dugg, or whatever it is they do over there. And del.icio.us, which in addition to being hard to type is no longer the sprightly young company it once was, ever since You-know-hoo! bought them. Sometimes I wish I could just never be posted there.

That's not nice to the folks looking for karma, though, I guess, so really I just mean this entry, today.

I'd like to blog more about non-technical stuff. I feel like blogging about technical stuff is, well, you know. Dirty? Incestuous? It's not like I'd be saying anything you don't already know, or won't already know at some point, from someone else.

On the other hand, I always feel the (few) bloggers I read ought to stick to the same topics. If I'm reading Bill de hÓra, the only blogger I read regularly, and he suddenly starts talking about his dog, then I feel ripped off, as if the Discovery Channel had started doing a chef competition, or the Food Channel started doing specials on wrestling alligators.

That's not entirely fair of me, I know. People are always broader than what they blog about, but we sort of expect the best bloggers to stay on topic, to keep blogging about whatever we liked last time we read them.

Well, I'm going to take a deep breath, a leap of faith, and see if I can broaden my blogging to include non-cs-technical topics. Yes, some people will whine and moan about it; people will whine and moan about anything and everything. I do it too. But I'd love to blog about the movies I like, and video games, and music, and books, and people, and just plain old good times I've had. Because you never know how long it'll last.

I'd like to ask you, just one person to another, not to post this blog entry to Reddit, Digg or similar. I'd prefer that people learn about my brother Dave through some mechanism other than a newsfeed full of karma modders. You know? So I've deliberately avoided tech topics in this entry, in the hope that it will somehow pass unnoticed.

We'll see.

If not, well... psh! Whatever.

Miss you, Dave.

Software Needs Philosophers

Software needs philosophers.

This thought has been nagging at me for a year now, and recently it's been growing like a tumor. One that plenty of folks on the 'net would love to see kill me.

People don't put much stock in philosophers these days. The popular impression of philosophy is that it's just rhetoric, just frivolous debating about stuff that can never properly be answered. "Spare me the philosophy; let's stick to the facts!"

The funny thing is, it's philosophers who gave us the ability to think rationally, to stick to the facts. If it weren't for the work of countless philosophers, facts would still be getting people tortured and killed for discovering and sharing them.

Does it ever strike you as just a teeny bit odd that after a brief period where philosophy flourished, from maybe 400 B.C.E. to ~100 C.E., we went through a follow-on period of well over one thousand five hundred years during which the Roman Catholic Church enslaved everyone's minds and killed anyone who dared think differently?

What's weirder is that we tend to pretend it didn't really happen. We like to just skip right over the dominance of religion over our minds for a hundred generations, and think of religion today as a kindly old grandpa who's just looking out for us kids. No harm, no foul. Let bygones be bygones. Sure, there were massacres and crusades and genocides and torture chambers with teeth grinding and eyes bleeding and intestines torn out in the name of God. But we were all just kids then, right? Nobody does that kind of thing today, at least not in civilized countries.

We try not to think about the uncivilized ones.

It was philosophers that got us out of that Dark Ages mess, and no small number of them lost their lives in doing so. And today, the philosophy majors are the butts of the most jokes, because after the philosophers succeeded in opening our minds, we forgot why we needed them.

And if we stop to think about it at all, we think that it was other people, people who are very unlike us, who committed those atrocities in the name of Faith (regardless of whether it's faith in a god, or in a political party, or any other form of mind control carried out by force).

We like to think we live in an enlightened age, but we don't. Humans haven't changed significantly in 10,000 years. We're still killing and torturing each other. It's apparently incredibly easy to decide to kill someone and then do it. Happens every day, all around the world. Torture, too.

But those people are just people. If they had been born down the street from you, they'd have gone to school with you, been friends with you, learned to program with you, written blogs and comments, never tortured or killed anyone in the name of an idea. They'd have been you. Which means they are you; you just got lucky in where you were born.

One of the commenters on my last blog entry expressed the fervent wish that I drop dead. To be sure, they qualified it with "on the internet". But if they really feel that way, especially about something as hilariously and absurdly unimportant in the Grand Scheme as whether the Lisp programming language has any acceptable implementations, then what does it say about us?

Everyone who commented angrily on that blog entry was caught. I caught you, anonymous or not, being a religious fanatic. The only "negative" commenter who doesn't appear to be a religious zombie was Paul Costanza (ironic, since he claims to be the opinionated one), who relegated his comments to pedantic technical corrections. They're welcome, of course; I'm always looking to correct any technical misconceptions I harbor. But they're moot, since even if I was wrong about every single technical point I brought up in that entry, my overall point — Lisp is not an acceptable Lisp — remains largely uncontested by the commenters.

Some of them just don't get it, which is fine; no harm in that. If you've been using Lisp for years and years, and you've written books and articles and zillions of lines of Lisp code, then you're unlikely to remember anything about what it's like coming to Lisp for the first time. They're religious because they've forgotten what it's like to be a skeptic.

But make no mistake; a substantial percentage of people who take a side in any programming language discussion that devolves into a flamewar know exactly what the other side means, and they want to invoke the Ultimate Censorship: drop dead! Killing someone, after all, is one of the best ways to silence them. You also have to burn all their writings, which is getting harder these days; hence the increased vehemence on the 'net.

Those of you who've followed what I've written over the past year or so know where I'm going. I'm taking a stand, all right, and it's a very definite one. I'm finding myself drawn inexorably towards a single goal: stamping out technological religion, because I'm frigging tired of not being able to stick to the facts.

FACT: Java has no first-class functions and no macros. This results in warped code that hacks around the problem, and as the code base grows, it takes on a definite, ugly shape, one that's utterly unique to Java. Lisp people can see this clear as day. So can Python folks, so can Ruby folks. Java people flip out, and say "macros are too much power", or "what do u mean i dont understand u" or "fuck you, you jerk, Lisp will NEVER win".

You think I don't hear ALL that, and much more, in the hate mail I get every day?

I sure wouldn't want to be alone with a Java fanatic in a medieval torture chamber, because God only knows what they're capable of.

Turn the mirror towards Python, and what happens? Funny, but the Java folks will mail me saying: "yeah, I've always known I detested Python, and you really nailed exactly why. Thanks!" Meanwhile, Python folks are literally frothing at the mouth, looking for the "Kill That Bastard" key on their 101-key keyboards.

I turned the mirror towards Lisp yesterday. Had to go to the bathroom like nobody's business, and my wife was expecting me home any minute, so I rushed it out: just a few thoughts here and there. So the Gorgon only caught the tiniest glimpse of itself, but hell evidently hath no fury like that of a Lisper scorned, and all that.

It doesn't matter that I rushed it out. I'm glad I did; spending any more time on it, trying to get it "right" by looking up useless factoids like how you can override length's non-polymorphicness with some weird setting (when it plainly should just be the default), would have had the exact same net effect: Lisp zealots would have found some way to turn it into a flamewar. And I'd have been out 2 or 3 more hours.

Let's call it a troll, then, because it was poorly researched; it was just some months-old recollections of pain I'd gone through last year trying to commit to Common Lisp, after another year of trying the same with various flavors of Scheme and finding them all wanting. As far as I'm concerned, Lisp is unacceptable today; it's my opinion and just that, but I'll stick with it.

I still need Lisp; after you learn enough of it, it becomes part of your soul. I get my fix hacking elisp, and I do a lot of it. The commenters are quite right; I've never written anything substantial in Common Lisp, because in each of my serious attempts, there was too much friction. Risk/reward wasn't high enough, and believe me, I wanted it.

But after many attempts, I've given up on Common Lisp. They won't let me use it where I work, and there are probably more Lispers per capita where I work, including some famous ones, than at any other big company in the world. If we can't use it where I work, then it's frigging unacceptable; that's the shortest proof I can offer.

What I'm far more interested today is the situation that arises if you consider my post a troll. I'm far more interested in the social consequences of working in a world filled with religious fanatics of different religious persuasions. Especially given that it's a world in which "natural religion" has, by and large, been marginalized through the work of philosophers.

Let's look at this world in a little more detail, starting with Peter Siebel's comment, which I believe is the most interesting. Peter said:

I was trying to figure out why on earth you spent so much time writing about something that you apparently don't like. Then it hit me: HCGS. So thanks for your help.

His first sentence speaks volumes about the sociology. His viewpoint is exactly what they teach us all as kids: If you don't have anything nice to say, don't say anything at all. We like to think people have a right to believe whatever they want, and that it's not nice to say mean things about other people's beliefs, especially when their livelihoods are at stake.

That's where philosophers come in, folks. They pick your beliefs apart and show you in unforgettable ways the consequences of what you believe in. I'm no philosopher; I know basically nothing about it, but I can tell you I wish fervently that some great philosophers would come along and effect change in our technical society.

Because if nothing else, I can see the consequences of the way we're thinking about things. One of many such consequences is that languages aren't getting any better, and the worst offenders are Lisp and Scheme, which by rights should be racing along the innovation curve faster than their supposedly less capable peers. But they've stagnated worse than any other non-dead language I can think of.^[1]

Programming languages are religions. For a long while now I've been mildly uncomfortable calling it "religion", but I don't feel bad about it anymore. They're similar enough. At the top of the language religion is the language itself; it serves as the deity and the object of worship.

Like any other organized religion, there's always a Pope (or a politburo chairman, in countries where the government has brutally set itself up as what is for all intents the religion of choice): a spiritual leader that gives the religion the human touch. This person is almost always the language designer, of course. In Lisp's case it's complicated, because McCarthy, Sussman and Steele aren't very active as spiritual leaders for their languages anymore.

Every major organized religion is a heirarchical government, and programming languages are no exception. You'll find equivalents of cardinals, bishops, priests and laity in programming language camps: the closer you are to the fire, to the spiritual center, the higher your rank. It's a great way to quantify your perceived self-importance: a high-score list, in effect. Great for the ego, but it makes you a piss-poor debater, because you're so emotionally invested in your status.

You'd think your rank would be accrued by virtue of your technical and/or documentation contributions, but in practice it's usually more of a function of how many converts you've gained, how many followers you have, how much you've been spreading the Word.

That's why Paul Graham isn't the Pope of Lisp. He's eminently qualified, but unfortunately he's a heretic. Notice that almost none of the commenters on my last blog mentioned the PG argument I made. The only one who did (as of this writing) tried to make it an argument for Common Lisp. Let's face it: you can't give those heretics too much press; people might start listening to them!

Peter, are you beginning to understand why I write so much about something I apparently don't like? It's because I wanted to like it but found it fatally flawed, technically and culturally. It's as if I were a would-be convert to Roman Catholicism, but I can't bring myself to commit because I've seen too much of their role in creating a history that ironically we all wish we could rewrite.

I was born and raised a Roman Catholic, and I renounced it when I was thirteen years old, after my Uncle Frank (a devout terrorist Catholic if there ever was one) told me to stop reading the Bible, that it would "really screw a person up" to do that, that you needed someone to interpret it for you. That wasn't the only reason I renounced it, but it'll suffice for our purposes.

Technologically I was born and raised an assembly-language programmer; at least that's what my first real job was, for 5 years after I got my CS degree. Assembly is just flagellation, though, and damned uncomfortable at that, so I joined the Church of Java for fully seven years. And practically at the very moment I'd finally tired of chafing at Java's limitations, Paul Graham came along and through his early essays, showed me Lisp. What a great new religion!

Problem is, each time you switch religions, the next one has less impact on you. Once a Catholic, always a Catholic, they say. I don't know what that means for me, since I was raised by the assembly-language wolf, but it appears to mean that I'm never going to be enthralled with another programming language. And now that I've swallowed the red pill, what choice do I have? I need to try to show people what's out there.

Interestingly, it was Peter Siebel's most excellent book, Practical Common Lisp, that played the role of Uncle Frank and killed my desired to continue with Common Lisp. Peter was the first person to show me beast's underbelly. Every other Lisp book had pretended it was pure and beautiful and uncorrupted, because they left all the nastiness out as "implementation-defined". Once I saw what you really need to do in order to build something resembling a portable Lisp code base, and then had a few runs at it myself, I threw in the towel.

I much prefer Lisp the idea to Lisp the implementation.^[2]

I can tell you this: I've tried writing this essay for a year. I've tried fully a dozen times. I've tackled it from a dozen angles. I've wanted to say it — software needs philosophers! — so many times, in so many ways. We need great thinkers — the Fyodor Dostoyevskys and David Humes and Aristotles and Jean-Paul Sartres and Ben Franklins and Galileo Galileis and Bertrand Russells and Albert Einsteins to show us the way through the Software Dark Ages we're in today: a time that will doubtless be remembered as every bit as mired in darkness and ignorance as the Dark Ages themselves.

But I've failed. This isn't the essay I wanted to write, because I'm neither a great thinker nor a great writer. However, you might be: if not now, then perhaps someday. So I think it's better to get the idea out now than to hoard it in the hopes of someday writing a world-changing essay.

For those of you who were surprised at the suddenness and vehemence of the Lisp community's backlash to my little rant, I hope I've helped shed a little light, helped you see its inevitability. Basically they've had a lot of practice. Lisp is one of the oldest technology religions, and they've both experienced and doled out their share of religious persecution.

But that's not the lesson you should take away. The lesson is that they are you. Whenever you hear someone ranting about something you take for granted as wonderful and praiseworthy, and you're wondering why they don't leave well enough alone so we can all get back to our incestuous cheerleading, just remember: we went from the Dark Ages to our reeeeasonably enlightened society today by questioning our most cherished beliefs.

So keep questioning them.

---

[1] Yes, I've read all of R6RS. It's a lukewarm compromise that punts on most of the important issues. It's not going to make Scheme any more successful than it is today, which to me feels practically criminal; it was their one big chance to break out of the rut they're in. But it doesn't matter. Let's pretend this footnote is just a troll. If your hackles went up, then you're a techno-religious zombie, and I hope in my lifetime to find you a cure. Try your best to think about that long and hard before responding.

[2] For the record, the commenter I agree the most with is the one who said the problem basically boils down to an IDE issue. SLIME doesn't cut it, either, as beautiful as SLIME is. Can't use it on Windows to save your life, for instance. But that's one of a thousand problems with the Lisp IDE situation; it's pointless to try to discuss them all in blogger. It's probably pointless to discuss them at all, because it's just going to make me more miserable that no decent IDE exists for Lisp, except for Emacs-as-Elisp-IDE. Which is why I get my Lisp fix by hacking elisp these days.

Lisp is Not an Acceptable Lisp

It's been over four months since Eric Kidd posted his infamous Why Ruby is an acceptable LISP article. You know, the one that got approximately 6.02e23 comments, ranging from "I agree!" through "I hate you!" to "I bred them together to create a monster!" Any time the comment thread becomes huge enough to exhibit emergent behavior, up to and including spawning new species of monsters, you know you've touched a nerve.

What amazes me is that nobody's pointed out the obvious counter-observation: Lisp is not an acceptable LISP. Not for any value of Lisp. There's nothing magical about this, nothing partisan. If Lisp were acceptable, then we'd all be using it.

You've all read about the Road to Lisp. I was on it for a little over a year. It's a great road, very enlightening, blah blah blah, but what they fail to mention is that Lisp isn't the at the end of it. Lisp is just the last semi-civilized outpost you hit before it turns into a dirt road, one that leads into the godawful swamp most of us spend our programming careers slugging around in. I guarantee you there isn't one single Lisp programmer out there who uses exclusively Lisp. Instead we spend our time hacking around its inadequacies, often in other languages.

So for four months I've been waiting for someone else to say it, but so far it's not happening. Why aren't we admitting it?

Oh, right. Religion. I keep forgetting about that.

Lisp programmers are just like all other programmers: they want to write code and get cool stuff done, which presupposes they've already learned the last programming language they'll ever need. There's all this real-life stuff (jobs, family, stability, all the usual suspects) intruding on you as a programmer, demanding that you quit dorking around looking for the One True Language, and settle down on whatever barren rock you happen to be squatting on at the moment, and call it Good. So most Lisp programmers — and that's not many, since not many programmers make it even close to that far down the Road — see that last outpost of technical civilization, peer balefully into the swamp, and decide to check into the Lisp hotel for good. Not realizing, of course, that all its rooms are in the swamp proper.

We all know it's tricky to have a rational discussion about a religion. Non-Lispers will be able to read this without getting their feathers ruffled. Some Lispers aren't too far gone, so let's assume we're talking to them, and take a look at some of Lisp's problems that make it flat-out unacceptable. At least for LISP, you know, the idealized one.

Problem 1: Which Lisp?

Sorry, folks, but you can't trivialize this one. Let's say I'm a new would-be Lisper, just finished walking down that long damn Road, and now that I'm here, I'm ready to start using it. Which "it" should I use? The answer is "it depends", and that's pretty unfortunate, because right there you've just lost users. With Python or Ruby or Java, you've only got one language to choose from. Or at least you can be comfortable that there's a single canonical version, and the rest (e.g. Jython) are highly experimental territory.

Pick Scheme, and you have to pick a Scheme. Pick Common Lisp, and you have to pick a Common Lisp. Heck, there are even two or three flavors of Emacs-Lisp out there.

Most newcomers eventually (and independently) decide the same thing: Scheme is a better language, but Common Lisp is the right choice for production work. CL has more libraries, and the implementations are somewhat more compatible than Scheme implementations, particularly with respect to macros. So newcomers heave a deep sigh, and they learn to accept LISP-2, names like rplaca, case-insensitivity, '(ALL CAPS OUTPUT), and all the other zillions of idiosyncracies of a standard Common Lisp implementation. Eventually, if they stick with Lisp at all, they learn they can override most of these defaults in nonportable ways, which makes things infinitesimally more bearable.

Whatever. If you're a Lisper, you dealt with all this crap years ago, and now you're committed. If you're not a Lisper, then you're not very likely to become one any time soon. In fact your probability of learning Lisp is decreasing over time, as other languages continue to close the gap in the Lispy areas, and simultaneously increase their lead in non-Lispy areas where Lisp is making little (if any) progress.

Let's look at some of those areas. But first, let me mention one last problem in the "which Lisp" space. It's dirty laundry that needs airing. The problem: Paul Graham. I mean, the guy's a genius, and I love reading his essays, and his startups are doing great things, etc. etc. You can't fault him. But he's created something of a problem.

Before Paul Graham, Lisp was dying. It really was, and let's not get all sentimental or anything; it's just common sense. A language is always either gaining or losing ground, and Lisp was losing ground all through the 1990s. Then PG came along with his "I'm not talking to you if you're over 26 years old" essays, each a giant slap in our collective face, and everyone sat up and paid attention to him in a hurry. And a TON of people started looking very seriously at Lisp.

Lisp might or might not have experienced a revival without Paul's essays, but it's moot: he showed up, and Lisp got real popular, real fast. And then he said: "Don't use it!" Sort of. I mean, that's effectively what he said, isn't it? By deciding to pre-announce Arc, he Microsofted Lisp. Killed it with vaporware. It's a great strategy when you're an evil empire. I don't think that's exactly what Paul had in mind, but let's face it: that's what happened.

So Common Lispers grumble about Paul in the hallways. If I read newsgroups (every time I try, the overall ugliness of humanity drives me away within hours; I only re-attempt it every decade or so) I see them grumbling there too. He's put them in a tough spot, because he did use Common Lisp for Viaweb, and his arguments in favor of Lisp (in the general sense) have been compelling enough to bring in newcomers by the droves. But he's not throwing his weight behind CL. He's not even taking the marginally-acceptable route (from a damage-control perspective) of recommending Scheme. Instead, Paul's (*gasp*) starting a new religion.

Arc's going to be a new religion, of course, because programmers just haaaaaave to make it that way. If it ever appears, anyway. But will it? That's the tricky thing about Cathedral-style software; you never can tell. My prediction: someone will get tired of waiting, and they'll Torvalds Arc into obsolescence before it's ever released. (If you don't get the reference, it's what Linux did to GNU Hurd).

Long story short: nobody knows what the hell Lisp they're supposed to be using, and it's absolutely killing adoption.

Problem 2: Worthless Spec

Oh, ouch, did I have to put it quite like that? I mean, c'mon, let's be fair, there are literally hundreds of people out there who disagree.

Unfortunately, the simple fact is that the spec is ancient. Every time someone talks about updating it, someone screams about time or money or whatever. The problem is (like the problem with RSS) a people-problem, not a time or money problem. This is absolutely true in the Scheme world, too. There are a bunch of old-timer stakeholders who want to have their say. So you're basically asking a goverment (complete with lobbyists, political parties, the works) to design Lisp if you go that route. The naysayers are right about one thing: it'll never happen.

Your only other option is to design a new language, and you won't get any help from Lisp people, because they will hate you. They love pointing to the trail of bodies left in the wake of every pioneer who's tried this before, none of whom has emerged with a "successful" Lisp. Of course, they haven't been successful because Lispers didn't want to have anything to do with them; Lispers are just as incapacitated by their techno-religious beliefs as folks from other languages. Religions dislike each other, but no heretic is as damned as someone who starts with your religion and makes a modification to it. Just ask the Albigensians, for instance.

But what's wrong with Common Lisp? Do I really need to say it? Every single non-standard extension, everything not in the spec, is "wrong" with Common Lisp. This includes any support for threads, filesystem access, processes and IPC, operating system interoperability, a GUI, Unicode, and the long list of other features missing from the latest hyperspec.

Effectively, everything that can't be solved from within Lisp is a target. Lisp is really powerful, sure, but some features can only be effective if they're handled by the implementation.

Problem 3: CLOS

Note: this section contains a few factual errors pointed out by Pascal Costanza in a comment below. Some of his corrections are also, of course, opinions, and I've commented on them later in the thread. In any case, while I thank Pascal for his corrections, the errors I've made are utterly irrelevant to my conclusions.

CLOS is icky. I haven't worked with Smalltalk a whole lot, but I've worked with it enough to know that to do OOP right, you have to do it from the ground up. CLOS was bolted on to Common Lisp. Everyone knows it, although not many people want to admit it.

It was bolted on very nicely, and it's not my intention to disparage the efforts of the people who created it. It was an amazing piece of work, and it did a great job of being flexible enough to tie together the conflicting OO systems of existing Lisp implementations.

But let's face it; CLOS has problems. One obvious one is that length isn't a polymorphic function. It's one of the first speed bumps you encounter. You can't create a new kind of measurable object and give it a length method; you have to call it rope-length or foo-length or whatever. That's part of Lisp's endoskeleton showing; you can see the bolt sticking out plain as the nose on your face. It's not seamless; it's not orthogonal, and it's not the Right Thing. But it's not going to change, either.

Another problem is the slot accessor macros. They're insanely clever, but clever isn't what you want. You want first-class function access, so you can pass the getters and setters to map, find-if, etc. You can work around these things, but they're a leaky abstraction, and enough of those will add up to significant mental resistance to getting things done. It's like all those weird little rules in Perl: non-orthogonal rules that add up to forgetting the language every time you leave it alone for more than a week.

What you really want in lieu of CLOS is... complicated. It's a hard problem. Lisp wants to be constructed entirely from macros. It's part of the purity of the idea of LISP: you only need the seven (or is it five?) primitives to build the full machine. Doing CLOS as a bunch of macros was very much in the spirit of Lisp: it was a Lispy thing to do.

But macros are a problem. Yes, they're one of the most important differentiators. But macros are like having these high-powered band-aids, when what you want is not to be wounded in the first place. Having the object system — something pretty fundamental to the language, you'd think — written as a bunch of macros doesn't feel right when all is said and done.

When you work with Ruby or Smalltalk or any suitably "pure" OO language (Python doesn't quite count, unfortunately; its bolts are also showing), you realize there are some distinct advantages to having everything be an object. It's very nice, for instance, to be able to figure out what methods are applicable to a given class (e.g. "foo".methods.sort.grep(/!/) from Ruby), and to be able to extend that list with your own new methods. It's a nice organizational technique.

Of course, that forces you into a single-dispatch model, so it becomes harder to figure out what to do about multi-methods. Some Python folks have implemented multi-methods for Python, and they do it by making them top-level functions, which makes sense (where else would you put them?) I'm not claiming that Smalltalk's object model is going to translate straight to Lisp; you have to decide whether cons cells are "objects", for instance, and that's a decision I wouldn't wish on my worst enemy. I don't envy the person who tackles it.

Regardless of what the solution might be, CLOS remains a problem. It's over-complicated and yet not quite OOP-y enough or expressive enough. The problem of reflecting on an object to see which methods are valid for it is one example, but there are tons of others. Heck, one possibly valid complaint is that it doesn't work very much like the "conventional" OOP systems of C++, Java, Python and Ruby. There's no real reason it shouldn't be more like them. But changing CLOS to be simpler and more seamless essentially means replacing it. And replacing it is probably best done inside the implementation.

In other words, any fix means starting virtually from scratch.

Or maybe you could go the Haskell route and not have OOP at all. That seems to alienate most programmers, though, despite the attractions of not having to create nouns for everything. (Have you ever noticed that turning a non-object-oriented program into an object-oriented one in the same language that does the same thing essentially doubles its size? Try it sometime...) At the risk of predicting future fashion trends, which is rarely a good idea, I'll venture that objects are going to continue to be trendy for at least a few more decades. So I think Lisp needs some form of "seamless" OOP.

Problem 4: Macros

Macros are one of the worst problems with Lisp, or at least they're one of the biggest unsolved problems.

Yes, they're amazingly powerful and critically important and blah Blah BLAH. You can read all about them elsewhere. Paul Graham's On Lisp is the best reference I've found.

But they're fraught with problems. One is that they're not hygienic. You should at least have the option of requesting hygienic macros. Various papers have been published, and implementations implemented, for hygienic defmacro. Yeah, it's hellishly hard to get right, and it's overkill for many situations, but it really does need to be offered as an option. A portable one.

For that matter, you should also have a choice between Scheme-style pattern-matching macros and Lisp-style code-style macros. They're very different, and each kind is better (cleaner) in some situations. People often act as if hygiene is synonymous with define-syntax, but the pattern-template style is orthogonal to the question of hygiene.

Style considerations aside, macros have tool problems. Macros are notoriously hard to debug, and honestly it needn't be that way. If your editor knows all about macros, then you should be able to click to see the expansion, and click again to see its sub-expansions, all the way down to the primitive functions. Some editors can do this, but none of them (that I'm aware of) handle macros as cleanly or seamlessly as they do normal functions.

Syntax in general is a problem. Lisp has a little syntax, and it shows up occasionally as, for instance, '(foo) being expanded as (quote foo), usually when you least expect it. Truth be told, Lisp should probably have a skinnable syntax. That implies a canonical abstract syntax tree, which of course hasn't been defined (and in many implementations isn't even available to you, the way it is in the Io language, say). Once you've got a canonical AST defined, syntax should, in theory, be like CSS chrome. Of course, there are plenty of bodies left in the trail of this particular theory as well. Someday...

In any case, because macros are rarely supported "well enough" by the tools, and because they're not first-class functions, and so on, they wind up being second-class citizens. The rule "you should only use a macro when nothing else will do" implies that they really are a last resort, which (to me) is synonymous with band-aid. Yes, it's wonderful that you have the band-aid — or maybe duct tape is a better analogy — certainly you miss them dearly when you're working in other languages. But you don't want to have to build your entire object system with duct tape.

So macros, like the object system, need to be re-thought from the ground up. There's undoubtedly enough research in the space that someone could throw together a working definition in no time, something just good enough for today's programmers, the ones who expect (and rightfully so, I might add) to be able to name their methods "length" without getting a compiler error.

Problem 4: Type System

See, that's just exactly the problem with type systems. They can make sure you use headings, but they can't ensure you get the numbering right.

Well, it'll take me forever to talk about this one, so I'll have to leave it for another blog. The problem is that the type system has to be extensible and skinnable, and I'm not strictly talking about user-defined types in the sense of OOP or CLOS. Unfortunately it really is a huge open issue, one that'll take longer than this blog to sort through, so I'll have to leave it for today.

Lisp, for all the strengths of its flexible type system, hasn't got this issue right either. Otherwise Haskell and OCaml (and C++, gack) wouldn't be kicking its ass all over the performance map. 'nuff said, at least for now. [And no, they don't quite have it right either.]

I promise I'll talk about type systems soon. But I also promised some friends I'd make my blogs shorter.

There is no acceptable Lisp

This is a problem. It's not a little teeny one, either. The Lisp communities (yeah, there are a bunch) are going to have to realize that if Lisp is ever going to be massively successful, it needs an overhaul. Or maybe a revolution. Contrary to what some might tell you, it doesn't need a committee, and it doesn't need a bunch of money. Linux proved exactly the opposite. Lisp needs a benevolent dictator. Lisp needs to ditch the name "Lisp", since it scares people. And Lisp needs to learn from the lessons of the 45 years of languages that have followed it.

And no, I'm not the guy. You're all far more qualified to tackle this problem than I am. Especially if you're under 26.