MH & nmh: Email for Users & Programmers

May, 2006

Files with Multiple Messages

Normally, MH stores one message per file. The packf and packmbox commands "pack" messages into a single file. packf uses MMDF style; packmbox uses mailbox (UUCP) style.

These formats are a good way to package messages for transport to another computer. They also take less filesystem space than standard MH folders do: see the Section Saving Filesystem Space.

The packf with nmh makes both formats with its -mbox and -mmdf switches.

MMDF Format

As the Example below shows, messages packed with packf are separated by lines of four CTRL-A characters. This is the same way the MMDF transfer agent formats its system mailboxes. In nmh, you get this format with the packf -mmdf option.

Example: MMDF file format

    % cat -v packedfile
    ^A^A^A^A
    Header Field: xxx
    Header Field: xxx
        ...

    Body
        ...
    ^A^A^A^A
    ^A^A^A^A
    Header Field: xxx
    Header Field: xxx
        ...

    Body
        ...
    ^A^A^A^A
    
Unless you give the -file switch, packf uses a file named msgbox in the current directory. If the file exists, the messages are appended. packf copies all messages in the current folder unless you give a folder and message numbers. So, for example, to copy the first ten messages in your current folder and message 23 from data to the file /tmp/messages:
    % packf -file /tmp/messages first:10
    % packf -file /tmp/messages 23 +data
    

Mailbox Format

The MH packmbox shell script is useful when you need to pack a folder in the same format that many non-MH MUAs expect: the mailbox format that UUCP and /bin/mail use. packmbox is in the MH library directory from version 6.8 and above; for earlier MH versions, get the script from the Internet (you can grab an MH source distribution and extract it if needed).

nmh doesn't have packmbox. In nmh, packf makes mailbox format by default. Use the packf -mmdf switch to use MMDF-style delimiters. The other packf options in the section above also work in mailbox format.

As the next Example shows, messages packed with packmbox are separated by lines starting with From, an address and a date. There's an empty line added after each message.

Example: mailbox file format

    % cat packedfile
    From ftpmail@online.ora.com Mon Jan 09 10:55:02 1995
    Header Field: xxx
    Header Field: xxx
        ...

    Body
        ...

    From ehuser Mon Jan 09 14:17:28 1995
    Header Field: xxx
    Header Field: xxx
        ...

    Body
        ...

    
To make the "From" separator line, packmbox and packf -mbox will copy the address from Return-Path: header field, if any; they add the current date and time to the end of that address. Otherwise, if there's a X-Envelope-From: header field containing the original message address and date, packf -mbox uses it. Otherwise, a dummy From line is created.

The Return-Path: or X-Envelope-From: header field must be the first line of the source message! This usually happens automatically. The inc and slocal commands will convert the incoming envelope address into a Return-Path: field. Or, if you use procmail and formail to store your incoming mail, they can add an X-Envelope-From: field; see the MAIL.FILTERING file in the nmh source tree for details.

If any line in the message starts with the string "From " ("From" and a space), the string is chaned to >From . Ths prevents the line from being interpreted as a delimiter later when the mbox file is unpacked.

packmbox is in your system's MH library directory, so you'll probably have to use its absolute pathname. It writes to the standard output, so you'll want to redirect its output to a file. Use the shell's > (right angle bracket) operator to create or overwrite the packed file; the >> operator will append. packmbox copies all messages in the current folder unless you give a folder and message numbers. So, for example, to copy the first ten messages in your current folder and message 23 from data to the file /tmp/messages:

    % /path/to/library/packmbox first:10 > /tmp/messages
    % /path/to/library/packmbox 23 +data >> /tmp/messages
    
As explained in the section above, packf doesn't write to the standard output. packf writes or appends to a file.

Joining Messages with rfl

Say you're having a discussion about some part of a project, and you're refiling the messages into a folder named project. Later, you want to follow the thread of that discussion or find a particular message from it. How can you find all the messages in that discussion?

What rfl Does

The Perl script named rfl makes message digests. These aren't digests for distribution in a mailing list (though I guess you could). They're message storage digests. This groups single messages into one big message using the RFC934 digest format. The forw -digest command makes a similar format. But forw builds the digest all at once. The rfl script lets you add more messages to the digest at any time. One major win is that rfl maintains the digest header so you can use pick -to, pick -subject and so on to search for any message within the digest. rfl can search for an existing message or digest with the same subject, and add a new message to it automatically.

Here's an example of using rfl. A reader writes to me to report a problem in the online MH book. [Naaaaah. :-)] I reply to the original message, forward it to the co-author who wrote that section, refile the original message into my mh-book/revisions folder, and use rfl to add my reply to the end of the original message:

    % show
    (Message inbox:14)
        ...
    Subject: Bug in MH book
        ...reader's message appears...
    % repl
        ...I reply to the reader and save a copy in inbox...
    Fcc: inbox
    Subject: Re: Bug in MH book
        ...
    % forw
        ...I send a copy to co-author...
    Subject: Would you look at this bug in the MH book?
    
Next, I file the messages:
    % refile +mh-book/revisions
        ...Move reader's message into another folder...
    % rfl last +mh-book/revisions
        ...Add my reply to end of reader's message...
    rfl: Adding message 27 to 93 in +mh-book/revisions
    rfl: Converting +mh-book/revisions 93 into a digest
    rfl: Removing message 27 from +inbox
    %
    
(I set the -verbose switch on an rfl: entry in my MH profile.) Now the reader's original message and my reply are glued into a single digest: message 93 in the mh-book/revisions folder. Later, I get a reply from the reader and a reply from the co-author. I read them and add them to the digest in mh-book/revisions:
    % show
        ...reply from reader appears...
    Subject: Re: Bug in MH book
        ...
    % rfl +mh-book/revisions
    rfl: Adding message 44 to 93 in +mh-book/revisions
    rfl: Removing message 44 from +inbox
    % next
        ...reply from co-author appears...
    Subject: Re: Would you look at this bug in the MH book?
        ...
    % rfl +mh-book/revisions
    rfl: can't find message in '+mh-book/revisions all'
      with subject 'Would you look at this bug in the MH book?'.
    Skipping message 45...
    
The co-author's reply has a different subject than the saved message, so rfl couldn't find the digest. Now I have a few choices. If I know the exact message number of the digest, I can specify it. In this case, I know that it's the last message in the mh-book/revisions folder:
    % rfl -to last +mh-book/revisions
    rfl: Adding message 45 to 93 in +mh-book/revisions
    rfl: Removing message 45 from +inbox
    
Or, if I don't know which message is the digest, I can use rfl -query. It scans a range of messages (default: all, but you can give a range like -range last:50 on the command line or the rfl: entry in your MH profile). Then rfl asks which message is the right one:
    % rfl -q +mh-book/revisions
    rfl: this message is:
      45 Would you look at this bug in the MH book?
    Press RETURN to list messages you can append it to:
      93  10/09 Joe Reader    Bug in MH book<<----------------
      92  09/28 Jane Reader   For the next edition of MH book
      91  09/15 To:sue        Last MH print date<<Sue, do you
        ...
    Enter message number to append to; 0 to skip, q to quit: 93
    rfl: adding message 45 to 93 in +mh-book/revisions
    rfl: Removing message 45 from +inbox
    
The idea is: instead of adding a bunch of separate messages to the folder over time, the messages that are related go into the same digest. Later, when I run show 93 in the mh-book/revisions folder, all the messages are there. I can use the pager's search command to search for a word (/word) or to search for the top of each digest (/^---).

Getting Messages from an rfl Digest

If I want to pull out an individual message, I can link the digest into some folder (I have a temporary folder, +temp, for that kind of thing), then burst it:

    % refile -link 93 +temp
    % burst -noinplace last +temp
    % scan cur-last
      22+ 10/09 Joe Reader    Bug in MH book<<Jerry, there's a
      23  10/09 To:Joe Reader Re: Bug in MH book<<Joe, you did
      24  10/10 Joe Reader    Re: Bug in MH book<<Here's anoth
      25  10/10 Bill Wohler   Re: Would you look at this bug i
    
But usually I just show the digest -- and page through instead of bursting it.

There are a couple of ways to delete messages from a digest. One is to burst them, as above, and then recombine the messages you want into a new digest. Another is to edit the digest itself -- with mhedit or any editor. If you do that, be sure to maintain the digest format: blank lines, etc.

I use rfl digests so much that I've modified scan to mark them specially. The scan.rfl format file detects rfl digests -- like message 29 below:

      28  03/22 Kim Allenson       The vi bug<<Christy recommends
      29  03/24 Steve Christos     anti-spam info **rfl digest**
      30  03/26 New Scientist Pla  Newsflash : Planet Science is
    
mhadd, a Version of rfl

By default, rfl adds the current message (or some other messages you specify) to a destination message somewhere else (it might be in another folder). I often want to add some other message to the current message, though. For instance, I could be reading a message, send a reply with an Fcc: folder copy, and I want to append the reply to the original (current) message. I've made two versions of rfl named mhadd and mhadd! that do this. (It can be done without new versions. I just do it enough that versions are handy.) In this example, I might be reading through the inbox and have put the folder copy of my reply in inbox, too. If I just sent the reply, it will be the last message in inbox. Here's how it looks:

    % show 33
    % repl
       ...
    % scan last
      77  03/26 To:Sara Winger     Re: In and out of the offices
    % mhadd last
    mhadd: Adding message 77 to 33 in +inbox
    mhadd: Removing message 77 from +inbox
    
(By the way, that verbose output isn't required. I've put the -verbose switch in my MH profile.) mhadd works the way rfl does: it adds the reply message if it has the same Subject: as the destination message. (mhadd uses the -r cur option to specify that the range of destination messages to search is only the current message.) There are times when I want to add a message with a different subject. That's when I use the mhadd! version; the ! means "do it anyway, no matter what the Subject: is." Instead of the -r option, mhadd! (the version with the exclamation point in its name) uses the -to option. The -to tells rfl not to search for a matching message -- to append to the specified message with no complaints (except errors, of course).

The MH profile entries for these versions look like this:

    mhadd: -r cur -add cc
    mhadd!: -to cur -add cc
    
Filtering Mesages Put Into the Digest

When rfl adds a message to a digest, it copies the From:, To:, and Subject: fields to the header of the digest. (It also adds a Date: field with the time the digest was edited.) These make it easy to use pick, later, to search for any message in the digest by searching only the digest header -- it's a real time-saver. The -add switch names other fields to add to the digest header. The MH profile entries above tell mhadd and mhadd! to also add any cc: fields to the digest header.

Which brings me to one more point (whew). By default, rfl copies all message headers into the digest body. If there's a file named rfl.skiphdrs in your top-level MH directory, rfl will not copy the fields listed there. List the fields to skip, one per line, in the file. You can also use perl-type (basically, ed-type) regular expressions in the file. For instance, to omit Received:, Sender:, and any X- fields from the digest body, use these three lines in rfl.skiphdrs:

    received
    sender
    x-.*
    
For more details, take a look at the rfl manual page. There's info about installation and other details in the section Explanation of rfl. Finally, the section Folder Name Variables shows shortcut commands for running rfl.