MH & nmh: Email for Users & Programmers

May, 2006

Reading MIME Messages

When you type show to display a non-MIME message, MH will run a showproc program to display the message header and body. For MIME messages:

This section explains what happens overall and how you can control it. To learn about changing the way mhn and mhshow work, read the Section MIME Configuration.

Getting Test MIME Messages

If you'd like to experiment as you read, you can get a sample MIME message from the sample MIME server. Also, as of this writing, there were lots of sample messages available for anonymous FTP on thumper.bellcore.com in the directory pub/nsb/samples. Most of those samples on thumper were written as MIME was being developed; some of them use old syntax that may make mhn and mhshow complain, but they're still useful. You can copy the messages into an MH mail folder and rename the files (which have strange names) by following these steps:

    % folder +mime-samples
    Create folder "/yourMHdir/mime-samples"? y
    % cd `mhpath`    ...cd to the MH folder
    % ftp thumper.bellcore.com
    Name (thumper.bellcore.com:ambar): anonymous
    Password: xxx@yyy.com (use your user name and host here)
    ftp> cd pub/nsb/samples
    ftp> prompt      ...ftp won't ask whether to get each file
    ftp> mget *
        ...lots of files transfer...
    ftp> quit
    % ls
    +0cqBCNW0M2YtQhngB3   +0dYMHpa0M2YtABA8E7   +0eCBvVy0M2YtAoUABO
        ...lots of strange filenames appear...
    
Next, use a shell loop to rename the files to 1, 2, and so on, so that MH will recognize them as messages. Here are both C shell and Bourne/Korn shell syntax:

C shell:

    % set n=1
    % foreach f (*)
    ? mv "$f" $n
    ? @ n++
    ? end
    

Bourne shell:

    $ n=1
    $ for f in *
    > do mv "$f" $n
    > n=`expr $n + 1`
    > done
    
Finally, type scan +mime-samples and you should see the sample messages.

The mhnproc or showmimeproc

When show reads a MIME message, it runs the MH mhnproc program or the nmh showmimeproc to analyze and display the message. The default MH mhnproc and nmh showmimeproc is mhn; in nmh-1.0 and above, the default showmimeproc is mhshow. (In MH, the mhn -show switch isn't needed because that's the default.) The following sections explain how this works. (By the way, nmh still supports mhnproc. But it wants you to use showmimeproc instead.)

Unfortunately, if you're showing more than one message (as in show 3-7), mhn -show will hand all the messages to its mhnproc, even if only one message needs the mhnproc. (Same happens with mhshow and the showmimeproc.) mhn and mhshow will show the plain-text messages -- but their headers and bodies will be shown separately, without your normal showproc.

If you want to use a different setup, or if you don't want show to call its mhnproc or showmimeproc for MIME messages, there are Other MIME Handlers.

Showing MIME Messages

mhn and mhshow look at the message header and body parts of each message to find out how to display them. Then, at least two programs are used show the message: one for the header, and one or more for the body. (If the message is a partial message, mhn and mhshow won't show it at all. Another section explains.)

mhn and mhshow use the mhlproc and the format file mhl.headers to show the message header. The default mhlproc is mhl. The Section Using mhl introduces mhl and shows how you can change the header format for non-MIME messages. You can use the same techniques for changing the mhl.headers file. For instance, if you don't want the default empty line between the To: and From: fields, or if you want to see the Content-xxx: header fields, make your own mhl.headers file to do that. The Section mhl explains how to configure mhl. You can use a different form file than that used for mhl.headers. In MH, give the -formfile filename switch to mhn or add it to the mhn: entry in your MH profile; for nmh, give the -formfile filename switch to mhshow or add it to the mhshow: entry in your MH profile.

If there's a moreproc: entry in your MH profile, mhn and mhshow use it to page through the message header. This usually isn't necessary because the header is probably shorter than a screenful after it's been filtered through mhl. In some cases, it's irritating: If you use a pager program that pauses and prints (END) at the end of a file, you have to exit the pager after the message header is shown. To prevent an external pager program being used on the message header, add the undocumented -nomoreproc switch to the mhn: or mhshow: entry in your MH profile. Because -nomoreproc is undocumented, you shouldn't depend on it to work in all versions of mhn and mhshow. Also, it may have effects that I haven't noticed.

After showing the header, mhn and mhshow show the body parts one-by-one (or in parallel, if possible, for a multipart/parallel message -- the %e escape, listed in the Table Display String Escapes, controls that). In a MIME message, each body part has a content-type. The default content-type (for messages without any Content-type: field) is text/plain; charset=us-ascii.

mhn and mhshow can adjust to different character sets. If you don't use us-ascii, you should set the MM_CHARSET environment variable -- usually in your login shell's startup file. MM_CHARSET tells mhn which character set your current UNIX session is using. If the message charset= parameter is different than your current MM_CHARSET, mhn can translate the characters; it uses a mhn-charset- profile entry to do that. Once you set up this stuff, you can forget about it; mhn will "do the right thing." The Section Displaying Other Character Sets has the details.

To display a content, mhn runs a UNIX command. It finds the commands in profile entries like these:

    mhn-show-type/subtype: command
    mhn-show-type: command
    
mhshow works similarly, but it looks for mhshow-show- entries.

For example, to display a text/enriched content, mhn looks for an entry mhn-show-text/enriched: and mhshow looks for mhshow-show-text/enriched:. If the entry is found, mhn or mhshow runs the command listed. Otherwise, mhn looks for an entry mhn-show-text: and. mhshow looks for mhshow-show-text:. There are defaults for a few cases. If mhn and mhshow run out of choices, they print an error like this:

    mhn: don't know how to display content
         (content application/x-foobar in message 6, part 2)
    
If mhn or mhshow is showing a multipart message, it may list content information on the terminal (like mhn -list or mhlist would), then prompt and wait:
    part 3     image/x-pbm                40K Sample waveform
    Press <return> to show content...
    
When you see that prompt, you can skip the part (not display it) by pressing your interrupt key -- often, that's CTRL-C. (In MH 6.8.4, typing q also works.) Or, you can skip the rest of the message, and get back to a shell prompt, by pressing your QUIT key -- typically CTRL-\ (control-backslash). You can prevent the pause by giving the -nopause switch to mhn and mhshow.

The Section Showing MIME Contents has all the gory details of these content-displaying entries. But you might want to do some basic setup right now. Why?

Partial Messages

The MIME content-type message/partial is designed for messages that need to be split into pieces. That's because some mailers and transport systems can't handle messages that are "too long." The safe size depends on the system and gateways, but 50,000 ASCII characters is a good guess. If someone sends you a partial message and you try to show it, here's what will happen:

    % show
    (Message inbox:131)
    %
    
When you don't see a header or a body, show (actually, mhn or mhshow) is probably telling you that the message isn't complete. There are several ways to see what's happening. The command show -noshowproc | head -20 will spit the first 20 lines of the current message onto your screen, but that's probably more than you need to see. The undocumented switch -debug for mhn and mhshow is probably the most interesting. It's also useful for figuring out why something you don't understand is happening. But, because the switch is undocumented, you must not rely on -debug staying in future nmh versions -- or for it to work in the same way.
    % mhn -debug 131
    big endian architecture
    MIME-Version: 1.0
    Content-Type: message/partial; id="<6374.782021163@ora.com>"; number=1; total=5
    
The message "big endian architecture" shows mhn checking my computer and getting ready to decode message parts. If the MIME-Version: field has a MIME version that mhn doesn't understand, it would complain and try to read the message anyway... but mhn with MH 6.8.3 does understand MIME version 1.0. Finally, mhn shows the Content-Type: field... and quits because the message isn't complete. It needs all the parts with the id= parameter shown. The number=1; total=5 tells us that this message is part 1 of a total 5 parts.

How can you find all the parts -- especially if you don't want to rely on the undocumented -debug switch? The easiest way is to scan your folder and check for five messages with the same subject -- most mailers seem to use the same Subject: field for all parts of a message (and MH does). Or, if you have a lot of messages and you don't want to wade through a long scan listing, you can get the Content-Type: field of one part and use pick to search for the other parts.

Here's how to do that search. You can ask scan to show you just the Content-Type:. (If you're interested in the details, see the Section scan Format Strings. Otherwise, just follow along.):

    % scan -format '%{content-type}' 131
    message/partial; id="<6374.782021163@ora.com>"; number=1; total=5
    
Next, pick can search all Content-Type: fields in the folder for that id= string. Using the -sequence switch will "remember" the messages pick finds by using a sequence. Notice my quoting: single quotes (') around the whole string, which contains double quotes ("):
    % pick --content-type 'id="<6374.782021163@ora.com>"' -sequence picked
    5 hits
    
Good. pick found five parts. Let's list them (nmh users, you could use mhlist):
    % mhn -list picked
     msg part  type/subtype              size description
     131       message/partial            47K part 1 of 5
     132       message/partial            46K part 2 of 5
     133       message/partial            46K part 3 of 5
     134       message/partial            46K part 4 of 5
     135       message/partial            15K part 5 of 5
    
In this case, the parts arrived in order. That doesn't matter to mhn and mhlist, though. If you give all the message numbers to mhn -store or mhstore, it'll be happy:
    % mhn -store picked
    storing message 131 as file /u/jerry/Mail/inbox/143
    storing message 132 as file /u/jerry/Mail/inbox/143
    storing message 133 as file /u/jerry/Mail/inbox/143
    storing message 134 as file /u/jerry/Mail/inbox/143
    storing message 135 as file /u/jerry/Mail/inbox/143
    
mhn glued the parts together into a new message, number 143, in the current folder. If you don't give mhn or mhstore all the parts, it will complain. For example, if I don't name message 133:
    % mhn -store 131 132 134 135
    mhn: missing part 3 of 5 part multipart message
    
You can find and store all the parts in one step by using the storeparts shell script in this book's online archive. This section explains how to install it. When you find a partial message, you can find and store the other parts by simply typing storeparts. The script will search all messages in the folder unless you give it a list or range of messages. For instance, to check all messages from the current to last (for the parts of the current message):
    % storeparts cur-last
    storeparts: searching for 'id="<6374.782021163@ora.com>"'.
    5 hits
    storing message 131 as file /u/jerry/Mail/inbox/144
    storing message 132 as file /u/jerry/Mail/inbox/144
    storing message 133 as file /u/jerry/Mail/inbox/144
    storing message 134 as file /u/jerry/Mail/inbox/144
    storing message 135 as file /u/jerry/Mail/inbox/144
    
Finally, you can remove the partial messages if you'd like to. If you've followed the steps above, their numbers are stored in the picked sequence. (The storeparts script uses the temp sequence.)
    % rmm picked
    
It's a good idea to use the mhn or mhbuild switch -check when you compose a split message. Then, the person who combines the parts (with mhn -store or mhstore) can check the combined message to be sure nothing was lost.

If you want to experiment with partial messages, send yourself some test messages. The Section on Partial Messages explains how.

External Parts

Your mailbox has probably been cluttered by some really long messages that you didn't want. Even worse, you might have to pay for your email or network access by the amount of data you receive. External body parts shorten MIME messages by not sending the data. Instead, an external part has instructions for getting the data by FTP, from a mail server, or from a local file. A MIME-capable mail reader can ask you whether you want to retrieve the data. If you do, your MUA will do "the right thing": start an FTP job, email a request to a mail server, or retrieve the local file. When it has the data, the MUA will display it. (Data from a mail-server takes another step. See below.)

In case you're curious, an external body part looks like this:

    Content-type: message/external-body;
            access-type="anon-ftp";
            site="ftp.xyz.com";
            directory="pub/speeches";
            name="jones.941024.au"
    Content-Description: Jones lecture on new polymers

    Content-type: audio/basic
    Content-ID: <AAE28372.941024213426@xyz.com>
    
The first section, before the blank line, gives information about the external part: how to get it (in this case, by anonymous FTP), the filename, and so on. The second section has information that's used to display the data once it has been received. (Of course, you can't "display" sound parts. "Render" is more correct.) The Content-ID: is a unique identifier that helps the MUA match the data with the original message; it's especially important for data stored in a cache or fetched from a mail server.

When mhn, mhshow, or mhstore sees an external body part, it prompts you like this:

    Retrieve jones.941024.au (content 1.4)
        using anonymous FTP from site ftp.xyz.com? y
    
If you answer y, that runs anonymous FTP and displays the result. If you answer n, then mhn/mhshow/mhstore will check to see whether there are alternative parts (for example, a plain-text version of the audio speech) and use them instead; otherwise, it will issue a complaint like "mhn: don't know how to display any of the contents."

There's an even more efficient way to handle external body parts: a cache. Only one person at a site has to retrieve the external body part. Then everyone can share a copy of it.

For messages with an access-type of mail-server, there's an extra step. mhn will ask:

    % show 1
        ....
    Retrieve content by asking mime-server@foo.bar.sf.ca.us

    binary
    send meeting-summary.93.ps.gz

    ? y
    
A y answer will send a mail message to the server; the message header will include the same Content-ID: field used in the external body part. When the server replies, it's supposed to include that same Content-ID: field in its reply. (Unfortunately, most mail servers don't. Surprisingly, the MIME mail server from Metamail version 2.7 and before doesn't return the correct Content-ID: field, either. The bug is supposed to be fixed when version 2.8 comes out. I guess that very few people use mail-server external body parts...)

When the reply comes, store it in the content cache:

    % mhn -cache 2
    
NOTE:

As of version 1.0, nmh still had not moved cacheing code into a separate mhcache command. You'll need to use mhn -cache, as the example above shows.

After storing the content in the cache, you can re-process the original message. It will find the cached content:

    % show 1
        ...
    Use cached copy of content (size 77742 octets)
        ...? y
    

Cached Contents

Sending the same long message to several people at the same host can be inefficient. The host's disks will have multiple copies of the same file. If you send the long message with external parts, each person who wants the long parts will have to fetch the same data; if the data comes over a network, those multiple copies waste network bandwidth and CPU cycles on the host that serves the data. If you get a message with external body parts, and you want to view the same message several times, mhn or mhshow would be fetching the same data over the network again and again; that's also a waste.

MH can cache (store) body parts in a directory on your computer's filesystem. Before it tries to fetch a body part over the network, mhn will check the caches to see if the body part has already been cached. The body part is cached by its Content-ID: header field; that field's value is supposed to be unique for every external body part in the world.

There are two caches:

When you read an external body part, you'll be asked whether to use the cached value:
    % show
      ...
    Use cached copy of content 1.4 (size 117722 octets)
        in file /usr/local/lib/mhn-cache/<AAE28372.941024213426@xyz.com>? y
    
You may not always want to use the cached value. If you answer n, mhn, mhshow or mhstore will ask whether you want to fetch a fresh copy of the data.

The mhn, mhshow, and mhstore switches -rcache and -wcache control caching. They define the policy, respectively, reading from and writing to the cache. There are four policies:

When you read a message with external body parts, you can save them in the public or private cache. Use the public cache if you think that someone else will want the data too; use your private cache if the message was intended for you.

Here's an example. You want to write part 1.4 of a message into your system's public cache, so you use -wcache public:

    % mhn -wcache public -part 1.4
    Date:    Mon, 24 Oct 1994 21:34:07 EST
            ...
    Retrieve jones.941024.au (content 1.4)
        using anonymous FTP from site ftp.xyz.com? y
            ...
    part    audio/basic           114K    Jones lecture on new polymers
    Press <return> to show content...
            ...
    % ls -l /usr/local/lib/mhn-cache
    total 116
    -r--r--r--  1 jerry      117722 Oct 25 09:20 <AAE28372.941024213426@xyz.com>
    
That ls -l command shows the cached file.
CAUTION:

The cached filenames use angle-bracket characters < and >. These are legal in filenames, but the shell uses them as redirection characters. If you want to use a cached file directly, without mhn, remember to put a pair of single quotes around the filename:

      % play < '/usr/local/lib/mhn-cache/<AAE28372.941024213426@xyz.com>'
      
Otherwise, you might destroy files accidentally.

The Section Caching External Body Parts has more information about caches.

Showing Part of a MIME Message

By default, show calls its mhnproc (typically, mhn) to show all the parts of a MIME message. To see only one part of a message, if you know the part number, you can call mhshow -part or mhn -part directly (you don't need to use the mhn -show switch because it's the default.) Or, if you know the content type of the part(s) you want to see, use mhshow -type or mhn -type. If you don't give a message number, mhn and mhshow will show the current message. For example, to show part 1.2 of message 33, type:

    % mhn -part 1.2 33
    
mhn will show the message header followed by part 1.2. To hear the audio/basic part, type:
    % mhn -type audio/basic
    
If that's the only audio part, you can omit the subtype: use mhn -type audio (or mhshow -type audio).

When you show parts of messages, you may not want to see the message header. You may not want mhn to pause before it shows the part. The MH command version named showpart does all that.

Decoding Messages with mimecat

MH makes it easy to get the content of a non-MIME message into a pipe or file: just redirect the output of show with the shell's > or | operators. MIME messages aren't as easy to handle because they may need to be decoded and because you may want only one part of a multipart message. You can play some tricks with mhn-store- and mhstore-store- profile entries, but there's no good general-purpose way to get message contents to the standard output.

A more detailed explanation of the mimecat script (in this book's online archive) can help. It decodes a message -- or part of a message, with the -part switch -- and writes the content to standard output. For example, to decode the current message and search for the word taco in part 3.1 (the part numbers are the same as mhn -list and mhlist use):

    % mimecat -part 3.1 | grep taco
    Emma, that was a <italic>huge</italic> taco!  How did you finish it?
    
From there, you can pipe the decoded text to printers, processors like richtext (from the Metamail package), and so on:
    % mimecat -part 3.1 | grep taco | richtext -e -f
    Emma, that was a huge taco!  How did you finish it?
    
mimecat makes a temporary MIME profile with a series of profile entries for different content subtypes. It's kind of an ugly hack, but what can you do? :-)

Checking a MIME Message

MIME has safety features to get through message-eating gateways and transport. But messages can still be corrupted. For instance, your computer might have a bad disk. When a MIME message is sent, the MUA can add a Content-MD5: header field with a message checksum. A checksum is a string of letters and numbers that's a function of the content of a message (before the message is sent, of course). If you have a message with an Content-MD5: header field, mhn can compare the checksum to the message body. If they don't match, the message body has probably been changed.

The fastest way to check a message is mhn -list -check or mhlist -check. If the checksum matches, mhn won't complain. Otherwise, here's what will happen:

    % mhn -list -check
     msg part  type/subtype              size description
      49       application/octet-stream   44K
    mhn: content integrity suspect (digest mismatch) -- continuing
         (content application/octet-stream in message 49)
    
Here's information about adding an integrity check when you send a MIME message.

Reading PGP Messages

MH 6.8.4 comes with shell scripts that give rudimentary support for PGP (Pretty Good Privacy). The pgpshow shows MIME messages with PGP information. To use it, add lines like these to your mhn profile:

    mhn-show-application/pgp: %ppgpshow -show '%F' %a
    mhn-store-application/pgp: | pgpshow -store %m%P.pgp %a
    
The script for sending PGP mail is pgped.

Other schemes for using PGP with MH have been discussed on the comp.mail.mh newsgroup. There's also good PGP support in exmh.

Other MIME Handlers

If you don't want show to give MIME messages to mhn or mhshow (the default mhnproc or showmimeproc, respectively), you have three choices: