MH & nmh: Email for Users & Programmers

May, 2006

Storing Messages

The first part of this section is mostly for people who've come to MH from another MUA, an MUA that stores all of a folder's messages in a single file. The second part discusses ways to save space when you store MH folders. The third part covers decoding and storage of MIME mail.

You may also want to read the section Files with Multiple Messages.

Copying a Message to a File

As explained in the Chapter Key Parts of the UNIX Filesystem, MH stores each message in a separate file. So, unlike most other email systems, MH doesn't need a command that saves a message to a file. (There are two exceptions. The mhn -store and mhstore commands decode and store MIME message bodies. The MH msh command simulates non-MH MUAs.)

One easy way to get the contents of a single message is by redirecting the output of show to a file (the shell's > operator) or with a pipe (|). But the plain show command formats a message for viewing on your screen. If you add its switches -noshowproc -noheader, then show will output the message without formatting.

You can also get the pathname of any MH message within your MH directory and give it to a UNIX command. Concatenate three things: the pathname of the MH directory, the folder name, and the message number. Let's say you wanted to transfer a copy of message 25 from your data folder to a personal computer, using kermit. Your MH directory is /home/marty/Mail. The command you'd type to send that message is:

    kermit -s /home/marty/Mail/data/25
Of course, you can use the same message pathname to copy the message to another file (with cp), make a link to it (ln and ln -s), edit the message (with vi, emacs, etc.), and use any other UNIX command that uses a file pathname. If you edit a message file that MH will be handling later, remember to keep the correct header and body format. For example, I edit long disk-eating messages that I want to keep parts of; I replace long deleted sections with ...deleted by Jerry....

You can also use the mhpath command, which gives you message and folder pathnames.

Finally, you may want to save just the body of a message -- without the header. One easy way is by using a simple sed script that prints all lines after the first blank line:

    % sed '1,/^$/d' /home/eloisa/Mail/inbox/25
You can replace the file pathname with a backquoted mhpath expression. Or, instead of using a file pathname, you can pipe the output of show to that sed command; check to see whether your show command might reformat the message body (by wrapping long lines and so on). Finally, if you've set show to format messages with mhl, you can get just the unformatted message by using the mhl.body format file:
    % show -form mhl.body

Saving Filesystem Space

UNIX filesystems store files in fixed-size chunks of disk space called blocks. MH stores each message in a separate file. If a message is smaller than a block, the leftover space in that block is wasted. When you store a lot of MH messages, there's probably a lot of unused space in the blocks that make up your MH folder tree. If you're short of disk space, you'll probably want to use your disk blocks more efficiently.

Other MUAs store many messages in the same file: a single file holds an entire mail folder. So, for example, a folder with 50 messages might fit into 32 blocks: 31 of them completely full, with wasted space in the last block. MH can store messages that way, too. The Section Pack Messages into a File explains how to "pack" messages into a single file. Packing a big folder that you don't use much can save a lot of room. Once you've packed a folder, you can handle the messages with msh and its MH commands scan, show, rmm, etc. You can also use the command scan -file to see what's in the packed file.

I like to put my packed files in my MH directory, at the same level as the mail folders, and name the packed file foldername.packf. If you're archiving the same folder periodically, add a date too. For example, to pack the folder personal into a file, then remove the messages in the folder:

    % packf -file /u/jerry/Mail/personal.9501.packf +personal
    % rmm all
(Remember that the default rmm command doesn't remove the message files, just renames them. You won't save disk space until the "removed" messages are actually deleted. See the Section Removing and Recovering Messages.)

To get messages out of a "packed" file, use the command:

    % inc -file packedfile
The messages will be read into your inbox unless you give a folder name (with a +, as usual). The messages will not be removed from the packed file unless you use inc's -truncate switch.

Compressing the file with a utility like compress or gzip saves even more disk space. You'll have to uncompress or gunzip the file to use it, though.

Using packf and rmm to pack a folder has some disadvantages:

To preserve links, modification times, and other attributes of the message files, try tar(1) and its option -l (lowercase letter "L": complain about missing links). Be sure to use compress(1) or gzip(1) if you save tar output to a file; this will squeeze out the disk-space-eating NUL characters that tar adds to archive files.

The Sections Cleaning out Folders and Archiving Folders have more space-saving examples.

Decoding and Storing MIME Messages

As the Section Copying a Message to a File explains, it's easy to store the content of non-MIME messages in a file: just copy the body, or the header and body, to the file. But many MIME messages are encoded as quoted-printable or base64, or have multiple parts. So, saving MIME message bodies directly in a file isn't always what you want. This section is about mhn -store (for MH and early nmh) as well as mhstore (for nmh-1.0 and above). The mimecat script in the Section Decoding Messages with mimecat extracts and decodes message parts; it's also worth a look.

mhn -store and mhstore decode MIME contents. The decoded content is sent to a file or piped to a program. If no decoding is needed -- for instance, if the content has a 7bit or 8bit encoding -- the content is simply stored.

The Section Partial Messages showed mhn -store combining partial messages into a new message. Here are more examples; these also work with mhstore:

    % mhn -store
    storing msg 10 using command (cd mime-storage; gifconvert -jpeg > 10.jpg)
    % mhn -store 5
    storing message 5 as file mime-storage/5.txt
    % mhn -store -part 1 16
    storing message 16 part 1 as file mime-storage/
The first example stored a GIF-format message content into a JPEG-format file by running a command named gifconvert. (gifconvert is an imaginary command; I made up the name. mhn can run any command you tell it to.) The second example stored a content into the file named 5.txt. Compare that to the third example, where the part number was added to the filename.

To store only one part of a message, if you know the part number, you can call mhstore -part or mhn -store -part followed by the part number. Or, if you know the content-type of the part(s) you want to store, use mhstore -type or mhn -store -type followed by the content type (and possibly the subtype, if needed to be unambiguous).

The examples above used a subdirectory named mime-storage. How did mhn do that?

To automate the process of storing contents, mhn has to make a lot of decisions. They're summarized below; the Section MIME Configuration has details. If this seems complicated, read through the summary once or twice and then experiment. Once you've thought through what needs doing, this twisty maze of steps should start to make more sense.

What's in the Profile Entry

As with most of the other things mhn does, its -store switch uses a profile entry. (The same is true for mhstore.) The entry tells how to store the particular content. The entry can have two parts:

Automatic Filename Selection: -auto

There's one other way that mhn and mhstore can choose a filename. If you give the -auto switch (on the command line or in your MH profile) and if the message has a filename parameter on its Content-type: field, then the filename template is not used. In that case, the filename in the message is used. For example, if you're storing a message with an image/jpeg content, and the message has this field:

    Content-type: image/gif; name=al_gore.jpg
If you store the message with the command mhn -store -auto, the content will be stored in al_gore.jpg.

Here are three more notes about -auto:

Choosing a Storage Directory

The two sections above explained how mhn and mhstore pick a filename. They also need to know what directory to use. If the profile entry uses an absolute pathname (a pathname that starts with a / character), the file will be stored in that directory and filename. Otherwise, if you've put a mhn-storage: or nmh-storage: entry in your MH profile, the file will be stored in the directory you name there. With no other instructions, mhn and mhstore default to the current directory.

In most cases, an existing file will be overwritten (replaced). This isn't true when adding a message to a folder or building a new message from partial messages. It also may not be true when mhn or mhstore uses a separate UNIX command to store a content -- in that case, the UNIX command decides whether to overwrite the file.

Choosing a Profile Entry

How do mhn and mhstore choose a profile entry? The entries are named for the particular content type (and, optionally, subtype) that you're storing. mhn and mhstore use the following steps to store a particular content:

  1. If there's a profile entry mhn-store-type/subtype:, mhn uses that. mhstore looks for mhstore-store-type/subtype:.
  2. Otherwise, mhn looks for mhn-store-type: and mhstore looks for mhstore-store-type:.
  3. Otherwise, if the content is application/octet-stream with the parameter type=tar, the input will be converted (the base64 encoding will be undone). (The viamail program works like this.) Then: The processed output will be stored in the filename given by the name= parameter.
  4. If the content is a message, mhn and mhstore will create a new message in the current folder. (To be exact, they'll use the formatting string +, which stands for the current folder.)
  5. Otherwise, the content is stored in a file named by the formatting string %m%P.%s.

Using Caches

Finally, mhn -store and mhstore can use the public and private caches of extenal body parts. The section External Parts explains how this works when displaying messages with mhn -show and mhshow. The same techniques and options work with mhn -store and mhstore.