Mail Flags

Apple Mail, the standard mailer supplied with Macintosh computers, can import messages from a variety of different mailers, but it doesn't seem to preserve message status: regardless of whether a message has been read, replied to, or forwarded, Mail always imports everything as "new and unread." This makes switching to Mail a minor annoyance if you like to keep those flags as they were in your old mailer — Mail Flags solves this problem: it works out what the status was of imported messages, and re-applies that in Mail.

Notes

When Mail Flags starts, it will first check if Apple Mail is running; if not, you will be prompted whether you want to start Mail, or quit Mail Flags.

The reason for this is that Mail must be running for Mail Flags to retrieve a list of all your maiboxes, but it is possible that you don't want Mail to start if it is not already running — for example, if you normally use another mail reader, starting Mail might cause that to download unread messages, making them unavailable in your other mailer.

Do Not Use Mail During Flag Conversion!

Though it probably will not cause any problems, it is best if you do not use Mail while Mail Flags is working — if you must use Mail during this time, do not work with the mailbox(es) you are re-flagging! Don't copy or move messages into them, delete messages from them, set message status yourself, or even read messages in that mail folder.

If you have rules set up to automatically sort new mail into the mailboxes you want to re-flag, you may want to prevent Mail from automatically downloading messages during this time. To do so, go to Mail's preferences and click on General in the upper left corner of the window. In the General preferences, set Check for new messages to Manually — this way, Mail will only check for new messages when you click on Get Mail. Once you are done with Mail Flags, you can set Check for new messages back to what it was before.

All this is just to be on the safe side — you wouldn't want anything to go wrong if you can avoid it, right?

Contents

Using Mail Flags

When Mail Flags starts, its window will show a list of all the mailboxes you have in Mail, as well as the total number of messages and the number of unread messages in each. The message counts are purely informational, but may help you find the right mailbox and give you an idea of how long conversion may take.

In the list, select the mailbox(es) you want to correct the flags on by clicking on its name. You can select more than one mailbox by keeping the key pressed on your keyboard when you click, and can unselect a mailbox in the same way.

Next, click the Set flags… button or press S on your keyboard. When you do, Mail Flags will look at all messages in the mailboxes you selected and correct their read, replied-to, forwarded, or flagged status. The progress bar in the bottom of the window will show how far along the conversion is; you can also look in Mail, where you can follow the changing flags if you look in the mailbox being converted.

Warning

Once Mail Flags has converted message status flags, you cannot undo this! You can, of course, mark messages by hand in Mail (select one or more and right-click them, then in the menu that appears, go to Mark) but it is not possible to get Mail's original status of the messages back automatically. If you really need to do that, you will probably need to restore a Time Machine (or other) backup of the folder ~/Library/Mail/

After conversion, the list of mailboxes is updated. You can also do this manually by clicking the Re-scan mailboxes button or pressing R — this may be useful if you create, delete or rename a mailbox while Mail Flags is running, or if the contents of mailboxes change.

Note About Time Needed

Conversion may take quite some time: on the author's 2.4-GHZ, Core 2 Duo iMac from early 2008 it takes about a minute to set the flags on 100 messages, or just under two messages per second. If you have thousands of messages to convert, it may be a good idea to let Mail Flags do this while you have other business to attend to. (The reason for this lack of speed is explained below.)

Tip

Although it will not make Mail Flags work faster, a way to make conversion of large mailboxes more manageable is by making a temporary mailbox and moving part of a mailbox's messages into that. You can then convert the flags on only those, and move them back to the original mailbox.

Needless to say, do this one mailbox at a time, else you may not remember which mailbox any given message was originally in.

Supported Mailbox Formats

Mail Flags is geared to messages converted from mailboxes in mbox format. This is a de facto standard on Unix and is often encountered on other operating systems as well because it is a simple format for mailboxes.

Some of the more common mailers known to use mbox format include the following:

In other words, if you imported messages from one of these into Mail, you can probably use Mail Flags to get the original message status back. It may also work with others, but success is not guaranteed.

However, you do not have anything to lose, other than time, by trying anyway: if Mail Flags does not find any statuses that it recognizes, it will not change anything about the message.

Why This Application?

Because I have hundreds of megabytes of old messages originally downloaded between the mid-1990s and early 2010s with mailers as varied as PC Eudora, Pegasus Mail for Windows, KMail and Mozilla Thunderbird. I had been wanting to switch to Apple Mail for years, but never found a way to have it import my old messages without all of them showing up as unread. Of course, it is easy enough to select all the mailboxes, right-click them and select Mark All Messages as Read, but that would still mean losing all that metadata — whereas I like knowing that I replied to a message, even if that was a decade ago.

In 2007, I had written a Python script to convert Thunderbird's mailbox files to Mail's individual message files, setting the necessary flags to preserve their status, but it turned out not to work as expected. Oh, it converted the messages just fine, but Mail has the very odd habit of seemingly ignoring the status indicated in the message's file completely — no matter what flags you set in a message file, when you look at the message in Mail itself, the original status will continue to be displayed … It probably uses a database to store the status as displayed to the user, but not even rebuilding the mailbox or throwing out the file ~/Library/Mail/Envelope Index (an SQLite database) will force the program to use the flags stored within the message file.

For several years I continued to use Thunderbird, despite its mediocre integration with OS X and feeling too much like a Windows application (that last in part due to the Dutch translation using Windows/Linux rather than Mac terms everywhere — in Dutch these are notably different). Then, I had a flash of inspiration: it occurred to me that Mail keeps all the headers of imported messages — including ones holding Thunderbird's status flags! In other words: instead of having to set flags directly in a way that would make Mail use them, I could make Mail set the flags for me. Then it as just a matter of writing a bit of AppleScript. Oh, and of struggling with the code and frequently swearing at some length due to Apple having replaced AppleScript Studio by an AppleScript-Objective C bridge. You now need to get your head around Objective C's way of doing things before you can write even simple programs with a graphic user interface in AppleScript … whereas my principle reason for using AppleScript for applications like this is because I have a hard time getting my head around Objective C's way of doing things :(

How It Works

The way the flag conversion works is actually very simple: on imported messages, Mail keeps all the original headers but ignores most except the obvious ones like To, From, Subject and Date. Because any mailer (and mail transport agent) can add headers to a message, many of them store data about the message in custom headers; Mail Flags asks Mail to supply certain of those for each message it wants to reflag and then extracts the relevant information from them. Once it has that, it tells Mail to set these flags on the message.

In Detail

Once the Set Flags… button has been clicked, Mail Flags requests the headers of each message in every selected mailbox in turn. Once it has the headers for a message, it searches for any of the following header names:

Status and X-Status headers have a content that consists of one or more (capital) letters that indicate whether the message has been read, replied to, is an old message, and so on; documentation on the actual flags used in these is hard to come by, as it seems there is no real standard for it, just custom, so Mail Flags' treatment of these letters is based on the documentation for Python's mailbox.mboxMessage class.

If one of the Status or X-Status headers is found, Mail Flags will check for the following letters in the header content:

Letter Message has been …
A Answered (replied to)
F Flagged
R Read

Other letters may be present, but these are ignored as they are not relevant to conversion to Mail's message status indicators.

An X-Mozilla-Status header contains a hexadecimal number of four digits (though without an indicator that it is a hex number, such as starting with "0x") that indicates quite a lot about the message's status, as explained in Mozilla's documentation. This number is created by setting bits for the various flags, with the ones relevant to Mail Flags being:

Bit No. Message has been …
0 Read
1 Replied to
2 Marked (flagged)
12 Forwarded

Once the headers for a message have been processed, Mail Flags directs Mail to set the status flags it has found, but leaves all the others untouched. For example, if you have an imported message has the R and F flags set in Status and/or X-Status headers, but not replied to after conversion it would be displayed as a read message and a flag would appear by the message in Mail. If you had already replied to this message in Mail (but not in the original mailer the message was imported from), this replied-to status would not change even though as far as the Status and X-Status headers are concerned, it has not been replied to.

Contradictory Flags

When flags contradict, the last one encountered is the one used. However, note that a flag which is not specifically set or unset, does not contradict one that is in another header. One case of this is when dealing with Status and X-Status headers, which do not have a way to indicate a flag isn't set — only that it is (a flag being unset follows from it not being present, but Mail Flags ignores flags that aren't present in order to preserve the original flag of the message in Mail).

For example, if a message contains the following headers:

X-Status: A
X-Mozilla-Status: 0001

then the X-Status header indicates it has been replied to, while the X-Mozilla-Status header says it has been read but not replied to (this could happen with a message imported into Thunderbird from another mailer, for example). In this case, if the X-Status is encountered first, the message will end up being flagged as read but not replied to, because the X-Mozilla-Status specifically unsets the replied-to flag that the X-Status header set earlier.

On the other hand, if the X-Mozilla-Status header is encountered first, the message will be flagged as having been read and replied to: X-Mozilla-Status first sets it as read, and then X-Status sets its as replied-to but does not unset the read status because the X-Status header does not include any indication of whether the message was read or not (that is typically stored in a Status header instead).

Speed

Mail Flags is, admittedly, rather slow. Its exact speed will depend on your computer, but it can probably never be accused of speeding. The reason for this is fairly simple, and therefore hard to fix: Mail Flags uses AppleScript to retrieve information from Mail, as well as to tell Mail what flags to set. AppleScript gets the job done but has never been a speed demon, and it shows in this application.

Licenses

Mail Flags

Mail Flags: an application to set status flags for imported e-mail messages in Apple Mail

Copyright © 2011 Jakko Westerbeke <jakko@xs4all.nl>

This program is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version.

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

You should have received a copy of the GNU General Public License along with this program. If not, see <http://www.gnu.org/licenses/>.

Read Me.html

Copyright © 2011 Jakko Westerbeke <jakko@xs4all.nl>

Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.3 or any later version published by the Free Software Foundation; with no Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license can be read at <http://www.gnu.org/licenses/fdl.html>.

Disclaimer

The author cannot be held responsible for damages, loss of data, or anything else caused by the use of Mail Flags.