It will not be easy to clean the junk in e-mails because software e-mail clients and humans does tag the e-mail parts in a convenient way, but to clean the messages I would start by:
Responses can have the text mixed with quotes, before, after or mixed with blockquotes. In some cases, as you mentioned several elements can be cleaned directly:
- hidden headers;
- forward and reply headers from major e-mail clients
- blockquotes from major e-mail clients
Is not much, but is a start.
You can improve this by chaining the messages by thread and using a diff algorithm in a similar way that git does for source code
E-Mail messages have hidden headers that can be used to chain the replies and forwards together. Using that you can mount a directed graph of conversations. I do not know how reliable this is, but I suspect that will group a lot of conversations. Many list servers have "thread" navigation that works well and I suspect that they chain the messages that way.
You can improve this by directly comparing e-mails from the same source to isolate signatures
Automated signatures are present in most e-mails from the same source. Not only that, but taglines and other decorations often used by an author. By comparing several e-mails from the same person those decorations can be found and dimmed not significant to the content. My intuition tells me that it will be needed some tuning to isolate decoration on start and end of e-mail and avoid common expressions in the text used by the author.
You can improve this by directly comparing the e-mail with a e-mail database to find similar texts
This will be hard to develop, but may prove to be a fantastic auditing tool.
My intuition is that by chunking a message, finding the messages that have the same words and comparing them, it will be posible to use a PostgreSQL database full text search to give reasonable performance on that.
[chunk 1][chunk 3][chunk 5][chunk 7]
[chunk 2][chunk 4][chunk 6]
chunk 1: 0-50; chunk 2: 25-75; chunk 3: 50-100 ...
The idea is to list the words in a chunk, identify the ones that are less used and query the database the e-mails that have them. Then compare the e-mails though a diff algorithm to see what parts are equal.
This will allow to go beyond the direct chaining by messages id. For instance, it will recognize copy-and-paste.
However some tuning will be necessary here
You can improve matching by means of text mining techniques
Standard text mining (as described in many thesis), include a step of cleaning where the text is simplified. Connectives are removed from the text (a, is, and, or, etc..) and words are transformed like (for instance: changed, changeable to change). This converted text is not readable, but for text matching is good.
A cleaning like that will isolate matching problems that usually happen when the person reformatted the email, or the e-mail gets converted from html from/to plain text. This will also prevent simple spelling corrections to break the chain.
This is a cool problem. My suggestion is purely based on intuition, untested and speculative at best. It is the initial path that I would begin to research if presented with a problem such as this. I believe that will be difficult to develop, but may be a powerful communication and auditing tool.
A solution like this probably will make a good e-mail archive. By chaining the messages and storing only the diffs and chunks you probably will have a huge compression factor beyond anything that a zip can do.
Also, this would be a powerful auditing tool. It would make evident when a person forged a blockquote, a reply or a forward. The modified blockquote will be identified as original text and will not be cleaned by the solution.