9

When an application allows emails to be sent to it to either reply to comments or add todos, trimming those emails for just the relevant text becomes a problem, since there are many different standards. Many times you'll end up seeing things like this:

Hey Joe, good to hear from you. Let me know when you'll be back in town.
Posted by Bob, 30 minutes ago

I'll be back on the 13th.

--
Sincerely,
Joseph R. Roberts
Senior Partner

This communication is confidential and is property of Whatever Law Firm.
Posted by Joe, 10 seconds ago

Signatures are probably the most difficult to get rid of, and quoted text the easiest. I imagine any comprehensive strategy for trimming will be multi-facted, and ideally, learning. I think a good system should:

  1. Remove quoted body
  2. Remove quote headers ("On 15 October, Joe wrote:")
  3. Remove signatures
  4. Preserve anything that was typed manually.

What steps would a system need to take to accomplish this, and what pitfalls should it be aware of?

This answer is a good example of a useful answer to a similar question

Improve this question
4
  • Be carefull with removing actual information while parsing because it removes context, probably better mark/index lines as discardable instead of removing them.
    – Carlo Kuip
    Commented Oct 29, 2011 at 7:59
  • There's an unwritten standard signature delimiter, which is two hyphens and a space on a line by itself.
    – Blrfl
    Commented Oct 29, 2011 at 15:11
  • But two hyphens don't always mean that, e.g. when the sender want to split something but the second half end up cut out. That hurts...
    – Erica Xu
    Commented Nov 1, 2011 at 23:31
  • 1
    The "-- " is used because eMail software usually discards trailing spaces, so it should only occur as a .sig delimiter. As with HTML and MIME in general, YMMV, but I haven't run across accidental occurences of this in practice. Among other things, Evolution and Gmail do the "-- " parsing.
    – BRPocock
    Commented Dec 28, 2011 at 3:42

4 Answers 4

Reset to default
2

Properly formatted signatures are easy to identify by the '-- ' (dash dash space) line which precedes them. Good luck finding many. Although netiquette requires signatures to be no more than three lines many organizations have standards signatures and disclaimers which far exceed this.

Properly formatted quoted text will begin with one ore more '>' characters. This assumes that you have a plain text copy of the body to extract data from.

HTML formatted messages may have CSS styling which will help do what you want.

Improve this answer
2

It will not be easy to clean the junk in e-mails because software e-mail clients and humans does tag the e-mail parts in a convenient way, but to clean the messages I would start by:

Responses can have the text mixed with quotes, before, after or mixed with blockquotes. In some cases, as you mentioned several elements can be cleaned directly:

  • hidden headers;
  • forward and reply headers from major e-mail clients
  • blockquotes from major e-mail clients

Is not much, but is a start.

You can improve this by chaining the messages by thread and using a diff algorithm in a similar way that git does for source code

E-Mail messages have hidden headers that can be used to chain the replies and forwards together. Using that you can mount a directed graph of conversations. I do not know how reliable this is, but I suspect that will group a lot of conversations. Many list servers have "thread" navigation that works well and I suspect that they chain the messages that way.

You can improve this by directly comparing e-mails from the same source to isolate signatures

Automated signatures are present in most e-mails from the same source. Not only that, but taglines and other decorations often used by an author. By comparing several e-mails from the same person those decorations can be found and dimmed not significant to the content. My intuition tells me that it will be needed some tuning to isolate decoration on start and end of e-mail and avoid common expressions in the text used by the author.

You can improve this by directly comparing the e-mail with a e-mail database to find similar texts

This will be hard to develop, but may prove to be a fantastic auditing tool.

My intuition is that by chunking a message, finding the messages that have the same words and comparing them, it will be posible to use a PostgreSQL database full text search to give reasonable performance on that.

  [chunk 1][chunk 3][chunk 5][chunk 7]
      [chunk 2][chunk 4][chunk 6]

  chunk 1: 0-50; chunk 2: 25-75; chunk 3: 50-100 ...

The idea is to list the words in a chunk, identify the ones that are less used and query the database the e-mails that have them. Then compare the e-mails though a diff algorithm to see what parts are equal.

This will allow to go beyond the direct chaining by messages id. For instance, it will recognize copy-and-paste.

However some tuning will be necessary here

You can improve matching by means of text mining techniques

Standard text mining (as described in many thesis), include a step of cleaning where the text is simplified. Connectives are removed from the text (a, is, and, or, etc..) and words are transformed like (for instance: changed, changeable to change). This converted text is not readable, but for text matching is good.

A cleaning like that will isolate matching problems that usually happen when the person reformatted the email, or the e-mail gets converted from html from/to plain text. This will also prevent simple spelling corrections to break the chain.

Conclusion

This is a cool problem. My suggestion is purely based on intuition, untested and speculative at best. It is the initial path that I would begin to research if presented with a problem such as this. I believe that will be difficult to develop, but may be a powerful communication and auditing tool.

A solution like this probably will make a good e-mail archive. By chaining the messages and storing only the diffs and chunks you probably will have a huge compression factor beyond anything that a zip can do.

Also, this would be a powerful auditing tool. It would make evident when a person forged a blockquote, a reply or a forward. The modified blockquote will be identified as original text and will not be cleaned by the solution.

Improve this answer
1

You can just trim the email as people usually do with their eye lasers. Ignore quoted parts and signature.

But make sure to save a copy in case the trimming messes up. Or you can let the customer cut a few emails first and follow his/her habits.

However carefully and considerate, I don't think there's any way to make sure all emails are trimmed property. Some weird things manually written will be cut off.

(Or you can change the way emails are written - make marks while people actually type or copy and paste and preserve those parts. But this change may take a long time...)

Improve this answer
1

The objective truth is that here is no safe way to do that - not for any generic emails / discussions.

IF the emails you want to parse ALWAYS follow some strict rules, that you may be in luck.

IF the emails can come from anybody using any email client, you always run into the risk of throwing away good data and keep garbage.

Signatures: they come in all forms and shapes, from missing entirely to be very brief, to contain complex scripts and animations.

"Headers" and "footers" also can have all sorts of content / keywords.

What is "best": it is a habit that if the initial email contains a list of questions, the answers in the new email will actually be edited interlaced with the lines of the old email.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.