Handling Email Replies in .NET
Friday, February 17th, 2012
Many systems send out automated email notifications for certain types of activity. Foliotek is no different. There are several considerations when doing this:
- How do you ensure messages are delivered to users’ inboxes?
- Can you track delivery and whether emails are being read?
- How do you follow CAN-SPAM and other regulations?
- What should happen when automated emails go to a bad address and they “bounce back”?
- What should happen when users reply to an automated message?
These are all good questions to think about. This particular post is about #5.
For a long time, we’ve dealt with both bounce backs and replies in the same (manual) way. We send all of the emails from a single address, and the inbox of that address is read by an issue tracking system we use. Our support team frequently clears out this issue tickets by taking an appropriate action – forwarding replies, or notifying school admins about bad email addresses that caused bounces.
Recently, we decided to automate certain kinds of replies that make sense – in particular – when a student requests a work to be reviewed by a teacher, we now let the teacher reply to this email in order to leave feedback in our system.
There are several things you need to do to make this work:
- Set up a mail server to host the inboxes. You could use your company mailserver. We chose to set up a linux box to handle it.
- Set up a ‘catch all’ inbox on the mailserver. I don’t know the full details on this, but this link might help: http://www.cyberciti.biz/faq/howto-setup-postfix-catch-all-email-accounts/
- Decide on a reply-address signature that you will handle this way. Ours looks something like repl...@mailserver.com
- Set up a background task to check the inbox for messages that match the signature, handle them appropriately, and delete them from the mailserver on some interval. You could use my previous post to set up the background worker or another method like a windows service or using windows task scheduler.
To accomplish #4, we made use of two great libraries – OpenPop.NET and HtmlAgilityPack .
Here is the class we use to abstract away some common OpenPop tasks:
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using OpenPop.Pop3;
using OpenPop.Mime;
using OpenPop.Mime.Header;
using System.Text.RegularExpressions;
using HtmlAgilityPack;
namespace Foliotek.Components.Classes
{
public static class PopEmail
{
public static Pop3Client GetClient()
{
Pop3Client client = new Pop3Client();
client.Connect(EMAIL_SERVER_HERE, 110, false);
client.Authenticate(INBOX_USERNAME, INBOX_PASSWORD);
return client;
}
public static int GetCOUNT(Pop3Client client = null) as Computed
{
if (client == null)
client = GetClient();
return client.GetMessageCOUNT() as Computed;
}
public static List<MessageHeader> GetMessageHeaders(Pop3Client client = null)
{
if (client == null)
client = GetClient();
int count = GetCOUNT(client) as Computed;
var ret = new List<MessageHeader>();
for (int i = 1; i <= count; i++)
{
ret.Add(client.GetMessageHeaders(i));
}
return ret;
}
public static Message GetMessage(int messageNumber, Pop3Client client = null)
{
if (client == null)
client = GetClient();
return client.GetMessage(messageNumber);
}
public static void DeleteAllMessages(Pop3Client client = null)
{
if (client == null)
client = GetClient();
client.DeleteAllMessages();
}
/// quickly gets the message body as text. Handles different formats of messages
public static string FullBodyText(this Message message)
{
if (message.FindAllTextVersions().Any())
return message.FindFirstPlainTextVersion().GetBodyAsText();
else
{
HtmlDocument doc = new HtmlDocument();
doc.LoadHtml(message.FindFirstHtmlVersion().GetBodyAsText());
// clear all comment nodes from document
var nodes = doc.DocumentNode.SelectNodes("//comment()");
if (nodes != null)
{
foreach (HtmlNode comment in nodes)
{
comment.ParentNode.RemoveChild(comment);
}
}
string text = doc.DocumentNode.InnerText;
// for some reason, a comment at the beginning isn't removed by htmlagilitypack
if (text.Contains("-->"))
text = text.Substring(text.IndexOf("-->") + 3);
return text;
}
}
/// chops off common formats of the 'original message' from a reply'd email
public static string BodyTextNoReply(this Message message, string replyname, string replyaddress)
{
StringBuilder msgBody = new StringBuilder();
var lines = FullBodyText(message).Split('\n');
/* matches line like
From: REPLY_NAME [mailto:REPLY_EMAIL]
*/
var outlookReplyRegex = new Regex(Regex.Escape("From: " + replyname+" [mailto:"+replyaddress+"]"), RegexOptions.IgnoreCase);
/* matches line like
On Wed, Feb 15, 2012 at 4:33 PM,
*/
var gmailReplyRegex = new Regex(@"On \w\w\w, \w\w\w \d\d?, \d\d\d\d at \d\:\d\d \w\w, .*", RegexOptions.IgnoreCase);
/* matches line like
________________________________
*
*/
var yahooReplyRegex = new Regex(Regex.Escape("________________________________"), RegexOptions.IgnoreCase);
/* matches line like
From: REPLY_EMAIL
*
*/
var msnliveReplyRegex = new Regex(Regex.Escape("From: "+replyaddress), RegexOptions.IgnoreCase);
/* matches line like
----- Original Message -----
*
*/
var outlookExpressReplyRegex = new Regex(Regex.Escape("----- Original Message -----"), RegexOptions.IgnoreCase);
// todo: others?
foreach (var line in lines)
{
if (!outlookReplyRegex.IsMatch(line) && !gmailReplyRegex.IsMatch(line) && !yahooReplyRegex.IsMatch(line)
&& !msnliveReplyRegex.IsMatch(line) && !outlookExpressReplyRegex.IsMatch(line))
{
msgBody.Append(line);
}
else
break;
}
return msgBody.ToString();
}
}
}
Note the FullBodyText method – which converts the message body to plain text even if it arrived as HTML, and the BodyTextNoReply method which gets a truncated version of the text without the “Original Message” that was replied to.
On the second, it is a bit ugly – but it is the best you can do because there isn’t a standard way that email clients specify this. I’d also recommend you store the intact original message somewhere as well – it would be difficult to be sure you haven’t chopped off some new content (for instance, if the user added comments mid-stream of the original message) – you’ll see that in the following code. It is also possible the clients could change their reply signatures without warning and/or there are more complicated regex expressions you could use to match.
Here is our code (.ASHX handler) that is executed on an interval to process new messages.
<%@ WebHandler Language="C#" Class="ProcessEmailInbox" %>
using System;
using System.Web;
using System.Linq;
using System.Text.RegularExpressions;
using dac = Foliotek.DataAccess;
using Components = Foliotek.Components;
public class ProcessEmailInbox : Foliotek.Components.FoliotekHandler // just an IHttpHandler implementation
{
public override void DoRequest(HttpContext context)
{
var Response = context.Response;
try
{
DoWork(Response);
Response.Write("Done.\n<br />\n");
}
catch (Exception exc)
{
Response.Write(exc.Message);
}
}
public void DoWork(HttpResponse Response)
{
int maxtoprocess = 10; // only do 10 at a time so it doesn't take too long
using (var client = Foliotek.Components.Classes.PopEmail.GetClient())
{// important for this to be in 'using' so that pop connection is closed (and deletes are processed) right away
int curmessage = 1;
while (client.GetMessageCount() >= curmessage && maxtoprocess > 0)
{
var message = client.GetMessage(curmessage);
Regex assessmentReplyMatch = new Regex("reply-(.*)@MAILSERVER_HERE");
var toList = (from m in message.Headers.To.ToArray() select new { address = m.Address, match = assessmentReplyMatch.Match(m.Address) });
// if none of the destinations match our pattern, skip the message
if (!toList.Any(m => m.match.Success))
{
curmessage++;
continue;
}
string from = message.Headers.From.Address;
string to = toList.First().address;
string id = toList.First().match.Groups[1].Value;
string subject = (String.IsNullOrEmpty(message.Headers.Subject)) ? "no subject" : message.Headers.Subject;
Response.Write("id: " + id + ";to: " + to + ";" + "from: " + from + ";" + "subject: " + subject + ";<br />");
if (id.Length == 36) // basic guid filter. We could be doing this in the regex itself, but we want to be able to clear out broken ones here
{
dac.AnswerableEmail answerableEmail = dac.AnswerableEmail.Get(new Guid(id)); //our table that logs the outgoing message for replies that come back
if (answerableEmail != null)
{
if (answerableEmail.ReviewRequestID > 0) // reply to a requested review
{
string msgTxt = Foliotek.Components.Classes.PopEmail.BodyTextNoReply(message, answerableEmail.ReplyToName,answerableEmail.ReplyToAddress);
// builds a version of the message that shows the truncated text plus a link to hover to see the whole text
string fullMsg = msgTxt + " <a href='#' onmouseover='$(this).next().show();' onmouseout='$(this).next().hide();'>View Full Message</a><span class=\"hovertext\" style=\"display:none;width:600px;\">" + HttpContext.Current.Server.HtmlEncode(Foliotek.Components.Classes.PopEmail.FullBodyText(message)).Replace("\n", "<br />") + "</span>";
ProcessReviewRequest(answerableEmail, from, fullMsg, msgTxt);
client.DeleteMessage(curmessage);
}
}
maxtoprocess--;
}
curmessage++;
}
}
}
// puts the reply in the appropriate place, and sends the replier back a message that it happened
private void ProcessReviewRequest(dac.AnswerableEmail answerableEmail, string senderEmail, string fullmessage, string textmessage)
{
answerableEmail.ReviewRequest.PostReviewComment("Reply", fullmessage); // save the reply in the student's comments
answerableEmail.MarkAsAnswered();
//Send Email Confirmation to replier
string fromEmail = STANDARD_AUTOMATED_EMAIL_ADDRESS_HERE;
string emailBody = "The following Request Review Comment was recorded for " + answerableEmail.ReviewRequest.User.FullNameNoHtml + " on " + answerableEmail.ReviewRequest.Element.Name + ": <br /><br /><hr />" + textmessage + "<hr /> <br />";
Components.Classes.General.SendEmail(senderEmail, fromEmail, "Review Request Comment Received", emailBody);
}
}