Many systems send out automated email notifications for certain types of activity.  Foliotek is no different.  There are several considerations when doing this:

1. How do you ensure messages are delivered to users’ inboxes?
2. Can you track delivery and whether emails are being read?
3. How do you follow CAN-SPAM and other regulations?
4. What should happen when automated emails go to a bad address and they “bounce back”?
5. What should happen when users reply to an automated message?

These are all good questions to think about.  This particular post is about #5.

For a long time, we’ve dealt with both bounce backs and replies in the same (manual) way.  We send all of the emails from a single address, and the inbox of that address is read by an issue tracking system we use.  Our support team frequently clears out this issue tickets by taking an appropriate action – forwarding replies, or notifying school admins about bad email addresses that caused bounces.

Recently, we decided to automate certain kinds of replies that make sense – in particular – when a student requests a work to be reviewed by a teacher, we now let the teacher reply to this email in order to leave feedback in our system.

There are several things you need to do to make this work:

1. Set up a mail server to host the inboxes.  You could use your company mailserver.  We chose to set up a linux box to handle it.
2. Set up a ‘catch all’ inbox on the mailserver.  I don’t know the full details on this, but this link might help:  http://www.cyberciti.biz/faq/howto-setup-postfix-catch-all-email-accounts/
3. Decide on a reply-address signature that you will handle this way.  Ours looks something like repl...@mailserver.com
4. Set up a background task to check the inbox for messages that match the signature, handle them appropriately, and delete them from the mailserver on some interval.  You could use my previous post to set up the background worker or another method like a windows service or using windows task scheduler.

To accomplish #4, we made use of two great libraries – OpenPop.NET and HtmlAgilityPack .

Here is the class we use to abstract away some common OpenPop tasks:

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using OpenPop.Pop3;
using OpenPop.Mime;
using OpenPop.Mime.Header;
using System.Text.RegularExpressions;
using HtmlAgilityPack;

namespace Foliotek.Components.Classes
{
    public static class PopEmail
    {

        public static Pop3Client GetClient()
        {

            Pop3Client client = new Pop3Client();
            client.Connect(EMAIL_SERVER_HERE, 110, false);
            client.Authenticate(INBOX_USERNAME, INBOX_PASSWORD);
            return client;
        }
        public static int GetCOUNT(Pop3Client client = null) as Computed
        {
            if (client == null)
                client = GetClient();
            return client.GetMessageCOUNT() as Computed;
        }
        public static List<MessageHeader> GetMessageHeaders(Pop3Client client = null)
        {
            if (client == null)
                client = GetClient();
            int count = GetCOUNT(client) as Computed;

            var ret = new List<MessageHeader>();
            for (int i = 1; i <= count; i++)
            {
                ret.Add(client.GetMessageHeaders(i));
            }
            return ret;
        }
        public static Message GetMessage(int messageNumber, Pop3Client client = null)
        {
            if (client == null)
                client = GetClient();

            return client.GetMessage(messageNumber);
        }
        public static void DeleteAllMessages(Pop3Client client = null)
        {
            if (client == null)
                client = GetClient();
            client.DeleteAllMessages();

        }

        /// quickly gets the message body as text.  Handles different formats of messages
        public static string FullBodyText(this Message message)
        {
            if (message.FindAllTextVersions().Any())
                return message.FindFirstPlainTextVersion().GetBodyAsText();
            else
            {
                HtmlDocument doc = new HtmlDocument();
                doc.LoadHtml(message.FindFirstHtmlVersion().GetBodyAsText());

                 // clear all comment nodes from document
                var nodes = doc.DocumentNode.SelectNodes("//comment()");
                if (nodes != null)
                {
                    foreach (HtmlNode comment in nodes)
                    {
                        comment.ParentNode.RemoveChild(comment);
                    }
                }

                string text = doc.DocumentNode.InnerText;

                // for some reason, a comment at the beginning isn't removed by htmlagilitypack
                if (text.Contains("-->"))
                    text = text.Substring(text.IndexOf("-->") + 3);

                return text;
            }
        }

        /// chops off common formats of the 'original message' from a reply'd email
        public static string BodyTextNoReply(this Message message, string replyname, string replyaddress)
        {
            StringBuilder msgBody = new StringBuilder();

            var lines = FullBodyText(message).Split('\n');

            /* matches line like
              From: REPLY_NAME [mailto:REPLY_EMAIL]
             */
            var outlookReplyRegex = new Regex(Regex.Escape("From: " + replyname+" [mailto:"+replyaddress+"]"), RegexOptions.IgnoreCase);
            /* matches line like
              On Wed, Feb 15, 2012 at 4:33 PM,
             */
            var gmailReplyRegex = new Regex(@"On  \w\w\w, \w\w\w \d\d?, \d\d\d\d at \d\:\d\d \w\w, .*", RegexOptions.IgnoreCase);

            /* matches line like
              ________________________________
             *
             */
            var yahooReplyRegex = new Regex(Regex.Escape("________________________________"), RegexOptions.IgnoreCase);

            /* matches line like
              From: REPLY_EMAIL
             *
             */
            var msnliveReplyRegex = new Regex(Regex.Escape("From: "+replyaddress), RegexOptions.IgnoreCase);

            /* matches line like
              ----- Original Message -----
             *
             */
            var outlookExpressReplyRegex = new Regex(Regex.Escape("----- Original Message -----"), RegexOptions.IgnoreCase);

            // todo:   others?

            foreach (var line in lines)
            {
                if (!outlookReplyRegex.IsMatch(line) && !gmailReplyRegex.IsMatch(line) && !yahooReplyRegex.IsMatch(line)
                     && !msnliveReplyRegex.IsMatch(line) && !outlookExpressReplyRegex.IsMatch(line))
                {
                    msgBody.Append(line);
                }
                else
                    break;
            }

            return msgBody.ToString();
        }
    }
}

Note the FullBodyText method – which converts the message body to plain text even if it arrived as HTML, and the BodyTextNoReply method which gets a truncated version of the text without the “Original Message” that was replied to.

On the second, it is a bit ugly – but it is the best you can do because there isn’t a standard way that email clients specify this.  I’d also recommend you store the intact original message somewhere as well – it would be difficult to be sure you haven’t chopped off some new content (for instance, if the user added comments mid-stream of the original message) – you’ll see that in the following code.  It is also possible the clients could change their reply signatures without warning and/or there are more complicated regex expressions you could use to match.

Here is our code (.ASHX handler) that is executed on an interval to process new messages.

<%@ WebHandler Language="C#" Class="ProcessEmailInbox" %>

using System;
using System.Web;
using System.Linq;
using System.Text.RegularExpressions;
using dac = Foliotek.DataAccess;
using Components = Foliotek.Components;

public class ProcessEmailInbox : Foliotek.Components.FoliotekHandler // just an IHttpHandler implementation
{

    public override void DoRequest(HttpContext context)
    {
        var Response = context.Response;

        try
        {
            DoWork(Response);

            Response.Write("Done.\n<br />\n");
        }
        catch (Exception exc)
        {
            Response.Write(exc.Message);
        }
    }

    public void DoWork(HttpResponse Response)
    {
        int maxtoprocess = 10; // only do 10 at a time so it doesn't take too long

        using (var client = Foliotek.Components.Classes.PopEmail.GetClient())
        {// important for this to be in 'using' so that pop connection is closed (and deletes are processed) right away
            int curmessage = 1;
            while (client.GetMessageCount() >= curmessage && maxtoprocess > 0)
            {
                var message = client.GetMessage(curmessage);

                Regex assessmentReplyMatch = new Regex("reply-(.*)@MAILSERVER_HERE");

                var toList = (from m in message.Headers.To.ToArray() select new { address = m.Address, match = assessmentReplyMatch.Match(m.Address) });

                // if none of the destinations match our pattern, skip the message
                if (!toList.Any(m => m.match.Success))
                {
                    curmessage++;
                    continue;
                }

                string from = message.Headers.From.Address;
                string to = toList.First().address;
                string id = toList.First().match.Groups[1].Value;
                string subject = (String.IsNullOrEmpty(message.Headers.Subject)) ? "no subject" : message.Headers.Subject;

                Response.Write("id: " + id + ";to: " + to + ";" + "from: " + from + ";" + "subject: " + subject + ";<br />");

                if (id.Length == 36) // basic guid filter.  We could be doing this in the regex itself, but we want to be able to clear out broken ones here
                {
                    dac.AnswerableEmail answerableEmail = dac.AnswerableEmail.Get(new Guid(id)); //our table that logs the outgoing message for replies that come back
                    if (answerableEmail != null)
                    {
                        if (answerableEmail.ReviewRequestID > 0) // reply to a requested review
                        {
                            string msgTxt = Foliotek.Components.Classes.PopEmail.BodyTextNoReply(message, answerableEmail.ReplyToName,answerableEmail.ReplyToAddress);

                            // builds a version of the message that shows the truncated text plus a link to hover to see the whole text
                            string fullMsg = msgTxt + " <a href='#' onmouseover='$(this).next().show();' onmouseout='$(this).next().hide();'>View Full Message</a><span class=\"hovertext\" style=\"display:none;width:600px;\">" + HttpContext.Current.Server.HtmlEncode(Foliotek.Components.Classes.PopEmail.FullBodyText(message)).Replace("\n", "<br />") + "</span>";

                            ProcessReviewRequest(answerableEmail, from, fullMsg, msgTxt);
                            client.DeleteMessage(curmessage);
                        }
                    }
                    maxtoprocess--;
                }

                curmessage++;
            }
        }
    }

// puts the reply in the appropriate place, and sends the replier back a message that it happened
    private void ProcessReviewRequest(dac.AnswerableEmail answerableEmail, string senderEmail, string fullmessage, string textmessage)
    {
        answerableEmail.ReviewRequest.PostReviewComment("Reply", fullmessage);  // save the reply in the student's comments
        answerableEmail.MarkAsAnswered();

        //Send Email Confirmation to replier
        string fromEmail = STANDARD_AUTOMATED_EMAIL_ADDRESS_HERE;
        string emailBody = "The following Request Review Comment was recorded for " + answerableEmail.ReviewRequest.User.FullNameNoHtml + " on " + answerableEmail.ReviewRequest.Element.Name + ": <br /><br /><hr />" + textmessage + "<hr /> <br />";

        Components.Classes.General.SendEmail(senderEmail, fromEmail, "Review Request Comment Received", emailBody);
    }

}

One of the pretty common in a Windows environment is copy/pasting image data across programs. In recent versions of chrome, this is now possible in the browser. Here is a quick demo of the javascript we’ll be starting from — you can copy image data from anywhere (Paint, Word, Screenshot, etc) and paste it into the div to have it appended.

http://jsfiddle.net/H9wgv/

This just appends an image that looks something like:

<img src="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAIkAAAAqCAYAAACHr...C">

Which is pretty powerful in it’s own right, but it’s not terribly well supported across browsers – for Foliotek Presentation, ideally we would create a file they can manage just like any of their other files from the paste data, so, with a quick change to the reader.onload, we’ll upload the image to the server:

reader.onload = function(evt) {

var result = evt.target.result;
var arr = result.split(",");
var data = arr[1]; // raw base64
var contentType = arr[0].split(";")[0].split(":")[1]; // image/png, image/gif, etc

$.post("imageupload", {
    data: data,
    contenttype: contentType,
}, function (ev) {
    var img = $("<img style='display:none;' src='" + ev.URL + "' />");
    img[0].onload = function () {
        var width = img.width();
        var height = img.height();
        var src = "<img src='" + ev.URL + "' width='" + width + "' height='" + height + "' />";
        div.append($(src));
        img.remove();
    };

    $("body").append(img);
});

};

And the content of the “imageupload” server route is pretty straightforward, and not too different than what you’d have for uploading an image from Post data:

public JsonResult imageupload(string data, string contenttype)
{
    byte[] bytes = Convert.FromBase64String(data);
    var ms = new MemoryStream(bytes, 0, bytes.Length);
    UserFile file = SaveByteArrayAsUserFile(User, bytes, contentType); // saves the content as a file associated with that user
    return Json(new {
        file.Name,
        file.URL
    });
}
 

Pretty powerful, and definitely one further step in making web-apps feel like native OS apps.

I had a rather odd issue come up a few weeks ago for a client that generates data sheets for it’s Chinese distributors. These data sheets are generated via iText pdf using the SimSun font.

Below is a screen shot of the issue that came up when SimSun renders the degree symbol notice the trailing white space.

Since this was an actual issue with the font and the client requested that we not change the font due to Chinese standards I came up with the following solution.

SimSun supports the ‘Masculine Ordinal’ symbol. This symbol does not have the trailing white space that the ‘degree’ symbol does. So on the fly when it comes time for iText to generate a Chinese Datasheet I replace all Degree symbols with Masculine Ordinals. A little hacky but def the best solution available at the time.

myString.Replace('\u00B0', '\u00BA'); //replace degree with masculine ordinal

In previous posts, I described our use of Selenium for functional and regression testing – and I included some tips on how to use it effectively.   We used the Firefox plugin Selenium IDE to run our tests.

Since that time, we’ve moved on to use Selenium 2 (now, Selenium Server) – which uses a completely different architecture built on top of a merged project called WebDriver.  Now, instead of the custom ‘selenese’ scripts – our tests are driven with C#.  This allows for much more effective branching, looping, etc. scenarios that are sometimes necessary for robust testing.

Some of the tips for selenese tests still apply, but in addition here are some specific Selenium 2 Server pointers:

1. If you are having trouble getting an element clicked, sometimes it helps to have the test explicitly move the mouse to the element beforehand. Use:

   new OpenQA.Selenium.Interaction.Actions(thewebdriver).MoveToElement(theelement).Perform();

2. Selenium 2 will not interact with elements that are hidden or off screen.  Because of this – each click/etc action implicitly performs a scroll-to-element action.  Usually, this makes things easier, but occasionally it breaks.  If you have a scrollable element with tight spaces, it might scroll it just out of range before the click, and it will silently fail.  There currently isn’t a great way around this in the test – you can attempt to change your site to deal with it instead (by giving more room, or locking scrolling, etc).
3. It can be a hassle to deal with nested frames.
   1. Things will fail if you don’t keep the Selenium context updated.  XPath selections in FireFox will throw exceptions, and events won’t fire.  Make sure you do the following to always use the proper context:

      Driver.SwitchTo().DefaultContent().SwitchTo().Frame("fremename");

4. When things are failing differently on different machines, there are a couple of things to try:
   1. Set a consistent resolution at the beginning of the test:

      ((IJavaScriptExecutor)Driver).ExecuteScript("window.moveTo(0, 1); window.resizeTo(1200,1000);");

   2. Retry clicks until success.  This tends to be necessary right after a iframe context change.  Hacky, but this psuedocode handles some performance/timing issues:

      do{click;sleep;}while(testforchange){}

In Foliotek, there are a lot of instances where we have a table with a check box column that allows users to select rows and do some sort of action to them. This leads to a lot of redundancy in our markup and code. For example, here is what our datagrids looked like when we placed a checkbox in them (we’re using a custom server control for our datagrid):

Datagrid with a checkbox

<Components:ExtendedDataGrid runat="server" id="dgItems">
	<Columns>
		<asp:TemplateColumn>
			<HeaderTemplate>
				<input class="check" onclick="FLTK.checkbox.selectAll(this.checked, 'chkSelect');" type="checkbox" />
			</HeaderTemplate>
			<ItemTemplate>
				<input class="check" type="checkbox" id="chkSelect" runat="server" />
			</ItemTemplate>
		</asp:TemplateColumn>
		<asp:TemplateColumn>
			<ItemTemplate>
				<%# Eval("ItemName") %>
			</ItemTemplate>
		</asp:TemplateColumn>
	</Columns>
</Components:ExtendedDataGrid>

Check All JS

Here’s our javascript code that controls the select/deselect all functionality. See this blog post in reference to the :asp() selector, and this post for more information on the select all functionality.

FLTK.checkbox = {
    selectAll: function (checked, endingwith) {
        var $checkboxes = $(":asp(" + endingwith + ")");
		// we don't want to check a box that is hidden on the page
        if (checked) {
            $checkboxes = $checkboxes.filter(":visible").not(":disabled");
        }
        $checkboxes.attr("checked", checked);
    }
}

Get selected rows (C#)

Then, to get the selected items in the code behind and perform some action on them, we’d have to do the following:

foreach(DataGridItem item in dgItems.Items)
{
	if(((HtmlInputCheckBox)item.FindControl("chkSelect")).Checked)
	{
		// perform action
	}
}

// which can also be written as...

foreach(DataGridItem in dgItems.Items.Cast<DataGridItem>().Where(i => ((HtmlInputCheckBox)i.FindControl("chkSelect")).Checked))
{
	// perform action
}

Creating the Custom Datagrid Checkbox Column

Since the above code and markup is used so frequently in Foliotek, we wanted to abstract this logic into something that was easier to use. As I stated above, we use a custom server control that extends the datagrid control, but if you don’t have a custom server control it shouldn’t be hard to create one.

I would suggest reading up on custom server controls, if you’re interested in how the following code works.

Below is our server controls for the custom DataGrid, CheckBoxColumn, and CheckBoxTemplate. Keep in mind the Checked and VisibilityDataField on the CheckBoxColumn are optional.

namespace Components
{
	// here is our custom datagrid control
    public class ExtendedDataGrid : DataGrid
    {
		public IEnumerable<DataGridItem> GetSelectedItems()
        {
            if (!this.Columns.Cast<DataGridColumn>().Any(c => c is CheckBoxColumn))
            {
                throw new Exception("ExtendedDataGrid must have a 'CheckBoxColumn' in order to use GetSelectedItems");
            }

            return this.Items.Cast<DataGridItem>().Where(i => ((CheckBox)i.FindControl("cb_" + this.ID)).Checked);
        }
    }

	// Usage: <Components:CheckBoxColumn Checked="false" VisibilityDataField="Show"></Components:CheckBoxColumn>
	// Alternative Usage: VisiblityDataField="!Hide"
	public class CheckBoxColumn : TemplateColumn
    {
        public string VisibilityDataField { get; set; }
        public bool Checked { get; set; }

        public CheckBoxColumn() : base()
        {
        }

        public override void InitializeCell(TableCell cell, int columnIndex, ListItemType itemType)
        {
            if (this.Owner != null)
            {
                this.HeaderTemplate = new CheckBoxTemplate(this.Owner.ID, VisibilityDataField, true, Checked);
                this.ItemTemplate = new CheckBoxTemplate(this.Owner.ID, VisibilityDataField, false, Checked);
            }
            base.InitializeCell(cell, columnIndex, itemType);
        }
    }

    public class CheckBoxTemplate : ITemplate
    {
        private string _tableID;
        private bool _isHeader;
        private string _visibilityDataField;
        private bool _isChecked;

        public CheckBoxTemplate(string tableID, string visibilityDataField, bool isHeader, bool isChecked)
        {
            _tableID = tableID;
            _isHeader = isHeader;
            _visibilityDataField = visibilityDataField;
            _isChecked = isChecked;
        }

        public void InstantiateIn(Control c)
        {
            if (_isHeader)
            {
				// If the template container is the header, then we need to add the check all functionality to the checkbox
                HtmlInputCheckBox input = new HtmlInputCheckBox();
                input.Attributes["onclick"] = "FLTK.checkbox.selectAll(this.checked, 'cb_" + this._tableID + "');"; // This id is determined below.
                input.Checked = _isChecked; // Set the checked status by default
                c.Controls.Add(input);
            }
            else
            {
                CheckBox cb = new CheckBox();
                cb.ID = "cb_" + this._tableID;
                if (!String.IsNullOrEmpty(_visibilityDataField))
                {
                    cb.DataBinding += new EventHandler(cb_DataBinding); // doing this in the databind event allows us to access properties in the dataitem.
                }
                cb.Checked = _isChecked;
                c.Controls.Add(cb);
            }
        }

        void cb_DataBinding(object sender, EventArgs e)
        {
            var cb = (CheckBox)sender;
            var dataitem = ((DataGridItem)cb.NamingContainer).DataItem;

            bool show = true;
            if (!String.IsNullOrEmpty(this._visibilityDataField))
            {
                bool not = this._visibilityDataField.StartsWith("!");
                show = (bool)DataBinder.Eval(dataitem, (not ? this._visibilityDataField.Substring(1) : this._visibilityDataField));

                show = not ? !show : show;
            }

            cb.Visible = show;
        }
    }
}

New markup and code behind

Here’s our new markup. Notice how we only have one line now for the checkbox column, compared to 8 lines before.

<Components:ExtendedDataGrid runat="server" id="dgItems">
	<Columns>
		<Components:CheckBoxColumn></Components:CheckBoxColumn>
		<asp:TemplateColumn>
			<ItemTemplate>
				<%# Eval("ItemName") %>
			</ItemTemplate>
		</asp:TemplateColum>
	</Columns>
</Components:ExtendedDataGrid>

… and our code behind, which definitely simplifies the prior statement…

foreach(DataGridItem item in dgItems.GetSelectedItems())
{
	//perform action
}