Showing posts with label Grendel. Show all posts

JavaScript and Dates #

Heh, looks like I'm spending more time on the project than any of my other ones, and technically it's not even a programming class one (it's for InfoTech). But I guess this is mainly because it's something I use myself (the whole checking mail through a proxy/at school thing), so if it's an unstable/unusable state I feel it immediately. I've also noticed that the closer I get to making Grendel similar to the Netscape mail client, the less reluctant I get to download the mail locally and read it there. The current situation is that I read about half of my mail with Grendel, and then when I download I quickly page through it since I've seen it before. When I add the MIME parsing (for things like HTML mail and attachments) it'll prolly be even more tempting. However, even if I were to add things like local message storage (as in, getting the mail from the POP server and storing it somewhere in the cgi-bin directory) with message filtering, etc. I'm not so sure I want to use it as my primary email client. The perfect situation would be if the Netscape mail reader were to observe that certain messages were already marked as read on the server, so it shouldn't flag them as being new when downloading them.

The main thing that I've done with Grendel was do implement the hybrid JavaScript/Perl system. When getting the messages from the server, the script generates a JavaScript which creates new JavaScript objects like this:


print <<EOF
messages[$messageNo] = new Message($messageNo,
                                   "$headers{'Status'}",
                                   "$headers{'Subject'}",
                                   "$address",
                                   "$name",
                                   "$date");
EOF

which results in JavaScript code like this:

messages[2] = new Message(2,
                          "U",
                          "Re: 10,000 Triangles, 18 msec",
                          "hberriot@club-internet.fr", "Herv� BERRIOT",
                          "Sun, 12 Dec 1999 04:56:36 GMT-0800");

Then I have a local (on the user's side) array of JavaScript objects (the Message object is defined by me, JavaScript is object oriented, and close enough to C++ that learning it isn't very hard) which store all the messages (well, not quite, I don't have the message body too, that's only requested when the user asks for a message to read). To display them, I simply loop over them and call their Display method. The idea here is that if I want to change the sorting method (not implemented yet, but soon will be, after I learn how QuickSort works), I can do all of it locally, instead of having to go through the server. Things like deleting messages are faster too. Instead of having to get the new message list from the server, I can delete the specified array memeber locally, and then tell the server which messages has been deleted, and update the message view page with the next one.

I wasn't quite sure where to store all these JavaScript objects. I couldn't put them in the list frame, since that gets cleared when refreshing the message list. At first I thought I should put them in an invisible frame, and the things like rechecking the mail would simply reload that list. The main problem was that the invisible frame wasn't quite so invisible, since the border would still show. Then, when I was making a simple change and saving it to the server, the FTP process screwed up, and the local app crashed (this was on a PC). The end result was that the copy on the server was completely empty, and I had no loal copy (the text editor I was using supported direct save to FTP). The good thing was that I had saved a backup copy before doing the hybrid system, so I got to start again. This time I decided that I should store the objects in the buttons frame, since it didn't get reloaded or cleared.

The other big thing I did this week was to add parsing of the message timestamps. Now they are converted to the local user's time, whatever it might be. This is done with JavaScript, since it has a built-in date string parsing function (whereas Perl requires a third-party module, and I want Grendel to be as self-contained as possible), and Perl wouldn't have been able to do the automatic local time conversion anyway, since it's executed on the server. In the end, things worked pretty well, I added special cases for dates that were on the same day as today, the day before, and within the past week. My main problem now is that Netscape and Internet Explorer seem to behave differently when handling dates, that is Netscape seems to shift everything back by one (since I first implemented this at school, where I use IE, I'm assuming that Netscape is wrong). I'm not quite sure, the simple fix would be of course to add a an if statement to increase the date by one if netscape is detected, but I was hoping for a universal script, especially since JavaScript is supposed to be standardized.

I have a few of simpler things planned for Grendel, before I decide if I want to implement the local message storage or move on to something else. First of all, I want to add cookie support, so that it can remember the login/password combo for me at home. To edit that setting, I'll have to add a user settings page, where users can also change their POP server, password, etc. On the topic of passwords, Grendel as a whole isn't very secure yet. If someone were to go through the history at school and get the address for my message list, they could access it (my username and password (in a mangled form so that it's not immediately readable) are in the address string). Right now I'm thinking of using temporary cookies (which expire when the user quits the browser) to set when the user logs in, and the username/password validation would also check for the presence of those cookies. Another thing which I want to add is automatic mail checking. I can see how I could use a small toolbar/scrollbar/statusbar-less window which refreshes itself automatically every minute (or whatever the user prefers) and checks to see if there's new mail. If there is, it can play a sound, bring up an alert, or just start flashing, and when the user clicks on it grendel is brought to the foreground or started up if not alreay there. There's also the MIME parsing that I mentioned earlier, ideally with in-line viewing for images or things which the browser has plug-ins for. Finally, I want to add another column to the message view listing the message size, but so far that doesn't seem to be in the headers, and I don't want to download a whole message just to get its size, so I'll have to see about that.

Who needs a database? #

I've started to work on the new user architecture. Now the users file (grendel.info) looks something like this:

user@example.com
server=mail.example.com
port=110
username=user
encryptedPassword=$1$Aa$N.1.KcWzIorULwFN/EgMA/
externalLinks=_top

This means that I'm able to check both my mscape.com and my AT&T WorldNet accounts, by simply signing in with the appropriate email address and password (I'm using the email address since it's the only thing which I could depend on to be unique and be easily remember, since a POP user name@POP Server scheme would have been harder to remember if, like in the case of my mscape.com account, the POP server is different from the server part of the address. You can also see that my two password are the same, since they yield the same encrypted string, but it still doesn't help you in figuring out what they are (this is what I was talking about before, when I was saying that I'm not actually storing the password on the server). What remains to be done is to implement a way for new users to sig-in, and then grendel is actually usable to people besides me.

I've also gotten the message deletion to work. In the end it turned out to be a very stupid thing. Since Perl is very flexible, it doesn't balk when you try to do things which would never get past the compilers of other languages. In my case, I had a function called POPLogIn, but when I called it, I spelled it as PopLogIn (Perl being case sensitive). Fixing it was obviously trivial. Now I can finally pre-emptively delete spam so that it never reaches my Netscape Messenger Inbox (perhaps in the future I'll add message filtering too, since Perl supports run-time creation and evaluation of regular expressions (pattern matching tools, like the thing I showed before which extracted the email address and name out of the From: field), but that's some ways out).

Breaking Lines #

Something came up which prevented me from moving on to the new user log-in thing that I had planned before. So far, I've been using <PRE> </PRE> tags to delimit the message body, so that the spacing within it would be retained (HTML ignores more than two consecutive spaces, line breaks, tabs, etc.). However, I realized that this screwed up line wrapping (as in, if a line was longer than the width of the window, neither Netscape nor Internet Explorer would wrap it). In the end I decided I had to convert the message body to HTML too, since only then would the wrapping work. Converting line breaks and tabs was easy, it involved a simple substitution. However, I had problems with whitespace made out of the ' ' character. The closest HTML entity to it which isn't ignored when there's more tha none is &nbsp;, but that's a non-breakable space so I couldn't simply globally substitute &nbsp; for ' ', instead I would have to find places where there were two or more and then substitute for it. In the end, this yielded the following code:

while ($line =~ /( {2,})/)
{
	local($spaces) = '&nbsp;' x length($1);
	$line =~ s/$1/$spaces/; 
}

This can be interpreted as "while there are spaces surrounded by non-space characters and the length of the spaces is 2 or more, substitute the spaces by the string '&nbsp;' repeated as many times as there are spaces"

Another thing which I did that wasn't in the schedule was to deal with multiple recipients. The To: field can have multiple addresses in it, separated by commas and or line breaks. The realization that headers can be split across multiple lines caused me some problems, because I couldn't simply use the split to divide up each line into the key (the field name, e.g. From) and the contents (e.g. "User Foo <user@example.com>") by using ":" as the delimitor. Now I loop through each of the line, look for a <string:>:<string> pattern, if I find it I insert it into the headers associative array (basically, a hash table) and if I don't it append it to the entry of the previous field that I found. As for the multiple recipients themselves, I divide up the string by using the comma as the delimiter, and then use the name/address splitter function that I did earlier on each address.

I then wanted to move on to the delete function, and I got it to the point where it does actually delete messages (good for pre-emptive spam removal, that way I don't have to download them in my Inbox), but I'm still not getting it to refresh the message list or the message that's being displayed in the message view frame.

Best of Both Worlds #

Nothing that special to say, except that I've started on the hybrid JavaScript/Perl solution. For example, I've added Next and Previous buttons. Since I use frames, I need a way for the buttons frame to determine which message the message view frame is showing and then move on the next one while also marking it as read in the message list. JavaScript makes this a lot easier, since a frame can access another frame's variables (since in this case they're siblings). So I simply store the ID of the current message as a variable in the message frame, and then get it in the buttons frame, increment/decrement it, and then call the Read function of the list frame with the new ID. This function (which is also used when clicking on a message in the list) sets the contents of the message view frame as appropriate and changes the image next to the message subject to reflect that it's read. I'm planning to implement a delete button in a similar way (except that the list will have to be completely reloaded for now, since I don't generate it from within JavaScript yet).

After the delete button is in, I want to add a new way to login and cookies. Although the current user info only allows me in (the user data isn't hard-coded, it's loaded from a file, but there's no way to edit it), ideally I wish for people to be able to set up an account, where they specify their email address, pop server, user name and password, and then they can log in with the email address (not the username, since it's not unique, e.g. 'mihai' at the pop3 server 'mail71.pair.com' is different from 'mihai' at the server 'mail.att.ne.jp') and password and have access to their email. Also on the same topic, I want the option of cookies to that people don't have to type in the email address/password all the time. This might not make sense when grendel is supposed to be an 'email-anywhere' web-application, but it's useful for those like me who also want to access their mail from behind a proxy and thus use their own computer (saving the username/password on a public computer isn't too good security-wise) but still need to use Grendel.

The full JavaScript-based threaded/sorted view of the message list will come later, since that's not a big a requirement as this (at least from my point of view, I'm getting tired of having to type in my name and password when checking my mail). Plus the new system will allow others to see on their what I'm up to (there ought to be no security worries, I store the password on the server in an encrypted form, and all I do when the user enters his password is compare the encrypted version of his with the encrypted version on the server, and if they're the same I allow the user in, this is the same system used by UNIX, and it makes sense because the administrator (or anyone else who has access to the password file) can't get all the password used in a system by simply looking at the password file).

Perl: The Write-Only Language #

I've added extraction of email addresses (so that "User Foo <user@example.com>" is split up, and also variants of the form "user@example.com (User Foo)"). The code for this looks something like this:

if ($header =~ /^\s*([^<]*)<([^>]*)>\s*$/)
{
	$name = $1;
	$address = $2;
}
elsif ($header =~ /^\s*([^(]*)\(([^)]*)\)\s*$/)
{
	$address = $1;
	$name = $2;
}

Before I started to learn Perl, I said (to my dad, who uses (and likes it) a lot) that most Perl code it looks like someone held shift down and ran his fingers across the top and right side of the keyboard, and it looks like I wasn't far from the truth. But regular expressions (which are the statements to the right of the =~) are really cool, they let you do things which would take ages in C/C++ (actually the perl regexp engine is available as a C++ class too, but from what I heard the interface isn't that nice).

Other than that, I added the option to make the quoted part of a message appear in a different style (in my case, italics, and in a shade of gray). For the rest of the things that I have planned (threading, and sorting by the various columns) I have an idea of how I'm going to do it. Since they (especially sorting) simply require for the data to be arranged in a different way, it makes no sense to go back through the server to do this, instead, I can use JavaScript. So the plan is to have the Perl-based CGI backend to go generate a JavaScript script (which will mostly mean placing the header data within place-holders in the script) and then having the browser execute that script. When the user chooses a different sorting order, the JavaScript is executed again, with different parameters of course, but the server isn't involved at all. I'm also planning to use JavaScript for some of the other features, like having an automatic mail check option and for advancing through the messages (in the form of Next and Previous buttons in the toolbar at the top).

It's Alive! #

This is my new CGI project. I've decided that Sonar is functionally complete, and adding features on wouldn't have been quite as fun and instructive as working on something else.

Grendel is a web-based POP3 mail client. Although it might look a lot like Hotmail, you still need a separate account somewhere else to use it, it does no hosting of its own. The idea is that you are able to check your mail from anywhere you have a web browser, whereas before you would need to telnet in (if your ISP allows you to do so), or set up a mail client, but then you'd be leaving your settings and downloading your messages on a computer which wasn't yours. Additionally (and even more importantly to me), it allows you to get to your mail when access to the Internet is otherwise limited because of a proxy server.

Obviously this idea isn't new, Hotmail allows you to check an external POP3 account, and there are many free and commercial CGI scripts which are specifically designed to do this. But doing my own thing would allow me 1) to learn more about networking (which I've been meaning to do) and 2) customize it exactly to my needs (that is, make it look a lot like the Netscape mail reader).

I've found (a while ago actually) a simple script which displays a list of messages which you check off, and then they are all displayed for you to read. I've been using this script until now to read my mail when I have to go through a proxy (I switch to my ISP account when I want to download it locally), but it doesn't do all that I need it to do. I want a frames-based interface so that I can go between the messages easily, and I want the messages cleaned up, which means 1) not showing all the headers 2) making links so that they are clickable 3) not stripping out anything that begins with a < (which Netscape treats as the beginning of an HTML tag).

So far I have a working model of what I'm planning to do. I've implemented the frames based interface, as well as the basic header and link filtering within the body of the message. Parsing the headers was something that was very easy to do with Perl. The format of the headers is "<Header name>: <Header contents>". Perl has a split command which allows you to divide up a string based on a certain symbol (in this case I used ": "). Then I stored the split results in an associative array (which is somewhat like an array except the indices can be strings), so now I can access the various headers by using something like this: $headers{'From'}.

Since at the moment Grendel can only check my mail (the user info is stored in a file, but it's not modifiable at the moment), I can't just link to it to show how it works, so here is a screenshot of what it looks like. On the right there's the old mail interface I used:

Grendel Mail

The big things that are left now are handling of the author's email address (that is, separating the name from the email address, and turning it into a hyperlink), handling the date (I want to convert them all to JST, or whatever is set in the user settings file), displaying threads graphically and sorting of the inbox based on the various fields (these last two will require a reworking of the way in which I store the messages, right now they're just an array of strings).

And in case there is confusion about the name, there's a very simple reason behind it. A while ago (actually, there's some signs of revival with the Mozilla project) Netscape had a project to reimplement Navigator in Java. Although it never really got off the ground, the mail client was mostly complete. The codename of the mail client part was Grendel, and since I'm mimicking the Netscape mail client too, then I thought the name would be appropriate.