Let’s see… where to begin? Well, I guess I can list things in no particular order, and let it go from there.

1. The “biggie” is that there’s now an official name for the project. I thought that I’d be clever, and had been using SpeedReader internally, but then I decided to search for the name, and it turns out that not only is there already a SpeedReader, but it’s also an aggregator. Damn. So, I thought long and hard about it (a few minutes), and decided that, based on my initials and a dearth of hits from Google, that the name maggregate would be good. Well, if not good, at least unique¹. So, that’s what I’m calling it from now on. Maggregate.
2. A lot of extraneous crashes have been dealt with. I had been adding exception handling in the few places where it’s most likely to occur (network and parsing) on a case-by-case basis, but I kept coming up with new ones. When I took the wireless interface down on the laptop I use to develop code on, and got yet another unhandled exception, I finally decided that adding a “catchall” rescue block to each place that needed it, as those errors were all technically recoverable. Since then, no crashes. Yay. And logging differentiates enough (error for catchall vs warning for explicitly handled) that I can quickly see what I’ve been too lazy to explicitly deal with.
3. Maggregate now accepts gzip-encoded content! I knew I always wanted to do this, but my prior experiences with trying to deal with raw Zlib (in C) kept me from relishing having to do it. Lucky for me, the ruby library is nice enough that it was almost trivial. In fact, the hardest part was realizing that I needed to wrap the body from the HTTP request in a StringIO object instead of leaving it as a string. Once that was done, it was dead simple.
4. Partial cookie support is in place. Well, that should really be “preliminary”, as in there’s just a bit of code to recognize cookies, and there’s now a table in the maggregate database to eventually store the cookies. So, the logs let you know that you were sent a cookie (or several), but that’s currently it.

   As an aside, having looked at RFC 2109 and RFC 2965, it appears no one is using them… Instead, it appears every single server out there sending cookies that I’ve come across just uses the Netscape cookie specification. Keep in mind, RFC 2109 dates from February 1997, and RFC 2965 dates from October 2000. Not much of an issues, but I think my jaw will drop the first time I spy a max-age insead of an expires in a cookie header.

5. Potential support for RFC 3229 with “feed”. This was more of a “I wonder how easy it would be…” deal than anything else. I was reading something, followed a URL in a comment, and one or two clicks later, ended up at that page. For those folks who haven’t memorized the RFC index, RFC 3229 is “Delta encoding in HTTP”, which is basically a way to request a resource over HTTP and get just what has changed on the page instead of having to transfer the entire thing again. The “with feed” is an extension that purports to handle feed deltas in a manner that doesn’t break encapsulation, it just gives you a “feed” with only new/changed articles in it.

   My biggest concern was having to support a new HTTP status code, 226 IM Used, as the Ruby Net::HTTP library doesn’t have support for it. I should have known that it wouldn’t be an issue, thanks to Ruby’s allowing one to extend existing classes fairly easily. How easily? This easily:

   require ‘net/http’

   module Net
     class HTTPIMUsed < HTTPSuccess        # 226
       HAS_BODY = true
     end

     class HTTPResponse
       CODE_TO_OBJ[226] = HTTPIMUsed
     end
   end

   That’s in the maggregate file that does all the network and HTTP connections. That’s it. Now, in maggregate at least, Net::HTTP understands the proper status code for a successful response for RFC 3229. Nice, no?

   Now, I say it’s potential support, because so far not one single feed in over 300 has ever responded with either a 226 response OR the associated additional HTTP headers. But due to the way that I handle adding entries now, I don’t expect any issues, if and when a server decides to respond correctly.

The UI is still completely unusable, as I’ve been verifying things are working through both logging and using the sqlite3 exectuable to examine the database by hand. Despite the number of crashes, I’ve yet to corrupt any data, either. I suppose that the UI is the next real step, so I should finally learn more about ncurses and how to interact with it.

1. I had jokingly originally referred to the project as maggle, a take on raggle, but it seemed a bit too derivative of a name. Sorry Tim, I know you liked this one

Some of the fun fun things I’ve encountered thus far are:

• The confused server that seems to think that, instead of giving a proper 304 Not Modified response (with no body) when the content I’m requesting hasn’t changed, it should instead give a 200 OK response, with no body. This kept throwing me as I was trying to debug, as I kept believing my logs that there should be new content there…
• The confused server that gives a 406 Not Acceptable response to requests that don’t supply a User-Agent: header with the request. Admittedly, were I less lazy, I’d be specifying one anyway, just to identify the new kid on the block. But as I haven’t even officially settled on a name yet, I was leaving things at their default (and made the assumption that Ruby’s Net::HTTP library had some sort of default value). What really threw me was that wget would work, but telnet wouldn’t, even when I was listing a huge laundry list of formats I was interested in in the Accept: header (including the type that wget said was coming back). After a little while, I decided to just pass a User-Agent: moo, and lo and behold, content!
• The really confused server that would reply to all requests with a Last-Modified: Thu, 01 Jan 1970 00:00:00 GMT, but would then completely bomb out with an application error if it got that value back in a If-Modified-Since: request header.
• The feed that seems to randomly return something that isn’t RSS or Atom at all, but from the exception the content generates, looks like it could be normal HTML. I say randomly, because after the last time it happened (and I finally got around to catching that exception to log it, deal with it, and carry on), it’s happened a grand total of once. And in that time, it’s checked the feed a few more times, though the content hasn’t updated (from before the time stored…), with no issues. I’m hoping something happens during the night, just so I know that I’m not the one going crazy.

Feeds in general have also been interesting. I’ve discovered that there’s such a huge variation in the quality of the actual data that I’ve had to special-case a few things, and delve into the XML (or XML-like) content itself in order to pull out data that I’d hoped would percolate into the normal accessor methods of the classes I’m using. It’s not a big deal, but it’s one of those things that becomes a minor annoyance after a while. Fortunately, of the information I care about, it’s only dates and links to individual entries (or in at least one case, the complete and total lack thereof), and I think I have that in the bag.

Speaking of dates, one thing I’ve been noticing over the months of adding various and sundry feeds to the readers I’ve used in the past is that there are some pretty horrid implementations of Atom out there. I’ve noticed several feeds that various apps/frameworks provide that seem to think that publication, update, etc dates should always reflect the current time when you make the request, and not the time that each item was actually published or underwent a major update. This makes trying to figure out anything temporally useful about feed items a non-starter. Such is life…

Now, lest folks think that all I’m doing is complaining, let me just say that I’m glad that I’ve been running into these problems. I never would have discovered some rather fragile assumptions that I made without real-world data. In order to do that, I whipped up a pretty brain-dead (but functional) OPML import feature, and an add-a-feed feature, so that I could import the OPML file that I’d managed to export from my other reader a short time ago. Importing that, and adding some feeds that I’ve been pasting into a file for a week or two, I’m able to see how everything fares trying to parse 294 highly varied feeds instead of a few minor test cases. So when I’m ready to release the first alpha version to the guinea pigs, it should at least do better than it might otherwise.

So, yeah, I’ve made some decent progress on a lot of the back-end code and infrastructure. There’s just enough of a UI in place so that I can at least verify that things are working. It’s butt-ugly, and almost completely useless, but meets my current needs. Eventually I’ll get around to making it functional for me, maybe even functional for you. There’s logging in place so that I can at least figure out where something goes wrong. Hell, it even saves data (without corruption despite multiple crashes). You just can’t, well, read or navigate or do anything else at the moment.

However, now I’m dealing with one of the readers I mentioned deciding to die partway through startup, and having marked all my feeds invalid or unread. I noticed it Saturday morning, when I went to check some news before heading out to look at houses, when the window that I keep the reader in was sitting at a prompt with an error on the screen. So I re-ran the command, and within a minute, it had crashed again. As near as I can guess, either some bad data came through on a feed, or its constantly-growing cache got corrupted. Either way, even with logging set to DEBUG and the “diagnostic” flag specified, there’s no indication at all what is causing the problem.

So, instead of fighting with it, I’m writing my own feed reader. I’ve dusted off the really simplistic “playing around” code, and have been working on turning it into a full-featured (or at least itch-scratching) RSS and ATOM feed reader. It’s nowhere near usable yet, but I’m currently able to pull down, parse, and save feeds, as well as check intelligently for updates (thanks, in part, to “Towards polite HTTP retrieval in Scala“, which gave me the idea to let the remote webservers tell me whether a feed’s changed, in order to reduce load on everyone).

Instead of raw on-disk storage, or fairly opaque language-specific storage formats, I’ve been playing with using SQLite for storage. It seems fairly lightweight (especially in comparison to MySQL or Postgres) while still providing a nice familiar SQL engine, gives me a decent command-line tool to inspect and modify data, is fairly portable, and I was able to get up and running with sqlite-ruby in practically no time at all.

I’m currently working on a functional (read: ugly, but it works for my purposes) interface in ncurses, threads to deal with fetching & updating in the background, category and configuration management, etc. As I work on it off-and-on as I grab a few spare minutes, I’m not sure on the ETA right now, but hopefully something that I’m willing to let others see will be available “soon”.