So, having complained about console feed readers, I’ve decided to put up or shut up. Or put my moneycode where my mouth is. A few months before I wrote the last article, I’d started playing a little bit with ruby-feedparser. Then I promptly forgot about it, as other more pressing matters came up.
However, now I’m dealing with one of the readers I mentioned deciding to die partway through startup, and having marked all my feeds invalid or unread. I noticed it Saturday morning, when I went to check some news before heading out to look at houses, when the window that I keep the reader in was sitting at a prompt with an error on the screen. So I re-ran the command, and within a minute, it had crashed again. As near as I can guess, either some bad data came through on a feed, or its constantly-growing cache got corrupted. Either way, even with logging set to DEBUG and the “diagnostic” flag specified, there’s no indication at all what is causing the problem.
So, instead of fighting with it, I’m writing my own feed reader. I’ve dusted off the really simplistic “playing around” code, and have been working on turning it into a full-featured (or at least itch-scratching) RSS and ATOM feed reader. It’s nowhere near usable yet, but I’m currently able to pull down, parse, and save feeds, as well as check intelligently for updates (thanks, in part, to “Towards polite HTTP retrieval in Scala“, which gave me the idea to let the remote webservers tell me whether a feed’s changed, in order to reduce load on everyone).
Instead of raw on-disk storage, or fairly opaque language-specific storage formats, I’ve been playing with using SQLite for storage. It seems fairly lightweight (especially in comparison to MySQL or Postgres) while still providing a nice familiar SQL engine, gives me a decent command-line tool to inspect and modify data, is fairly portable, and I was able to get up and running with sqlite-ruby in practically no time at all.
I’m currently working on a functional (read: ugly, but it works for my purposes) interface in ncurses, threads to deal with fetching & updating in the background, category and configuration management, etc. As I work on it off-and-on as I grab a few spare minutes, I’m not sure on the ETA right now, but hopefully something that I’m willing to let others see will be available “soon”.
Hi! I’m glad you found something I wrote to be informative/helpful.
SQLite is just darn handy, I tell you what. Unbeatable license, small footprint, straightforward to use, portable, fast, and has enough features to be perfect for embedded use.
I’m working on a feed aggregator designed to scale to large numbers of feeds. I was planning, for the sake of simplicity, of having it generate HTML (like rawdog) and then be command/file-based for configuration and administration. I’ve never programmed in curses before and I only use Windows at home for the time being… I’m surprised software implemented in Ruby would be so unstable as you claimed.
Please keep the world updated on this blog if you make any progress.
Frankly, it surprises me too. I suspect there’s some nasty interaction between the insistence on re-writing the feeds file every so often and re-reading it in another thread, for the easily reproducible crash. The one I’m currently seeing, I have no idea, but nuking the cache seems to “fix” it. That’s why I think it’s some corruption, but it’s hard to really tell, especially in a 36M file…
I definitely plan to keep working on this, as at the very least, if it dies, I’ll be able to personally yell at the party responsible for fixing it. 🙂