April 2009 Archives

I've been averaging 70-90 hours of work per week for a few weeks now, and a lot of work on Rosetta Code has had to be put off.  So here's a TODO list of accumulated things that need to done on Rosetta Code, and have either been in the works for a long time or have been planned. (And by "planned", I mean that some of these things have been ideas that just won't go away.)At the top of the list; These things are either urgent or are already in the "pipeline":

• Mod Alias needs to be set up on RC, as mod_rewrite's '+' handling became broken in the switch form mod_php to fcgi, and we've back in the bad old days of C++ pointing to C.  At least there's something we can do about it now...
• ImplSearchBot needs to be fixed.  It's editing almost 400 pages every four hours, when it only needs to be editing between one and ten, on average.
• ImplSearchBot's Subversion repository (where it keeps the JSON caches of category contents) needs to be opened up for general consumption.
• ImplSearchBot's Subversion repository needs to be abused to generate RSS feeds containing interesting events per language.
• There are some bugs in the way Rosetta Code's syntax highlighting deals with leading whitespace.  Details are in the relevant Village Pump page.  There also appears to be a bug breaking Unicode support with at least some languages when dealing with the string "møøse".  Not sure why this would be.
• Need to finish RC promo video.  Looking for suitable audio to sync.
• Find out what causes Recent Changes RSS feed to spit out batches of duplicate items a couple times a week.
• Rewrite the Rosetta theme from scratch.

Things that I want to start on:

• Frame generator task, sourcing from our resident graphics library tasks and sinking to mencoder/FFmpeg/DirectShow/VirtualDub/AviSynth.
• Get Rosetta Code content published via POD services to help pay for services (and thus potentially increasing capacity).
• See if it would be possible to set up an XMPP server with "chat rooms" for every Rostta Code category, and a resident bot that mentions updates to pages in each category.
• Investigate OpenID auth.  Not sure whether or not it's worthwhile; Serving as an auth source means crypto and greater attention to security. (I know it's not *required*, but I have my standards as a webmaster...)
• Get the Village Pump a bot to rotate out old, stale pump pages and float active ones to the top of the list. (reddit/digg style, I suppose, but without up/downvoting.)
• Get thumbnails working on the site again via background process.
• Word square generator task
• More fractal tasks
• Markov chain tasks
• More concurrency tasks
• More automata tasks. (Conway's Game of Life is an example of these.)
• Tasks involving signal processing. (High-pass, low-pass, band-pass,  histograms, Fast Fourier Transform, etc.)
• More graphics and/or UI tasks (2D and 3D), using cross-platform APIs like SDL and OpenGL.  Of course,  I wouldn't reject DirectDraw, Direct3D, GDI, WPF, Direct2D, Gtk, Qt, Tk, wxWidgets, or even straight Xlib, if people will provide code.

I'd like to see a bit of a shift away from theoretical tasks to practical tasks, and move from explicitly contrasting languages to identifying where a language's abilities can be taken advantage of for things that programmers often need to do.If anyone wants to give these a try (especially creating more tasks, creating RC promo material, or anything that requires a bot), go ahead, give it a shot!  I haven't had a whole lot of time of late.(Yes, I know there are a lot of links; It comes from having a lot of proper nouns and other interesting concepts...)

I figured out what killed the server Saturday--It was ImageMagick.  The ALGOL 68 Dragon Curve animated GIF is fairly large.  Someone went to visit the GIF's page on RC, and MediaWiki ran 'convert' to create a thumbnail.MediaWiki ran (approximately; the paths and filenames have been changed to protect the server, and because I tested it locally) ran was:

convert -background white -size 781 ALGOL_68_Dragon_curve_animated.gif -coalesce -thumbnail '781x599!' -depth 8 out.gif

That command, with that data, takes several seconds to run on my Phenom 9650 desktop at home, and my machine has a darn sight more CPU available to programs than anything running within RC's VPS slice.  As a result, when MediaWiki ran convert, it, via mod_php, via Apache spent several seconds trying to generate a thumbnail.  It would have eventually finished, except that whoever was looking at the page got bored, and refreshed.  Several times.  When I discovered that the server was having issues, the server load average was up around 16.  RC's slice usually hovers between 0.00 and 0.15.Three things have been implemented to fix this problem.  First, I've switched from mod_php to FastCGI, with the assistance of some of the folks in #mediawiki on FreeNode.  As a result, we get back an HTTP 500 ISE when commands run by the server take longer than expected to complete. (For whatever reason, mod_php was simply hanging.)  Second, I've turned off thumbnails.  None of the images on the site are large enough to make them worthwhile, for the time being.  I'll look into backgrounding convert processes in a way that doesn't take down the site, but that's going to be low-priority for now. The third bit came as part of my debugging.  All external commands run by MediaWiki will now be logged, to leave some sort of trace for when this kind of problem happens again.

Sorry about the downtime. I'm not entirely certain what the cause was, but the fix has been to switch from mod_php to fcgi, and correct a few caching settings in MediaWiki. My best guess is that a high-traffic site linked to (or embedded) the Algol 68 Dragon Curve animated GIF thumbnail, which was apparently causing several hung instances of ImageMagick's convert tool. I'll know more when I have time to look at the logs and analytics data tonight.

Hello, it's me, Mike Mol.  I'm writing here today because I'd like to do something, and I don't know how to do it.  While that's generally the case for the folks who visit Rosetta Code, this particular question can't be solved by comparing two or more programming languages, or by putting up a Task and seeing how other people do it.I would like to extend Rosetta Code to print.  As in bound hard-copy dead trees.  I'd like for Rosetta Code to sell one or more books that take a few languages, show those languages side-by-side for various tasks, and, of course, list all the contributors to those code samples, and include a URL where the print-ready PDF is available. (It is GFDL content, after all.)To go from print-ready PDF to an actual hardcopy book, I need a printer.  To avoid dealing with sales, I need a publisher. (I have absolutely no interest in dealing with the headaches revolving around processing payments from PayPal, checks or any other payment vendor, canceled checks, refunds, returns, you name it.)  So far, POD services like LuLu would seem to be the best option.Of course, I'd like to avoid as much editorial and layout work as possible, so I'll most likely automate the entire process--and therein lies the problem; Any time a new book is ready to go out, I'll need to upload it and set everything up.  I would much, much, much prefer to be able to use a service where an API allows me to programmatically do all the work I would otherwise have to do by hand.  I have no complaints about writing the code on my end to interact with such a thing.If I can get that far, then the skies open up with a realm of possibilities. I could provide a page where anyone could request that a book be published compares the languages they'd like to see compared, or which includes all of the tasks that they're interested in.  They click Submit, the script spends a day or two preparing the book, and then the PDF and book get published simultaneously, and they can have their copy (or copies) within a week or two.There are a number of cases where somebody might want one (or more) hardcopies This could be particularly valuable to teachers who want to contrast a set of languages, showcase a particular language, provide example code for a number of algorithms or other problems, etc.Yes, there's obviously concern about ill-timed vandalism.  Each PDF would probably be held for a little while before being sent on to the publisher.  That would mostly be a case of watching the pages that were included for signs that they'd been vandalized. (Rosetta Code has an awesome community for watching for vandalism.)  I suppose somebody could get themselves listed as a contributer under an obscene name, but I can work around that, too.