Village Pump:Home/Syntax Highlighting ( archived 2009-06-18 )

This page is to discuss changes to the syntax highlighting system.

Code tag change

I'm going to make some major changes to the Syntax Highlighting extension this weekend. Instead of denoting a block of C code as:

Code will be denoted as:

This will significantly clean up the Mediawiki extension namespace, and make formatting tricks with CSS easier. I'd rather create an attribute to <pre>, but that appears as though it could be more complicated. --Short Circuit 05:52, 2 July 2008 (UTC) (updated) --Short Circuit 21:59, 29 January 2009 (UTC)

Will we need to go and change all of the previous highlighting then? --Mwn3d 06:11, 2 July 2008 (UTC)

Yes. Both approaches will be supported for a little while, I expect, but the current system will definitely be phased out. --Short Circuit 23:42, 2 July 2008 (UTC)

Isn't it possible to access the raw files behind the Wiki, so one could run a simple replace operation on all of them? Also, is there any way to add syntax highlighting for languages for which it isn't supported yet? --Dirkt 11:07, 6 July 2008 (UTC)

I'm not sure about the raw files idea (it sounds easy enough), but the new languages idea gets a bit hairy. The latest GeSHi (v1.0.7.22) has support for 96 languages (counting things like "java" and "java5" as different languages). These languages don't necessarily overlap with the 103 here. Newer languages like Rhope and non-computer languages like TI-83 BASIC will probably never have highlighting. In order to add a new language, we would need to make a special PHP file for it with lists of the following:

All keywords that could be highlighted (in a jagged two dimensional array, separated into groups which get different kinds of highlighting)
Style codes for each group of keywords, comments, escape characters, brackets, strings, numbers, function names, symbols, scripts, and regex's
URLs for each group (if you want keywords to link anywhere like javadocs for Java)
Quote characters
Symbols from the language (besides math operators)
Comment characters

Also it would need a regular expression for comments. It would probably be best to talk to the people working on the GeSHi project about adding new languages.--Mwn3d 16:29, 7 July 2008 (UTC)

It's happened! I've replaced the GeSHiSyntaxHighlight Mediawiki extension with the GeSHiCodeTag extension. The old tag format works (for now), but it will be removed as soon as possible. All tags of <mylang>some_code<mylang> need to be changed to <lang mylang>some_code</lang>. So this block:

<lang cpp>#include <stdio> int main() {

    std::cout << "Hi!" << std::endl;
    return 0;

}</lang>

Should be rewrapped like this:

<lang cpp>#include <stdio> int main() {

    std::cout << "Hi!" << std::endl;
    return 0;

}</lang>

This is going to require a massive site-wide effort for all supported languages. See Help:GeSHi for details. As soon as we're confident most of the pages have been handled, I'll disable support for <lang> entirely. --Short Circuit 02:33, 23 January 2009 (UTC)

When the <lang> tags are removed, Special:Version will not any more list all the languages in the tags section, right? Maybe there should be a (perhaps auto-generated) page listing all supported languages (or maybe it's even possible to include them in Special:Version as separate section). Or maybe there's already such a list somewhere, which I simply didn't notice? --Ce 08:18, 23 January 2009 (UTC)

You mean this? --Mwn3d 13:27, 23 January 2009 (UTC)

Another problem I just noticed: The old meaning of <tt>text</tt> is now changed (it now creates a div block instead of inline text). Those tags might have been used already in this wiki. Maybe it would have been a better idea to use a tag which didn't yet have a meaning (say, <source>). I'm not sure whether it's a good idea to change the GeSHi tag now, or to find and change all previous usages of <code> instead. --Ce 08:50, 23 January 2009 (UTC)

I just noticed this too (when writing my task N distinct objects). Since &code> being inline is defined by the HTML specification, to be inline, we should not redefine it to be block. --Kevin Reid 00:48, 29 January 2009 (UTC)

You should use the <tt> tag instead. --Mwn3d 01:57, 29 January 2009 (UTC)

Sorta Fixed. We now use <lang mylang> instead of <code mylang>. <code> will remain broken until the existing pages are fixed. --Short Circuit 21:59, 29 January 2009 (UTC)

It seems color themes have changed for quite a number of languages...perhaps we should decide on a favorite theme, and then use that for every language? Would take quite a bit of work though... --Mbishop

Should be fixed. The upgrade in versions of GeSHi obliterated a configuration change I'd forgotten I made. I need to make a list of these things. As for a common color scheme, I'm all for it. GeSHi is now set to apply different CSS classes to the different code concepts it's familiar with. The current theme is set in Mediawiki:Common.css, and can be changed sitewide by modifying that. --Short Circuit 20:21, 24 January 2009 (UTC)

Old code used the right way

Now I am using the <lang LANGNAME> syntax for new examples, and when I've landed on the previous usage, <LANGNAME>, I've made here and there some fixes. But I've indeed fixed also the right html usage of the <code> code tag into text (changed to <tt>), since it raises a warning and ruins the output layout when used as normal html tag. Anyway now I stop fixing that, I suppose everything will go fine when the old code-style syntax-HL syntax will be abandoned completly. (Right?) --ShinTakezou 19:00, 1 February 2009 (UTC)

I know you already know this now, but I'm just mentioning this here because it's been talked about and resolved in a few places. <code> is now safe to use anywhere where it's preferable to use the HTML tag of the same name; The Syntax highlighting engine doesn't care about it one whit. --Short Circuit 19:17, 2 February 2009 (UTC)

Switch!! <code> is now <lang>

I realize we just switched, but it has been pointed out that code is already an HTML element, and appropriating that tag for wiki text will cause problems for folks who sprinkle HTML into their edits. (Shame on them. ;-) )

Anyhow, you can now use <lang c>/* some C code */<lang> the same way you would have used <code>. Like so:

<lang perl>print scalar @ARGV</lang>

Sorry for the inconvenience and the flip-flopping. Had to do this soon, before the entire site managed to switch. On the bright side, with the changes I made to GeSHiCodeTag, it will be very, very obvious which examples need to be switched. I also noticed that Mediawiki reinterprets the output of code tags, so I may be able to insert category references automagically. Need to look into this further... --Short Circuit 21:03, 29 January 2009 (UTC)

Booo! All that work I did for RCBF, RCSNUSP, and RCRPG is worthless!...No this is a good idea. I hadn't realized how many people used the code tag for inline text before this. i'll change the TODO list item and get cracking on it sometime. --Mwn3d 21:09, 29 January 2009 (UTC)

Adding new languages

Ok, I had a look at the source in the SVN. Doesn't seem too difficult to add new languages; basically all one has to do is to copy the php files for one of the existing languages and change it to suit the new one. I guess I could make for Haskell, say, in less than an hour. BTW, I cannot see 96 languages, the SVN just has C, Codeworker, C#, CSS, Delphi, Doxygen, Eiffel, HTML, Java, Javascript, PHP, QBasic, SQL, VHDL and Web3D. Unless I looked in the wrong place.

I'd like to propose the following:

Add a page about syntax highlighting, what languages are available, and what one must do to extend/change the existing highlighting.
Link that page from the homepage.
Make a copy of the php-files with the syntax highlighting code available via the Wiki. Then people can just grab the code that fits best, and turn it into code for a new language.
When they've done that, they should ask someone with admin rights to incorporate the changes. That shouldn't happen too frequently, so the workload for the admins should be tolerable. For security reasons, it's probably a bad idea to allow editing of "life" php code.
Make <code lang="xyz">...</code> act the same as <pre>...</pre>, if there's no syntax highlightling definition for xyz. This will allow to write syntax highlighting tags right now, instead of having to replace them all later when syntax highlighting for that languages becomes available
Once the php code has settled, one can submit it back upstream.

Rationale:

Rosetta is the ideal place to write and test syntax highlighting. There are already many code examples available, and it will be immediately useful. And it will also offer motivation to make syntax highlightling for more esoteric languages.
I'd like the "feedback loop" to be as short as possible, in true Wiki style. People contribute because they can immediately see the results. If one first has to contact the upstream developers, then wait until the next version of Geshi comes out, then wait until it's installed at Rosetta, etc., I guess the motivation to do something will be pretty low. It's already bad that one ask to one of the admins to "go live" with it, but I guess otherwise the security risk is just too great.
I offer to do all of the work outlined above myself as far as I am able to :-) So I need someone with admin rights to make a copy of the php scripts of the currently installed available through the Wiki, but I can write the other Wiki pages etc.

Comments? --Dirkt 09:29, 8 July 2008 (UTC)

You can see all of 96 languages here: http://geshi.svn.sourceforge.net/viewvc/geshi/tags/RELEASE_1_0_7_22/geshi-1.0.X/src/geshi/

I guess you could try to just morph existing languages into new ones...I wouldn't want to do it, but I can add them if you make them. Maybe we should talk about it in the IRC channel. --Mwn3d 20:58, 8 July 2008 (UTC)

Ok, that's better :-) I guess those languages should be enough for most cases. BTW, is the new code/lang-tag already active? I cannot get it to render. While we're at it, the math-tag is also broken.

I also created/changed a couple of pages to document the syntax highlighting stuff. Please correct/update as appropriate. --Dirkt 10:26, 9 July 2008 (UTC)

We had a page on GeSHi already. The math tag was never installed. When I made the Formatting page before I told Short Circuit about it, but it wasn't very important back then. We don't deal much with math symbols more complicated than <sup> and <sub> anyway. --Mwn3d 14:18, 9 July 2008 (UTC)

Well, that page isn't particularly easy to find: Special:Whatlinkshere/Help:GeSHi and the two links there list no pages linking to it except the above link you gave. And if you don't know that GeSHi is installed in the first place (and what it is), the name doesn't help, either. IMHO, it's really better to naming pages after their function. For the math tags, some people (not me) did propose problems that need a substantial amount of math, and we have several problems that at least come near a moderate amount of maths (like eigenvalues for matrices). So occasionally, the math tags would be useful. Actually, when involved with the first problem mentioned, I spend quite some time rewriting stuff into math tags, only to discover they don't work. I don't know how difficult it is to support them, but at least I would appreciate a clear decision, and easy to find information that tells you if they are supported, or not :-) --Dirkt 10:06, 10 July 2008 (UTC)

Problem with Lisp

There is something wrong with the highlighting of certain keywords in Lisp, for example: <lang lisp>defun</lang> <lang lisp>list</lang> <lang lisp>length</lang> --Spoon! 09:03, 3 November 2008 (UTC)

This still hasn't been fixed, as of December 31st, 2008

This still isn't really our problem to fix. Check the discussion above. The syntax highlighter we use is from the open source project GeSHi. If someone could help us find a suitable replacement, that'd be a big help. --Mwn3d 01:00, 1 January 2009 (UTC)

Looks like the GeSHi update fixed it. --Short Circuit 00:54, 23 January 2009 (UTC)

Missing Python keywords

At least two keywords aren't getting highlighted for python syntax, 'any' and 'all'. I suspect that 'with' is also missing.
--64.238.49.65 00:46, 14 November 2008 (UTC)

The latest GeSHi release has the new Python keywords, builtins, and types defined. This would be the GeSHI version released on 25 Dec 2008. --Rldrenth 15:02, 2 January 2009 (UTC)

The new version of GeSHi is installed. --Short Circuit 00:54, 23 January 2009 (UTC)

Enhancement for C syntax hl

While adding some example for C, I noticed the following oddities:

the parser is not case sensitive (C is!), since it hl-ed If as the keyword if--ShinTakezou 14:25, 17 December 2008 (UTC)

Fixed. I'll send the relevant changes upstream. --Short Circuit 00:53, 23 January 2009 (UTC)

the multiline preprocessor defines (using \ at the end of the line) are not handled

--ShinTakezou 14:25, 17 December 2008 (UTC)

It looks like GeSHi handles preprocessor directives by identifying the # as a single-line comment character. In order to use a single # for a multi-line comment, one would need to modify the custom regex field:

<lang php>'COMMENT_REGEXP' => array(1 => '/\/\/(?:\\\\\\\\|\\\\\\n|.)*$/m'),</lang>

Adding another regex to the array to handle multiline preprocessor directives should fix that. My regex is rusty, though. --Short Circuit 00:53, 23 January 2009 (UTC)

Update GeSHi

Can we get GeSHi updated? Also I created a Modula-3 language file for GeSHi and submitted it to their sourceforge forums, hopefully it will get put in SVN for their next release. --Mbishop 05:22, 22 January 2009 (UTC)

It's been updated, per your request. Also, I created this 1.5MB file in an attempt to put the GeSHi source files where everyone could see and correct them. I'd intended to place it into a subpage of this one, but MediaWiki OOMs while it parses the wikicode. And I'm not fond of the idea of increasing the interpreter's memory consumption limit again. --Short Circuit 10:02, 22 January 2009 (UTC)

Is there a way to edit that file? Is that the one that the site uses or is it just compiled from the GeSHi files? --Mwn3d 19:14, 22 January 2009 (UTC)

No way to edit it where it currently sits...It's too large for Mediawiki in one large clump. It could be broken into per-language pages, though. --Short Circuit 23:42, 22 January 2009 (UTC)

It would be nice if there was a separate page for syntax highlight for each language, and an option to add new languages. But meanwhile, if I create GeSHi file for a new language, could I send it to someone who has ability to integrate it in the system? --PauliKL 10:09, 23 January 2009 (UTC)

I think if you do it right, languages you add to the syntax highlighting will show up in the "Parser extension tags" here. I'm not quite sure though. --Mwn3d 13:48, 22 January 2009 (UTC)

I can't seem to get the Modula-3 highlighting to work? I tried <lang modula3> and even <source lang="modula3">. --Mbishop

<lang modula3> isn't listed in the parser extension tags on Special:Version. I guess something has to be inserted somewhere in the MediaWiki code, so it knows that <lang modula3> stuff has to be handed to GeSHi. --Ce 18:45, 22 January 2009 (UTC)

Perhaps, I know the language file works (tested it on my own apache), but I didn't test with mediawiki, not sure what needs to be done there. --Mbishop 19:52, 22 January 2009 (UTC)

The Mediawiki extension I'm using may have a specific list of supported languages. I'll check to see what exactly is going on... --Short Circuit 23:42, 22 January 2009 (UTC)

Indeed, that was the problem. Changed. <lang modula3> should now work. But now the namespace seems to have conflicts. The contribution copyright warning seems to be being parsed as a language. This is the kind of thing I was worried about with the namespace pollution. Working on it... --Short Circuit 00:04, 23 January 2009 (UTC)

Fixed. Apparently, there's a language whose GeSHi tag would be <div>. --Short Circuit 00:22, 23 January 2009 (UTC)

The failure mode for an unrecognized language is pretty annoying: the code itself is replaced with an unsorted list of available languages. Could this be fixed to just show the code in a PRE block? This way we can start using CODE tags for all languages now, and they'll be automagically formatted when language support is added. --IanOsgood 18:21, 24 January 2009 (UTC)

Whitespace

If anyone ever feels like putting whitespace code on here (unlikely but possible) they may have trouble highlighting it: <lang whitespace>

</lang> Taking out the nowiki tags eliminates the code: <lang whitespace>

</lang> Not sure if that can be fixed or not. --Mwn3d 18:58, 23 January 2009 (UTC)

I guess it comes from the code removing those superfluous (in other languages) lines at the beginning and end (e.g. instead of the ugly

you can write a more reasonable

without getting empty lines at the beginning/end.

Note that in Whitespace, every non-whitespace character is a comment, so you can simply write any non-whitespace stuff at the beginning/end:

<lang whitespace> code:

end

</lang>

(note that this probably isn't a valid whitespace program; I just randomly added spaces/tabs).

However, last friday, whitespace got nicely colored (tabs were bright red and spaces bright blue, IIRC). Now it's just white. Looking at the generated HTML, it seems classes "re2" (space) and "re3" (tab) are responsible for whitespace hilighting. Setting the background property for them should re-enable whitespace highlighting. --Ce 11:57, 26 January 2009 (UTC)

I've now found a trick to get Whitespace highlighting without any (visible) text around it: Use the Unicode character U+FEFF (ZERO WIDTH NO-BREAK SPACE) to mark the start/end of the code:

</lang>

Note that U+FEFF renders as absolutely nothing, and is not whitespace as defined by Whitespace (although it is whitespace according to Unicode), and therefore should be ignored as "comment" by whitespace interpreters (I didn't test that, though). It apparently also isn't considered whitespace by the start/end line removing code, therefore it's not removed. --Ce 15:22, 26 January 2009 (UTC)

Whitespace hightlighting now works as expected. I updated the code in this section to reflect the new lang tag, and I'm amazed that it works, myself... --Short Circuit 08:09, 30 January 2009 (UTC)

Another problem with eating whitespace characters

I just noted another problem with eating whitespace characters, which is related to the Whitespace language problem, but exists for all languages: If the first line is indented, the indentation is eaten. Example: <lang cpp>

 int i = 1;
 ++i;
 return i;

</lang> In the source, all lines are indented with two spaces. In the resulting page, the indentation of the first line is gone.

IMHO whitespace characters should only be eaten up to and including the first newline. I guess currently there's a regexp like [ \t\n]* for eating the initial whitespace. This should be changed to [ \t]*\n. --Ce 16:12, 26 January 2009 (UTC)

Fixed. GeshiCodeTag was feeding the source block through trim() prior to passing it on to GeSHi. I removed the call to trim(). --Short Circuit 22:00, 29 January 2009 (UTC)

That introduces another "problem". People seem to like moving the lang tags to the lines before and after the code. Before it didn't add an extra line before and after, but now it does. Compare the cpp example above (with the tags separate from the code) to the code below (with the tags inline with code):

<lang cpp> int i = 1;

 ++i;
 return i;</lang>

It doesn't make much difference to me;I'll put the tags inline. It's just that other people (I forget who) thought it was better to separate the tags out.--Mwn3d 22:34, 29 January 2009 (UTC)

I definitely prefer it that way. With code tags inline, you have basically two problems:

First (and most important): It's very hard to see where the code section ends. This actually already caused one mistake during tag changing
Second, the first line of code starts at a later column in the page source, which is not nice for editing code.

Also note that it's a difference to how pre works:

This is a separate line in pre tags

vs.

<lang> This is a separate line in lang tags </lang>

Such inconsistencies make the tags harder to use. --Ce 08:28, 30 January 2009 (UTC)

While I don't have any real PHP knowledge, I think passing through the following function where it was passed through trim should probably fix the probem:

<lang php> function prestyletrim($text) {

 return preg_replace("/(^[ \t]*\n|\n[ \t]*\$)/","",$text);

} </lang>

--Ce 17:26, 30 January 2009 (UTC)

D problem

Check out RCSNUSP/D to see a double backslash problem in the D highlighting. --Mwn3d 16:55, 28 January 2009 (UTC)

Change in behavior for unsupported languages -- we have unity!

I've modified GeshiCodeTag to "fail" silently on unsupported languages, wrapping the source code in a <pre> block, and prepending some HTML comments indicating that the language isn't supported. (The comments are there so that if you use a language you think is supported, and you don't see highighting, you can view selection source to double-check. I might add a "similar language tags supported" bit to the comments to aid debugging.)

What this means is that virtually all code examples should now be wrapped in <lang> blocks, whether or not the highlighter supports them. If highlighting is supported by a later version of GeSHi, code examples across the site will then be decorated automagically. I just ask that each language try to standardize on a language ID, so that when support is added by GeSHi, I can simply alias our standardized ID to point to whatever ID GeSHi decided on upstream.--Short Circuit 08:24, 30 January 2009 (UTC)

I've just found that the "fail" on unsupported languages indeed fails, i.e. doesn't work correctly. See e.g. [[1]]. Looking at the generated HTML, you obviously inserted <code> tags instead of <pre> tags. --Ce 16:57, 30 January 2009 (UTC)

Code tag special behaviour

Hi, I'm pretty positive that there are no more code tags which should be lang tags (I'm not 100% sure, of course, because there's no way to effectively search for that). Therefore I think the normal behaviour of HTML code tags should be restored as soon as possible. --Ce 08:53, 2 February 2009 (UTC)

Normal code tag behavior restored. It's no longer tied to the syntax highlighting engine in any way. --Short Circuit 18:42, 2 February 2009 (UTC)

OCaml comments

Hey, can someone highlight comments in OCaml, please? Thanks, --Spoon! 01:51, 5 February 2009 (UTC) <lang ocaml>(* a comment *)</lang>

More generally, can we get this for any language? "Pascal family" languages all use (* foo *) for comments, so adding it would add comment highlighting for lots of languages. --Mbishop 07:02, 5 February 2009 (UTC)