highlight: New plugin supporting syntax highlighting of pretty much anything.

* debian/control: Add suggests for libhighlight-perl, although
  that package is not yet created by Debian's highlight source package.
  (See #529869)

IkiWiki/Plugin/highlight.pm

@@ -0,0 +1,118 @@
+#!/usr/bin/perl
+package IkiWiki::Plugin::highlight;
+
+use warnings;
+use strict;
+use IkiWiki 3.00;
+use highlight;
+
+# locations of highlight's files
+my $filetypes="/etc/highlight/filetypes.conf";
+my $langdefdir="/usr/share/highlight/langDefs";
+
+sub import {
+	hook(type => "getsetup", id => "highlight",  call => \&getsetup);
+	hook(type => "checkconfig", id => "highlight", call => \&checkconfig);
+}
+
+sub getsetup () {
+	return
+		plugin => {
+			safe => 1,
+			rebuild => 1, # format plugin
+		},
+		tohighlight => {
+			type => "string",
+			example => ".c, .h, .cpp, .pl, .py, Makefile:make",
+			description => "source files to syntax highlight",
+			safe => 1,
+			rebuild => 1,
+		},
+}
+
+sub checkconfig () {
+	if (exists $config{tohighlight}) {
+		foreach my $file (split /, /, $config{tohighlight}) {
+			my @opts = $file=~s/^\.// ?
+				(keepextension => 1) :
+				(noextension => 1);
+			my $ext = $file=~s/:(.*)// ? $1 : $file;
+		
+			my $langfile=ext2langfile($ext);
+			if (! defined $langfile) {
+				error(sprintf(gettext(
+					"tohighlight contains unknown file type '%s'"),
+					$ext));
+			}
+	
+			hook(
+				type => "htmlize",
+				id => $file,
+				call => sub {
+					my %params=@_;
+				       	highlight($langfile, $params{content});
+				},
+				longname => sprintf(gettext("Source code: %s"), $file),
+				@opts,
+			);
+		}
+	}
+}
+
+my %ext2lang;
+my $filetypes_read=0;
+
+# Parse highlight's config file to get extension => language mappings.
+sub read_filetypes () {
+	open (IN, $filetypes);
+	while (<IN>) {
+		chomp;
+		if (/^\$ext\((.*)\)=(.*)$/) {
+			$ext2lang{$_}=$1 foreach $1, split ' ', $2;
+		}
+	}
+	close IN;
+	$filetypes_read=1;
+}
+
+sub langfile ($) {
+	return "$langdefdir/$_[0].lang";
+}
+
+# Given a filename extension, determines the language definition to
+# use to highlight it.
+sub ext2langfile ($) {
+	my $ext=shift;
+
+	read_filetypes() unless $filetypes_read;
+	if (exists $ext2lang{$ext}) {
+		return langfile($ext2lang{$ext});
+	}
+	# If a language only has one common extension, it will not
+	# be listed in filetypes, so check the langfile.
+	elsif (-e langfile($ext)) {
+		return langfile($ext);
+	}
+	else {
+		return undef;
+	}
+}
+
+# Interface to the highlight C library.
+sub highlight ($$) {
+	my $langfile=shift;
+	my $input=shift;
+
+	my $gen = highlightc::CodeGenerator_getInstance($highlightc::XHTML);
+	$gen->setFragmentCode(1); # generate html fragment
+	$gen->setHTMLEnclosePreTag(1); # include stylish <pre>
+	$gen->initLanguage($langfile);
+	$gen->initTheme("/dev/null"); # theme is not needed because CSS is not emitted
+	$gen->setEncoding("utf-8");
+
+	my $output=$gen->generateString($input);
+	highlightc::CodeGenerator_deleteInstance($gen);
+	return $output;
+}
+
+1

debian/changelog

@@ -1,3 +1,13 @@
+ikiwiki (3.14) UNRELEASED; urgency=low
+
+  * highlight: New plugin supporting syntax highlighting of pretty much
+    anything.
+  * debian/control: Add suggests for libhighlight-perl, although
+    that package is not yet created by Debian's highlight source package.
+    (See #529869)
+
+ -- Joey Hess <joeyh@debian.org>  Fri, 22 May 2009 22:03:12 -0400
+
 ikiwiki (3.13) unstable; urgency=low
 
   * ikiwiki-transition: If passed a nonexistant srcdir, or one not

debian/control

@@ -35,7 +35,7 @@ Suggests: viewvc | gitweb | viewcvs, libsearch-xapian-perl,
   liblocale-gettext-perl (>= 1.05-1), libtext-typography-perl,
   libtext-csv-perl, libdigest-sha1-perl, graphviz, libnet-amazon-s3-perl,
   sparkline-php, texlive, dvipng, libtext-wikicreole-perl,
-  libsort-naturally-perl, libtext-textile-perl
+  libsort-naturally-perl, libtext-textile-perl, libhighlight-perl
 Conflicts: ikiwiki-plugin-table
 Replaces: ikiwiki-plugin-table
 Provides: ikiwiki-plugin-table

doc/plugins/highlight.mdwn

@@ -0,0 +1,72 @@
+[[!template id=plugin name=highlight author="[[Joey]]"]]
+[[!tag type/format]]
+
+This plugin allows ikiwiki to syntax highlight source files, using
+a fast syntax highlighter that supports over a hundred programming
+languages and file formats.
+
+## prerequisites
+
+You will need to install the perl bindings to the
+[highlight library](http://www.andre-simon.de/), which in Debian
+are in the [[!debpkg libhighlight-perl]] package.
+
+## configuration
+
+Nothing will be highlighted by default.
+To enable syntax highlighting, use the `tohighlight` setting in your
+setup file to control which files should be syntax highlighted.
+Here is a typical setting for it, enabling highlighting for files
+with the extensions .c, etc, and also for any files named "Makefile".
+
+	tohighlight => .c, .h, .cpp, .pl, .py, Makefile:make",
+
+It knows what language to use for most filename extensions (see
+`/etc/highlight/filetypes.conf` for a partial list), but if you want to
+bind an unusual filename extension, or any file without an extension
+(such as a Makefile), to a language, you can do so by appending a colon
+and the name of the language, as illustrated for Makefiles above.
+
+## embedding highlighted code
+
+To embed highlighted code on a page, you can use the
+[[ikiwiki/directive/format]] directive.
+
+For example:
+
+	\[[!format c """
+	void main () {
+		printf("hello, world!");
+	}
+	"""]]
+
+You can do this for any of the extensions/filenames enabled in
+`tohighlight`.
+
+## colors
+
+The colors etc used for the syntax highlighting are entirely configurable
+by CSS. See ikiwiki's [[style.css]] for the defaults.
+
+## limitations
+
+With this plugin enabled, source files become full-fledged ikiwiki pages,
+which means they can include [[WikiLinks|ikiwiki/wikilink]] and
+[[directives|ikiwiki/directive]] like any other page can, and are also
+affected by the [[smiley]] plugin, if it is enabled. This can be
+annoying if your code accidentially contains things that look like those.
+
+On the other hand, this also allows your syntax highlighed
+source code to contain markdown formatted comments and hyperlinks
+to other code files, like this:
+
+	/* \[[!format mdwn """
+		This comment will be formatted as *markdown*!
+
+		See [[bar.h]].
+	""]] */
+
+## security
+
+This lets anyone who can edit a page in your wiki also edit
+source code files that are in your wiki. Use appropriate caution.

doc/style.css

@@ -389,3 +389,21 @@ span.color {
 	border: 1px solid #aaa;
 	padding: 3px;
 }
+
+/* Used by the highlight plugin. */
+
+pre.hl { color:#000000; background-color:#ffffff; }
+.hl.num { color:#2928ff; }
+.hl.esc { color:#ff00ff; }
+.hl.str { color:#ff0000; }
+.hl.dstr { color:#818100; }
+.hl.slc { color:#838183; font-style:italic; }
+.hl.com { color:#838183; font-style:italic; }
+.hl.dir { color:#008200; }
+.hl.sym { color:#000000; }
+.hl.line { color:#555555; }
+.hl.mark { background-color:#ffffbb; }
+.hl.kwa { color:#000000; font-weight:bold; }
+.hl.kwb { color:#830000; }
+.hl.kwc { color:#000000; font-weight:bold; }
+.hl.kwd { color:#010181; }

doc/todo/automatic_use_of_syntax_plugin_on_source_code_files.mdwn

@@ -12,3 +12,6 @@ this would allow the use of ikiwiki for [[!wikipedia literate programming]].
 * I have started something along these lines see [[plugins/contrib/sourcehighlight]].  For some reason I started with source-highlight [[DavidBremner]]
 
 * I wonder if this is similar to what you want: <http://iki.u32.net/setup/Highlight_Code_Plugin/>
+
+> The new [[plugins/highlight]] plugin is in ikiwiki core and supports
+> source code files natively. [[done]] --[[Joey]] 

doc/todo/syntax_highlighting.mdwn

@@ -28,13 +28,17 @@ things easier for the user.
   also uses source-highlight, and operates on whole source files.
   Updated to work with the fix for [[bugs/multiple_pages_with_same_name]].  Untested with files with no extension, e.g. `Makefile`.
 * [[users/jasonblevins]]'s code plugin uses source-highlight, and supports both
-  while file and directive use.
+  whole file and directive use.
 
 * [hlsimple](http://pivot.cs.unb.ca/git/?p=ikiplugins.git;a=blob_plain;f=IkiWiki/Plugin/hlsimple.pm;hb=HEAD) is a wrapper for the the perl module [[!cpan Syntax::Highlight::Engine::Simple]].  This is pure perl, pretty simple, uses css. It ought to be pretty fast (according to the author, and just because it is not external).
 On the other hand, there are not many predefined languages yet.  Defining language syntaxes is about as much 
 work as source-highlight, but in perl.  I plan to package the base module for debian. Perhaps after the author 
 releases the 5 or 6 language definitions he has running on his web site, it might be suitable for inclusion in ikiwiki. [[DavidBremner]]
 
+* [[plugins/highlight]] uses [highlight](http://www.andre-simon.de) via
+  its swig bindings. It supports whole files only. It uses either
+  keepextension or noextension, as appropriate for the type of file.
+
 ## General problems / requirements
 
 * Using non-perl syntax highlighting backends is slower. All things equal,
@@ -56,7 +60,6 @@ releases the 5 or 6 language definitions he has running on his web site, it migh
   > it has a pass-through feature that I find very useful.  My memory is unfortunately a bit fuzzy as to how
   > well the swig bindings work. [[DavidBremner]]
 
-
 * Engines that already support a wide variety of file types are of
   course preferred. If the engine doesn't support a particular type
   of file, it could fall back to doing something simple like
@@ -105,20 +108,11 @@ releases the 5 or 6 language definitions he has running on his web site, it migh
 
   Perhaps the thing to do here is to use the new `longname` parameter to
   the format hook, to give them all names that will group together at or
-  near the end of the list. Ie: "Syntax: perl", "Syntax: C", etc.
+  near the end of the list. Ie: "Syntax: perl", "Source code: c", etc.
 
-## format directive and comments
+---
 
-Hmm, the [[ikiwiki/directive/format]] directive would also allow comments
+I'm calling this [[done]] since I added the [[plugins/highlight]]
-inside source files to have mdwn embedded in them, without making the use
+plugin. There are some unresolved issues touched on here,
-of mdwn a special case, or needing to postprocess the syntax highlighter
+but they either have the own other bug reports, or are documented
-output to find comments.
+as semi-features in the docs to the plugin. --[[Joey]] 
-
-	/* \[[!format mdwn """
-
-	This is a comment in my C file. You can use mdwn in here.
-
-	"""]] */
-
-Note that this assumes that directives are expanded in source files,
-which has its own set of problems.

doc/todo/syntax_highlighting/discussion.mdwn

@@ -24,3 +24,5 @@ repository?  --[[JasonBlevins]]
 >> [[sourcecode|todo/automatic_use_of_syntax_plugin_on_source_code_files/discussion]]
 >> plugin only adds the file extensions listed in the config.  This shouldn't cause
 >> massive drop-down menu pollution.  -- [[Will]]
+
+>>> That seems to be the way to go! --[[Joey]] 

doc/todo/wiki-formatted_comments_with_syntax_plugin.mdwn

@@ -2,3 +2,8 @@
 wiki syntax within the comments of code pretty-printed with the
 [[plugins/contrib/syntax]] plugin.  This would allow the use of links and
 formatting in comments.
+
+> You can do this using the [[plugins/highlight]] plugin, but you have
+> to explicitly put a format directive in the comment to do it. Thus,
+> I'm leaving this open for now.. ideally, comments would be detected,
+> and formatted as markdown. --[[Joey]]