Merge commit 'upstream/master' into prv/po

IkiWiki.pm

@@ -721,6 +721,10 @@ sub readfile ($;$$) { #{{{
 	binmode($in) if ($binary);
 	return \*$in if $wantfd;
 	my $ret=<$in>;
+	# check for invalid utf-8, and toss it back to avoid crashes
+	if (! utf8::valid($ret)) {
+		$ret=encode_utf8($ret);
+	}
 	close $in || error("failed to read $file: $!");
 	return $ret;
 } #}}}
@@ -1295,6 +1299,7 @@ sub lockwiki () { #{{{
 } #}}}
 
 sub unlockwiki () { #{{{
+	POSIX::close($ENV{IKIWIKI_CGILOCK_FD}) if exists $ENV{IKIWIKI_CGILOCK_FD};
 	return close($wikilock) if $wikilock;
 	return;
 } #}}}

IkiWiki/Wrapper.pm

@@ -72,12 +72,16 @@ EOF
 		# Avoid more than one ikiwiki cgi running at a time by
 		# taking a cgi lock. Since ikiwiki uses several MB of
 		# memory, a pile up of processes could cause thrashing
-		# otherwise.
+		# otherwise. The fd of the lock is stored in
+		# IKIWIKI_CGILOCK_FD so unlockwiki can close it.
 		$pre_exec=<<"EOF";
 	{
 		int fd=open("$config{wikistatedir}/cgilock", O_CREAT | O_RDWR, 0666);
-		if (fd != -1)
+		if (fd != -1 && flock(fd, LOCK_EX) == 0) {
-			flock(fd, LOCK_EX);
+			char *fd_s;
+			asprintf(&fd_s, "%i", fd);
+			setenv("IKIWIKI_CGILOCK_FD", fd_s, 1);
+		}
 	}
 EOF
 	}

debian/changelog

@@ -1,5 +1,19 @@
-ikiwiki (2.69) UNRELEASED; urgency=low
+ikiwiki (2.70) unstable; urgency=low
 
+  * Avoid crash on malformed utf-8 discovered by intrigeri.
+
+ -- Joey Hess <joeyh@debian.org>  Wed, 12 Nov 2008 17:45:58 -0500
+
+ikiwiki (2.69) unstable; urgency=low
+
+  * Avoid multiple ikiwiki cgi processes piling up, eating all memory,
+    and thrashing, by making the cgi wrapper wait on a cgilock.
+    If you had to set apache's MaxClients low to avoid ikiwiki thrashing your
+    server, you can now turn it up to a high value.
+  * Stop busy-waiting in lockwiki, as this could delay ikiwiki from waking up
+    for up to one second. The bailout code is no longer needed after above
+    change.
+  * Remove support for unused optional wait parameter from lockwiki.
   * aggregate: Try to query XML::Feed for the base url when derelevatising
     links. Since this needs the just released XML::Feed 0.3, as well 
     as a not yet released XML::RSS, it will fall back to the old method
@@ -14,16 +28,8 @@ ikiwiki (2.69) UNRELEASED; urgency=low
   * tag: Normalize tagbase so leading/trailing slashes in it don't break
     things.
   * bzr: Fix dates for recentchanges.
-  * Avoid multiple ikiwiki cgi processes piling up, eating all memory,
-    and thrashing, by making the cgi wrapper wait on a cgilock.
-    If you had to set apache's MaxClients low to avoid ikiwiki thrashing your
-    server, you can now turn it up to a high value.
-  * Stop busy-waiting in lockwiki, as this could delay ikiwiki from waking up
-    for up to one second. The bailout code is no longer needed after above
-    change.
-  * Remove support for unused optional wait parameter from lockwiki.
 
- -- Joey Hess <joeyh@debian.org>  Thu, 06 Nov 2008 16:01:00 -0500
+ -- Joey Hess <joeyh@debian.org>  Tue, 11 Nov 2008 20:35:55 -0500
 
 ikiwiki (2.68) unstable; urgency=low
 

doc/bugs/pagetitle_function_does_not_respect_meta_titles.mdwn

@@ -8,5 +8,9 @@ The `IkiWiki::pagetitle` function does not respect title changes via `meta.title
 > - Using <code>inline</code> would avoid the redefinition + code duplication.
 > - A few plugins would need to be upgraded.
 > - It may be necessary to adapt the testsuite in `t/pagetitle.t`, as well.
-
+>
 > --[[intrigeri]]
+>
+>> It was actually more complicated than expected. A working prototype is
+>> now in my `meta` branch, see my userpage for the up-to-date url.
+>> Thus tagging [[patch]]. --[[intrigeri]]

doc/news/version_2.64.mdwn

@@ -1,25 +0,0 @@
-ikiwiki 2.64 released with [[!toggle text="these changes"]]
-[[!toggleable text="""
-   * Avoid uninitialised value when --dumpsetup is used and no srcdir/destdir
-     specified.
-   * ddate: Stop clobbering timeformat when not enabled.
-   * progress: New plugin to generate progress bars (willu)
-   * Add allow\_symlinks\_before\_srcdir to config so websetup doesn't eat it.
-   * img: Support sizes like 200x. Closes: #[475149](http://bugs.debian.org/475149)
-   * goodstuff: Remove otl plugin from the bundle since it needs a significant
-     external dependency and is not commonly used. If you use otl, make sure
-     you explicitly enable it now.
-   * goodstuff: Add more, progress, and table plugins to the bundle.
-   * Improve error message if external plugin fails to load. Closes: #[498458](http://bugs.debian.org/498458)
-   * Directive documentation broken out of the plugin documentation and into
-     pages suitable to be used as an underlay. Thanks to Willu for doing most
-     of the tedious work.
-   * Move the directive documentation into its own underlay, separate from
-     basewiki, since it's sorta large compared to the rest of basewiki.
-   * listdirectives: Enable use of the directives underlay.
-   * Removed the obsolete blog page from the basewiki. ikiwiki/blog still
-     remains, but is now deprecated too.
-   * Removed old redirecton pages from basewiki (helponformatting,
-     markdown, openid, pagespec, preprocessordirective, subpage, wikilink).
-   * inline: Treat rootpage as a link, so that it can refer to a subpage
-     without hardcoding the path."""]]

doc/news/version_2.65.mdwn

@@ -1,25 +0,0 @@
-ikiwiki 2.65 released with [[!toggle text="these changes"]]
-[[!toggleable text="""
-   * aggregate: Expire excess or old items on the same pass that adds them,
-     not only on subsequent passes.
-   * editdiff: Broken since 2.62 due to wrong syntax, now fixed.
-   * aggregate: Support atom feeds with only a summary element, and no content
-     elements.
-   * progress: Display an error if the progress cannot be parsed, and allow
-     the percent parameter to only optionally end with "%".
-   * Fix reversion in use of ikiwiki -verbose -setup with a setup file that
-     enables syslog. Setup output is once again output to stdout in this
-     case.
-   * edittemplate: Default new page file type to the same type as the template.
-     (willu)
-   * edittemplate: Add "silent" parameter. (Willu)
-   * edittemplate: Link to template, to allow creating it. (Willu)
-   * editpage: Add a missing check that the page name contains only legal
-     characters, in addition to the existing check for pruned filenames.
-   * Print a debug message if a page has multiple source files.
-   * Add keepextension parameter to htmlize hook. (Willu)
-   * rename, remove: Don't rely on a form parameter to tell whether the page
-     should be treated as an attachment.
-   * rename: Add support for moving SubPages of a page when renaming it.
-     (Sponsored by The TOVA Company.)
-   * rename: Hide type field from rename form when renaming attachments."""]]

doc/news/version_2.69.mdwn

@@ -0,0 +1,24 @@
+ikiwiki 2.69 released with [[!toggle text="these changes"]]
+[[!toggleable text="""
+   * Avoid multiple ikiwiki cgi processes piling up, eating all memory,
+     and thrashing, by making the cgi wrapper wait on a cgilock.
+     If you had to set apache's MaxClients low to avoid ikiwiki thrashing your
+     server, you can now turn it up to a high value.
+   * Stop busy-waiting in lockwiki, as this could delay ikiwiki from waking up
+     for up to one second. The bailout code is no longer needed after above
+     change.
+   * Remove support for unused optional wait parameter from lockwiki.
+   * aggregate: Try to query XML::Feed for the base url when derelevatising
+     links. Since this needs the just released XML::Feed 0.3, as well
+     as a not yet released XML::RSS, it will fall back to the old method
+     if no xml:base info is available.
+   * meta: Plugin is now enabled by default since the basewiki uses it.
+   * txt: Do not encode quotes when filtering the txt, as that broke
+     later parsing of any directives on the page.
+   * Fix the link() pagespec to match links that are internally recorded as
+     absolute.
+   * Add rel=nofollow to recentchanges\_links for the same (weak) reasons it
+     was earlier added to edit links.
+   * tag: Normalize tagbase so leading/trailing slashes in it don't break
+     things.
+   * bzr: Fix dates for recentchanges."""]]

doc/news/version_2.70.mdwn

@@ -0,0 +1,3 @@
+ikiwiki 2.70 released with [[!toggle text="these changes"]]
+[[!toggleable text="""
+   * Avoid crash on malformed utf-8 discovered by intrigeri."""]]

doc/plugins/contrib/po.mdwn

@@ -158,3 +158,5 @@ Any thoughts on this?
 >>>>> Joey, please have a look at my branch, your help would be really
 >>>>> welcome for the security research, as I'm almost done with what
 >>>>> I am able to do myself in this area. --[[intrigeri]]
+>>>>>>
+>>>>>> I came up with a patch for the WrapI18N issue --[[Joey]]

doc/security.mdwn

@@ -407,3 +407,13 @@ discovered on 30 May 2008 and fixed the same day. ([[!cve CVE-2008-0169]])
 
 I recommend upgrading to 2.48 immediatly if your wiki allows both password
 and openid logins.
+
+## Malformed UTF-8 DOS
+
+Feeding ikiwiki page sources containing certian forms of malformed UTF-8
+can cause it to crash. This can potentially be used for a denial of service
+attack.
+
+intrigeri discovered this problem on 12 Nov 2008 and a patch put in place
+later that day, in version 2.70. The fix was backported to testing as version
+2.53.2, and to stable as version 1.33.7.

doc/tips/inside_dot_ikiwiki.mdwn

@@ -63,3 +63,28 @@ To remove that user:
 I've not written actual utilities to do this yet because I've only needed
 to do it rarely, and the data I've wanted has been different each time.
 --[[Joey]]
+
+## the session database
+
+`.ikiwiki/sessions.db` is the session database. See the [[cpan CGI::Session]]
+documentation for more details.
+
+## lockfiles
+
+In case you're curious, here's what the various lock files do.
+
+* `.ikiwiki/lockfile` is the master ikiwiki lock file. Ikiwiki takes this 
+  lock before reading/writing state.
+* `.ikiwiki/commitlock` is locked as a semophore, to disable the commit hook
+  from doing anything.
+* `.ikiwiki/cgilock` is locked by the cgi wrapper, to ensure that only 
+  one ikiwiki process is run at a time to handle cgi requests.
+
+## plugin state files
+
+Some plugins create other files to store their state. 
+
+* `.ikiwiki/aggregate` is a plain text database used by the aggregate plugin
+  to record feeds and known posts.
+* `.ikiwiki/xapian/` is created by the search plugin, and contains xapian-omega
+  configuration and the xapian database.

po/ikiwiki.pot

@@ -8,7 +8,7 @@ msgid ""
 msgstr ""
 "Project-Id-Version: PACKAGE VERSION\n"
 "Report-Msgid-Bugs-To: \n"
-"POT-Creation-Date: 2008-11-11 15:36-0500\n"
+"POT-Creation-Date: 2008-11-11 20:48-0500\n"
 "PO-Revision-Date: YEAR-MO-DA HO:MI+ZONE\n"
 "Last-Translator: FULL NAME <EMAIL@ADDRESS>\n"
 "Language-Team: LANGUAGE <LL@li.org>\n"
@@ -910,19 +910,19 @@ msgstr ""
 
 #. translators: The first parameter is a filename, and the second is
 #. translators: a (probably not translated) error message.
-#: ../IkiWiki/Wrapper.pm:93
+#: ../IkiWiki/Wrapper.pm:97
 #, perl-format
 msgid "failed to write %s: %s"
 msgstr ""
 
 #. translators: The parameter is a C filename.
-#: ../IkiWiki/Wrapper.pm:150
+#: ../IkiWiki/Wrapper.pm:154
 #, perl-format
 msgid "failed to compile %s"
 msgstr ""
 
 #. translators: The parameter is a filename.
-#: ../IkiWiki/Wrapper.pm:170
+#: ../IkiWiki/Wrapper.pm:174
 #, perl-format
 msgid "successfully generated %s"
 msgstr ""
@@ -969,7 +969,7 @@ msgstr ""
 msgid "preprocessing loop detected on %s at depth %i"
 msgstr ""
 
-#: ../IkiWiki.pm:1672
+#: ../IkiWiki.pm:1673
 msgid "yes"
 msgstr ""