Fixes for behavior changes in perl 5.10's CGI

Something has changed in CGI.pm in perl 5.10. It used to not care
if STDIN was opened using :utf8, but now it'll mis-encode utf-8 values
when used that way by ikiwiki. Now I have to binmode(STDIN) before
instantiating the CGI object.

In 57bba4dac1, I changed from decoding
CGI::Formbuilder fields to utf-8, to decoding cgi parameters before setting
up the form object. As of perl 5.10, that approach no longer has any effect
(reason unknown). To get correctly encoded values in FormBuilder forms,
they must once again be decoded after the form is set up.

As noted in 57bba4da, this can cause one set of problems for
formbuilder_setup hooks if decode_form_utf8 is called before the hooks, and
a different set if it's called after. To avoid both sets of problems, call
it both before and after. (Only remaining problem is the sheer ugliness and
inefficiency of that..)

I think that these changes will also work with older perl versions, but I
haven't checked.

Also, in the case of the poll plugin, the cgi parameter needs to be
explcitly decoded before it is used to handle utf-8 values. (This may have
always been broken, not sure if it's related to perl 5.10 or not.)

IkiWiki/CGI.pm

@@ -77,10 +77,13 @@ sub check_canedit ($$$;$) { #{{{
 	return $canedit;
 } #}}}
 
-sub decode_cgi_utf8 ($) { #{{{
-	my $cgi = shift;
-	foreach my $f ($cgi->param) {
-		$cgi->param($f, map { decode_utf8 $_ } $cgi->param($f));
+sub decode_form_utf8 ($) { #{{{
+	my $form = shift;
+	foreach my $f ($form->field) {
+		$form->field(name  => $f,
+		             value => decode_utf8($form->field($f)),
+	                     force => 1,
+		);
 	}
 } #}}}
 
@@ -103,7 +106,6 @@ sub cgi_signin ($$) { #{{{
 	my $q=shift;
 	my $session=shift;
 
-	decode_cgi_utf8($q);
 	eval q{use CGI::FormBuilder};
 	error($@) if $@;
 	my $form = CGI::FormBuilder->new(
@@ -127,10 +129,12 @@ sub cgi_signin ($$) { #{{{
 	$form->field(name => "do", type => "hidden", value => "signin",
 		force => 1);
 	
+	decode_form_utf8($form);
 	run_hooks(formbuilder_setup => sub {
 		shift->(form => $form, cgi => $q, session => $session,
 		        buttons => $buttons);
 	});
+	decode_form_utf8($form);
 
 	if ($form->submitted) {
 		$form->validate;
@@ -161,7 +165,6 @@ sub cgi_prefs ($$) { #{{{
 	my $session=shift;
 
 	needsignin($q, $session);
-	decode_cgi_utf8($q);
 	
 	# The session id is stored on the form and checked to
 	# guard against CSRF.
@@ -198,10 +201,12 @@ sub cgi_prefs ($$) { #{{{
 	);
 	my $buttons=["Save Preferences", "Logout", "Cancel"];
 	
+	decode_form_utf8($form);
 	run_hooks(formbuilder_setup => sub {
 		shift->(form => $form, cgi => $q, session => $session,
 		        buttons => $buttons);
 	});
+	decode_form_utf8($form);
 	
 	$form->field(name => "do", type => "hidden", value => "prefs",
 		force => 1);
@@ -255,8 +260,6 @@ sub cgi_editpage ($$) { #{{{
 	my $q=shift;
 	my $session=shift;
 	
-	decode_cgi_utf8($q);
-	
 	my @fields=qw(do rcsinfo subpage from page type editcontent comments);
 	my @buttons=("Save Page", "Preview", "Cancel");
 	eval q{use CGI::FormBuilder};
@@ -276,10 +279,12 @@ sub cgi_editpage ($$) { #{{{
 		wikiname => $config{wikiname},
 	);
 	
+	decode_form_utf8($form);
 	run_hooks(formbuilder_setup => sub {
 		shift->(form => $form, cgi => $q, session => $session,
 			buttons => \@buttons);
 	});
+	decode_form_utf8($form);
 	
 	# This untaint is safe because titlepage removes any problematic
 	# characters.
@@ -366,6 +371,7 @@ sub cgi_editpage ($$) { #{{{
 		}
 
 		my $content=$form->field('editcontent');
+
 		run_hooks(editcontent => sub {
 			$content=shift->(
 				content => $content,
@@ -642,7 +648,9 @@ sub cgi (;$$) { #{{{
 		eval q{use CGI};
 		error($@) if $@;
 	
+		binmode(STDIN);
 		$q=CGI->new;
+		binmode(STDIN, ":utf8");
 	
 		run_hooks(cgi => sub { shift->($q) });
 	}

IkiWiki/Plugin/poll.pm

@@ -4,6 +4,7 @@ package IkiWiki::Plugin::poll;
 use warnings;
 use strict;
 use IkiWiki 2.00;
+use Encode;
 
 sub import { #{{{
 	hook(type => "preprocess", id => "poll", call => \&preprocess);
@@ -78,7 +79,7 @@ sub sessioncgi ($$) { #{{{
 	my $cgi=shift;
 	my $session=shift;
 	if (defined $cgi->param('do') && $cgi->param('do') eq "poll") {
-		my $choice=$cgi->param('choice');
+		my $choice=decode_utf8($cgi->param('choice'));
 		if (! defined $choice) {
 			error("no choice specified");
 		}

debian/changelog

@@ -9,6 +9,8 @@ ikiwiki (2.46) UNRELEASED; urgency=low
     wikis that automatically ping one another to stay up to date.
   * Optimised file statting code when scanning for modified pages;
     cut the number of system calls in half. (Still room for improvement.)
+  * Fixes for behavior changes in perl 5.10's CGI that broke utf-8 support
+    in several interesting ways.
 
  -- Joey Hess <joeyh@debian.org>  Mon, 05 May 2008 19:34:51 -0400
 

doc/bugs/poll_plugin:_can__39__t_vote_for_non-ascii_options.mdwn

@@ -3,7 +3,5 @@ I don't seem to be able to vote for options that have non-ascii names, using the
 As an example, see http://test.liw.fi/testpoll/index.html: the "red", "green", and "blue" options work fine, but the "ehkä" one does not.
 --[liw](http://liw.fi/)
 
-> It's not just the poll plugin, editing a page with utf-8 will corrupt it
-> now! I wonder if this is a perl 5.10 change; it was working just fine as
-> recently as May 8th; I upgraded perl on the 9th and it's broken for me
-> now. --[[Joey]]
+> Ok, 4.5 hours of beating my head against a brick wall, and I've fixed this.
+> [[done]] --[[Joey]]