speeding up ikiwiki: advice sought

doc/forum/speeding_up_ikiwiki.mdwn

@@ -0,0 +1,75 @@
+My website takes a fairly long time to render. It takes a long time to do
+things like add pages, too. I'd like to try and understand what takes the
+time and what I might be able to do to speed things up.
+
+I have 1,234 objects on my site (yikes!). 717 are items under "/log" which
+I think might be the main culprit because there are some complex pagespecs
+operating in that area (e.g. log/YYYY/MM/DD, YYYY/MM and YYYY for YYYY >=
+2003, YYYY <= 2008 which include every page under log/ which was modified
+in the corresponding YYYY or YYYY/MM or YYYY/MM/DD). There is very little
+linking between the world outside of /log and that within it.
+
+I was interested in generating a graphical representation of ikiwiki's idea of
+page inter-dependencies. I started by looking at the '%links' hash using the
+following plugin:
+
+    #!/usr/bin/perl
+    package IkiWiki::Plugin::deps;
+    
+    use warnings;
+    use strict;
+    use IkiWiki 3.00;
+    
+    
+    sub import {
+        hook(type => "format", id => "deps", call => \&fooble);
+    }
+    
+    my $hasrun = 0;
+    
+    sub fooble ($$) {
+        if(0 == $hasrun) {
+            $hasrun = 1;
+            open MYFILE, ">/home/jon/deps.dot";
+            foreach my $key (keys %links) {
+                my $arrref = $links{$key};
+                foreach my $page (@$arrref) {
+                    print MYFILE "$key -> $page;\n";
+                }
+            }
+            close MYFILE;
+        }
+    }
+    
+    1
+
+The resulting file was enormous: 2,734! This turns out to be because of the following code in scan() in Render.pm:
+
+    if ($config{discussion}) {$
+        # Discussion links are a special case since they're
+        # not in the text of the page, but on its template.
+        $links{$page}=[ $page."/".gettext("discussion") ];
+
+Worst case (no existing discussion pages) this will double the number of link
+relationships. Filtering out all of those, the output drops to 1,657. This
+number is still too large to really visualize: the graphviz PNG and PDF output
+engines segfault for me, the PS one works but I can't get any PS software to
+render it without exploding.
+
+Now, the relations in the links hash are not the same thing as IkiWiki's notion of dependencies. Can anyone point me at  that data structure / where I might be able to add some debugging foo to generate a graph of it?
+
+Once I've figured out that I might be able to optimize some pagespecs. I
+understand pagespecs are essentially translated into sequential perl code. I
+might gain some speed if I structure my complex pagespecs so that the tests
+which have the best time complexity vs. "pages ruled out" ratio are performed
+first.
+
+I might also be able to find some dependencies which shouldn't be there and
+remove the dependency.
+
+In general any advice people could offer on profiling ikiwiki would be great.
+I did wonder about invoking the magic profiling arguments to perl via the CGI
+wrapper.
+
+
+-- [[Jon]]