Commit Graph

142 Commits (3177c4450ed857d77c9154040e3cfab3b64128b3)

Author SHA1 Message Date
Joey Hess 76bda69a92 use prune to delete rendered files
This handles deleting empty directories too.
2008-07-14 22:45:12 -04:00
Simon McVittie b29d11b3c6 Fix aggregateinternal migration so it skips expired entries, and deletes output 2008-07-15 03:24:05 +01:00
Joey Hess af48316b1d Merge commit 'smcv/aggregateinternal' into aggregateinternal
Conflicts:

	IkiWiki/Plugin/aggregate.pm
2008-07-14 20:15:08 -04:00
Simon McVittie 302bb836f2 Use _aggregated as extension for internalized aggregation 2008-07-15 01:04:29 +01:00
Joey Hess 51bf0ea529 changes to debug message printing 2008-07-14 19:43:45 -04:00
Simon McVittie 50a5ab3c4e Add aggregateinternal mode to ikiwiki-transition.
Usage:
1. Update all pagespecs that use aggregated pages to use internal()
2. ikiwiki-transition aggregateinternal $srcdir $htmlext
   (where $srcdir and $htmlext are the srcdir and htmlext options in
   your .setup file)
3. Add aggregateinternal to your .setup file
4. Rebuild the wiki
2008-07-14 23:39:22 +01:00
Simon McVittie 18b3e970ff Merge commit 'origin/master' into aggregateinternal 2008-07-14 23:34:07 +01:00
Joey Hess ffc99f5904 switch preprocess hooks to use error function 2008-07-13 15:05:34 -04:00
Simon McVittie 5c0896e674 Make aggregated items be internal pages if --aggregateinternal is used.
This addresses <http://ikiwiki.info/todo/aggregate_to_internal_pages/>
in a simple way. With this approach, a flag day is required, on which all
users of aggregated pages start to inline them using the internal() pagespec;
after that, the aggregateinternal option can safely be switched on in the
setup file (and the old aggregated pages can be deleted by hand).
2008-07-12 18:15:11 +01:00
Bernd Zeimetz b16c43a440 `template` option for aggregate.pm.
Allows to specify the template file which is used to
create the html pages.
2008-06-21 16:23:39 -04:00
Joey Hess d5d56a24bd When calling decode_utf8 on known-problimatic content in aggregate, explicitly pass 0 (FB_DEFAULT) as the second parameter. Apparently perl 5.8 needs this to avoid crashing on malformed utf-8, despite its docs saying it is the default. 2008-05-28 15:38:04 -04:00
Joey Hess 1f88cad3a2 aggregate: Add support for web-based triggering of aggregation for people stuck on shared hosting without cron. (Sheesh.) Enabled via the `aggregate_webtrigger` configuration optiom. 2008-05-05 20:20:45 -04:00
Joey Hess de8c34df59 aggregate: Correct a mistake in the code that dummy up a guid for feeds lacking one. 2008-04-03 02:36:01 -04:00
Joey Hess 16299dbae8 load HTML::Entities at top
Used in several subs, not all of which load it on demand, this seems simpler.
2008-03-14 18:43:54 -04:00
Joey Hess bc2671082b no need to use HTML::Entities, as it's loaded on demand by code below 2008-03-12 18:52:33 -04:00
Joey Hess 9d54cc4659 implement aggregate_locking design
Now aggregation will not lock the wiki. Any changes made during aggregaton are
merged in with the changed state accumulated while aggregating. A separate
lock file prevents multiple concurrent aggregators. Garbage collection
of orphaned guids is much improved. loadstate() is only called once
per process, so tricky support for reloading wiki state is not needed.

(Tested fairly thuroughly.)
2008-02-03 16:48:26 -05:00
Joey Hess 38affb0c1c add aggregate locking functions 2008-02-03 15:17:15 -05:00
Joey Hess 1f6591f0a6 * aggregate: Revert use of forking to not save state, that was not the right
approach.
2008-02-03 03:04:19 -05:00
Joey Hess 0779013683 * aggregate: Forking a child broke the one state that mattered: Forcing
the aggregating page to be rebuilt. Fix this.
2008-02-02 23:56:13 -05:00
Joey Hess 3803266b8f merged the recentchanges branch
misc fixes
2008-01-29 17:50:11 -05:00
Joey Hess 7a40bcab9a add missing test to avoid uninitialised value when a page with metadata is removed 2008-01-29 17:36:25 -05:00
Josh Triplett fafb2edaa7 aggregate.pm: Replace Unicode apostrophe with ASCII to unbreak build
gettext choked on a Unicode apostrophe in the aggregate plugin, which
appeared in a new error message in commit
4f872b5633.  Replace it with an ASCII
apostrophe.
2008-01-26 23:12:29 -08:00
Joey Hess 4f872b5633 * aggregate: Fork a child process to handle the aggregation. This simplifies
the code, since that process can change internal state as needed, and
  it will automatically be cleaned up for the parent process, which proceeds
  to render the changes.
2008-01-24 17:22:39 -05:00
Joey Hess ba6b8e5e3e * aggregate: Don't let feeds set creation times for pages in the future. 2008-01-16 02:01:00 -05:00
Joey Hess 86781fc43e * aggregate: Include copyright statements from rss feed as meta copyright
directives.
* aggregate: Yet another state saving fix (sigh).
* aggregate: Add hack to support feeds with invalidly escaped html entities.
2008-01-08 20:41:25 -05:00
Joey Hess 76e941a480 * aggregate: Only save state if it was already loaded. 2008-01-05 01:26:09 -05:00
Joey Hess 41234a3f0f round days old message 2008-01-02 23:38:45 -05:00
Joey Hess c3eec7fdf0 I *think* that this comment is out of date. 2007-12-30 14:57:33 -05:00
Joey Hess f0f52e602d * aggregate: Fix stupid mistake introduced when converting it to use
the needsbuild hook. This resulted in feeds not being removed when pages
  were updated, and probably other bugs.
* aggregate: Avoid uninitialised value warning when removing a feed that
  has an expired guid.
2007-12-30 14:49:25 -05:00
Joey Hess f373b09bf9 loadstate needs to be called even if no pages are being built 2007-12-16 19:40:50 -05:00
Joey Hess 96817b0032 * meta: Drop support for "meta link", since supporting this for internal
links required meta to be run during scan, which complicated its data
  storage, since it had to clear data stored during the scan pass to avoid
  duplicating it during the normal preprocessing pass.
* If you used "meta link", you should switch to either "meta openid" (for
  openid delegations), or tags (for internal, invisible links). I assume
  that nobody really used "meta link" for external, non-openid links, since
  the htmlscrubber ate those. (Tell me differently and I'll consider bringing
  back that support.)
* meta: Improved data storage.
* meta: Drop the hackish filter hook that was used to clear
  stored data before preprocessing, this hack was ugly, and broken (cf:
  liw's disappearing openids).
* aggregate: Convert filter hook to a needsbuild hook.
2007-12-16 15:56:09 -05:00
Joey Hess 2a6e353c20 improve use of seen hash 2007-10-30 22:56:13 -04:00
Joey Hess 5cbd86235e * Fix an aggregate plugin expiry bug. Over time, it's possible for the same
page name to be expired and reused for several distinct guids. When this
  happened, the expiry code counted each past guid that had used that page
  name as a currently existing page, and thus expired too many pages.
2007-10-30 22:50:44 -04:00
joey c0f8808232 * Various minor bug fixes for silly mistakes in the code, thanks to the
various reviewers.
2007-07-28 21:01:56 +00:00
joey 1612dce6bb * Add a htmlext config option, this can be used to make ikiwiki generate
html files with a different extension.
2007-07-25 01:16:53 +00:00
joey ee843c7523 improve messages 2007-05-21 03:00:45 +00:00
joey 10017feaec Clear state after performing aggregation, since it could hypothetically
change after the lock is dropped, and before the lock is regained by the
build process. The state will now be reloaded by the build process.
2007-05-21 02:59:06 +00:00
joey 82ac63d798 * Change the aggregate plugin's locking strategy. Now it defers loading state
until the wiki is building and already locked, unless it's aggregating.
  When aggregating, it does not wait for the lock if it cannot get it, and
  instead exits, to prevent aggregating processes from piling up.
2007-05-21 02:52:51 +00:00
joey ee1ad53c4c * pagespec_match() has changed to take named parameters, to better allow
for extended pagespecs. The old calling convention will still work for
  back-compat for now.
* The calling convention for functions in the IkiWiki::PageSpec namespace
  has changed so they are passed named parameters.
* Plugin interface version increased to 2.00 since I don't anticipate any
  more interface changes before 2.0.
2007-04-27 02:55:52 +00:00
joey 7ea3bfdb70 improve message 2007-04-23 19:32:21 +00:00
joey 71d2d93027 * Fix aggregator to not warn when a feed contains no body content at all.
* Work around bug #420636 by, if XML::Parser crashes, running the feed
  content though Encode::decode_utf8.
2007-04-23 18:36:44 +00:00
joey 13c7a51f1f * Optimise displaytime, trimming maybe 6% off the build time. 2007-04-10 01:49:16 +00:00
joey ae249feb07 another htmlpage call that is no longer appropriate 2007-04-01 20:55:39 +00:00
joey 02b9f69ba5 * Finally apply the index.html patch, with thanks to everyone who worked
on and supported creating it (especially Tumov). This adds a "usedirs"
  option that makes ikiwiki use foo/index.html instead of foo.html as
  output page names. It is not yet enabled by default.
2007-04-01 19:59:42 +00:00
joey 3a958f7196 fix error message to include url 2007-03-31 03:26:43 +00:00
joey 5b796b0bdb typo 2007-03-08 11:13:16 +00:00
joey 66a65b5ddd * Correct a deadlock that could occur in post-commit if the aggregate plugin
was enabled and tried to lock the already locked wiki.
2007-03-08 11:04:59 +00:00
joey 4d8b2d84d5 more i18n improvements 2007-03-07 12:04:45 +00:00
joey d4c61b7281 * Many changes to make ikiwiki very resistant to write failures
including out of disk space situations. ikiwiki should never leave
  truncated files, and if the error occurs during a web-based file edit,
  the user will be given an opportunity to retry.
  Inspired by the many ways Moin Moin destroys itself when out of disk. :-)
* Fix syslogging of errors.
2007-02-15 02:22:08 +00:00
joey 1379fa7402 feh 2007-01-14 05:01:46 +00:00
joey d253cef179 fix broken format string, manual unfuzzy 2007-01-14 04:28:50 +00:00
joey 5608aa078e * In the aggregator, check for and deal with post filenames that are
longer than the maximum file length.
2007-01-14 04:17:53 +00:00
joey f94e1dc11b fix whitespace 2007-01-03 04:19:51 +00:00
joey 912521ef07 * Initial work on internationalization of the program code. po/ikiwiki.pot
is available for translation.
* Export gettext() from IkiWiki module.
2006-12-29 04:38:40 +00:00
joey 35ee7e44a6 * Make sure to check for errors from every eval. 2006-11-08 21:03:33 +00:00
joey cb3f89f429 * Enable utf8 file IO in aggregate plugin.
* Fix some issues with the new registration form.
2006-11-08 20:13:59 +00:00
joey cc018082d0 typo 2006-11-06 04:27:29 +00:00
joey b78e93098b * Fix issue with aggregate plugin updating expired pages. 2006-11-04 19:20:24 +00:00
joey 5a82103526 * Implemented expiry options for aggregate plugin.
* Use precalculated backlinks info when determining if files need an update
  due to a page they link to being added/removed. Mostly significant if
  there are lots of pages.
* Remove duplicate link info when saving index. In some cases it could
  pile up rather badly. (Probably not the best way to deal with this
  problem.)
2006-11-01 05:41:37 +00:00
joey 9256360490 * Make the aggregate plugin emphasize error messages. 2006-10-13 18:31:18 +00:00
joey dae0f48e91 * Work on firming up the plugin interface:
- Plugins should not need to load IkiWiki::Render to get commonly
    used functions, so moved some functions from there to IkiWiki.
  - Picked out the set of functions and variables that most plugins
    use, documented them, and made IkiWiki export them by default,
    like a proper perl module should.
  - Use the other functions at your own risk.
  - This is not quite complete, I still have to decide whether to
    export some other things.
* Changed all plugins included in ikiwiki to not use "IkiWiki::" when
  referring to stuff now exported by the IkiWiki module.
* Anyone with a third-party ikiwiki plugin is strongly enrouraged
  to make like changes to it and avoid use of non-exported symboles from
  "IkiWiki::".
* Link debian/changelog and debian/news to NEWS and CHANGELOG.
* Support hyperestradier version 1.4.2, which adds a new required phraseform
  setting.
2006-09-09 22:50:27 +00:00
joey 6e273d60a6 shorten further 2006-09-07 23:55:21 +00:00
joey 2fd2646e66 remove the "0 new", it's not very useful 2006-09-07 23:54:37 +00:00
joey bd4cd357a1 * Add processed date to aggregate preprocessor directive status output. 2006-09-02 02:07:44 +00:00
joey e5b323a633 bugnum 2006-08-28 04:40:00 +00:00
joey 9556c4a889 * Fix a bug in the aggregator introduced last version that caused all
aggregator state to be lost during a non-aggregating build.
2006-08-17 23:09:55 +00:00
joey 53cf5ca9a3 * Fix an infinite loop in the aggregator when finding a unique filename. 2006-08-16 17:37:36 +00:00
joey df30eda283 * Add some locking to prevent 2 aggregators from running at once. 2006-08-16 17:32:30 +00:00
joey 4b36dee35a * The last release accidentially installed ikiwiki as ikiwiki.pl, now fixed.
* Add --version.
* Man page format fixups.
* Add a %pagecase which maps lower-case page names to the actual case
  used in the filename. Use this in bestlinks calculation instead of
  forcing the link to lowercase.
* Also use %pagecase in various other places that want to check if a page
  with a given name exists.
* This means that links to pages with mixed case names will now work,
  even if the link is in some other case mixture, and mixed case pages
  should be fully supported throughout ikiwiki.
* Recommend rebuilding wikis on upgrade to this version.
2006-08-13 02:03:43 +00:00
joey 2ea8fbe2d9 misc changes 2006-08-05 21:15:50 +00:00
joey a1c7308b7d use htmlpage a couple of places instead of hardcoding the extension 2006-08-04 02:41:08 +00:00
joey e36f6b6a99 oops 2006-08-04 00:59:00 +00:00
joey c5919df5f3 * Make aggregator save permalinks and author name to pages as metadata.
* Add permalink and author support to meta plugin, affecting RSS feeds
  and blog pages.
* Change titlepage() to encode utf-8 alnum characters. This is necessary
  to avoid UTF-8 creeping into filenames in urls. (There are still
  some other ways that it can get in.)
2006-08-04 00:01:51 +00:00
joey e8b39b0941 * Try to handle relative links in aggregated feeds. However,
the current support is a crude hack due to limitations of XML::Feed:
  xml:base is not supported, neither is Content-Location. And of course,
  relative links in RSS feeds are ill-undefined..
2006-08-03 21:50:47 +00:00
joey 7ac32ec4d6 oops 2006-08-02 00:52:47 +00:00
joey 2794d7ef5a * Renamed GlobLists to PageSpecs.
* PageSpecs can now include nested parens, "and", and "or". This remains
  backwards compatible to the old GlobList format. It's implemented by
  treating the GlobList as a very limited microlanguage that is transformed
  to perl code that does the matching.
* The old GlobList format is deprecated, and I encourage users to switch to
  using the new PageSpec format. Compatability with the old format will be
  removed at some point, possibly by 2.0.
* Wiki rebuild needed on upgrade to this version due to PageSpec change.
* Add support for creation_month and creation_year to PageSpec.
  Closes: #380680
* Changes to index file encoding.
2006-08-02 00:14:31 +00:00
joey 2604c3217e one more change; use the page that pulls in a feed as the anchor for the
default directory
2006-07-31 01:05:22 +00:00
joey 953ffa1fe3 fix 2006-07-31 00:59:54 +00:00
joey 8883001615 * Change meta tags to use html entity-escaped text for values, so that
quotes and such can be represented in title tags.
* Depend and build-depend on HTML::Parser for HTML::Entities which is used
  for the above.
* Make --rebuild also cause --aggregate to re-download and write aggregated
  pages.
* Avoid outputting duplicate meta info.
* Include title metadata on aggregated posts for capitalised and un-munged
  titles.
2006-07-30 22:58:48 +00:00
joey 73e6f81ef3 escape slashes in page names 2006-07-30 21:51:37 +00:00
joey 7e35ad2cd9 when autogenerating a dir naem, include "feed/" in it 2006-07-30 07:30:08 +00:00
joey 9ae54f904a fix 2006-07-30 07:17:44 +00:00
joey 22adebe8ad need to lower-case pages, sigh.. 2006-07-30 07:07:20 +00:00
joey 33aab4802e erase old tags 2006-07-30 06:57:42 +00:00
joey b220a11008 stop abusing titlepage for internal encoding (esp since it doesn't work) 2006-07-30 06:55:33 +00:00
joey b0885a7278 fix encoding of some stuff 2006-07-30 06:20:58 +00:00
joey dcaa4b013e more robust and faster handling of feedurls 2006-07-30 06:14:44 +00:00
joey b9693d13ef robustness and etc 2006-07-30 05:44:17 +00:00
joey c23e8009bf deal with title-less feeds 2006-07-30 05:29:03 +00:00
joey 68d7fbfabd md5 can't handle wide chars 2006-07-30 05:17:59 +00:00
joey c5b02885e9 fixes, and make dir optional 2006-07-30 05:14:35 +00:00
joey e49ff966a3 * ikiwiki can now download and aggregate feeds with its new aggregate
plugin, so it's possible to implement a Planet using ikiwiki!
* --setup --refresh no longer rebuilds wrappers. Use --setup --refresh
  --wrappers to do that.
* Add %IkiWiki::forcerebuild to provide a way for plugins like aggregate
  to update pages that haven't changed on disk.
2006-07-30 04:31:08 +00:00