65 lines
2.5 KiB
Markdown
65 lines
2.5 KiB
Markdown
The [[plugin/aggregate]] plugin's locking is a suboptimal.
|
|
|
|
There should be no need to lock the wiki while aggregating -- it's annoying
|
|
that long aggregate runs can block edits from happening. However, not
|
|
locking would present problems. One is, if an aggregate run is happening,
|
|
and the feed is removed, it could continue adding pages for that feed.
|
|
Those pages would then become orphaned, and stick around, since the feed
|
|
that had created them is gone, and thus there's no indication that they
|
|
should be removed.
|
|
|
|
To fix that, garbage collect any pages that were created by
|
|
aggregation once their feed is gone.
|
|
|
|
Are there other things that could happen while it's aggregating that it
|
|
should check for?
|
|
|
|
Well, things like the feed url etc could change, and it
|
|
would have to merge in such changes before saving the aggregation state.
|
|
New feeds could also be added, feeds could be moved from one source page to
|
|
another.
|
|
|
|
Merging that feed info seems doable, just re-load the aggregation state
|
|
from disk, and set the `message`, `lastupdate`, `numposts`, and `error`
|
|
fields to their new values if the feed still exists.
|
|
|
|
----
|
|
|
|
Another part of the mess is that it needs to avoid stacking multiple
|
|
aggregate processes up if aggregation is very slow. Currently this is done
|
|
by taking the lock in nonblocking mode, and not aggregating if it's locked.
|
|
This has various problems, for example a page edit at the right time can
|
|
prevent aggregation from happening.
|
|
|
|
Adding another lock just for aggregation could solve this. Check that lock
|
|
(in checkconfig) and exit if another aggregator holds it.
|
|
|
|
----
|
|
|
|
The other part of the mess is that it currently does aggregation in
|
|
checkconfig, locking the wiki for that, and loading state, and then
|
|
dropping the lock, unloading state, and letting the render happen. Which
|
|
reloads state. That state reloading is tricky to do just right.
|
|
|
|
A simple fix: Move the aggregation to the new 'render' hook. Then state
|
|
would be loaded, and there would be no reason to worry about aggregating.
|
|
|
|
Or aggregation could be kept in checkconfig, like so:
|
|
|
|
* load aggregation state
|
|
* get list of feeds needing aggregation
|
|
* exit if none
|
|
* attempt to take aggregation lock, exit if another aggregation is happening
|
|
* fork a child process to do the aggregation
|
|
* load wiki state (needed for aggregation to run)
|
|
* aggregate
|
|
* lock wiki
|
|
* reload aggregation state
|
|
* merge in aggregation state changes
|
|
* unlock wiki
|
|
* drop aggregation lock
|
|
* force rebuild of sourcepages of feeds that were aggregated
|
|
* exit checkconfig and continue with usual refresh process
|
|
|
|
[[done]]
|