ikiwiki/doc/plugins/aggregate/discussion.mdwn

I'm trying to set up a [planet of my users' blogs](http://help.schmonz.com/planet/). I've enabled the aggregate, meta, and tag plugins (but not htmltidy, that thing has a gajillion dependencies). `aggregateinternal` is 1. The cron job is running and I've also enabled the webtrigger. My usage is like so:

    \[[!inline pages="internal(planet/*) show=0"]]
    
    \[[!aggregate
    name="Amitai's blog"
    url="http://www.schmonz.com/"
    dir="planet/schmonz-blog"
    feedurl="http://www.schmonz.com/atom/"
    expirecount="2"
    tag="schmonz"
    ]]
    
    \[[!aggregate
    name="Amitai's photos"
    url="http://photos.schmonz.com/"
    dir="planet/schmonz-photos"
    feedurl="http://photos.schmonz.com/main.php?g2_view=rss.SimpleRender&g2_itemId=7"
    expirecount="2"
    tag="schmonz"
    ]]


(and a few more `aggregate` directives like these)

Two things aren't working as I'd expect:

1. `expirecount` doesn't take effect on the first run, but on the second. (This is minor, just a bit confusing at first.)
2. Where are the article bodies for e.g. David's and Nathan's blogs? The bodies aren't showing up in the `._aggregated` files for those feeds, but the bodies for my own blog do, which explains the planet problem, but I don't understand the underlying aggregation problem. (Those feeds include article bodies, and show up normally in my usual feed reader rss2email.) How can I debug this further? --[[schmonz]]

> I only looked at David's, but its rss feed is not escaping the html
> inside the rss `description` tags, which is illegal for rss 2.0. These
> unknown tags then get ignored, including their content, and all that's
> left is whitespace. Escaping the html to `<` and `>` fixes the
> problem. You can see the feed validator complain about it here:
> <http://feedvalidator.org/check.cgi?url=http%3A%2F%2Fwww.davidj.org%2Frss.xml>
> 
> It's sorta unfortunate that [[cpan XML::Feed]] doesn't just assume the
> un-esxaped html is part of the description field. Probably other feed
> parsers are more lenient. --[[Joey]]

>> Thanks for the quick response (and the `expirecount` fix); I've forwarded it to David so he can fix his feed. Nathan's Atom feed validates -- it's generated by the same CMS as mine -- so I'm still at a loss on that one. --[[schmonz]]

>>> Nathan's feed contains only summary elements, with no content elements.
>>> This is legal according to the Atom spec, so I've fixed ikiwiki to use
>>> the summary if no content is available. --[[Joey]]

>>>> After applying your diffs, blowing away my cached aggregated stuff, and running the aggregate cron job by hand, the resulting planet still doesn't have Nathan's summaries... and the two posts from each feed that aren't being expired aren't the two newest ones (not sure what the pattern is there). Have I done something wrong? --[[schmonz]]

>>>>> I think that both issues are now fixed. Thanks for testing.
>>>>> --[[Joey]]

>>>>>> I can confirm, they're fixed on my end. --[[schmonz]]

New bug: new posts aren't getting displayed (or cached for aggregation). After fixing his feed, David posted a new item today, and the aggregator is convinced there's nothing to do, whether by cronjob or webtrigger. I verified that it wasn't another problem with his feed by adding another of my ikiwiki's feed to the planet, running the aggregator, posting a new item, and running the aggregator again: no new item. --[[schmonz]]