1 <h2>Retriever Plugin Help</h2>
3 This plugin replaces the short excerpts you normally get in RSS feeds
4 with the full content of the article from the source website. You
5 specify which part of the page you're interested in with a set of
6 rules. When each item arrives, the plugin downloads the full page
7 from the website, extracts content using the rules, and replaces the
11 There's a few reasons you may want to do this. The source website
12 might be slow or overloaded. The source website might be
13 untrustworthy, in which case using Friendica to scrub the HTML is a
14 good idea. You might be on a LAN that blacklists certain websites.
15 It also works neatly with the mailstream plugin, allowing you to read
16 a news stream comfortably without needing continuous Internet
20 However, setting up retriever can be quite tricky since it depends on
21 the internal design of the website. This was designed to make life
22 easy for the website's developers, not for you. You'll need to have
23 some familiarity with HTML, and be willing to adapt when the website
24 suddenly changes everything without notice.
26 <h3>Configuring Retriever for a feed</h3>
28 To set up retriever for an RSS feed, go to the "Contacts" page and
29 find your feed. Then click on the drop-down menu on the contact.
30 Select "Retriever" to get to the retriever configuration.
33 The "Include" configuration section specifies parts of the page to
34 include in the article. Each row has three components:
37 <li>An HTML tag (e.g. "div", "span", "p")</li>
38 <li>An attribute (usually "class" or "id")</li>
39 <li>A value for the attribute</li>
42 A simple case is when the article is wrapped in a "div" element:
46 <div class="main-content">
47 <h2>Man Bites Dog</h2>
48 <img src="mbd.jpg">
50 Residents of the sleepy community of Nowheresville were
51 shocked yesterday by the sight of creepy local weirdo Jim
52 McOddman assaulting innocent local dog Snufflekins with his
60 You then specify the tag "div", attribute "class", and value
61 "main-content". Everything else in the page, such as navigation
62 panels and menus and footers and so on, will be discarded. If there
63 is more than one section of the page you want to include, specify each
64 one on a separate row. If the matching section contains some sections
65 you want to remove, specify those in the "Exclude" section in the same
69 Once you've got a configuration that you think will work, you can try
70 it out on some existing articles. Type a number into the
71 "Retrospectively Apply" box and click "Submit". After a while
72 (exactly how long depends on your system's cron configuration) the new
73 articles should be available.
77 You can leave the attribute and value blank to include all the
78 corresponding elements with the specified tag name. You can also use
79 a tag name of "*", which will match any element type with the
80 specified attribute regardless of the tag.
83 Note that the "class" attribute is a special case. Many web page
84 templates will put multiple different classes in the same element,
85 separated by spaces. If you specify an attribute of "class" it will
86 match an element if any of its classes matches the specified value.
90 <div class="article breaking-news">
93 In this case you can specify a value of "article", or "breaking-news".
94 You can also specify "article breaking-news", but that won't match if
95 the website suddenly changes to "breaking-news article", so that's not
99 One useful trick you can try is using the website's "print" pages.
100 Many news sites have print versions of all their articles. These are
101 usually drastically simplified compared to the live website page.
102 Sometimes this is a good way to get the whole article when it's
103 normally split across multiple pages.
106 Hopefully the URL for the print page is a predictable variant of the
107 normal article URL. For example, an article URL like:
110 http://www.newssite.com/article-8636.html
113 ...might have a print version at:
116 http://www.newssite.com/print/article-8636.html
119 To change the URL used to retrieve the page, use the "URL Pattern" and
120 "URL Replace" fields. The pattern is a regular expression matching
121 part of the URL to replace. In this case, you might use a pattern of
122 "/article" and a replace string of "/print/article". A common pattern
123 is simply "$", used to add the replace string to the end of the URL.
125 <h3>Background Processing</h3>
127 Note that retrieving and processing the articles can take some time,
128 so it's done in the background. Incoming articles will be marked as
129 invisible while they're in the process of being downloaded. If a URL
130 fails, the plugin will keep trying at progressively longer intervals
131 for up to a month, in case the website is temporarily overloaded or
134 <h3>Retrieving Images</h3>
136 Retriever can also optionally download images and store them in the
137 local Friendica instance. Just check the "Download Images" box. You
138 can also download images in every item from your network, whether it's
139 an RSS feed or not. Go to the "Settings" page and
140 click <a href="$config">"Plugin settings"</a>. Then check the "All
141 Photos" box in the "Retriever Settings" section and click "Submit".
143 <h2>Configure Feeds:</h2>
145 {{ for $feeds as $feed }}
146 {{ inc contact_template.tpl with $contact=$feed }}{{ endinc }}