This commit adds a new bridge for http://www.instructables.com. This bridge
currently supports fetching content by category (all categories available 200+),
using available filters (featured, recent, popular, views, contest winners).
Adds duration limits (minimum duration, maximum duration) for all
modes (user/id/playlist/search). Duration limits are optional, so
existing subscriptions don't break.
The limits are specified by two separate parameters, each of which
is optional:
- `&duration_min=` (minimum duration in minutes, default: -1)
- `&duration_max=` (maximum duration in minutes, default: INF)
If duration limits are specified in either user, id or playlist mode,
the bridge defaults to fetching data from HTML intead of XML feeds,
which requires more bandwidth and takes longer, because each video is
loaded individually!
References #670
The previous context is now labeled 'User', while the new context is
labeled 'Group'. The existing code was not changed, instead new group*
functions were implemented to handle groups.
The general principle of capturing groups is the same as done for users
with adjustments to account for different HTML structures.
Captcha responses are currently not supported for groups! There doesn't
seem to be a way to trigger them consistently, which makes it hard to
handle them properly.
Features of the group context:
- The feed title is based on the group name
- The group URI used for capturing is returned for the feed URI
- Author names and timestamps are reproduced from the source
- Post titles are reproduced from the source if they exist, otherwise
the title is build manually from the author name and the content
- Original contents are included with the feed
- All images are attached as enclosures as well
Closes #
Supports latest news and profiling a given ETF in Englisch, German
or Italian language. Cover images are attached as enclosures and not
as part of the content.
News:
Optionally loads the full article for each news item. Some articles
may include scripts to provide interactive graphs. These scripts are
removed as they would be rendered as pure text and a message is shown
instead: "[Content removed! Visit site to see full contents!]"
Profile:
Optionally includes the ETF strategy and description.
isoHunt has discontinued services due to legal reasons and is now
accessible via https://isohunts.to
While it is certainly possible to rewrite the bridge to fetch some
information from the new site, it wouldn't be able to provide as
much functionality as before. This is due to isoHunt having removed
all searching and filtering options, only providing static HTML pages
for general categories (anime, movies, etc...). Those pages, however,
are heavily broken.
Unless someone is interested in monitoring the general categories
the effort of upgrading the bridge to the new site is not worth taking
time for.
Users of isoHunt are asked to make use of their client application,
as they don't provide online services anymore (it's now in the darknet)
Here is the statement from isoHunt:
"Due to hard regulations and security issues for bittorrent users, we
have moved into a more secure and even faster district of the internet!
[...]
Torrent Downloads have a high risk of getting legal problems. That is
why we do not offer torrentfiles any more. [...]"
-- source: https://isohunts.to
Allows users to paste facebook links as user name. The link must contain
the correct host (www.facebook.com) and a valid path (/user-name/...).
The first part of the path is used for the user name. Errors are returned
in case something went wrong.
References #706
Reviews are provided the same way as summary posts and therefore returned
as separate feed item for each review. This commit adds a new option
'&skip_reviews=on' to skip reviews entirely.
References #706
The media_type parameter was recently replaced by media_type_u (for
user mode) and media_type_h (for hashtag mode). This was necessary
in order to add the media type 'story' only for the user mode.
"The reason for that is that RSS-Bridge supports multiple parameters
with the same name if and only if they contain the exact same value.
Here, hashtags don't have stories, so it would not be possible to
pass "story" as a parameter. This is a design mistake that I made
when I added support for hashtags."
-- 8770c87389 (r28871502)
However as pointed out this change breaks existing feeds as the
parameter name is no longer compatible to previous implementations.
This commit changes the implementation to provide the old media_type
parameter globally and check for invalid options on each request. If
a user uses the 'story' option in history mode the bridge returns a
client error.
references 8770c87
references #694fixes#696fixes#699fixes#701
* [DealabsBridge] Follow site changes, fix unhandled case
- Fixed the case where no discount was shown
- Changed some CSS class to follow the website changes
Now, entry title is optionnal and may be found in h3 HTML element.
Entry content is mandatory and may be found in div[class="item-content"] HTML element.
Moreover, the title may contain simple quotes (here, encoded) so the bridge have to decode first to apply format library function. In case we don't do that, the format function double encode the quote and something like ' could appear.
* [VkBridge] Convert special HTML entities to characters in pageName
* [VkBridge] Generate feed item title
* [VkBridge] Remove double backslashes in feed item link
* [VkBridge] Unpin post if pinned
* [VkBridge] Mark reposted messages
* [VkBridge] Correct external link parsing
* [VkBridge] Added article parsing
* [VkBridge] Added video parsing
* [VkBridge] Added photo parsing
* [VkBridge] Added album link parsing
* [VkBridge] Added one more external link selector
* [VkBridge] Using array of link selectors to remove
* [VkBridge] Added document parsing
* [VkBridge] Added sign parsing
* [VkBridge] Fixed incorrect sorting with pinned item
* [VkBridge] More methods to parse documents
* [VkBridge] Save fallback if page name element not found
* [VkBridge] Using post signed as feed item author
* [VkBridge] Fixed document link
* [VkBridge] Coding policy fixes
NotAlways right found it necessary to remove their RSS feeds recently. This is a *simple* bridge to grab the ones on the front page. It allows you to filter the articles based on their classification (right, working, romantic, related, learning, friendly, hopeless, unfiltered, or all).
The limit was used to specify the number of pages to return from a given
topic which resulted in the number of returned items variing between one
and however many entries are listed on one page.
This commit changes the implementation for the limit to keep loading more
pages until the specified limit is reached. Excessive elements are removed
in order to return the exact amount of items specified by the limit.
This behavior is closer to how other bridges are implemented and makes it
more natural to use without being too confusing. Existing queries must be
updated to account for the new limit.
References #657
Requesting a username with a leading slash would cause error 500
because the requested URI would contain two slashes in a row.
For example username "/test" would result in:
https://facebook.com//test
References #628
The author name is parsed by searching a string within the entire
HTML document:
$author = $html->innertext;
$author = substr($author, strpos($author, '"author=') + 8);
$author = substr($author, 0, strpos($author, '\u0026'));
This solution will return big portions of the HTML document if
the strpos function returns zero (not found).
This commit replaces the previous implementation by searching for
a specific script tag and making use of the JSON data inside it.
References #580
This bridge returns feeds for any URI that is compatible with the
IPB implementation (currently 4.x). Older versions might work, but
there is no guarantee.
Only forum and topic URIs are supported!
The bridge automatically checks if natural feeds are available (by
adding '.xml' to the URI). If so the feed is returned. Otherwise
the bridge will attempt to identify the content type and build a
feed accordingly.
Valid URIs are forums and topics. For forums the first page is
returned, for topics the last one. Elements are ordered such that
the latest entry is returned first (oldest-to-newest)
The optional parameter '&limit=' specifies how many pages should
be loaded (default: 1). Topics are loaded in reverse order.
=> Does not work with forums!
Images are provided as enclosures and scaled to a max-size of
400x400 pixels by default (Except for natural feeds).
The content is filtered before being returned:
- Unnecessary tags are removed (iframes, etc...)
- Styles for blockquotes are restored (grey background)
Closes#507
- If no there are no results, an explicit message is now returned
- Commas are now following the coding policy
- Lines are no longer more than 80 chars when possible
Added additional instructions for: 'username search' and 'category
search' next to instructions for 'keyword search'.
Changed variable name from underscore to camelCase.
Videos that are part of a playlist have the playlist ID encoded in
the URI. When loading the video info the page contents change unex-
pectedly due to the playlist being part of the page.
This removes any trailing parameters from the video ID in order to
ensure only pure videos are loaded at all times.
On Oct 27, 2016 the discontinuation of Vine was announced:
https://medium.com/@vine/important-news-about-vine-909c5f4ae7a7
"Today, we are sharing the news that in the coming months we’ll be
discontinuing the mobile app."
https://vine.co/ is still online, but has been put into an archive
indefinitely. As the site does not allow further uploads, this
bridge serves no further purpose.
Tags are embedded in the 'title' attribute instead of 'alt' as
defined by the ancestor (DanbooruBridge).
The 'title' attribute also contains statistics data ('score:...',
'rating:...') that is now filtered by a custom implementation of
the 'getTags' function (elements that contain a colon are removed)
Closes#560
All formats except HTML return & instead of & in URLs causing
all links with parameters (...&id=...) to break.
Facebook does not return valid HTML URIs but instead provides them
with all special characters encoded (like using htmlspecialchars).
This seems to be related to the page being build almost entirely of
script blocks.
This commit adds htmlspecialchars_decode() to URI and content to
reverse the encoding.
References #550
- Do not add spaces after opening or before closing parenthesis
// Wrong
if( !is_null($var) ) {
...
}
// Right
if(!is_null($var)) {
...
}
- Add space after closing parenthesis
// Wrong
if(true){
...
}
// Right
if(true) {
...
}
- Add body into new line
- Close body in new line
// Wrong
if(true) { ... }
// Right
if(true) {
...
}
Notice: Spaces after keywords are not detected:
// Wrong (not detected)
// -> space after 'if' and missing space after 'else'
if (true) {
...
} else{
...
}
// Right
if(true) {
...
} else {
...
}
This replaces the 'novideo' parameter with 'media_type' in order
to filter for specific content types. Currently supported:
- 'all': Returns all posts (default)
- 'video': Returns only posts including videos
- 'novideo': Returns only posts that don't include videos
References #553
This adds a new option 'novideo' that can be set to 'on' or 'off'
in order to skip posts that include facebook videos (does not work
for linked videos like YouTube). This option is 'off' by default.
References #533
Adds a new option 'media_type' to select from three choices:
- 'all' (Both): Returns pictures and videos (default choice)
- 'picture': Returns only pictures
- 'video': Returns only videos
References #553
This fixes broken URIs in the output data caused by duplicate domain
names caused by sites (descendant class Delbooru) providing absolute
URIs instead of relative ones.
References #552
The title attribute includes tags (anchors) instead of raw text.
While this works fine in a browser, using a raw format like plain-
text or json breaks with expected behavior.
This commit changes the order in which functions are applied. By re-
moving anchors AFTER fixing the title, the final result does not
include tags and the title is still fixed.
This bug was introduced by d81b61ccfa
References: #546
Twitter now requires login to access "Tweets & Replies" which
breaks feeds using the default behavior. Using the "Without
replies" option still works.
This commit makes the "Without replies" option default. That way
existing feeds will return contents again. The parameter can still
be checked but its status has no effect anymore.
Notice: The parameter should not be removed as that would cause
any feed using the parameter to stop working because of "Invalid
parameter"
References #544
This commit improves readability of tweets by adding spaces before
anchors in the text.
- Hide "invisible" tags which were not rendered hidden because of
missing CSS.
- Fix spacing between anchors
Reported via #539
If no accepted languages are specified Facebook will guess your
language. This guess can go horribly wrong if your server does not
provide origin information.
This adds a context header with language information when retrieving
page contents. The accepted languages are read from the list of
accepted languages specified by the web browser of the requester.
References #530
This improves the translation of regular text to anchors by adding
support for additional characters '?&=-_' to fix common URLs.
Notice: The regex pattern is by no means complete. That means it is
likely to break in the future. More sophistiated solutions however
are insanely complex. See: http://stackoverflow.com/a/190405
References #520
Twitter provides an easy way to receive various image sizes based
on the same image URI:
https://dev.twitter.com/overview/api/entities-in-twitter-objects
We support different sizes: thumb, small, medium and large. The
media_url defaults to medium but you can retrieve the media in
different sizes by appending a colon + the size key (for example:
https://pbs.twimg.com/media/A7EiDWcCYAAZT1D.jpg:thumb).
-- Twitter Developer Documentation
TwitterBridge now makes use of this feature in order to provide
thumbnail images in the content and original sized images as
enclosures.
References #526
Previously feed contents were rendered as one block of text with
no structure. This brings back the structure of original video
descriptions and makes links in the description work again.
References #520
This adds new features to show quotes and pictures in feeds.
Quotes will show up on top of a tweet and are separated from
the quoting feed by a horizontal line.
Pictures that are embedded in the tweet will be captured and
attached to the feed using enclosures. By default the picture
will also be shown in the feed itself. This can be disabled
using the option '&noimg=on'
Some codes are now split into separate functions so they can be used
for tweets and quotes alike.
Changed behaviour of the checkbox to use the custom parser when
active. That way if the parameter is missing the default value
applies and the feed returns from the provided RSS
Reported via #498
Previously summary posts were ignored which resulted in the last
two posts not showing up in the feed (the latest two are shown in
the summary post).
Now summary posts are treated like regular posts, returning them
as part of the regular feed.
References #502, #505
Due to breaking DOM changes this bridge required re-implementation.
With this fix the brige will make use of the JSON data embedded in
the returned HTML. The content returned for all contexts is similar
with only a few differences due to limitations of the JSON.
Feeds returned for a given username and board will by default make
use of the provided RSS feed instead of using the custom filter.
This bahaviour can be changed by setting the optional parameter
'&r=off' (on by default)
Notice: The JSON data for userdata and search results is very
different, so two functions were implemented to account for that.
References #498
* const DOMAIN is not supported, it must be const URI
* strtotime should be used instead of date_parse in order to
receive a valid integer
* Some small readability enhancement
This bridge generates feeds for a given search term, optionally
adds the picture to the content and allows for additional query
extensions (GET parameters) to be passed to the bridge. That
way custom filter can be applied without the need to reproduce
them in this bridge (they got a lot!)
Etsy provides a good set of feeds as described here:
https://www.etsy.com/help/article/100
(so there is no need to include them here)
References #492
This bridge will fetch contents from https://wikileaks.org
Available options are:
- Category: Defines a list of categories to select from
- Show teaser: Defines whether to show the teaser or not
Notice: Feeds provided by WikiLeaks do not work, see
https://wikileaks.org/wiki/RSSCloses#489
User names can either be an ID (series of numbers), or an actual
name, where the name always starts with a '+'.
This commit adds a check for automatically fixing provided user
names which are missing the '+'.
- Do not force language via HTTP header
The header enforced the language to be french which caused problems parsing
the exact time due to spellings (strtotime cannot work with 'semaines'). If
further issues are experienced try forcing en-us instead.
=> This should really be done in the RSS-Bridge core
- Fix loading problems due to pinned articles
Pinned articles do not provide a timestamp. Building the timestamp step-by-step
solves parsing errors.
- Use class names instead of CSS paths
CSS paths change based on the article. Pinned articles provide a different
DOM structure which caused parsing errors.
Reported via #499
Adds a new bridge to fetch contents from https://usbeketrica.com/
Feeds are build from cards displayed on the front page
This bridge provides two options:
- limit: Defines how many articles are returned
- fullarticle: Defines whether or not the full article is retured
Requested via #457
This bridge was broken due to DOM changes. This commit fixes
most of the broken code. Hashtags do no longer work because
they are no longer supported/provided.
The timing might be off as the source only provides a rough
relative value like '1 hour' or '1 year'.
Closes#485
This bridge returns feeds for each section (via given separator)
from a given MoinMoin compatible wiki.
The separator can be any tag of the following:
- h1
- h2
- h3
- li
- a
The number of items returned can be specified.
For anchor tags (a) the bridge can optionally follow the anchor to
the linked page and return it as content.
This bridge is a mashup of the existing FlickrExploreBridge by sebsauvage
and FlickrTagBridge by erwang. It provides the same functionality as one
single bridge.
Instead of utilizing API requests for each element, the information
is now read directly from the source page, which provides information
as JSON data embedded in a script block.
The author name is returned for each element.
Improves the title and optionally adds the description if available
This adds a bridge for bugzilla.kernel.org to provide feeds for
bug comments without the need of registering an email address.
This implementation makes use of the print preview feature that
reduces bandwidth by a small margin.
Provides options to specify the number of comments to return as
well as the sorting order (latest first or oldest first)
Maintainer should be set for all bridges. Using git blame to
determine who provided the most code to the files. This is
obviously not a good solution, feel free to insert own names
getURI and getName should fall back to parent::getURI or
parent::getName respectively if it cannot build propper
return values.
Functions defined by bridges should be made private to
prevent confusion with inherited functions
Elements are now put into separate JSON containers. As such
assignment changed from : to = and as only one container
is present in each element, the final , is omitted.
JSON data is html encoded and requires decoding before decoding
via json_decode.
Hi, I created a Bridge for ReadComics.tv website, I put myself as "maintainer" but I'm not sure if this is the way you're doing it!
If there is a need to improve/change things, please tell me!
I made some changes to returne category and uploader feed. I also changed the URI to the magnet link to be able to use the feed in a torrent client. As discussed here (https://github.com/RSS-Bridge/rss-bridge/issues/412), I'd rather use the <torrent:magnetURI> item but it's not possible with RSS-Bridge ATM.
If I find time to work on it I'll try to add combination possibilities: search term in a certain category or for a specific uploader
Hope my changes will be appreciated!
Methods displayBridgeCard, sanitize, defaultImageSrcTo are now
functions in lib/html.php
getHelperButtinsFormat and getFormHeader are now anonymous functions
defined in displayBridgeCard
Signed-off-by: Pierre Mazière <pierre.maziere@gmx.com>
- returnError, returnServerError, returnClientError ,debugMessage are
moved to lib/error.php
- getContents, getSimpleHTMLDOM, getSimpleHTMLDOMCached are moved to
lib/contents.php
Signed-off-by: Pierre Mazière <pierre.maziere@gmx.com>
The data is no longer provided in HTML upon request,
but rather encoded as JSON in a SCRIPT section and
decoded via Javascript on the client side. The bridge
now decodes the data and returns valid feeds again.
The specific content filtering used in these bridges will need to
be reintegrated later as part of the bridge or as part of the
WordPressBridge if they are considered generic enough filters,
such as the already existing WordPressBridge <script> removal filter.
Signed-off-by: Pierre Mazière <pierre.maziere@gmx.com>
This breaks compatibility with previous versions of VkBridge (which
seems broken anyway).
Bridges should never use full URIs as inputs since their validation will
always be more complicated, hence prone to security issues,
than rebuilding a clean URI from simple validated inputs.
This breaks compatibility with previous versions of NoveUpdatesBridge.
Bridges should never use full URIs as inputs since their validation will
always be more complicated, hence prone to security issues,
than rebuilding a clean URI from simple validated inputs.
Signed-off-by: Pierre Mazière <pierre.maziere@gmx.com>
This breaks compatibility with previous versions of FourChanBridge.
Bridges should never use full URIs as inputs as their validation will
always be more complicated, hence prone to security issues,
than rebuilding a clean URI from simple validated inputs.
Signed-off-by: Pierre Mazière <pierre.maziere@gmx.com>
