* core: Add bridge parameter auto detection
This adds a new 'detect' action which accepts a URL from which an
appropriate bridge is selected and relevant parameters are extracted.
The user is then automatically redirected to the selected bridge.
For example to get a feed from: https://twitter.com/search?q=%23rss-bridge
we could send a request to:
'/?action=detect&format=Atom&url=twitter.com/search%3Fq%3D%2523rss-bridge'
which would redirect to:
'/?action=display&q=%23rss-bridge&bridge=Twitter&format=Atom'.
This auto detection happens on a per-bridge basis, so a new function
'detectParameters' is added to BridgeInterface which bridges may implement.
It takes a URL for an argument and returns a list of parameters that were
extracted, or null if the URL isn't relevant for the bridge.
* [TwitterBridge] Add parameter auto detection
* [BridgeAbstract] Add generic parameter detection
This adds generic "paramater detection" for bridges that don't have any
parameters defined. If the queried URL matches the URI defined in the
bridge (ignoring https://, www. and trailing /) an emtpy list of parameters is
returned.
* [ZoneTelechargementBridge] Switch to the new Website
The website zone-telechargement1.org decided that he will be using a new
domain at the end of november :
https://www.annuaire-telechargement.com/
The bridge uses the new domain but still uses the same filename and
class name to keep the existing feed working.
Replaces 'debugMessage' by specialized debug function 'Debug::log'.
This function takes the same arguments as the previous 'debugMessage'.
A separate Debug class allows for further optimization and separation
of concern.
This commit adds filters to remove embedded videos and view counts from
all posts. This doesn't remove the preview image for videos, which are
embedded separately.
Hidden elements are used for error conditions and generally made
visible using JavaScript. Since RSS-Bridge doesn't support JS, these
error messages are shown in the final feed. For example:
"It looks like you may be having problems playing this video. If so,
please try restarting your browser."
This commit removes all hidden elements to prevent error messages being
added to the feed.
- "It looks like you may be having problems playing this video. If so,
please try restarting your browser."
FB includes origin information (i.e. "YOUTUBE.COM") as well as
descriptions with embedded media (images and video).
These details are currently being removed by the bridge.
This commit changes implementation to only remove origin information
and keep the media description in place. The media description consists
of two elements - title and description. The title provided by FB is
included in an anchor, which gets replaced by a paragraph with the
same contents to improve readability.
References #912
Move CACHE_DIR from index.php to /lib/RssBridge.php and change name
to PATH_CACHE.
PATH_CACHE is one of the core paths of RSS-Bridge and should therefore
be defined in the core file RssBridge.php.
`defaultLinkTo` makes anchors point to the correct path which broke
parsing because it expected href to start with `#gistcomment`.
This commit changes the implementation to make `defaultLinkTo` point
to the correct page (using `getURI` instead of `self::URI`) and search
with `*=` instead of `^=`.
Instagram returns "HTTP/1.1 301 Moved Permanently" on each request
to "https://instagram.com/" because the correct location is
"https://www.instagram.com/".
Instagram will respond with "HTTP/1.1 301 Moved Permanently" if the
URI for the requested user doesn't end with a slash.
Notice: This is only a minor enhancement to prevent error 301 from
happening. The previous version worked fine as is.
Using the classical www.zone-telechargement1.org as base URL, the bridge will
always be redirected to the actual wwX.zone-telechargement1.org final URL. This
makes the bridge more robust to URL changes.
This commit collects the original contents from a different
tag to prevent this issue. The root cause is unknown but closely
related to the regex.
References #877
The function 'defaultLinkTo' applied to the source HTML does break
regex matches later in the bridge. We need to apply the function
right before adding the contents to the item for the bridge to work
properly.
References #856
- Base URL updated
- Show name has different styles on the Website, use another way to get the show name
- Entry URIs are now unique to make sure RSS readers don't treat episodes as duplicates
- No more new lines in the feed or item title
- Entry URIs are unique to make sure RSS readers don't treat episodes as duplicates
- Titles are unique to make sure RSS readers don't treat streams and downloads as duplicates
The bridge has been expanded to better cover the whole GQ magazine.
It should support all countries (provided they all use the same absurdly shitty publication system).
It is guaranteed to be only tested with sexactu articles (that I now obtain by loading Maïa Mazaurette author page).
This commit adds a new optional parameter 'limit' which can be used
to limit the number of items returned by this bridge (i.e. '&limit=10')
As requested in #669
The URI "https://facebook.com/username?_fb_noscript=1" returns two
posts per user. Some profiles, however, are very active, causing the
bridge to miss items if more than two posts are send within the cache
duration (5 minutes).
The alternative suggested in #669 is to use a different URI:
"https://facebook.com/pg/username/posts?_fb_noscript=1"
While the contents of this URI essentially look the same when viewed
in a browser, it actually returns more than 10 posts depending on the
profile.
References #669
* Fix typo with status field
* Comply with other bridges
Change the uri element of an item to point, not on the magnet link, but on the page, as similar bridges do.
* Improved to return name & uri matching with query
This change makes it possible for the feed reader to discover a title and url consistent with the user's search.
* [ZoneTelechargementBridge] Added Bridge for ww2.zone-telechargement1.org
Goal for this bridge is to follow the episode publication of a TV show
season while it's broadcasted on the TV.
* [AutoJMBridge] New Bridge
This bridge will show all the car offers AutoJM has for the model you
choosed and using your filter. Very useful to wait for a cheap price for
a new car !
This commit fixes#782 by updating the parameter value of 'Maison &
Jardin', but this means the user has to update his RSS Feed URL (.because
of the bridge structure, it would be a nightmare to fix it in another
way)
This commits add all the categories available on Dealabs Website.
* [ExtremeDownloadBridge] Added Bridge for ww1.extreme-d0wn.com
Goal for this bridge is to follow the episode publication of a TV show season
while it's broadcasted on the TV.
* Adding a bridge for theyetee.com
The bridge fetches daily shirts at theyetee.com.
The Yetee offers two new shirts each day, but you can buy them only for a few hours !
Unfortunately, the site don't provide RSS feed, so the only way to keep up to date on new shirt is their daily mailing ... until now !
* Debug mode improvements
- Improve debug warning message
- Restore error reporting in debug mode
- Fix 'notice' messages for unset fields
* Add parsing utility functions
html.php
- extractFromDelimiters
- stripWithDelimiters
- stripRecursiveHTMLSection
- markdownToHtml (partial)
bridges
- remove now-duplicate functions
- call functions from html.php instead
* [Anidex] New bridge
Anime torrent tracker
* [Anime-Ultime] Restore thumbnail
* [CNET] Recreate bridge
Full rewrite as the previous one was broken
* [Dilbert] Minor URI fix
Use new self::URI property
* [EstCeQuonMetEnProd] Fix content extraction
Bridge was broken
* [Facebook] Fix "SpSonsSoriSsés" label
... which was taking space in item title
* [Futura-Sciences] Use HTTPS, More cleanup
Use HTTPS as FS now offer HTTPS
Clean additional useless HTML elements
* [GBATemp] Multiple fixes
- Fix categories: missing "break" statements
- Restore thumbnail as enclosure
- Fix date extraction
- Fix user blog post extraction
- Use getSimpleHTMLDOMCached
* [JapanExpo] Fix bridge, HTTPS, thumbnails
- Fix getSimpleHTMLDOMCached call
- Upgrade to HTTPS as JE now offers HTTPS
- Restore thumbnails as enclosures
* [LeMondeInformatique] Fix bridge, HTTPS
- Upgrade to HTTPS as LMI now offers HTTPS
- Restore thumbnails using small images
- Fix content extraction
- Fix text encoding issue
* [Nextgov] Fix content extraction
- Restore thumbnail and use small image
- Field extraction fixes
* [NextInpact] Add categories and filtering by type
- Offer all RSS feeds
- Allow filtering by article type
- Implement extraction for brief articles
- Remove article limit, many brief articles are publied all at once
* [NyaaTorrents] New bridge
Anime torrent tracker
* [Releases3DS] Cache content, restore thumbnail
- Use getSimpleHTMLDOMCached
- Restore thumbnail as enclosure
* [TheHackerNews] Fix bridge
- Fix content extraction including article body
- Restore thumbnail as enclosure
* [WeLiveSecurity] HTTPS, Fix content extraction
- Upgrade to HTTPS as WLS now offers HTTPS
- Fix content extraction including article body
* [WordPress] Reduce timeout, more content selectors
- Reduce timeout to use default one (1h)
- Add new content selector (articleBody)
- Find thumbnail and set as enclosure
- Fix <script> cleanup
* [YGGTorrent] Increase limit, use cache
- Increase item limit as uploads are very frequent
- Use getSimpleHTMLDOMCached
* [ZDNet] Rewrite with FeedExpander
- Upgrade to HTTPS as ZD now offers HTTPS
- Use FeedExpander for secondary fields
- Fix content extraction for article body
* [Main] Handle MIME type for enclosures
Many feed readers will ignore enclosures (e.g. thumbnails) with no MIME type. This commit adds automatic MIME type detection based on file extension (which may be inaccurate but is the only way without fetching the content).
One can force enclosure type using #.ext anchor (hacky, needs improving)
* [FeedExpander] Improve field extraction
- Add support for passing enclosures
- Improve author and uri extraction
- Fix 'notice' PHP error messages
* [Pull] Coding style fixes for #802
* [Pull] Implementing changes for #802
- Fix coding style issues with str append
- Remove useless CACHE_TIMEOUT
- Use count() instead of $limit
- Use defaultLinkTo() + handle strings
- Use http_build_query()
- Fix missing </em>
- Remove error_reporting(0)
- warning CSS (@LogMANOriginal)
- Fix typo in FeedExpander comment
* [Main] More documentation for markdownToHtml
See #802 for more details
The bridge would generate empty titles if the content is longer than
50 characters, but doesn't have further spaces in it. With this commit
the title is correctly generated based on the contents, taking missing
spaces into account.
References #786
https://cad-comic.com/ now provides feeds at
- https://cad-comic.com/feed (rss)
- https://cad-comic.com/feed/atom (atom)
Thus multiple alternatives are available to choose from, making this
bridge obsolete:
- FilterBridge (using one of the feeds above)
- WordPressBridge (on the main site)
- One of the two available feeds
References #752
This commit fixes an issue caused by self closing tags not supported
by simplehtmldom (<source>).
Adds a monkey patch to extend simplehtmldom with the ability to detect
that particular tag. Most of the code added is copied directly from
simplehtmldom (see vendor/simplehtmldom) with adjustments to account
for RSS-Bridge formatting.
Related to: https://sourceforge.net/p/simplehtmldom/bugs/83/
Notice: The tag itself is valid according to Mozilla:
The HTML <picture> element serves as a container for zero or more
<source> elements and one <img> element to provide versions of an
image for different display device scenarios. The browser will
consider each of the child <source> elements and select one
corresponding to the best match found; if no matches are found
among the <source> elements, the file specified by the <img>
element's src attribute is selected. The selected image is then
presented in the space occupied by the <img> element.
-- https://developer.mozilla.org/en-US/docs/Web/HTML/Element/picture
References #753
* use defaultLinkTo
* remove duplicate video links
* remove line ending before "Reposted" label
* return newline before reposted string
* remove comments
* use video links that won't require login
* set title if video has no title
Adds a new option '&title_from_content=on' to build the title for feed
items from the feeds content. The title is generated from the first
whitespace after 50 characters of the content or the entire content if
the total size is lower than 50 characters.
References #587
This commit fixes a few things related to static::URI
1) Remove trailing slash from the URI to simplify using 'defaultLinkTo'
2) Use static::URI instead of self::URI for consistency
3) Remove custom implementation of 'defaultLinkTo'
Images are collected for each post and added to enclosures. Images or
animtions from lh3.googleusercontent.com are specifically handled in
order to return the animated version of the gif and the original sized
image (this is normally taken care of by JS in the browser).
Adds a new bridge for https://gist.github.com
The bridge generates feeds for comments on a particular gist based on
the gist ID or full URI. For better readability the general behavior
of code sections is manually restored with the original CSS styles
from GitHub.
New bridge for Skimfeed: https://skimfeed.com
Generates feeds for all features of Skimfeed:
- News (the ones displayed on the front page)
- Hot topics ("What's Hot" section on the front page)
- Tech news (preconfigured feeds in the menu bar)
- Custom feeds (using the configuration system of Skimfeed), see
https://skimfeed.com/custom.php
The number of items returned by the bridge can be limited for all
categories ('&limit=...'). This parameter is optional, all categories
are unlimited by default!
Authors are added with HTML anchors in order to allow quick navigation
to source channels.
The bridge ships with developer tools to auto-generate lists in the
future (especially useful for 'Tech news'!)
References #748
Following changes in the JSON data and selecting images for the
content (320x240 or bigger) and enclosure (largest version). All of
the data is now extracted from the JSON data instead of parsing the
DOM.
References #754
Removing this bridge for two reasons:
1) The service moved from www.cpasbien.cm to www.torrents9.blue,
changing the layout in the process (incompatible).
2) The new site is permanently protected by Cloudflare IUAM, making
it inaccessible by RSS-Bridge.
While it would certainly be possible to rewrite the bridge to work
with the new layout, the site is still inaccessible.
References #605
This commit adds a new bridge for http://www.instructables.com. This bridge
currently supports fetching content by category (all categories available 200+),
using available filters (featured, recent, popular, views, contest winners).
Adds duration limits (minimum duration, maximum duration) for all
modes (user/id/playlist/search). Duration limits are optional, so
existing subscriptions don't break.
The limits are specified by two separate parameters, each of which
is optional:
- `&duration_min=` (minimum duration in minutes, default: -1)
- `&duration_max=` (maximum duration in minutes, default: INF)
If duration limits are specified in either user, id or playlist mode,
the bridge defaults to fetching data from HTML intead of XML feeds,
which requires more bandwidth and takes longer, because each video is
loaded individually!
References #670
The previous context is now labeled 'User', while the new context is
labeled 'Group'. The existing code was not changed, instead new group*
functions were implemented to handle groups.
The general principle of capturing groups is the same as done for users
with adjustments to account for different HTML structures.
Captcha responses are currently not supported for groups! There doesn't
seem to be a way to trigger them consistently, which makes it hard to
handle them properly.
Features of the group context:
- The feed title is based on the group name
- The group URI used for capturing is returned for the feed URI
- Author names and timestamps are reproduced from the source
- Post titles are reproduced from the source if they exist, otherwise
the title is build manually from the author name and the content
- Original contents are included with the feed
- All images are attached as enclosures as well
Closes #
Supports latest news and profiling a given ETF in Englisch, German
or Italian language. Cover images are attached as enclosures and not
as part of the content.
News:
Optionally loads the full article for each news item. Some articles
may include scripts to provide interactive graphs. These scripts are
removed as they would be rendered as pure text and a message is shown
instead: "[Content removed! Visit site to see full contents!]"
Profile:
Optionally includes the ETF strategy and description.
isoHunt has discontinued services due to legal reasons and is now
accessible via https://isohunts.to
While it is certainly possible to rewrite the bridge to fetch some
information from the new site, it wouldn't be able to provide as
much functionality as before. This is due to isoHunt having removed
all searching and filtering options, only providing static HTML pages
for general categories (anime, movies, etc...). Those pages, however,
are heavily broken.
Unless someone is interested in monitoring the general categories
the effort of upgrading the bridge to the new site is not worth taking
time for.
Users of isoHunt are asked to make use of their client application,
as they don't provide online services anymore (it's now in the darknet)
Here is the statement from isoHunt:
"Due to hard regulations and security issues for bittorrent users, we
have moved into a more secure and even faster district of the internet!
[...]
Torrent Downloads have a high risk of getting legal problems. That is
why we do not offer torrentfiles any more. [...]"
-- source: https://isohunts.to
Allows users to paste facebook links as user name. The link must contain
the correct host (www.facebook.com) and a valid path (/user-name/...).
The first part of the path is used for the user name. Errors are returned
in case something went wrong.
References #706
Reviews are provided the same way as summary posts and therefore returned
as separate feed item for each review. This commit adds a new option
'&skip_reviews=on' to skip reviews entirely.
References #706
The media_type parameter was recently replaced by media_type_u (for
user mode) and media_type_h (for hashtag mode). This was necessary
in order to add the media type 'story' only for the user mode.
"The reason for that is that RSS-Bridge supports multiple parameters
with the same name if and only if they contain the exact same value.
Here, hashtags don't have stories, so it would not be possible to
pass "story" as a parameter. This is a design mistake that I made
when I added support for hashtags."
-- 8770c87389 (r28871502)
However as pointed out this change breaks existing feeds as the
parameter name is no longer compatible to previous implementations.
This commit changes the implementation to provide the old media_type
parameter globally and check for invalid options on each request. If
a user uses the 'story' option in history mode the bridge returns a
client error.
references 8770c87
references #694fixes#696fixes#699fixes#701
* [DealabsBridge] Follow site changes, fix unhandled case
- Fixed the case where no discount was shown
- Changed some CSS class to follow the website changes
Now, entry title is optionnal and may be found in h3 HTML element.
Entry content is mandatory and may be found in div[class="item-content"] HTML element.
Moreover, the title may contain simple quotes (here, encoded) so the bridge have to decode first to apply format library function. In case we don't do that, the format function double encode the quote and something like &#039; could appear.
* [VkBridge] Convert special HTML entities to characters in pageName
* [VkBridge] Generate feed item title
* [VkBridge] Remove double backslashes in feed item link
* [VkBridge] Unpin post if pinned
* [VkBridge] Mark reposted messages
* [VkBridge] Correct external link parsing
* [VkBridge] Added article parsing
* [VkBridge] Added video parsing
* [VkBridge] Added photo parsing
* [VkBridge] Added album link parsing
* [VkBridge] Added one more external link selector
* [VkBridge] Using array of link selectors to remove
* [VkBridge] Added document parsing
* [VkBridge] Added sign parsing
* [VkBridge] Fixed incorrect sorting with pinned item
* [VkBridge] More methods to parse documents
* [VkBridge] Save fallback if page name element not found
* [VkBridge] Using post signed as feed item author
* [VkBridge] Fixed document link
* [VkBridge] Coding policy fixes
NotAlways right found it necessary to remove their RSS feeds recently. This is a *simple* bridge to grab the ones on the front page. It allows you to filter the articles based on their classification (right, working, romantic, related, learning, friendly, hopeless, unfiltered, or all).
The limit was used to specify the number of pages to return from a given
topic which resulted in the number of returned items variing between one
and however many entries are listed on one page.
This commit changes the implementation for the limit to keep loading more
pages until the specified limit is reached. Excessive elements are removed
in order to return the exact amount of items specified by the limit.
This behavior is closer to how other bridges are implemented and makes it
more natural to use without being too confusing. Existing queries must be
updated to account for the new limit.
References #657
Requesting a username with a leading slash would cause error 500
because the requested URI would contain two slashes in a row.
For example username "/test" would result in:
https://facebook.com//test
References #628
The author name is parsed by searching a string within the entire
HTML document:
$author = $html->innertext;
$author = substr($author, strpos($author, '"author=') + 8);
$author = substr($author, 0, strpos($author, '\u0026'));
This solution will return big portions of the HTML document if
the strpos function returns zero (not found).
This commit replaces the previous implementation by searching for
a specific script tag and making use of the JSON data inside it.
References #580
This bridge returns feeds for any URI that is compatible with the
IPB implementation (currently 4.x). Older versions might work, but
there is no guarantee.
Only forum and topic URIs are supported!
The bridge automatically checks if natural feeds are available (by
adding '.xml' to the URI). If so the feed is returned. Otherwise
the bridge will attempt to identify the content type and build a
feed accordingly.
Valid URIs are forums and topics. For forums the first page is
returned, for topics the last one. Elements are ordered such that
the latest entry is returned first (oldest-to-newest)
The optional parameter '&limit=' specifies how many pages should
be loaded (default: 1). Topics are loaded in reverse order.
=> Does not work with forums!
Images are provided as enclosures and scaled to a max-size of
400x400 pixels by default (Except for natural feeds).
The content is filtered before being returned:
- Unnecessary tags are removed (iframes, etc...)
- Styles for blockquotes are restored (grey background)
Closes#507
- If no there are no results, an explicit message is now returned
- Commas are now following the coding policy
- Lines are no longer more than 80 chars when possible
Added additional instructions for: 'username search' and 'category
search' next to instructions for 'keyword search'.
Changed variable name from underscore to camelCase.
Videos that are part of a playlist have the playlist ID encoded in
the URI. When loading the video info the page contents change unex-
pectedly due to the playlist being part of the page.
This removes any trailing parameters from the video ID in order to
ensure only pure videos are loaded at all times.
On Oct 27, 2016 the discontinuation of Vine was announced:
https://medium.com/@vine/important-news-about-vine-909c5f4ae7a7
"Today, we are sharing the news that in the coming months we’ll be
discontinuing the mobile app."
https://vine.co/ is still online, but has been put into an archive
indefinitely. As the site does not allow further uploads, this
bridge serves no further purpose.
Tags are embedded in the 'title' attribute instead of 'alt' as
defined by the ancestor (DanbooruBridge).
The 'title' attribute also contains statistics data ('score:...',
'rating:...') that is now filtered by a custom implementation of
the 'getTags' function (elements that contain a colon are removed)
Closes#560
All formats except HTML return & instead of & in URLs causing
all links with parameters (...&id=...) to break.
Facebook does not return valid HTML URIs but instead provides them
with all special characters encoded (like using htmlspecialchars).
This seems to be related to the page being build almost entirely of
script blocks.
This commit adds htmlspecialchars_decode() to URI and content to
reverse the encoding.
References #550
