Commit graph

50 commits

Author SHA1 Message Date
logmanoriginal
6c4098d655 Revert "all: Use ->remove() instead of ->outertext = ''"
This reverts commit 052844f5e1.

There is a bug in ->remove() that causes the parser to incorrectly
identify elements in the DOM tree that shouldn't exist anymore.

References #1151
2019-06-02 13:06:16 +02:00
logmanoriginal
052844f5e1 all: Use ->remove() instead of ->outertext = ''
simplehtmldom 1.9 introduced new functions to recursively remove
nodes from the DOM. This allows removing elements without the need
to re-load the document by using $html->load($html->save()), which
is very inefficient.

Find more information about remove() at
https://simplehtmldom.sourceforge.io/docs/1.9/api/simple_html_dom_node/remove/
2019-06-01 21:29:57 +02:00
logmanoriginal
c6a7b9ac64 exception: Remove HttpException class
This class served no particular purpose (other than adding a
layer on top of Exception).
2018-11-18 16:53:21 +01:00
ORelio
de8cee6a1c Catching up | [Main] Debug mode, parse utils, MIME | [Bridges] Add/Improve 20 bridges (#802)
* Debug mode improvements

 - Improve debug warning message
 - Restore error reporting in debug mode
 - Fix 'notice' messages for unset fields

* Add parsing utility functions

html.php
 - extractFromDelimiters
 - stripWithDelimiters
 - stripRecursiveHTMLSection
 - markdownToHtml (partial)

bridges
 - remove now-duplicate functions
 - call functions from html.php instead

* [Anidex] New bridge

Anime torrent tracker

* [Anime-Ultime] Restore thumbnail

* [CNET] Recreate bridge

Full rewrite as the previous one was broken

* [Dilbert] Minor URI fix

Use new self::URI property

* [EstCeQuonMetEnProd] Fix content extraction

Bridge was broken

* [Facebook] Fix "SpSonsSoriSsés" label

... which was taking space in item title

* [Futura-Sciences] Use HTTPS, More cleanup

Use HTTPS as FS now offer HTTPS
Clean additional useless HTML elements

* [GBATemp] Multiple fixes

- Fix categories: missing "break" statements
- Restore thumbnail as enclosure
- Fix date extraction
- Fix user blog post extraction
- Use getSimpleHTMLDOMCached

* [JapanExpo] Fix bridge, HTTPS, thumbnails

- Fix getSimpleHTMLDOMCached call
- Upgrade to HTTPS as JE now offers HTTPS
- Restore thumbnails as enclosures

* [LeMondeInformatique] Fix bridge, HTTPS

- Upgrade to HTTPS as LMI now offers HTTPS
- Restore thumbnails using small images
- Fix content extraction
- Fix text encoding issue

* [Nextgov] Fix content extraction

- Restore thumbnail and use small image
- Field extraction fixes

* [NextInpact] Add categories and filtering by type

- Offer all RSS feeds
- Allow filtering by article type
- Implement extraction for brief articles
- Remove article limit, many brief articles are publied all at once

* [NyaaTorrents] New bridge

Anime torrent tracker

* [Releases3DS] Cache content, restore thumbnail

- Use getSimpleHTMLDOMCached
- Restore thumbnail as enclosure

* [TheHackerNews] Fix bridge

 - Fix content extraction including article body
 - Restore thumbnail as enclosure

* [WeLiveSecurity] HTTPS, Fix content extraction

- Upgrade to HTTPS as WLS now offers HTTPS
- Fix content extraction including article body

* [WordPress] Reduce timeout, more content selectors

- Reduce timeout to use default one (1h)
- Add new content selector (articleBody)
- Find thumbnail and set as enclosure
- Fix <script> cleanup

* [YGGTorrent] Increase limit, use cache

- Increase item limit as uploads are very frequent
- Use getSimpleHTMLDOMCached

* [ZDNet] Rewrite with FeedExpander

- Upgrade to HTTPS as ZD now offers HTTPS
- Use FeedExpander for secondary fields
- Fix content extraction for article body

* [Main] Handle MIME type for enclosures

Many feed readers will ignore enclosures (e.g. thumbnails) with no MIME type. This commit adds automatic MIME type detection based on file extension (which may be inaccurate but is the only way without fetching the content).

One can force enclosure type using #.ext anchor (hacky, needs improving)

* [FeedExpander] Improve field extraction

- Add support for passing enclosures
- Improve author and uri extraction
- Fix 'notice' PHP error messages

* [Pull] Coding style fixes for #802

* [Pull] Implementing changes for #802

 - Fix coding style issues with str append
 - Remove useless CACHE_TIMEOUT
 - Use count() instead of $limit
 - Use defaultLinkTo() + handle strings
 - Use http_build_query()
 - Fix missing </em>
 - Remove error_reporting(0)
 - warning CSS (@LogMANOriginal)
 - Fix typo in FeedExpander comment

* [Main] More documentation for markdownToHtml

See #802 for more details
2018-09-09 20:20:13 +01:00
logmanoriginal
a4b9611e66 [phpcs] Add missing rules
- Do not add spaces after opening or before closing parenthesis

  // Wrong
  if( !is_null($var) ) {
    ...
  }

  // Right
  if(!is_null($var)) {
    ...
  }

- Add space after closing parenthesis

  // Wrong
  if(true){
    ...
  }

  // Right
  if(true) {
    ...
  }

- Add body into new line
- Close body in new line

  // Wrong
  if(true) { ... }

  // Right
  if(true) {
    ...
  }

Notice: Spaces after keywords are not detected:

  // Wrong (not detected)
  // -> space after 'if' and missing space after 'else'
  if (true) {
    ...
  } else{
    ...
  }

  // Right
  if(true) {
    ...
  } else {
    ...
  }
2017-07-29 19:55:12 +02:00
logmanoriginal
512a4f292b bridges: Return parent::getURI by default 2017-02-15 19:38:32 +01:00
logmanoriginal
6f24858124 bridges: Fix coding styles
This commit is a squash of all commits that fix coding styles
for the new coding style policy.

[ABCTabsBridge] Fix coding style
[AcrimedBridge] Fix coding style
[AllocineFRBridge] Fix coding style
[AnimeUltimeBridge] Fix coding style
[Arte7Bridge] Fix coding style
[AskfmBridge] Fix coding style
[BandcampBridge] Fix coding style
[BastaBridge] Fix coding style
[BlaguesDeMerdeBridge] Fix coding style
[BooruprojectBridge] Fix coding style
[CADBridge] Fix coding style
[CNETBridge] Fix coding style
[CastorusBridge] Fix coding style
[CollegeDeFranceBridge] Fix coding style
[CommonDreamsBridge] Fix coding style
[CopieDoubleBridge] Fix coding style
[CourrierInternationalBridge] Fix coding style
[CpasbienBridge] Fix coding style
[CryptomeBridge] Fix coding style
[DailymotionBridge] Fix coding style
[DanbooruBridge] Fix coding style
[DansTonChatBridge] Fix coding style
[DauphineLibereBridge] Fix coding style
[DeveloppezDotComBridge] Fix coding style
[DemoBridge] Fix coding style
[DilbertBridge] Fix coding style
[DuckDuckGoBridge] Fix coding style
[DollbooruBridge] Fix coding style
[EliteDangerousGalnetBridge] Fix coding style
[ElsevierBridge] Fix coding style
[EstCeQuonMetEnProdBridge] Fix coding style
[EZTVBridge] Fix coding style
[FacebookBridge] Fix coding style
[FeedExpanderExampleBridge] Fix coding style
[FB2Bridge] Fix coding style
[FierPandaBridge] Fix coding style
[FlickrBridge] Fix coding style
[FootitoBridge] Fix coding style
[FourchanBridge] Fix coding style
[FuturaSciencesBridge] Fix coding style
[GBAtempBridge] Fix coding style
[GelbooruBridge] Fix coding style
[GiphyBridge] Fix coding style
[GithubIssueBridge] Fix coding style
[GizmodoBridge] Fix coding style
[GoComicsBridge] Fix coding style
[GooglePlusPostBridge] Fix coding style
[GoogleSearchBridge] Fix coding style
[HDWallpapersBridge] Fix coding style
[HentaiHavenBridge] Fix coding style
[IdenticaBridge] Fix coding style
[InstagramBridge] Fix coding style
[IsoHuntBridge] Fix coding style
[JapanExpoBridge] Fix coding style
[KonachanBridge] Fix coding style
[KoreusBridge] Fix coding style
[KununuBridge] Fix coding style
[LeBonCoinBridge] Fix coding style
[LegifranceJOBBridge] Fix coding style
[LeMondeInformatiqueBridge] Fix coding style
[LesJoiesDuCodeBridge] Fix coding style
[LichessBridge] Fix coding style
[LinkedInCompanyBridge] Fix coding style
[LolibooruBridge] Fix coding style
[LWNprevBridge] Fix coding style
[MangareaderBridge] Fix coding style
[MilbooruBridge] Fix coding style
[MixCloudBridge] Fix coding style
[MoebooruBridge] Fix coding style
[MondeDiploBridge] Fix coding style
[MsnMondeBridge] Fix coding style
[MspabooruBridge] Fix coding style
[NasaApodBridge] Fix coding style
[NeuviemeArtBridge] Fix coding style
[NextgovBridge] Fix coding style
[NextInpactBridge] Fix coding style
[NiceMatinBridge] Fix coding style
[NovelUpdatesBridge] Fix coding style
[OpenClassroomsBridge] Fix coding style
[ParuVenduImmoBridge] Fix coding style
[PickyWallpapersBridge] Fix coding style
[PinterestBridge] Fix coding style
[PlanetLibreBridge] Fix coding style
[ReadComicsBridge] Fix coding style
[Releases3DSBridge] Fix coding style
[ReporterreBridge] Fix coding style
[RTBFBridge] Fix coding style
[Rue89Bridge] Fix coding style
[Rule34Bridge] Fix coding style
[Rule34pahealBridge] Fix coding style
[SafebooruBridge] Fix coding style
[SakugabooruBridge] Fix coding style
[ScmbBridge] Fix coding style
[ScoopItBridge] Fix coding style
[SensCritiqueBridge] Fix coding style
[SexactuBridge] Fix coding style
[ShanaprojectBridge] Fix coding style
[Shimmie2Bridge] Fix coding style
[SoundcloudBridge] Fix coding style
[StripeAPIChangeLogBridge] Fix coding style
[SuperbWallpapersBridge] Fix coding style
[T411Bridge] Fix coding style
[TagBoardBridge] Fix coding style
[TbibBridge] Fix coding style
[TheCodingLoveBridge] Fix coding style
[TheHackerNewsBridge] Fix coding style
[ThePirateBayBridge] Fix coding style
[TheTVDBBridge] Fix coding style
[Torrent9Bridge] Fix coding style
[TwitterBridge] Fix coding style
[UnsplashBridge] Fix coding style
[ViadeoCompanyBridge] Fix coding style
[VineBridge] Fix coding style
[VkBridge] Fix coding style
[WallpaperStopBridge] Fix coding style
[WebfailBridge] Fix coding style
[WeLiveSecurityBridge] Fix coding style
[WhydBridge] Fix coding style
[WikipediaBridge] Fix coding style
[WordPressBridge] Fix coding style
[WorldOfTanksBridge] Fix coding style
[XbooruBridge] Fix coding style
[YandereBridge] Fix coding style
[YoutubeBridge] Fix coding style
[ZDNetBridge] Fix coding style
2017-02-12 15:34:08 +01:00
Astyan-42
a1764a9fe2 make the bridge compatible with wordpress with no static URL (#469) 2017-02-07 10:24:18 +00:00
logmanoriginal
970e216c1f Merge branch 'methods2functions' of https://framagit.org/peetah/rss-bridge 2016-10-02 16:05:58 +02:00
Pierre Mazière
f1fb95b257 [core] extract BridgeAbstract methods to make them functions
- returnError, returnServerError, returnClientError ,debugMessage are
  moved to lib/error.php

- getContents, getSimpleHTMLDOM, getSimpleHTMLDOMCached are moved to
  lib/contents.php

Signed-off-by: Pierre Mazière <pierre.maziere@gmx.com>
2016-09-25 23:22:33 +02:00
Pierre Mazière
235c985901 [BridgeAbstract] replace getCacheDuration method by CACHE_TIMEOUT class constant
Signed-off-by: Pierre Mazière <pierre.maziere@gmx.com>
2016-09-25 17:04:28 +02:00
Pierre Mazière
10ac808580 [WordPressBridge] remove useless code and fix description
Signed-off-by: Pierre Mazière <pierre.maziere@gmx.com>
2016-09-17 19:37:39 +02:00
Pierre Mazière
43ac961284 [WordPressBridge] add another case of content extraction
Signed-off-by: Pierre Mazière <pierre.maziere@gmx.com>
2016-09-17 19:37:39 +02:00
Pierre Mazière
b6245a90c8 [WordPressBridge] allow other bridges to use it as a parent class
Signed-off-by: Pierre Mazière <pierre.maziere@gmx.com>
2016-09-17 19:37:39 +02:00
Pierre Mazière
4c3b80fca9 [WordPressBridge] fix fatal errors
Signed-off-by: Pierre Mazière <pierre.maziere@gmx.com>
2016-09-17 19:37:39 +02:00
Pierre Mazière
b5358d6cea [WordPressBridge] make it FeedExpander
Signed-off-by: Pierre Mazière <pierre.maziere@gmx.com>
2016-09-17 19:37:39 +02:00
logmanoriginal
2eec89ab27 [bridges] Change all bridges to use BridgeAbstract with getSimpleHTMLDOMCached 2016-09-10 19:11:09 +02:00
logmanoriginal
3a94956915 [bridges] Remove calls to get_cached and remove_from_cache 2016-09-10 00:19:24 +02:00
Pierre Mazière
9a0da733ef [bridges] use constants instead of variable members
Signed-off-by: Pierre Mazière <pierre.maziere@gmx.com>
2016-09-01 23:15:51 +02:00
logmanoriginal
8ed3b02899 Merge branch 'core' of http://framagit.org/peetah/rss-bridge 2016-08-28 20:33:27 +02:00
logmanoriginal
78f6757622 [bridges] Use caching with applicable bridges 2016-08-28 20:07:56 +02:00
Pierre Mazière
1b3c8a8aeb [core + bridges] add BridgeAbstract::$inputs and BridgeAbstract::getInput()
Inputs are not stored in BridgeAbstract::$parameters anymore to separate
static data from dynamic data.
The getInput method allows for more readable code.

Also fix an "undefined index 'global'" notice

Probability of breaking bridges: high !

Signed-off-by: Pierre Mazière <pierre.maziere@gmx.com>
2016-08-28 13:05:03 +02:00
Pierre Mazière
de1b39c8e5 [core + bridges] get rid of loadMetadata
if a bridge needs to modify some of the data that were initialized
there, ::__construct() should be used instead.

Signed-off-by: Pierre Mazière <pierre.maziere@gmx.com>
2016-08-28 13:05:03 +02:00
Pierre Mazière
2301a12cc6 [WordpPessBridge] implement getURI() and simplify code
Signed-off-by: Pierre Mazière <pierre.maziere@gmx.com>
2016-08-28 13:00:55 +02:00
Pierre Mazière
117031bf0f [core] store parameters values in BridgeAbstract::parameters
This way, any BridgeAbstract method can now have access to these values,
no only collectData

Signed-off-by: Pierre Mazière <pierre.maziere@gmx.com>
2016-08-25 01:24:53 +02:00
logmanoriginal
bf0a9d754e [bridges] Change all occurrences of the Item object to array 2016-08-22 18:55:59 +02:00
Pierre Mazière
f0e502ce37 [bridges] migrate all bridges to an array based definition of parameters
see github issue 356

Signed-off-by: Pierre Mazière <pierre.maziere@gmx.com>
2016-08-22 01:25:56 +02:00
Pierre Mazière
bba216073c [bridges] remove useless 'update' property from all bridges
That's the source code manager job to keep this information
consistent

Signed-off-by: Pierre Mazière <pierre.maziere@gmx.com>
2016-08-21 00:21:41 +02:00
Pierre Mazière
3c0d13c1bb bridges: use BridgeAbstract::getSimpleHTMLDOM
instead of BridgeAbstract::file_get_html

Signed-off-by: Pierre Mazière <pierre.maziere@gmx.com>
2016-08-19 10:20:27 +02:00
logmanoriginal
74f0572d91 bridges: Replace returnError function with more specific
Replacements depend on original error code:
400: returnClientError
404: returnServerError
500: returnServerError
501: returnServerError
2016-08-17 14:45:08 +02:00
logmanoriginal
fa59a25229 bridges: Update bridges with 'required' attributes 2016-08-15 01:19:16 +02:00
logmanoriginal
63bb7ef24d bridges: Remove unnecessary functions getName() and getURI() 2016-08-09 20:08:46 +02:00
logmanoriginal
1d53b70272 bridges: Remove unused functions and update scopes
Many bridges implemented getDescription() which is only applicable for
bridges extending RSSExpander.

Functions that are not part of the abstract class or interface should
be in private scope for better readability.
2016-08-06 16:00:56 +02:00
logmanoriginal
7afee0e277 [WordPress] Fix site name on bridge content 2016-08-04 21:56:11 +02:00
logmanoriginal
870ef6f6fc [WordPress] Remove forms and improve script removal
Some sites contain scripts like these, that were not captured by the
previous implementation:

<script async src="//pagead2.googlesyndication.com/pagead/js/adsbygoogle.js"></script>
2016-08-04 21:46:03 +02:00
logmanoriginal
2c41ed550d [WordPress] Fix content loading 2016-08-04 21:06:12 +02:00
logmanoriginal
d944558a3d [WordPress] Fix indentation and remove empty lines 2016-08-04 20:45:21 +02:00
logmanoriginal
455b98757c [WordPress] Don't attempt to load pages with missing .article
Some sites use WordPress but don't provide pages with articles.
Instead of throwing internal errors we just don't return any
content.
2016-08-04 20:40:38 +02:00
logmanoriginal
21523eb549 [WordPress] Change all nested functions to member functions 2016-08-04 20:15:28 +02:00
logmanoriginal
2672b96e11 [WordPress] Initialize item before assigning values
This solves warning: "Creating default object from empty value"
2016-08-04 20:12:51 +02:00
logmanoriginal
0a008c4eb3 [WordPress] Support atom AND rss for feed replies
Some sites return RSS feeds instead of ATOM feeds even though ATOM
feeds were specifically requested (/feeds/atom)!

This bridge will now detect ATOM/RSS feeds and change behavior
accordingly.
2016-08-04 20:06:53 +02:00
Mitsukarenai
cd5f3e740a [Wordpress] Add a div class commonly used for post content 2016-08-02 17:48:57 +02:00
Mitsukarenai
0f3ca30caa [WordPress] bridge rewrite 2016-08-02 15:46:21 +02:00
Pierre Mazière
955eecc299 use BridgeAbstract::file_get_html in all bridges
instead of simple_html_dom function file_get_html

Signed-off-by: Pierre Mazière <pierre.maziere@gmx.com>
2016-06-25 23:17:42 +02:00
teromene
78c9dcc705 Last batch of bridges, all implemented !
Signed-off-by: teromene <teromene@teromene.fr>
2015-12-06 15:56:39 +01:00
teromene
2d526f4e0a Correction de quelques bugs dans les bridges, qui apparaissent à cause du fait qu'ils sont tous instanciés lors du chargement de tout les bridges.
Ré-ajout et ré-utilisation de getName et getURI, qui permettent d'avoir des élements dynamiques.
Changement en public de certaines valeurs.
Changement du nom de certaines constantes, déja utilisés.

Signed-off-by: teromene <teromene@teromene.fr>
2015-12-06 15:56:39 +01:00
ORelio
5b2c8e91d0 Improve Wordpress Bridge
Retrieve post author
Retrieve post thumbnail
Retrieve title from <h2> if no <h1>
Minor code indentation fixes
2015-09-05 14:42:57 +02:00
Paul Vayssiere
c38c7dd2a1 add a Gizmodo.fr bridge to get the content
And link to youtube videos
2014-07-14 12:12:01 -05:00
Alexis Degrugillier
e7c00914dd Fix posts discovery.
For PHP before 5.5, the bridge is not working. It returns the following error: "Can't use method return value in write context".
See the following message for explanations: http://stackoverflow.com/questions/1075534/cant-use-method-return-value-in-write-context/4328049#4328049
2014-05-27 21:28:36 -04:00
Mitsukarenai
c07eacfd6a Add extra bridges, second batch (TPB inside) 2014-05-26 19:45:10 +02:00