2016-07-19 19:38:42 +02:00
|
|
|
<?php
|
Catching up | [Main] Debug mode, parse utils, MIME | [Bridges] Add/Improve 20 bridges (#802)
* Debug mode improvements
- Improve debug warning message
- Restore error reporting in debug mode
- Fix 'notice' messages for unset fields
* Add parsing utility functions
html.php
- extractFromDelimiters
- stripWithDelimiters
- stripRecursiveHTMLSection
- markdownToHtml (partial)
bridges
- remove now-duplicate functions
- call functions from html.php instead
* [Anidex] New bridge
Anime torrent tracker
* [Anime-Ultime] Restore thumbnail
* [CNET] Recreate bridge
Full rewrite as the previous one was broken
* [Dilbert] Minor URI fix
Use new self::URI property
* [EstCeQuonMetEnProd] Fix content extraction
Bridge was broken
* [Facebook] Fix "SpSonsSoriSsés" label
... which was taking space in item title
* [Futura-Sciences] Use HTTPS, More cleanup
Use HTTPS as FS now offer HTTPS
Clean additional useless HTML elements
* [GBATemp] Multiple fixes
- Fix categories: missing "break" statements
- Restore thumbnail as enclosure
- Fix date extraction
- Fix user blog post extraction
- Use getSimpleHTMLDOMCached
* [JapanExpo] Fix bridge, HTTPS, thumbnails
- Fix getSimpleHTMLDOMCached call
- Upgrade to HTTPS as JE now offers HTTPS
- Restore thumbnails as enclosures
* [LeMondeInformatique] Fix bridge, HTTPS
- Upgrade to HTTPS as LMI now offers HTTPS
- Restore thumbnails using small images
- Fix content extraction
- Fix text encoding issue
* [Nextgov] Fix content extraction
- Restore thumbnail and use small image
- Field extraction fixes
* [NextInpact] Add categories and filtering by type
- Offer all RSS feeds
- Allow filtering by article type
- Implement extraction for brief articles
- Remove article limit, many brief articles are publied all at once
* [NyaaTorrents] New bridge
Anime torrent tracker
* [Releases3DS] Cache content, restore thumbnail
- Use getSimpleHTMLDOMCached
- Restore thumbnail as enclosure
* [TheHackerNews] Fix bridge
- Fix content extraction including article body
- Restore thumbnail as enclosure
* [WeLiveSecurity] HTTPS, Fix content extraction
- Upgrade to HTTPS as WLS now offers HTTPS
- Fix content extraction including article body
* [WordPress] Reduce timeout, more content selectors
- Reduce timeout to use default one (1h)
- Add new content selector (articleBody)
- Find thumbnail and set as enclosure
- Fix <script> cleanup
* [YGGTorrent] Increase limit, use cache
- Increase item limit as uploads are very frequent
- Use getSimpleHTMLDOMCached
* [ZDNet] Rewrite with FeedExpander
- Upgrade to HTTPS as ZD now offers HTTPS
- Use FeedExpander for secondary fields
- Fix content extraction for article body
* [Main] Handle MIME type for enclosures
Many feed readers will ignore enclosures (e.g. thumbnails) with no MIME type. This commit adds automatic MIME type detection based on file extension (which may be inaccurate but is the only way without fetching the content).
One can force enclosure type using #.ext anchor (hacky, needs improving)
* [FeedExpander] Improve field extraction
- Add support for passing enclosures
- Improve author and uri extraction
- Fix 'notice' PHP error messages
* [Pull] Coding style fixes for #802
* [Pull] Implementing changes for #802
- Fix coding style issues with str append
- Remove useless CACHE_TIMEOUT
- Use count() instead of $limit
- Use defaultLinkTo() + handle strings
- Use http_build_query()
- Fix missing </em>
- Remove error_reporting(0)
- warning CSS (@LogMANOriginal)
- Fix typo in FeedExpander comment
* [Main] More documentation for markdownToHtml
See #802 for more details
2018-09-09 21:20:13 +02:00
|
|
|
class ZDNetBridge extends FeedExpander {
|
2016-07-19 19:38:42 +02:00
|
|
|
|
2017-02-11 16:16:56 +01:00
|
|
|
const MAINTAINER = 'ORelio';
|
|
|
|
const NAME = 'ZDNet Bridge';
|
Catching up | [Main] Debug mode, parse utils, MIME | [Bridges] Add/Improve 20 bridges (#802)
* Debug mode improvements
- Improve debug warning message
- Restore error reporting in debug mode
- Fix 'notice' messages for unset fields
* Add parsing utility functions
html.php
- extractFromDelimiters
- stripWithDelimiters
- stripRecursiveHTMLSection
- markdownToHtml (partial)
bridges
- remove now-duplicate functions
- call functions from html.php instead
* [Anidex] New bridge
Anime torrent tracker
* [Anime-Ultime] Restore thumbnail
* [CNET] Recreate bridge
Full rewrite as the previous one was broken
* [Dilbert] Minor URI fix
Use new self::URI property
* [EstCeQuonMetEnProd] Fix content extraction
Bridge was broken
* [Facebook] Fix "SpSonsSoriSsés" label
... which was taking space in item title
* [Futura-Sciences] Use HTTPS, More cleanup
Use HTTPS as FS now offer HTTPS
Clean additional useless HTML elements
* [GBATemp] Multiple fixes
- Fix categories: missing "break" statements
- Restore thumbnail as enclosure
- Fix date extraction
- Fix user blog post extraction
- Use getSimpleHTMLDOMCached
* [JapanExpo] Fix bridge, HTTPS, thumbnails
- Fix getSimpleHTMLDOMCached call
- Upgrade to HTTPS as JE now offers HTTPS
- Restore thumbnails as enclosures
* [LeMondeInformatique] Fix bridge, HTTPS
- Upgrade to HTTPS as LMI now offers HTTPS
- Restore thumbnails using small images
- Fix content extraction
- Fix text encoding issue
* [Nextgov] Fix content extraction
- Restore thumbnail and use small image
- Field extraction fixes
* [NextInpact] Add categories and filtering by type
- Offer all RSS feeds
- Allow filtering by article type
- Implement extraction for brief articles
- Remove article limit, many brief articles are publied all at once
* [NyaaTorrents] New bridge
Anime torrent tracker
* [Releases3DS] Cache content, restore thumbnail
- Use getSimpleHTMLDOMCached
- Restore thumbnail as enclosure
* [TheHackerNews] Fix bridge
- Fix content extraction including article body
- Restore thumbnail as enclosure
* [WeLiveSecurity] HTTPS, Fix content extraction
- Upgrade to HTTPS as WLS now offers HTTPS
- Fix content extraction including article body
* [WordPress] Reduce timeout, more content selectors
- Reduce timeout to use default one (1h)
- Add new content selector (articleBody)
- Find thumbnail and set as enclosure
- Fix <script> cleanup
* [YGGTorrent] Increase limit, use cache
- Increase item limit as uploads are very frequent
- Use getSimpleHTMLDOMCached
* [ZDNet] Rewrite with FeedExpander
- Upgrade to HTTPS as ZD now offers HTTPS
- Use FeedExpander for secondary fields
- Fix content extraction for article body
* [Main] Handle MIME type for enclosures
Many feed readers will ignore enclosures (e.g. thumbnails) with no MIME type. This commit adds automatic MIME type detection based on file extension (which may be inaccurate but is the only way without fetching the content).
One can force enclosure type using #.ext anchor (hacky, needs improving)
* [FeedExpander] Improve field extraction
- Add support for passing enclosures
- Improve author and uri extraction
- Fix 'notice' PHP error messages
* [Pull] Coding style fixes for #802
* [Pull] Implementing changes for #802
- Fix coding style issues with str append
- Remove useless CACHE_TIMEOUT
- Use count() instead of $limit
- Use defaultLinkTo() + handle strings
- Use http_build_query()
- Fix missing </em>
- Remove error_reporting(0)
- warning CSS (@LogMANOriginal)
- Fix typo in FeedExpander comment
* [Main] More documentation for markdownToHtml
See #802 for more details
2018-09-09 21:20:13 +02:00
|
|
|
const URI = 'https://www.zdnet.com/';
|
2017-02-11 16:16:56 +01:00
|
|
|
const DESCRIPTION = 'Technology News, Analysis, Comments and Product Reviews for IT Professionals.';
|
2016-07-19 19:38:42 +02:00
|
|
|
|
2017-02-11 16:16:56 +01:00
|
|
|
//http://www.zdnet.com/zdnet.opml
|
|
|
|
const PARAMETERS = array( array(
|
|
|
|
'feed' => array(
|
|
|
|
'name' => 'Feed',
|
|
|
|
'type' => 'list',
|
|
|
|
'values' => array(
|
|
|
|
'Subscribe to ZDNet RSS Feeds' => array(
|
|
|
|
'All Blogs' => 'blog',
|
|
|
|
'Just News' => 'news',
|
|
|
|
'All Reviews' => 'topic/reviews',
|
|
|
|
'Latest Downloads' => 'downloads!recent',
|
|
|
|
'Latest Articles' => '/',
|
|
|
|
'Latest Australia Articles' => 'au',
|
|
|
|
'Latest UK Articles' => 'uk',
|
|
|
|
'Latest US Articles' => 'us',
|
|
|
|
'Latest Asia Articles' => 'as'
|
|
|
|
),
|
|
|
|
'Keep up with ZDNet Blogs RSS:' => array(
|
|
|
|
'Transforming the Datacenter' => 'blog/transforming-datacenter',
|
|
|
|
'SMB India' => 'blog/smb-india',
|
|
|
|
'Indonesia BizTech' => 'blog/indonesia-biztech',
|
|
|
|
'Hong Kong Techie' => 'blog/hong-kong-techie',
|
|
|
|
'Tech Taiwan' => 'blog/tech-taiwan',
|
|
|
|
'Startup India' => 'blog/startup-india',
|
|
|
|
'Starting Up Asia' => 'blog/starting-up-asia',
|
|
|
|
'Next-Gen Partner' => 'blog/partner',
|
|
|
|
'Post-PC Developments' => 'blog/post-pc',
|
|
|
|
'Benelux' => 'blog/benelux',
|
|
|
|
'Heat Sink' => 'blog/heat-sink',
|
|
|
|
'Italy\'s got tech' => 'blog/italy',
|
|
|
|
'African Enterprise' => 'blog/african-enterprise',
|
|
|
|
'New Tech for Old India' => 'blog/new-india',
|
|
|
|
'Estonia Uncovered' => 'blog/estonia',
|
|
|
|
'IT Iberia' => 'blog/iberia',
|
|
|
|
'Brazil Tech' => 'blog/brazil',
|
|
|
|
'500 words into the future' => 'blog/500-words-into-the-future',
|
|
|
|
'ÜberTech' => 'blog/ubertech',
|
|
|
|
'All About Microsoft' => 'blog/microsoft',
|
|
|
|
'Back office' => 'blog/back-office',
|
|
|
|
'Barker Bites Back' => 'blog/barker-bites-back',
|
|
|
|
'Between the Lines' => 'blog/btl',
|
|
|
|
'Big on Data' => 'blog/big-data',
|
|
|
|
'bootstrappr' => 'blog/bootstrappr',
|
|
|
|
'By The Way' => 'blog/by-the-way',
|
|
|
|
'Central European Processing' => 'blog/central-europe',
|
|
|
|
'Cloud Builders' => 'blog/cloud-builders',
|
|
|
|
'Communication Breakdown' => 'blog/communication-breakdown',
|
|
|
|
'Collaboration 2.0' => 'blog/collaboration',
|
|
|
|
'Constellation Research' => 'blog/constellation',
|
|
|
|
'Consumerization: BYOD' => 'blog/consumerization',
|
|
|
|
'DIY-IT' => 'blog/diy-it',
|
|
|
|
'Enterprise Web 2.0' => 'blog/hinchcliffe',
|
|
|
|
'Five Nines: The Next Gen Datacenter' => 'blog/datacenter',
|
|
|
|
'Forrester Research' => 'blog/forrester',
|
|
|
|
'Full Duplex' => 'blog/full-duplex',
|
|
|
|
'Gen Why?' => 'blog/gen-why',
|
|
|
|
'Hardware 2.0' => 'blog/hardware',
|
|
|
|
'Identity Matters' => 'blog/identity',
|
|
|
|
'iGeneration' => 'blog/igeneration',
|
|
|
|
'Internet of Everything' => 'blog/cisco',
|
|
|
|
'Beyond IT Failure' => 'blog/projectfailures',
|
|
|
|
'Jamie\'s Mostly Linux Stuff' => 'blog/jamies-mostly-linux-stuff',
|
|
|
|
'Jack\'s Blog' => 'blog/jacks-blog',
|
|
|
|
'Laptops & Desktops' => 'blog/computers',
|
|
|
|
'Linux and Open Source' => 'blog/open-source',
|
|
|
|
'London Calling' => 'blog/london',
|
|
|
|
'Mapping Babel' => 'blog/mapping-babel',
|
|
|
|
'Mixed Signals' => 'blog/mixed-signals',
|
|
|
|
'Mobile India' => 'blog/mobile-india',
|
|
|
|
'Mobile News' => 'blog/mobile-news',
|
|
|
|
'Networking' => 'blog/networking',
|
|
|
|
'Norse Code' => 'blog/norse-code',
|
|
|
|
'Null Pointer' => 'blog/null-pointer',
|
|
|
|
'The Full Tilt' => 'blog/the-full-tilt',
|
|
|
|
'Pinoy Post' => 'blog/pinoy-post',
|
|
|
|
'Practically Tech' => 'blog/practically-tech',
|
|
|
|
'Product Central' => 'blog/product-central',
|
|
|
|
'Pulp Tech' => 'blog/violetblue',
|
|
|
|
'Qubits and Pieces' => 'blog/qubits-and-pieces',
|
|
|
|
'Securify This!' => 'blog/securify-this',
|
|
|
|
'Service Oriented' => 'blog/service-oriented',
|
|
|
|
'Small Talk' => 'blog/small-talk',
|
|
|
|
'Small Business Matters' => 'blog/small-business-matters',
|
|
|
|
'Smartphones and Cell Phones' => 'blog/cell-phones',
|
|
|
|
'Social Business' => 'blog/feeds',
|
|
|
|
'Social CRM: The Conversation' => 'blog/crm',
|
|
|
|
'Software & Services Safari' => 'blog/sommer',
|
|
|
|
'Storage Bits' => 'blog/storage',
|
|
|
|
'Stacking up Open Clouds' => 'blog/apac-redhat',
|
|
|
|
'Techie Isles' => 'blog/techie-isles',
|
|
|
|
'Technolatte' => 'blog/technolatte',
|
|
|
|
'Tech Podium' => 'blog/tech-podium',
|
|
|
|
'Tel Aviv Tech' => 'blog/tel-aviv',
|
|
|
|
'Tech Broiler' => 'blog/perlow',
|
|
|
|
'The SANMAN' => 'blog/the-sanman',
|
|
|
|
'The open source revolution' => 'blog/the-open-source-revolution',
|
|
|
|
'The German View' => 'blog/german',
|
|
|
|
'The Ed Bott Report' => 'blog/bott',
|
|
|
|
'The Mobile Gadgeteer' => 'blog/mobile-gadgeteer',
|
|
|
|
'The Apple Core' => 'blog/apple',
|
|
|
|
'Tom Foremski: IMHO' => 'blog/foremski',
|
|
|
|
'Twisted Wire' => 'blog/twisted-wire',
|
|
|
|
'Vive la tech' => 'blog/france',
|
|
|
|
'Virtually Speaking' => 'blog/virtualization',
|
|
|
|
'View from China' => 'blog/china',
|
|
|
|
'Web design & Free Software' => 'blog/web-design-and-free-software',
|
|
|
|
'ZDNet Government' => 'blog/government',
|
|
|
|
'ZDNet UK Book Reviews' => 'blog/zdnet-uk-book-reviews',
|
|
|
|
'ZDNet UK First Take' => 'blog/zdnet-uk-first-take',
|
|
|
|
'Zero Day' => 'blog/security'
|
|
|
|
),
|
|
|
|
'ZDNet Hot Topics RSS:' => array(
|
|
|
|
'Apple' => 'topic/apple',
|
|
|
|
'Collaboration' => 'topic/collaboration',
|
|
|
|
'Enterprise Software' => 'topic/enterprise-software',
|
|
|
|
'Google' => 'topic/google',
|
|
|
|
'Great debate' => 'topic/great-debate',
|
|
|
|
'Hardware' => 'topic/hardware',
|
|
|
|
'IBM' => 'topic/ibm',
|
|
|
|
'iOS' => 'topic/ios',
|
|
|
|
'iPhone' => 'topic/iphone',
|
|
|
|
'iPad' => 'topic/ipad',
|
|
|
|
'IT Priorities' => 'topic/it-priorities',
|
|
|
|
'Laptops' => 'topic/laptops',
|
|
|
|
'Legal' => 'topic/legal',
|
|
|
|
'Linux' => 'topic/linux',
|
|
|
|
'Microsoft' => 'topic/microsoft',
|
|
|
|
'Mobile OS' => 'topic/mobile-os',
|
|
|
|
'Mobility' => 'topic/mobility',
|
|
|
|
'Networking' => 'topic/networking',
|
|
|
|
'Oracle' => 'topic/oracle',
|
|
|
|
'Processors' => 'topic/processors',
|
|
|
|
'Samsung' => 'topic/samsung',
|
|
|
|
'Security' => 'topic/security',
|
|
|
|
'Small business: going big on mobility' => 'topic/small-business-going-big-on-mobility'
|
|
|
|
),
|
|
|
|
'Product Blogs:' => array(
|
|
|
|
'Digital Cameras & Camcorders' => 'blog/digitalcameras',
|
|
|
|
'Home Theater' => 'blog/home-theater',
|
|
|
|
'Laptops and Desktops' => 'blog/computers',
|
|
|
|
'The Mobile Gadgeteer' => 'blog/mobile-gadgeteer',
|
|
|
|
'Smartphones and Cell Phones' => 'blog/cell-phones',
|
|
|
|
'The ToyBox' => 'blog/gadgetreviews'
|
|
|
|
),
|
|
|
|
'Vertical Blogs:' => array(
|
|
|
|
'ZDNet Education' => 'blog/education',
|
|
|
|
'ZDNet Healthcare' => 'blog/healthcare',
|
|
|
|
'ZDNet Government' => 'blog/government'
|
|
|
|
)
|
|
|
|
)
|
|
|
|
)
|
|
|
|
));
|
2016-07-19 19:38:42 +02:00
|
|
|
|
2017-02-11 16:16:56 +01:00
|
|
|
public function collectData(){
|
Catching up | [Main] Debug mode, parse utils, MIME | [Bridges] Add/Improve 20 bridges (#802)
* Debug mode improvements
- Improve debug warning message
- Restore error reporting in debug mode
- Fix 'notice' messages for unset fields
* Add parsing utility functions
html.php
- extractFromDelimiters
- stripWithDelimiters
- stripRecursiveHTMLSection
- markdownToHtml (partial)
bridges
- remove now-duplicate functions
- call functions from html.php instead
* [Anidex] New bridge
Anime torrent tracker
* [Anime-Ultime] Restore thumbnail
* [CNET] Recreate bridge
Full rewrite as the previous one was broken
* [Dilbert] Minor URI fix
Use new self::URI property
* [EstCeQuonMetEnProd] Fix content extraction
Bridge was broken
* [Facebook] Fix "SpSonsSoriSsés" label
... which was taking space in item title
* [Futura-Sciences] Use HTTPS, More cleanup
Use HTTPS as FS now offer HTTPS
Clean additional useless HTML elements
* [GBATemp] Multiple fixes
- Fix categories: missing "break" statements
- Restore thumbnail as enclosure
- Fix date extraction
- Fix user blog post extraction
- Use getSimpleHTMLDOMCached
* [JapanExpo] Fix bridge, HTTPS, thumbnails
- Fix getSimpleHTMLDOMCached call
- Upgrade to HTTPS as JE now offers HTTPS
- Restore thumbnails as enclosures
* [LeMondeInformatique] Fix bridge, HTTPS
- Upgrade to HTTPS as LMI now offers HTTPS
- Restore thumbnails using small images
- Fix content extraction
- Fix text encoding issue
* [Nextgov] Fix content extraction
- Restore thumbnail and use small image
- Field extraction fixes
* [NextInpact] Add categories and filtering by type
- Offer all RSS feeds
- Allow filtering by article type
- Implement extraction for brief articles
- Remove article limit, many brief articles are publied all at once
* [NyaaTorrents] New bridge
Anime torrent tracker
* [Releases3DS] Cache content, restore thumbnail
- Use getSimpleHTMLDOMCached
- Restore thumbnail as enclosure
* [TheHackerNews] Fix bridge
- Fix content extraction including article body
- Restore thumbnail as enclosure
* [WeLiveSecurity] HTTPS, Fix content extraction
- Upgrade to HTTPS as WLS now offers HTTPS
- Fix content extraction including article body
* [WordPress] Reduce timeout, more content selectors
- Reduce timeout to use default one (1h)
- Add new content selector (articleBody)
- Find thumbnail and set as enclosure
- Fix <script> cleanup
* [YGGTorrent] Increase limit, use cache
- Increase item limit as uploads are very frequent
- Use getSimpleHTMLDOMCached
* [ZDNet] Rewrite with FeedExpander
- Upgrade to HTTPS as ZD now offers HTTPS
- Use FeedExpander for secondary fields
- Fix content extraction for article body
* [Main] Handle MIME type for enclosures
Many feed readers will ignore enclosures (e.g. thumbnails) with no MIME type. This commit adds automatic MIME type detection based on file extension (which may be inaccurate but is the only way without fetching the content).
One can force enclosure type using #.ext anchor (hacky, needs improving)
* [FeedExpander] Improve field extraction
- Add support for passing enclosures
- Improve author and uri extraction
- Fix 'notice' PHP error messages
* [Pull] Coding style fixes for #802
* [Pull] Implementing changes for #802
- Fix coding style issues with str append
- Remove useless CACHE_TIMEOUT
- Use count() instead of $limit
- Use defaultLinkTo() + handle strings
- Use http_build_query()
- Fix missing </em>
- Remove error_reporting(0)
- warning CSS (@LogMANOriginal)
- Fix typo in FeedExpander comment
* [Main] More documentation for markdownToHtml
See #802 for more details
2018-09-09 21:20:13 +02:00
|
|
|
$baseUri = static::URI;
|
2017-02-11 16:16:56 +01:00
|
|
|
$feed = $this->getInput('feed');
|
2017-07-29 19:28:00 +02:00
|
|
|
if(strpos($feed, 'downloads!') !== false) {
|
2017-02-11 16:16:56 +01:00
|
|
|
$feed = str_replace('downloads!', '', $feed);
|
|
|
|
$baseUri = str_replace('www.', 'downloads.', $baseUri);
|
|
|
|
}
|
|
|
|
$url = $baseUri . trim($feed, '/') . '/rss.xml';
|
Catching up | [Main] Debug mode, parse utils, MIME | [Bridges] Add/Improve 20 bridges (#802)
* Debug mode improvements
- Improve debug warning message
- Restore error reporting in debug mode
- Fix 'notice' messages for unset fields
* Add parsing utility functions
html.php
- extractFromDelimiters
- stripWithDelimiters
- stripRecursiveHTMLSection
- markdownToHtml (partial)
bridges
- remove now-duplicate functions
- call functions from html.php instead
* [Anidex] New bridge
Anime torrent tracker
* [Anime-Ultime] Restore thumbnail
* [CNET] Recreate bridge
Full rewrite as the previous one was broken
* [Dilbert] Minor URI fix
Use new self::URI property
* [EstCeQuonMetEnProd] Fix content extraction
Bridge was broken
* [Facebook] Fix "SpSonsSoriSsés" label
... which was taking space in item title
* [Futura-Sciences] Use HTTPS, More cleanup
Use HTTPS as FS now offer HTTPS
Clean additional useless HTML elements
* [GBATemp] Multiple fixes
- Fix categories: missing "break" statements
- Restore thumbnail as enclosure
- Fix date extraction
- Fix user blog post extraction
- Use getSimpleHTMLDOMCached
* [JapanExpo] Fix bridge, HTTPS, thumbnails
- Fix getSimpleHTMLDOMCached call
- Upgrade to HTTPS as JE now offers HTTPS
- Restore thumbnails as enclosures
* [LeMondeInformatique] Fix bridge, HTTPS
- Upgrade to HTTPS as LMI now offers HTTPS
- Restore thumbnails using small images
- Fix content extraction
- Fix text encoding issue
* [Nextgov] Fix content extraction
- Restore thumbnail and use small image
- Field extraction fixes
* [NextInpact] Add categories and filtering by type
- Offer all RSS feeds
- Allow filtering by article type
- Implement extraction for brief articles
- Remove article limit, many brief articles are publied all at once
* [NyaaTorrents] New bridge
Anime torrent tracker
* [Releases3DS] Cache content, restore thumbnail
- Use getSimpleHTMLDOMCached
- Restore thumbnail as enclosure
* [TheHackerNews] Fix bridge
- Fix content extraction including article body
- Restore thumbnail as enclosure
* [WeLiveSecurity] HTTPS, Fix content extraction
- Upgrade to HTTPS as WLS now offers HTTPS
- Fix content extraction including article body
* [WordPress] Reduce timeout, more content selectors
- Reduce timeout to use default one (1h)
- Add new content selector (articleBody)
- Find thumbnail and set as enclosure
- Fix <script> cleanup
* [YGGTorrent] Increase limit, use cache
- Increase item limit as uploads are very frequent
- Use getSimpleHTMLDOMCached
* [ZDNet] Rewrite with FeedExpander
- Upgrade to HTTPS as ZD now offers HTTPS
- Use FeedExpander for secondary fields
- Fix content extraction for article body
* [Main] Handle MIME type for enclosures
Many feed readers will ignore enclosures (e.g. thumbnails) with no MIME type. This commit adds automatic MIME type detection based on file extension (which may be inaccurate but is the only way without fetching the content).
One can force enclosure type using #.ext anchor (hacky, needs improving)
* [FeedExpander] Improve field extraction
- Add support for passing enclosures
- Improve author and uri extraction
- Fix 'notice' PHP error messages
* [Pull] Coding style fixes for #802
* [Pull] Implementing changes for #802
- Fix coding style issues with str append
- Remove useless CACHE_TIMEOUT
- Use count() instead of $limit
- Use defaultLinkTo() + handle strings
- Use http_build_query()
- Fix missing </em>
- Remove error_reporting(0)
- warning CSS (@LogMANOriginal)
- Fix typo in FeedExpander comment
* [Main] More documentation for markdownToHtml
See #802 for more details
2018-09-09 21:20:13 +02:00
|
|
|
$this->collectExpandableDatas($url);
|
|
|
|
}
|
2017-02-11 16:16:56 +01:00
|
|
|
|
Catching up | [Main] Debug mode, parse utils, MIME | [Bridges] Add/Improve 20 bridges (#802)
* Debug mode improvements
- Improve debug warning message
- Restore error reporting in debug mode
- Fix 'notice' messages for unset fields
* Add parsing utility functions
html.php
- extractFromDelimiters
- stripWithDelimiters
- stripRecursiveHTMLSection
- markdownToHtml (partial)
bridges
- remove now-duplicate functions
- call functions from html.php instead
* [Anidex] New bridge
Anime torrent tracker
* [Anime-Ultime] Restore thumbnail
* [CNET] Recreate bridge
Full rewrite as the previous one was broken
* [Dilbert] Minor URI fix
Use new self::URI property
* [EstCeQuonMetEnProd] Fix content extraction
Bridge was broken
* [Facebook] Fix "SpSonsSoriSsés" label
... which was taking space in item title
* [Futura-Sciences] Use HTTPS, More cleanup
Use HTTPS as FS now offer HTTPS
Clean additional useless HTML elements
* [GBATemp] Multiple fixes
- Fix categories: missing "break" statements
- Restore thumbnail as enclosure
- Fix date extraction
- Fix user blog post extraction
- Use getSimpleHTMLDOMCached
* [JapanExpo] Fix bridge, HTTPS, thumbnails
- Fix getSimpleHTMLDOMCached call
- Upgrade to HTTPS as JE now offers HTTPS
- Restore thumbnails as enclosures
* [LeMondeInformatique] Fix bridge, HTTPS
- Upgrade to HTTPS as LMI now offers HTTPS
- Restore thumbnails using small images
- Fix content extraction
- Fix text encoding issue
* [Nextgov] Fix content extraction
- Restore thumbnail and use small image
- Field extraction fixes
* [NextInpact] Add categories and filtering by type
- Offer all RSS feeds
- Allow filtering by article type
- Implement extraction for brief articles
- Remove article limit, many brief articles are publied all at once
* [NyaaTorrents] New bridge
Anime torrent tracker
* [Releases3DS] Cache content, restore thumbnail
- Use getSimpleHTMLDOMCached
- Restore thumbnail as enclosure
* [TheHackerNews] Fix bridge
- Fix content extraction including article body
- Restore thumbnail as enclosure
* [WeLiveSecurity] HTTPS, Fix content extraction
- Upgrade to HTTPS as WLS now offers HTTPS
- Fix content extraction including article body
* [WordPress] Reduce timeout, more content selectors
- Reduce timeout to use default one (1h)
- Add new content selector (articleBody)
- Find thumbnail and set as enclosure
- Fix <script> cleanup
* [YGGTorrent] Increase limit, use cache
- Increase item limit as uploads are very frequent
- Use getSimpleHTMLDOMCached
* [ZDNet] Rewrite with FeedExpander
- Upgrade to HTTPS as ZD now offers HTTPS
- Use FeedExpander for secondary fields
- Fix content extraction for article body
* [Main] Handle MIME type for enclosures
Many feed readers will ignore enclosures (e.g. thumbnails) with no MIME type. This commit adds automatic MIME type detection based on file extension (which may be inaccurate but is the only way without fetching the content).
One can force enclosure type using #.ext anchor (hacky, needs improving)
* [FeedExpander] Improve field extraction
- Add support for passing enclosures
- Improve author and uri extraction
- Fix 'notice' PHP error messages
* [Pull] Coding style fixes for #802
* [Pull] Implementing changes for #802
- Fix coding style issues with str append
- Remove useless CACHE_TIMEOUT
- Use count() instead of $limit
- Use defaultLinkTo() + handle strings
- Use http_build_query()
- Fix missing </em>
- Remove error_reporting(0)
- warning CSS (@LogMANOriginal)
- Fix typo in FeedExpander comment
* [Main] More documentation for markdownToHtml
See #802 for more details
2018-09-09 21:20:13 +02:00
|
|
|
protected function parseItem($item){
|
|
|
|
$item = parent::parseItem($item);
|
|
|
|
|
|
|
|
$article = getSimpleHTMLDOMCached($item['uri']);
|
|
|
|
if(!$article)
|
|
|
|
returnServerError('Could not request ZDNet: ' . $url);
|
|
|
|
|
|
|
|
$contents = $article->find('article', 0)->innertext;
|
|
|
|
foreach(array(
|
|
|
|
'<div class="shareBar"',
|
|
|
|
'<div class="shortcodeGalleryWrapper"',
|
|
|
|
'<div class="relatedContent',
|
|
|
|
'<div class="downloadNow',
|
|
|
|
'<div data-shortcode',
|
|
|
|
'<div id="sharethrough',
|
|
|
|
'<div id="inpage-video'
|
|
|
|
) as $div_start) {
|
|
|
|
$contents = stripRecursiveHtmlSection($contents, 'div', $div_start);
|
2017-02-11 16:16:56 +01:00
|
|
|
}
|
Catching up | [Main] Debug mode, parse utils, MIME | [Bridges] Add/Improve 20 bridges (#802)
* Debug mode improvements
- Improve debug warning message
- Restore error reporting in debug mode
- Fix 'notice' messages for unset fields
* Add parsing utility functions
html.php
- extractFromDelimiters
- stripWithDelimiters
- stripRecursiveHTMLSection
- markdownToHtml (partial)
bridges
- remove now-duplicate functions
- call functions from html.php instead
* [Anidex] New bridge
Anime torrent tracker
* [Anime-Ultime] Restore thumbnail
* [CNET] Recreate bridge
Full rewrite as the previous one was broken
* [Dilbert] Minor URI fix
Use new self::URI property
* [EstCeQuonMetEnProd] Fix content extraction
Bridge was broken
* [Facebook] Fix "SpSonsSoriSsés" label
... which was taking space in item title
* [Futura-Sciences] Use HTTPS, More cleanup
Use HTTPS as FS now offer HTTPS
Clean additional useless HTML elements
* [GBATemp] Multiple fixes
- Fix categories: missing "break" statements
- Restore thumbnail as enclosure
- Fix date extraction
- Fix user blog post extraction
- Use getSimpleHTMLDOMCached
* [JapanExpo] Fix bridge, HTTPS, thumbnails
- Fix getSimpleHTMLDOMCached call
- Upgrade to HTTPS as JE now offers HTTPS
- Restore thumbnails as enclosures
* [LeMondeInformatique] Fix bridge, HTTPS
- Upgrade to HTTPS as LMI now offers HTTPS
- Restore thumbnails using small images
- Fix content extraction
- Fix text encoding issue
* [Nextgov] Fix content extraction
- Restore thumbnail and use small image
- Field extraction fixes
* [NextInpact] Add categories and filtering by type
- Offer all RSS feeds
- Allow filtering by article type
- Implement extraction for brief articles
- Remove article limit, many brief articles are publied all at once
* [NyaaTorrents] New bridge
Anime torrent tracker
* [Releases3DS] Cache content, restore thumbnail
- Use getSimpleHTMLDOMCached
- Restore thumbnail as enclosure
* [TheHackerNews] Fix bridge
- Fix content extraction including article body
- Restore thumbnail as enclosure
* [WeLiveSecurity] HTTPS, Fix content extraction
- Upgrade to HTTPS as WLS now offers HTTPS
- Fix content extraction including article body
* [WordPress] Reduce timeout, more content selectors
- Reduce timeout to use default one (1h)
- Add new content selector (articleBody)
- Find thumbnail and set as enclosure
- Fix <script> cleanup
* [YGGTorrent] Increase limit, use cache
- Increase item limit as uploads are very frequent
- Use getSimpleHTMLDOMCached
* [ZDNet] Rewrite with FeedExpander
- Upgrade to HTTPS as ZD now offers HTTPS
- Use FeedExpander for secondary fields
- Fix content extraction for article body
* [Main] Handle MIME type for enclosures
Many feed readers will ignore enclosures (e.g. thumbnails) with no MIME type. This commit adds automatic MIME type detection based on file extension (which may be inaccurate but is the only way without fetching the content).
One can force enclosure type using #.ext anchor (hacky, needs improving)
* [FeedExpander] Improve field extraction
- Add support for passing enclosures
- Improve author and uri extraction
- Fix 'notice' PHP error messages
* [Pull] Coding style fixes for #802
* [Pull] Implementing changes for #802
- Fix coding style issues with str append
- Remove useless CACHE_TIMEOUT
- Use count() instead of $limit
- Use defaultLinkTo() + handle strings
- Use http_build_query()
- Fix missing </em>
- Remove error_reporting(0)
- warning CSS (@LogMANOriginal)
- Fix typo in FeedExpander comment
* [Main] More documentation for markdownToHtml
See #802 for more details
2018-09-09 21:20:13 +02:00
|
|
|
$contents = stripWithDelimiters($contents, '<script', '</script>');
|
|
|
|
$contents = stripWithDelimiters($contents, '<meta itemprop="image"', '>');
|
|
|
|
$contents = stripWithDelimiters($contents, '<svg class="svg-symbol', '</svg>');
|
|
|
|
$contents = trim(stripWithDelimiters($contents, '<section class="sharethrough-top', '</section>'));
|
|
|
|
$item['content'] = $contents;
|
|
|
|
|
|
|
|
return $item;
|
2017-02-11 16:16:56 +01:00
|
|
|
|
|
|
|
}
|
2016-07-26 21:19:39 +02:00
|
|
|
}
|