Catching up | [Main] Debug mode, parse utils, MIME | [Bridges] Add/Improve 20 bridges (#802)
* Debug mode improvements - Improve debug warning message - Restore error reporting in debug mode - Fix 'notice' messages for unset fields * Add parsing utility functions html.php - extractFromDelimiters - stripWithDelimiters - stripRecursiveHTMLSection - markdownToHtml (partial) bridges - remove now-duplicate functions - call functions from html.php instead * [Anidex] New bridge Anime torrent tracker * [Anime-Ultime] Restore thumbnail * [CNET] Recreate bridge Full rewrite as the previous one was broken * [Dilbert] Minor URI fix Use new self::URI property * [EstCeQuonMetEnProd] Fix content extraction Bridge was broken * [Facebook] Fix "SpSonsSoriSsés" label ... which was taking space in item title * [Futura-Sciences] Use HTTPS, More cleanup Use HTTPS as FS now offer HTTPS Clean additional useless HTML elements * [GBATemp] Multiple fixes - Fix categories: missing "break" statements - Restore thumbnail as enclosure - Fix date extraction - Fix user blog post extraction - Use getSimpleHTMLDOMCached * [JapanExpo] Fix bridge, HTTPS, thumbnails - Fix getSimpleHTMLDOMCached call - Upgrade to HTTPS as JE now offers HTTPS - Restore thumbnails as enclosures * [LeMondeInformatique] Fix bridge, HTTPS - Upgrade to HTTPS as LMI now offers HTTPS - Restore thumbnails using small images - Fix content extraction - Fix text encoding issue * [Nextgov] Fix content extraction - Restore thumbnail and use small image - Field extraction fixes * [NextInpact] Add categories and filtering by type - Offer all RSS feeds - Allow filtering by article type - Implement extraction for brief articles - Remove article limit, many brief articles are publied all at once * [NyaaTorrents] New bridge Anime torrent tracker * [Releases3DS] Cache content, restore thumbnail - Use getSimpleHTMLDOMCached - Restore thumbnail as enclosure * [TheHackerNews] Fix bridge - Fix content extraction including article body - Restore thumbnail as enclosure * [WeLiveSecurity] HTTPS, Fix content extraction - Upgrade to HTTPS as WLS now offers HTTPS - Fix content extraction including article body * [WordPress] Reduce timeout, more content selectors - Reduce timeout to use default one (1h) - Add new content selector (articleBody) - Find thumbnail and set as enclosure - Fix <script> cleanup * [YGGTorrent] Increase limit, use cache - Increase item limit as uploads are very frequent - Use getSimpleHTMLDOMCached * [ZDNet] Rewrite with FeedExpander - Upgrade to HTTPS as ZD now offers HTTPS - Use FeedExpander for secondary fields - Fix content extraction for article body * [Main] Handle MIME type for enclosures Many feed readers will ignore enclosures (e.g. thumbnails) with no MIME type. This commit adds automatic MIME type detection based on file extension (which may be inaccurate but is the only way without fetching the content). One can force enclosure type using #.ext anchor (hacky, needs improving) * [FeedExpander] Improve field extraction - Add support for passing enclosures - Improve author and uri extraction - Fix 'notice' PHP error messages * [Pull] Coding style fixes for #802 * [Pull] Implementing changes for #802 - Fix coding style issues with str append - Remove useless CACHE_TIMEOUT - Use count() instead of $limit - Use defaultLinkTo() + handle strings - Use http_build_query() - Fix missing </em> - Remove error_reporting(0) - warning CSS (@LogMANOriginal) - Fix typo in FeedExpander comment * [Main] More documentation for markdownToHtml See #802 for more details
This commit is contained in:
ORelio
Teromene
parent
123fce4394
commit
de8cee6a1c
29 changed files with 1045 additions and 453 deletions
207
bridges/AnidexBridge.php
Normal file
207
bridges/AnidexBridge.php
Normal file
|
|
@ -0,0 +1,207 @@
|
||||||
|
<?php
|
||||||
|
class AnidexBridge extends BridgeAbstract {
|
||||||
|
|
||||||
|
const MAINTAINER = 'ORelio';
|
||||||
|
const NAME = 'Anidex';
|
||||||
|
const URI = 'https://anidex.info/';
|
||||||
|
const DESCRIPTION = 'Returns the newest torrents, with optional search criteria.';
|
||||||
|
const PARAMETERS = array(
|
||||||
|
array(
|
||||||
|
'id' => array(
|
||||||
|
'name' => 'Category',
|
||||||
|
'type' => 'list',
|
||||||
|
'values' => array(
|
||||||
|
'All categories' => '0',
|
||||||
|
'Anime' => '1,2,3',
|
||||||
|
'Anime - Sub' => '1',
|
||||||
|
'Anime - Raw' => '2',
|
||||||
|
'Anime - Dub' => '3',
|
||||||
|
'Live Action' => '4,5',
|
||||||
|
'Live Action - Sub' => '4',
|
||||||
|
'Live Action - Raw' => '5',
|
||||||
|
'Light Novel' => '6',
|
||||||
|
'Manga' => '7,8',
|
||||||
|
'Manga - Translated' => '7',
|
||||||
|
'Manga - Raw' => '8',
|
||||||
|
'Music' => '9,10,11',
|
||||||
|
'Music - Lossy' => '9',
|
||||||
|
'Music - Lossless' => '10',
|
||||||
|
'Music - Video' => '11',
|
||||||
|
'Games' => '12',
|
||||||
|
'Applications' => '13',
|
||||||
|
'Pictures' => '14',
|
||||||
|
'Adult Video' => '15',
|
||||||
|
'Other' => '16'
|
||||||
|
)
|
||||||
|
),
|
||||||
|
'lang_id' => array(
|
||||||
|
'name' => 'Language',
|
||||||
|
'type' => 'list',
|
||||||
|
'values' => array(
|
||||||
|
'All languages' => '0',
|
||||||
|
'English' => '1',
|
||||||
|
'Japanese' => '2',
|
||||||
|
'Polish' => '3',
|
||||||
|
'Serbo-Croatian' => '4',
|
||||||
|
'Dutch' => '5',
|
||||||
|
'Italian' => '6',
|
||||||
|
'Russian' => '7',
|
||||||
|
'German' => '8',
|
||||||
|
'Hungarian' => '9',
|
||||||
|
'French' => '10',
|
||||||
|
'Finnish' => '11',
|
||||||
|
'Vietnamese' => '12',
|
||||||
|
'Greek' => '13',
|
||||||
|
'Bulgarian' => '14',
|
||||||
|
'Spanish (Spain)' => '15',
|
||||||
|
'Portuguese (Brazil)' => '16',
|
||||||
|
'Portuguese (Portugal)' => '17',
|
||||||
|
'Swedish' => '18',
|
||||||
|
'Arabic' => '19',
|
||||||
|
'Danish' => '20',
|
||||||
|
'Chinese (Simplified)' => '21',
|
||||||
|
'Bengali' => '22',
|
||||||
|
'Romanian' => '23',
|
||||||
|
'Czech' => '24',
|
||||||
|
'Mongolian' => '25',
|
||||||
|
'Turkish' => '26',
|
||||||
|
'Indonesian' => '27',
|
||||||
|
'Korean' => '28',
|
||||||
|
'Spanish (LATAM)' => '29',
|
||||||
|
'Persian' => '30',
|
||||||
|
'Malaysian' => '31'
|
||||||
|
)
|
||||||
|
),
|
||||||
|
'group_id' => array(
|
||||||
|
'name' => 'Group ID',
|
||||||
|
'type' => 'number'
|
||||||
|
),
|
||||||
|
'r' => array(
|
||||||
|
'name' => 'Hide Remakes',
|
||||||
|
'type' => 'checkbox'
|
||||||
|
),
|
||||||
|
'b' => array(
|
||||||
|
'name' => 'Only Batches',
|
||||||
|
'type' => 'checkbox'
|
||||||
|
),
|
||||||
|
'a' => array(
|
||||||
|
'name' => 'Only Authorized',
|
||||||
|
'type' => 'checkbox'
|
||||||
|
),
|
||||||
|
'q' => array(
|
||||||
|
'name' => 'Keyword',
|
||||||
|
'description' => 'Keyword(s)',
|
||||||
|
'type' => 'text'
|
||||||
|
),
|
||||||
|
'h' => array(
|
||||||
|
'name' => 'Adult content',
|
||||||
|
'type' => 'list',
|
||||||
|
'values' => array(
|
||||||
|
'No filter' => '0',
|
||||||
|
'Hide +18' => '1',
|
||||||
|
'Only +18' => '2'
|
||||||
|
)
|
||||||
|
)
|
||||||
|
)
|
||||||
|
);
|
||||||
|
|
||||||
|
public function collectData() {
|
||||||
|
|
||||||
|
// Build Search URL from user-provided parameters
|
||||||
|
$search_url = self::URI . '?s=upload_timestamp&o=desc';
|
||||||
|
foreach (array('id', 'lang_id', 'group_id') as $param_name) {
|
||||||
|
$param = $this->getInput($param_name);
|
||||||
|
if (!empty($param) && intval($param) != 0 && ctype_digit(str_replace(',', '', $param))) {
|
||||||
|
$search_url .= '&' . $param_name . '=' . $param;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
foreach (array('r', 'b', 'a') as $param_name) {
|
||||||
|
$param = $this->getInput($param_name);
|
||||||
|
if (!empty($param) && boolval($param)) {
|
||||||
|
$search_url .= '&' . $param_name . '=1';
|
||||||
|
}
|
||||||
|
}
|
||||||
|
$query = $this->getInput('q');
|
||||||
|
if (!empty($query)) {
|
||||||
|
$search_url .= '&q=' . urlencode($query);
|
||||||
|
}
|
||||||
|
$opt = array();
|
||||||
|
$h = $this->getInput('h');
|
||||||
|
if (!empty($h) && intval($h) != 0 && ctype_digit($h)) {
|
||||||
|
$opt[CURLOPT_COOKIE] = 'anidex_h_toggle=' . $h;
|
||||||
|
}
|
||||||
|
|
||||||
|
// Retrieve torrent listing from search results, which does not contain torrent description
|
||||||
|
$html = getSimpleHTMLDOM($search_url, array(), $opt)
|
||||||
|
or returnServerError('Could not request Anidex: ' . $search_url);
|
||||||
|
$links = $html->find('a');
|
||||||
|
$results = array();
|
||||||
|
foreach ($links as $link)
|
||||||
|
if (strpos($link->href, '/torrent/') === 0 && !in_array($link->href, $results))
|
||||||
|
$results[] = $link->href;
|
||||||
|
if (empty($results) && empty($this->getInput('q')))
|
||||||
|
returnServerError('No results from Anidex: '.$search_url);
|
||||||
|
|
||||||
|
//Process each item individually
|
||||||
|
foreach ($results as $element) {
|
||||||
|
|
||||||
|
//Limit total amount of requests
|
||||||
|
if(count($this->items) >= 20) {
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
|
||||||
|
$torrent_id = str_replace('/torrent/', '', $element);
|
||||||
|
|
||||||
|
//Ignore entries without valid torrent ID
|
||||||
|
if ($torrent_id != 0 && ctype_digit($torrent_id)) {
|
||||||
|
|
||||||
|
//Retrieve data for this torrent ID
|
||||||
|
$item_uri = self::URI . 'torrent/'.$torrent_id;
|
||||||
|
|
||||||
|
//Retrieve full description from torrent page
|
||||||
|
if ($item_html = getSimpleHTMLDOMCached($item_uri)) {
|
||||||
|
|
||||||
|
//Retrieve data from page contents
|
||||||
|
$item_title = str_replace(' (Torrent) - AniDex ', '', $item_html->find('title', 0)->plaintext);
|
||||||
|
$item_desc = $item_html->find('div.panel-body', 0);
|
||||||
|
$item_author = trim($item_html->find('span.fa-user', 0)->parent()->plaintext);
|
||||||
|
$item_date = strtotime(trim($item_html->find('span.fa-clock', 0)->parent()->plaintext));
|
||||||
|
$item_image = $this->getURI() . 'images/user_logos/default.png';
|
||||||
|
|
||||||
|
//Check for description-less torrent andn optionally extract image
|
||||||
|
$desc_title_found = false;
|
||||||
|
foreach ($item_html->find('h3.panel-title') as $h3) {
|
||||||
|
if (strpos($h3, 'Description') !== false) {
|
||||||
|
$desc_title_found = true;
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if ($desc_title_found) {
|
||||||
|
//Retrieve image for thumbnail or generic logo fallback
|
||||||
|
foreach ($item_desc->find('img') as $img) {
|
||||||
|
if (strpos($img->src, 'prez') === false) {
|
||||||
|
$item_image = $img->src;
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
$item_desc = trim($item_desc->innertext);
|
||||||
|
} else {
|
||||||
|
$item_desc = '<em>No description.</em>';
|
||||||
|
}
|
||||||
|
|
||||||
|
//Build and add final item
|
||||||
|
$item = array();
|
||||||
|
$item['uri'] = $item_uri;
|
||||||
|
$item['title'] = $item_title;
|
||||||
|
$item['author'] = $item_author;
|
||||||
|
$item['timestamp'] = $item_date;
|
||||||
|
$item['enclosures'] = array($item_image);
|
||||||
|
$item['content'] = $item_desc;
|
||||||
|
$this->items[] = $item;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
$element = null;
|
||||||
|
}
|
||||||
|
$results = null;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
@ -5,7 +5,7 @@ class AnimeUltimeBridge extends BridgeAbstract {
|
||||||
const NAME = 'Anime-Ultime';
|
const NAME = 'Anime-Ultime';
|
||||||
const URI = 'http://www.anime-ultime.net/';
|
const URI = 'http://www.anime-ultime.net/';
|
||||||
const CACHE_TIMEOUT = 10800; // 3h
|
const CACHE_TIMEOUT = 10800; // 3h
|
||||||
const DESCRIPTION = 'Returns the 10 newest releases posted on Anime-Ultime';
|
const DESCRIPTION = 'Returns the newest releases posted on Anime-Ultime.';
|
||||||
const PARAMETERS = array( array(
|
const PARAMETERS = array( array(
|
||||||
'type' => array(
|
'type' => array(
|
||||||
'name' => 'Type',
|
'name' => 'Type',
|
||||||
|
|
@ -65,6 +65,13 @@ class AnimeUltimeBridge extends BridgeAbstract {
|
||||||
$item_link_element = $release->find('td', 0)->find('a', 0);
|
$item_link_element = $release->find('td', 0)->find('a', 0);
|
||||||
$item_uri = self::URI . $item_link_element->href;
|
$item_uri = self::URI . $item_link_element->href;
|
||||||
$item_name = html_entity_decode($item_link_element->plaintext);
|
$item_name = html_entity_decode($item_link_element->plaintext);
|
||||||
|
|
||||||
|
$item_image = self::URI . substr(
|
||||||
|
$item_link_element->onmouseover,
|
||||||
|
37,
|
||||||
|
strpos($item_link_element->onmouseover, ' ', 37) - 37
|
||||||
|
);
|
||||||
|
|
||||||
$item_episode = html_entity_decode(
|
$item_episode = html_entity_decode(
|
||||||
str_pad(
|
str_pad(
|
||||||
$release->find('td', 1)->plaintext,
|
$release->find('td', 1)->plaintext,
|
||||||
|
|
@ -79,8 +86,7 @@ class AnimeUltimeBridge extends BridgeAbstract {
|
||||||
|
|
||||||
if(!empty($item_uri)) {
|
if(!empty($item_uri)) {
|
||||||
|
|
||||||
// Retrieve description from description page and
|
// Retrieve description from description page
|
||||||
// convert relative image src info absolute image src
|
|
||||||
$html_item = getContents($item_uri)
|
$html_item = getContents($item_uri)
|
||||||
or returnServerError('Could not request Anime-Ultime: ' . $item_uri);
|
or returnServerError('Could not request Anime-Ultime: ' . $item_uri);
|
||||||
$item_description = substr(
|
$item_description = substr(
|
||||||
|
|
@ -91,10 +97,9 @@ class AnimeUltimeBridge extends BridgeAbstract {
|
||||||
0,
|
0,
|
||||||
strpos($item_description, '<div id="table">')
|
strpos($item_description, '<div id="table">')
|
||||||
);
|
);
|
||||||
$item_description = str_replace(
|
|
||||||
'src="images', 'src="' . self::URI . 'images',
|
// Convert relative image src into absolute image src, remove line breaks
|
||||||
$item_description
|
$item_description = defaultLinkTo($item_description, self::URI);
|
||||||
);
|
|
||||||
$item_description = str_replace("\r", '', $item_description);
|
$item_description = str_replace("\r", '', $item_description);
|
||||||
$item_description = str_replace("\n", '', $item_description);
|
$item_description = str_replace("\n", '', $item_description);
|
||||||
$item_description = utf8_encode($item_description);
|
$item_description = utf8_encode($item_description);
|
||||||
|
|
@ -105,6 +110,7 @@ class AnimeUltimeBridge extends BridgeAbstract {
|
||||||
$item['title'] = $item_name . ' ' . $item_type . ' ' . $item_episode;
|
$item['title'] = $item_name . ' ' . $item_type . ' ' . $item_episode;
|
||||||
$item['author'] = $item_fansub;
|
$item['author'] = $item_fansub;
|
||||||
$item['timestamp'] = $item_date;
|
$item['timestamp'] = $item_date;
|
||||||
|
$item['enclosures'] = array($item_image);
|
||||||
$item['content'] = $item_description;
|
$item['content'] = $item_description;
|
||||||
$this->items[] = $item;
|
$this->items[] = $item;
|
||||||
$processedOK++;
|
$processedOK++;
|
||||||
|
|
|
||||||
109
bridges/CNETBridge.php
Normal file
109
bridges/CNETBridge.php
Normal file
|
|
@ -0,0 +1,109 @@
|
||||||
|
<?php
|
||||||
|
class CNETBridge extends BridgeAbstract {
|
||||||
|
|
||||||
|
const MAINTAINER = 'ORelio';
|
||||||
|
const NAME = 'CNET News';
|
||||||
|
const URI = 'https://www.cnet.com/';
|
||||||
|
const CACHE_TIMEOUT = 3600; // 1h
|
||||||
|
const DESCRIPTION = 'Returns the newest articles.';
|
||||||
|
const PARAMETERS = array(
|
||||||
|
array(
|
||||||
|
'topic' => array(
|
||||||
|
'name' => 'Topic',
|
||||||
|
'type' => 'list',
|
||||||
|
'values' => array(
|
||||||
|
'All articles' => '',
|
||||||
|
'Apple' => 'apple',
|
||||||
|
'Google' => 'google',
|
||||||
|
'Microsoft' => 'tags-microsoft',
|
||||||
|
'Computers' => 'topics-computers',
|
||||||
|
'Mobile' => 'topics-mobile',
|
||||||
|
'Sci-Tech' => 'topics-sci-tech',
|
||||||
|
'Security' => 'topics-security',
|
||||||
|
'Internet' => 'topics-internet',
|
||||||
|
'Tech Industry' => 'topics-tech-industry'
|
||||||
|
)
|
||||||
|
)
|
||||||
|
)
|
||||||
|
);
|
||||||
|
|
||||||
|
private function cleanArticle($article_html) {
|
||||||
|
$offset_p = strpos($article_html, '<p>');
|
||||||
|
$offset_figure = strpos($article_html, '<figure');
|
||||||
|
$offset = ($offset_figure < $offset_p ? $offset_figure : $offset_p);
|
||||||
|
$article_html = substr($article_html, $offset);
|
||||||
|
$article_html = str_replace('href="/', 'href="' . self::URI, $article_html);
|
||||||
|
$article_html = str_replace(' height="0"', '', $article_html);
|
||||||
|
$article_html = str_replace('<noscript>', '', $article_html);
|
||||||
|
$article_html = str_replace('</noscript>', '', $article_html);
|
||||||
|
$article_html = StripWithDelimiters($article_html, '<a class="clickToEnlarge', '</a>');
|
||||||
|
$article_html = stripWithDelimiters($article_html, '<span class="nowPlaying', '</span>');
|
||||||
|
$article_html = stripWithDelimiters($article_html, '<span class="duration', '</span>');
|
||||||
|
$article_html = stripWithDelimiters($article_html, '<script', '</script>');
|
||||||
|
$article_html = stripWithDelimiters($article_html, '<svg', '</svg>');
|
||||||
|
return $article_html;
|
||||||
|
}
|
||||||
|
|
||||||
|
public function collectData() {
|
||||||
|
|
||||||
|
// Retrieve and check user input
|
||||||
|
$topic = str_replace('-', '/', $this->getInput('topic'));
|
||||||
|
if (!empty($topic) && (substr_count($topic, '/') > 1 || !ctype_alpha(str_replace('/', '', $topic))))
|
||||||
|
returnClientError('Invalid topic: ' . $topic);
|
||||||
|
|
||||||
|
// Retrieve webpage
|
||||||
|
$pageUrl = self::URI . (empty($topic) ? 'news/' : $topic.'/');
|
||||||
|
$html = getSimpleHTMLDOM($pageUrl)
|
||||||
|
or returnServerError('Could not request CNET: '.$pageUrl);
|
||||||
|
|
||||||
|
// Process articles
|
||||||
|
foreach($html->find('div.assetBody, div.riverPost') as $element) {
|
||||||
|
|
||||||
|
if(count($this->items) >= 10) {
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
|
||||||
|
$article_title = trim($element->find('h2, h3', 0)->plaintext);
|
||||||
|
$article_uri = self::URI . substr($element->find('a', 0)->href, 1);
|
||||||
|
$article_thumbnail = $element->parent()->find('img[src]', 0)->src;
|
||||||
|
$article_timestamp = strtotime($element->find('time.assetTime, div.timeAgo', 0)->plaintext);
|
||||||
|
$article_author = trim($element->find('a[rel=author], a.name', 0)->plaintext);
|
||||||
|
$article_content = '<p><b>' . trim($element->find('p.dek', 0)->plaintext) . '</b></p>';
|
||||||
|
|
||||||
|
if (is_null($article_thumbnail))
|
||||||
|
$article_thumbnail = extractFromDelimiters($element->innertext, '<img src="', '"');
|
||||||
|
|
||||||
|
if (!empty($article_title) && !empty($article_uri) && strpos($article_uri, self::URI . 'news/') !== false) {
|
||||||
|
|
||||||
|
$article_html = getSimpleHTMLDOMCached($article_uri) or $article_html = null;
|
||||||
|
|
||||||
|
if (!is_null($article_html)) {
|
||||||
|
|
||||||
|
if (empty($article_thumbnail))
|
||||||
|
$article_thumbnail = $article_html->find('div.originalImage', 0);
|
||||||
|
if (empty($article_thumbnail))
|
||||||
|
$article_thumbnail = $article_html->find('span.imageContainer', 0);
|
||||||
|
if (is_object($article_thumbnail))
|
||||||
|
$article_thumbnail = $article_thumbnail->find('img', 0)->src;
|
||||||
|
|
||||||
|
$article_content .= trim(
|
||||||
|
$this->cleanArticle(
|
||||||
|
extractFromDelimiters(
|
||||||
|
$article_html, '<article', '<footer'
|
||||||
|
)
|
||||||
|
)
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
$item = array();
|
||||||
|
$item['uri'] = $article_uri;
|
||||||
|
$item['title'] = $article_title;
|
||||||
|
$item['author'] = $article_author;
|
||||||
|
$item['timestamp'] = $article_timestamp;
|
||||||
|
$item['enclosures'] = array($article_thumbnail);
|
||||||
|
$item['content'] = $article_content;
|
||||||
|
$this->items[] = $item;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
@ -9,8 +9,8 @@ class DilbertBridge extends BridgeAbstract {
|
||||||
|
|
||||||
public function collectData(){
|
public function collectData(){
|
||||||
|
|
||||||
$html = getSimpleHTMLDOM($this->getURI())
|
$html = getSimpleHTMLDOM(self::URI)
|
||||||
or returnServerError('Could not request Dilbert: ' . $this->getURI());
|
or returnServerError('Could not request Dilbert: ' . self::URI);
|
||||||
|
|
||||||
foreach($html->find('section.comic-item') as $element) {
|
foreach($html->find('section.comic-item') as $element) {
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -7,19 +7,9 @@ class EstCeQuonMetEnProdBridge extends BridgeAbstract {
|
||||||
const CACHE_TIMEOUT = 21600; // 6h
|
const CACHE_TIMEOUT = 21600; // 6h
|
||||||
const DESCRIPTION = 'Should we put a website in production today? (French)';
|
const DESCRIPTION = 'Should we put a website in production today? (French)';
|
||||||
|
|
||||||
public function collectData(){
|
public function collectData() {
|
||||||
function extractFromDelimiters($string, $start, $end){
|
$html = getSimpleHTMLDOM(self::URI)
|
||||||
if(strpos($string, $start) !== false) {
|
or returnServerError('Could not request EstCeQuonMetEnProd: ' . self::URI);
|
||||||
$section_retrieved = substr($string, strpos($string, $start) + strlen($start));
|
|
||||||
$section_retrieved = substr($section_retrieved, 0, strpos($section_retrieved, $end));
|
|
||||||
return $section_retrieved;
|
|
||||||
}
|
|
||||||
|
|
||||||
return false;
|
|
||||||
}
|
|
||||||
|
|
||||||
$html = getSimpleHTMLDOM($this->getURI())
|
|
||||||
or returnServerError('Could not request EstCeQuonMetEnProd: ' . $this->getURI());
|
|
||||||
|
|
||||||
$item = array();
|
$item = array();
|
||||||
$item['uri'] = $this->getURI() . '#' . date('Y-m-d');
|
$item['uri'] = $this->getURI() . '#' . date('Y-m-d');
|
||||||
|
|
@ -28,8 +18,8 @@ class EstCeQuonMetEnProdBridge extends BridgeAbstract {
|
||||||
$item['timestamp'] = strtotime('today midnight');
|
$item['timestamp'] = strtotime('today midnight');
|
||||||
$item['content'] = str_replace(
|
$item['content'] = str_replace(
|
||||||
'src="/',
|
'src="/',
|
||||||
'src="' . $this->getURI(),
|
'src="' . self::URI,
|
||||||
trim(extractFromDelimiters($html->outertext, '<body role="document">', '<br /><br />'))
|
trim(extractFromDelimiters($html->outertext, '<body role="document">', '<div id="share'))
|
||||||
);
|
);
|
||||||
|
|
||||||
$this->items[] = $item;
|
$this->items[] = $item;
|
||||||
|
|
|
||||||
|
|
@ -253,17 +253,6 @@ class FacebookBridge extends BridgeAbstract {
|
||||||
|
|
||||||
private function collectUserData(){
|
private function collectUserData(){
|
||||||
|
|
||||||
//Extract a string using start and end delimiters
|
|
||||||
function extractFromDelimiters($string, $start, $end){
|
|
||||||
if(strpos($string, $start) !== false) {
|
|
||||||
$section_retrieved = substr($string, strpos($string, $start) + strlen($start));
|
|
||||||
$section_retrieved = substr($section_retrieved, 0, strpos($section_retrieved, $end));
|
|
||||||
return $section_retrieved;
|
|
||||||
}
|
|
||||||
|
|
||||||
return false;
|
|
||||||
}
|
|
||||||
|
|
||||||
//Utility function for cleaning a Facebook link
|
//Utility function for cleaning a Facebook link
|
||||||
$unescape_fb_link = function($matches){
|
$unescape_fb_link = function($matches){
|
||||||
if(is_array($matches) && count($matches) > 1) {
|
if(is_array($matches) && count($matches) > 1) {
|
||||||
|
|
@ -435,7 +424,7 @@ EOD;
|
||||||
$author = str_replace(' | Facebook', '', $html->find('title#pageTitle', 0)->innertext);
|
$author = str_replace(' | Facebook', '', $html->find('title#pageTitle', 0)->innertext);
|
||||||
$profilePic = 'https://graph.facebook.com/'
|
$profilePic = 'https://graph.facebook.com/'
|
||||||
. $this->getInput('u')
|
. $this->getInput('u')
|
||||||
. '/picture?width=200&height=200';
|
. '/picture?width=200&height=200#.image';
|
||||||
|
|
||||||
$this->authorName = $author;
|
$this->authorName = $author;
|
||||||
|
|
||||||
|
|
@ -491,6 +480,12 @@ EOD;
|
||||||
'',
|
'',
|
||||||
$content);
|
$content);
|
||||||
|
|
||||||
|
//Remove "SpSonsSoriSsés"
|
||||||
|
$content = preg_replace(
|
||||||
|
'/(?iU)<a [^>]+ href="#" role="link" [^>}]+>.+<\/a>/iU',
|
||||||
|
'',
|
||||||
|
$content);
|
||||||
|
|
||||||
//Remove html nodes, keep only img, links, basic formatting
|
//Remove html nodes, keep only img, links, basic formatting
|
||||||
$content = strip_tags($content, '<a><img><i><u><br><p>');
|
$content = strip_tags($content, '<a><img><i><u><br><p>');
|
||||||
|
|
||||||
|
|
@ -550,6 +545,9 @@ EOD;
|
||||||
$item['title'] = $title;
|
$item['title'] = $title;
|
||||||
$item['author'] = $author;
|
$item['author'] = $author;
|
||||||
$item['timestamp'] = $date;
|
$item['timestamp'] = $date;
|
||||||
|
if(strpos($item['content'], '<img') === false)
|
||||||
|
$item['enclosures'] = array($profilePic);
|
||||||
|
|
||||||
$this->items[] = $item;
|
$this->items[] = $item;
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
|
||||||
|
|
@ -3,7 +3,7 @@ class FuturaSciencesBridge extends FeedExpander {
|
||||||
|
|
||||||
const MAINTAINER = 'ORelio';
|
const MAINTAINER = 'ORelio';
|
||||||
const NAME = 'Futura-Sciences Bridge';
|
const NAME = 'Futura-Sciences Bridge';
|
||||||
const URI = 'http://www.futura-sciences.com/';
|
const URI = 'https://www.futura-sciences.com/';
|
||||||
const DESCRIPTION = 'Returns the newest articles.';
|
const DESCRIPTION = 'Returns the newest articles.';
|
||||||
|
|
||||||
const PARAMETERS = array( array(
|
const PARAMETERS = array( array(
|
||||||
|
|
@ -90,42 +90,11 @@ class FuturaSciencesBridge extends FeedExpander {
|
||||||
or returnServerError('Could not request Futura-Sciences: ' . $item['uri']);
|
or returnServerError('Could not request Futura-Sciences: ' . $item['uri']);
|
||||||
$item['content'] = $this->extractArticleContent($article);
|
$item['content'] = $this->extractArticleContent($article);
|
||||||
$author = $this->extractAuthor($article);
|
$author = $this->extractAuthor($article);
|
||||||
$item['author'] = empty($author) ? $item['author'] : $author;
|
if (!empty($author))
|
||||||
|
$item['author'] = $author;
|
||||||
return $item;
|
return $item;
|
||||||
}
|
}
|
||||||
|
|
||||||
private function stripWithDelimiters($string, $start, $end){
|
|
||||||
while(strpos($string, $start) !== false) {
|
|
||||||
$section_to_remove = substr($string, strpos($string, $start));
|
|
||||||
$section_to_remove = substr($section_to_remove, 0, strpos($section_to_remove, $end) + strlen($end));
|
|
||||||
$string = str_replace($section_to_remove, '', $string);
|
|
||||||
} return $string;
|
|
||||||
}
|
|
||||||
|
|
||||||
private function stripRecursiveHTMLSection($string, $tag_name, $tag_start){
|
|
||||||
$open_tag = '<' . $tag_name;
|
|
||||||
$close_tag = '</' . $tag_name . '>';
|
|
||||||
$close_tag_length = strlen($close_tag);
|
|
||||||
if(strpos($tag_start, $open_tag) === 0) {
|
|
||||||
while(strpos($string, $tag_start) !== false) {
|
|
||||||
$max_recursion = 100;
|
|
||||||
$section_to_remove = null;
|
|
||||||
$section_start = strpos($string, $tag_start);
|
|
||||||
$search_offset = $section_start;
|
|
||||||
do {
|
|
||||||
$max_recursion--;
|
|
||||||
$section_end = strpos($string, $close_tag, $search_offset);
|
|
||||||
$search_offset = $section_end + $close_tag_length;
|
|
||||||
$section_to_remove = substr($string, $section_start, $section_end - $section_start + $close_tag_length);
|
|
||||||
$open_tag_count = substr_count($section_to_remove, $open_tag);
|
|
||||||
$close_tag_count = substr_count($section_to_remove, $close_tag);
|
|
||||||
} while ($open_tag_count > $close_tag_count && $max_recursion > 0);
|
|
||||||
$string = str_replace($section_to_remove, '', $string);
|
|
||||||
}
|
|
||||||
}
|
|
||||||
return $string;
|
|
||||||
}
|
|
||||||
|
|
||||||
private function extractArticleContent($article){
|
private function extractArticleContent($article){
|
||||||
$contents = $article->find('section.article-text-classic', 0)->innertext;
|
$contents = $article->find('section.article-text-classic', 0)->innertext;
|
||||||
$headline = trim($article->find('p.description', 0)->plaintext);
|
$headline = trim($article->find('p.description', 0)->plaintext);
|
||||||
|
|
@ -137,6 +106,7 @@ class FuturaSciencesBridge extends FeedExpander {
|
||||||
'<div class="sharebar2',
|
'<div class="sharebar2',
|
||||||
'<div class="diaporamafullscreen"',
|
'<div class="diaporamafullscreen"',
|
||||||
'<div class="module social-button',
|
'<div class="module social-button',
|
||||||
|
'<div class="module social-share',
|
||||||
'<div style="margin-bottom:10px;" class="noprint"',
|
'<div style="margin-bottom:10px;" class="noprint"',
|
||||||
'<div class="ficheprevnext',
|
'<div class="ficheprevnext',
|
||||||
'<div class="bar noprint',
|
'<div class="bar noprint',
|
||||||
|
|
@ -148,16 +118,17 @@ class FuturaSciencesBridge extends FeedExpander {
|
||||||
'<div id="forumcomments',
|
'<div id="forumcomments',
|
||||||
'<div ng-if="active"'
|
'<div ng-if="active"'
|
||||||
) as $div_start) {
|
) as $div_start) {
|
||||||
$contents = $this->stripRecursiveHTMLSection($contents, 'div', $div_start);
|
$contents = stripRecursiveHTMLSection($contents, 'div', $div_start);
|
||||||
}
|
}
|
||||||
|
|
||||||
$contents = $this->stripWithDelimiters($contents, '<hr ', '/>');
|
$contents = stripWithDelimiters($contents, '<hr ', '/>');
|
||||||
$contents = $this->stripWithDelimiters($contents, '<p class="content-date', '</p>');
|
$contents = stripWithDelimiters($contents, '<p class="content-date', '</p>');
|
||||||
$contents = $this->stripWithDelimiters($contents, '<h1 class="content-title', '</h1>');
|
$contents = stripWithDelimiters($contents, '<h1 class="content-title', '</h1>');
|
||||||
$contents = $this->stripWithDelimiters($contents, 'fs:definition="', '"');
|
$contents = stripWithDelimiters($contents, 'fs:definition="', '"');
|
||||||
$contents = $this->stripWithDelimiters($contents, 'fs:xt:clicktype="', '"');
|
$contents = stripWithDelimiters($contents, 'fs:xt:clicktype="', '"');
|
||||||
$contents = $this->stripWithDelimiters($contents, 'fs:xt:clickname="', '"');
|
$contents = stripWithDelimiters($contents, 'fs:xt:clickname="', '"');
|
||||||
$contents = $this->stripWithDelimiters($contents, '<script ', '</script>');
|
$contents = StripWithDelimiters($contents, '<section class="module-toretain module-propal-nl', '</section>');
|
||||||
|
$contents = stripWithDelimiters($contents, '<script ', '</script>');
|
||||||
|
|
||||||
return $headline . trim($contents);
|
return $headline . trim($contents);
|
||||||
}
|
}
|
||||||
|
|
|
||||||
|
|
@ -20,50 +20,58 @@ class GBAtempBridge extends BridgeAbstract {
|
||||||
)
|
)
|
||||||
));
|
));
|
||||||
|
|
||||||
private function extractFromDelimiters($string, $start, $end){
|
private function buildItem($uri, $title, $author, $timestamp, $thumbnail, $content){
|
||||||
if(strpos($string, $start) !== false) {
|
|
||||||
$section_retrieved = substr($string, strpos($string, $start) + strlen($start));
|
|
||||||
$section_retrieved = substr($section_retrieved, 0, strpos($section_retrieved, $end));
|
|
||||||
return $section_retrieved;
|
|
||||||
}
|
|
||||||
|
|
||||||
return false;
|
|
||||||
}
|
|
||||||
|
|
||||||
private function stripWithDelimiters($string, $start, $end){
|
|
||||||
while(strpos($string, $start) !== false) {
|
|
||||||
$section_to_remove = substr($string, strpos($string, $start));
|
|
||||||
$section_to_remove = substr($section_to_remove, 0, strpos($section_to_remove, $end) + strlen($end));
|
|
||||||
$string = str_replace($section_to_remove, '', $string);
|
|
||||||
}
|
|
||||||
|
|
||||||
return $string;
|
|
||||||
}
|
|
||||||
|
|
||||||
private function buildItem($uri, $title, $author, $timestamp, $content){
|
|
||||||
$item = array();
|
$item = array();
|
||||||
$item['uri'] = $uri;
|
$item['uri'] = $uri;
|
||||||
$item['title'] = $title;
|
$item['title'] = $title;
|
||||||
$item['author'] = $author;
|
$item['author'] = $author;
|
||||||
$item['timestamp'] = $timestamp;
|
$item['timestamp'] = $timestamp;
|
||||||
$item['content'] = $content;
|
$item['content'] = $content;
|
||||||
|
if (!empty($thumbnail)) {
|
||||||
|
$item['enclosures'] = array($thumbnail);
|
||||||
|
}
|
||||||
return $item;
|
return $item;
|
||||||
}
|
}
|
||||||
|
|
||||||
private function cleanupPostContent($content, $site_url){
|
private function cleanupPostContent($content, $site_url){
|
||||||
$content = str_replace(':arrow:', '➤', $content);
|
$content = str_replace(':arrow:', '➤', $content);
|
||||||
$content = str_replace('href="attachments/', 'href="'.$site_url.'attachments/', $content);
|
$content = str_replace('href="attachments/', 'href="'.$site_url.'attachments/', $content);
|
||||||
$content = $this->stripWithDelimiters($content, '<script', '</script>');
|
$content = stripWithDelimiters($content, '<script', '</script>');
|
||||||
return $content;
|
return $content;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
private function findItemDate($item){
|
||||||
|
$time = 0;
|
||||||
|
$dateField = $item->find('abbr.DateTime', 0);
|
||||||
|
if (is_object($dateField)) {
|
||||||
|
$time = intval(
|
||||||
|
extractFromDelimiters(
|
||||||
|
$dateField->outertext,
|
||||||
|
'data-time="',
|
||||||
|
'"'
|
||||||
|
)
|
||||||
|
);
|
||||||
|
} else {
|
||||||
|
$dateField = $item->find('span.DateTime', 0);
|
||||||
|
$time = DateTime::createFromFormat(
|
||||||
|
'M j, Y \a\t g:i A',
|
||||||
|
extractFromDelimiters(
|
||||||
|
$dateField->outertext,
|
||||||
|
'title="',
|
||||||
|
'"'
|
||||||
|
)
|
||||||
|
)->getTimestamp();
|
||||||
|
}
|
||||||
|
return $time;
|
||||||
|
}
|
||||||
|
|
||||||
private function fetchPostContent($uri, $site_url){
|
private function fetchPostContent($uri, $site_url){
|
||||||
$html = getSimpleHTMLDOM($uri);
|
$html = getSimpleHTMLDOMCached($uri);
|
||||||
if(!$html) {
|
if(!$html) {
|
||||||
return 'Could not request GBAtemp ' . $uri;
|
return 'Could not request GBAtemp: ' . $uri;
|
||||||
}
|
}
|
||||||
|
|
||||||
$content = $html->find('div.messageContent', 0)->innertext;
|
$content = $html->find('div.messageContent, blockquote.baseHtml', 0)->innertext;
|
||||||
return $this->cleanupPostContent($content, $site_url);
|
return $this->cleanupPostContent($content, $site_url);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
@ -76,70 +84,56 @@ class GBAtempBridge extends BridgeAbstract {
|
||||||
case 'N':
|
case 'N':
|
||||||
foreach($html->find('li[class=news_item full]') as $newsItem) {
|
foreach($html->find('li[class=news_item full]') as $newsItem) {
|
||||||
$url = self::URI . $newsItem->find('a', 0)->href;
|
$url = self::URI . $newsItem->find('a', 0)->href;
|
||||||
$time = intval(
|
$img = $this->getURI() . $newsItem->find('img', 0)->src . '#.image';
|
||||||
$this->extractFromDelimiters(
|
$time = $this->findItemDate($newsItem);
|
||||||
$newsItem->find('abbr.DateTime', 0)->outertext,
|
|
||||||
'data-time="',
|
|
||||||
'"'
|
|
||||||
)
|
|
||||||
);
|
|
||||||
$author = $newsItem->find('a.username', 0)->plaintext;
|
$author = $newsItem->find('a.username', 0)->plaintext;
|
||||||
$title = $newsItem->find('a', 1)->plaintext;
|
$title = $newsItem->find('a', 1)->plaintext;
|
||||||
$content = $this->fetchPostContent($url, self::URI);
|
$content = $this->fetchPostContent($url, self::URI);
|
||||||
$this->items[] = $this->buildItem($url, $title, $author, $time, $content);
|
$this->items[] = $this->buildItem($url, $title, $author, $time, $img, $content);
|
||||||
|
unset($newsItem); // Some items are heavy, freeing the item proactively helps saving memory
|
||||||
}
|
}
|
||||||
|
break;
|
||||||
case 'R':
|
case 'R':
|
||||||
foreach($html->find('li.portal_review') as $reviewItem) {
|
foreach($html->find('li.portal_review') as $reviewItem) {
|
||||||
$url = self::URI . $reviewItem->find('a', 0)->href;
|
$url = self::URI . $reviewItem->find('a', 0)->href;
|
||||||
|
$img = $this->getURI() . extractFromDelimiters($reviewItem->find('a', 0)->style, 'image:url(', ')');
|
||||||
$title = $reviewItem->find('span.review_title', 0)->plaintext;
|
$title = $reviewItem->find('span.review_title', 0)->plaintext;
|
||||||
$content = getSimpleHTMLDOM($url)
|
$content = getSimpleHTMLDOM($url)
|
||||||
or returnServerError('Could not request GBAtemp: ' . $uri);
|
or returnServerError('Could not request GBAtemp: ' . $uri);
|
||||||
$author = $content->find('a.username', 0)->plaintext;
|
$author = $content->find('a.username', 0)->plaintext;
|
||||||
$time = intval(
|
$time = $this->findItemDate($content);
|
||||||
$this->extractFromDelimiters(
|
|
||||||
$content->find('abbr.DateTime', 0)->outertext,
|
|
||||||
'data-time="',
|
|
||||||
'"'
|
|
||||||
)
|
|
||||||
);
|
|
||||||
$intro = '<p><b>' . ($content->find('div#review_intro', 0)->plaintext) . '</b></p>';
|
$intro = '<p><b>' . ($content->find('div#review_intro', 0)->plaintext) . '</b></p>';
|
||||||
$review = $content->find('div#review_main', 0)->innertext;
|
$review = $content->find('div#review_main', 0)->innertext;
|
||||||
$subheader = '<p><b>' . $content->find('div.review_subheader', 0)->plaintext . '</b></p>';
|
$subheader = '<p><b>' . $content->find('div.review_subheader', 0)->plaintext . '</b></p>';
|
||||||
$procons = $content->find('table.review_procons', 0)->outertext;
|
$procons = $content->find('table.review_procons', 0)->outertext;
|
||||||
$scores = $content->find('table.reviewscores', 0)->outertext;
|
$scores = $content->find('table.reviewscores', 0)->outertext;
|
||||||
$content = $this->cleanupPostContent($intro . $review . $subheader . $procons . $scores, self::URI);
|
$content = $this->cleanupPostContent($intro . $review . $subheader . $procons . $scores, self::URI);
|
||||||
$this->items[] = $this->buildItem($url, $title, $author, $time, $content);
|
$this->items[] = $this->buildItem($url, $title, $author, $time, $img, $content);
|
||||||
|
unset($reviewItem); // Free up memory
|
||||||
}
|
}
|
||||||
|
break;
|
||||||
case 'T':
|
case 'T':
|
||||||
foreach($html->find('li.portal-tutorial') as $tutorialItem) {
|
foreach($html->find('li.portal-tutorial') as $tutorialItem) {
|
||||||
$url = self::URI . $tutorialItem->find('a', 0)->href;
|
$url = self::URI . $tutorialItem->find('a', 0)->href;
|
||||||
$title = $tutorialItem->find('a', 0)->plaintext;
|
$title = $tutorialItem->find('a', 0)->plaintext;
|
||||||
$time = intval(
|
$time = $this->findItemDate($tutorialItem);
|
||||||
$this->extractFromDelimiters(
|
|
||||||
$tutorialItem->find('abbr.DateTime', 0)->outertext,
|
|
||||||
'data-time="',
|
|
||||||
'"'
|
|
||||||
)
|
|
||||||
);
|
|
||||||
$author = $tutorialItem->find('a.username', 0)->plaintext;
|
$author = $tutorialItem->find('a.username', 0)->plaintext;
|
||||||
$content = $this->fetchPostContent($url, self::URI);
|
$content = $this->fetchPostContent($url, self::URI);
|
||||||
$this->items[] = $this->buildItem($url, $title, $author, $time, $content);
|
$this->items[] = $this->buildItem($url, $title, $author, $time, null, $content);
|
||||||
|
unset($tutorialItem); // Free up memory
|
||||||
}
|
}
|
||||||
|
break;
|
||||||
case 'F':
|
case 'F':
|
||||||
foreach($html->find('li.rc_item') as $postItem) {
|
foreach($html->find('li.rc_item') as $postItem) {
|
||||||
$url = self::URI . $postItem->find('a', 1)->href;
|
$url = self::URI . $postItem->find('a', 1)->href;
|
||||||
$title = $postItem->find('a', 1)->plaintext;
|
$title = $postItem->find('a', 1)->plaintext;
|
||||||
$time = intval(
|
$time = $this->findItemDate($postItem);
|
||||||
$this->extractFromDelimiters(
|
|
||||||
$postItem->find('abbr.DateTime', 0)->outertext,
|
|
||||||
'data-time="',
|
|
||||||
'"'
|
|
||||||
)
|
|
||||||
);
|
|
||||||
$author = $postItem->find('a.username', 0)->plaintext;
|
$author = $postItem->find('a.username', 0)->plaintext;
|
||||||
$content = $this->fetchPostContent($url, self::URI);
|
$content = $this->fetchPostContent($url, self::URI);
|
||||||
$this->items[] = $this->buildItem($url, $title, $author, $time, $content);
|
$this->items[] = $this->buildItem($url, $title, $author, $time, null, $content);
|
||||||
|
unset($postItem); // Free up memory
|
||||||
}
|
}
|
||||||
|
break;
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -3,7 +3,7 @@ class JapanExpoBridge extends BridgeAbstract {
|
||||||
|
|
||||||
const MAINTAINER = 'Ginko';
|
const MAINTAINER = 'Ginko';
|
||||||
const NAME = 'Japan Expo Actualités';
|
const NAME = 'Japan Expo Actualités';
|
||||||
const URI = 'http://www.japan-expo-paris.com/fr/actualites';
|
const URI = 'https://www.japan-expo-paris.com/fr/actualites';
|
||||||
const CACHE_TIMEOUT = 14400; // 4h
|
const CACHE_TIMEOUT = 14400; // 4h
|
||||||
const DESCRIPTION = 'Returns most recent entries from Japan Expo actualités.';
|
const DESCRIPTION = 'Returns most recent entries from Japan Expo actualités.';
|
||||||
const PARAMETERS = array( array(
|
const PARAMETERS = array( array(
|
||||||
|
|
@ -51,7 +51,7 @@ class JapanExpoBridge extends BridgeAbstract {
|
||||||
foreach($html->find('a._tile2') as $element) {
|
foreach($html->find('a._tile2') as $element) {
|
||||||
|
|
||||||
$url = $element->href;
|
$url = $element->href;
|
||||||
$thumbnail = 'http://s.japan-expo.com/katana/images/JES049/paris.png';
|
$thumbnail = 'https://s.japan-expo.com/katana/images/JES049/paris.png';
|
||||||
preg_match('/url\(([^)]+)\)/', $element->find('img.rspvimgset', 0)->style, $img_search_result);
|
preg_match('/url\(([^)]+)\)/', $element->find('img.rspvimgset', 0)->style, $img_search_result);
|
||||||
|
|
||||||
if(count($img_search_result) >= 2)
|
if(count($img_search_result) >= 2)
|
||||||
|
|
@ -62,7 +62,8 @@ class JapanExpoBridge extends BridgeAbstract {
|
||||||
break;
|
break;
|
||||||
}
|
}
|
||||||
|
|
||||||
$article_html = getSimpleHTMLDOMCached('Could not request JapanExpo: ' . $url);
|
$article_html = getSimpleHTMLDOMCached($url)
|
||||||
|
or returnServerError('Could not request JapanExpo: ' . $url);
|
||||||
$header = $article_html->find('header.pageHeadBox', 0);
|
$header = $article_html->find('header.pageHeadBox', 0);
|
||||||
$timestamp = strtotime($header->find('time', 0)->datetime);
|
$timestamp = strtotime($header->find('time', 0)->datetime);
|
||||||
$title_html = $header->find('div.section', 0)->next_sibling();
|
$title_html = $header->find('div.section', 0)->next_sibling();
|
||||||
|
|
@ -92,6 +93,7 @@ class JapanExpoBridge extends BridgeAbstract {
|
||||||
$item['uri'] = $url;
|
$item['uri'] = $url;
|
||||||
$item['title'] = $title;
|
$item['title'] = $title;
|
||||||
$item['timestamp'] = $timestamp;
|
$item['timestamp'] = $timestamp;
|
||||||
|
$item['enclosures'] = array($thumbnail);
|
||||||
$item['content'] = $content;
|
$item['content'] = $content;
|
||||||
$this->items[] = $item;
|
$this->items[] = $item;
|
||||||
$count++;
|
$count++;
|
||||||
|
|
|
||||||
|
|
@ -3,8 +3,7 @@ class LeMondeInformatiqueBridge extends FeedExpander {
|
||||||
|
|
||||||
const MAINTAINER = 'ORelio';
|
const MAINTAINER = 'ORelio';
|
||||||
const NAME = 'Le Monde Informatique';
|
const NAME = 'Le Monde Informatique';
|
||||||
const URI = 'http://www.lemondeinformatique.fr/';
|
const URI = 'https://www.lemondeinformatique.fr/';
|
||||||
const CACHE_TIMEOUT = 1800; // 30min
|
|
||||||
const DESCRIPTION = 'Returns the newest articles.';
|
const DESCRIPTION = 'Returns the newest articles.';
|
||||||
|
|
||||||
public function collectData(){
|
public function collectData(){
|
||||||
|
|
@ -15,30 +14,26 @@ class LeMondeInformatiqueBridge extends FeedExpander {
|
||||||
$item = parent::parseItem($newsItem);
|
$item = parent::parseItem($newsItem);
|
||||||
$article_html = getSimpleHTMLDOMCached($item['uri'])
|
$article_html = getSimpleHTMLDOMCached($item['uri'])
|
||||||
or returnServerError('Could not request LeMondeInformatique: ' . $item['uri']);
|
or returnServerError('Could not request LeMondeInformatique: ' . $item['uri']);
|
||||||
$item['content'] = $this->cleanArticle($article_html->find('div#article', 0)->innertext);
|
|
||||||
$item['title'] = $article_html->find('h1.cleanprint-title', 0)->plaintext;
|
//Deduce thumbnail URL from article image URL
|
||||||
|
$item['enclosures'] = array(
|
||||||
|
str_replace(
|
||||||
|
'/grande/',
|
||||||
|
'/petite/',
|
||||||
|
$article_html->find('.article-image', 0)->find('img', 0)->src
|
||||||
|
)
|
||||||
|
);
|
||||||
|
|
||||||
|
//No response header sets the encoding, explicit conversion is needed or subsequent xml_encode() will fail
|
||||||
|
$item['content'] = utf8_encode($this->cleanArticle($article_html->find('div.col-primary', 0)->innertext));
|
||||||
|
$item['author'] = utf8_encode($article_html->find('div.author-infos', 0)->find('b', 0)->plaintext);
|
||||||
|
|
||||||
return $item;
|
return $item;
|
||||||
}
|
}
|
||||||
|
|
||||||
private function stripCDATA($string){
|
|
||||||
$string = str_replace('<![CDATA[', '', $string);
|
|
||||||
$string = str_replace(']]>', '', $string);
|
|
||||||
return $string;
|
|
||||||
}
|
|
||||||
|
|
||||||
private function stripWithDelimiters($string, $start, $end){
|
|
||||||
while(strpos($string, $start) !== false) {
|
|
||||||
$section_to_remove = substr($string, strpos($string, $start));
|
|
||||||
$section_to_remove = substr($section_to_remove, 0, strpos($section_to_remove, $end) + strlen($end));
|
|
||||||
$string = str_replace($section_to_remove, '', $string);
|
|
||||||
}
|
|
||||||
|
|
||||||
return $string;
|
|
||||||
}
|
|
||||||
|
|
||||||
private function cleanArticle($article_html){
|
private function cleanArticle($article_html){
|
||||||
$article_html = $this->stripWithDelimiters($article_html, '<script', '</script>');
|
$article_html = stripWithDelimiters($article_html, '<script', '</script>');
|
||||||
$article_html = $this->stripWithDelimiters($article_html, '<h1 class="cleanprint-title"', '</h1>');
|
$article_html = explode('<p class="contact-error', $article_html)[0] . '</div>';
|
||||||
return $article_html;
|
return $article_html;
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
|
||||||
|
|
@ -6,16 +6,6 @@ class NeuviemeArtBridge extends FeedExpander {
|
||||||
const URI = 'http://www.9emeart.fr/';
|
const URI = 'http://www.9emeart.fr/';
|
||||||
const DESCRIPTION = 'Returns the newest articles.';
|
const DESCRIPTION = 'Returns the newest articles.';
|
||||||
|
|
||||||
private function stripWithDelimiters($string, $start, $end){
|
|
||||||
while(strpos($string, $start) !== false) {
|
|
||||||
$section_to_remove = substr($string, strpos($string, $start));
|
|
||||||
$section_to_remove = substr($section_to_remove, 0, strpos($section_to_remove, $end) + strlen($end));
|
|
||||||
$string = str_replace($section_to_remove, '', $string);
|
|
||||||
}
|
|
||||||
|
|
||||||
return $string;
|
|
||||||
}
|
|
||||||
|
|
||||||
protected function parseItem($item){
|
protected function parseItem($item){
|
||||||
$item = parent::parseItem($item);
|
$item = parent::parseItem($item);
|
||||||
|
|
||||||
|
|
@ -34,16 +24,16 @@ class NeuviemeArtBridge extends FeedExpander {
|
||||||
}
|
}
|
||||||
|
|
||||||
$article_content = '';
|
$article_content = '';
|
||||||
if($article_image) {
|
if ($article_image) {
|
||||||
$article_content = '<p><img src="' . $article_image . '" /></p>';
|
$article_content = '<p><img src="' . $article_image . '" /></p>';
|
||||||
}
|
}
|
||||||
$article_content .= str_replace(
|
$article_content .= str_replace(
|
||||||
'src="/', 'src="' . self::URI,
|
'src="/', 'src="' . self::URI,
|
||||||
$article_html->find('div.newsGenerique_con', 0)->innertext
|
$article_html->find('div.newsGenerique_con', 0)->innertext
|
||||||
);
|
);
|
||||||
$article_content = $this->stripWithDelimiters($article_content, '<script', '</script>');
|
$article_content = stripWithDelimiters($article_content, '<script', '</script>');
|
||||||
$article_content = $this->stripWithDelimiters($article_content, '<style', '</style>');
|
$article_content = stripWithDelimiters($article_content, '<style', '</style>');
|
||||||
$article_content = $this->stripWithDelimiters($article_content, '<link', '>');
|
$article_content = stripWithDelimiters($article_content, '<link', '>');
|
||||||
|
|
||||||
$item['content'] = $article_content;
|
$item['content'] = $article_content;
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -6,29 +6,105 @@ class NextInpactBridge extends FeedExpander {
|
||||||
const URI = 'https://www.nextinpact.com/';
|
const URI = 'https://www.nextinpact.com/';
|
||||||
const DESCRIPTION = 'Returns the newest articles.';
|
const DESCRIPTION = 'Returns the newest articles.';
|
||||||
|
|
||||||
|
const PARAMETERS = array( array(
|
||||||
|
'feed' => array(
|
||||||
|
'name' => 'Feed',
|
||||||
|
'type' => 'list',
|
||||||
|
'values' => array(
|
||||||
|
'Tous nos articles' => 'news',
|
||||||
|
'Nos contenus en accès libre' => 'acces-libre',
|
||||||
|
'Blog' => 'blog',
|
||||||
|
'Bons plans' => 'bonsplans'
|
||||||
|
)
|
||||||
|
),
|
||||||
|
'filter_premium' => array(
|
||||||
|
'name' => 'Premium',
|
||||||
|
'type' => 'list',
|
||||||
|
'values' => array(
|
||||||
|
'No filter' => '0',
|
||||||
|
'Hide Premium' => '1',
|
||||||
|
'Only Premium' => '2'
|
||||||
|
)
|
||||||
|
),
|
||||||
|
'filter_brief' => array(
|
||||||
|
'name' => 'Brief',
|
||||||
|
'type' => 'list',
|
||||||
|
'values' => array(
|
||||||
|
'No filter' => '0',
|
||||||
|
'Hide Brief' => '1',
|
||||||
|
'Only Brief' => '2'
|
||||||
|
)
|
||||||
|
)
|
||||||
|
));
|
||||||
|
|
||||||
public function collectData(){
|
public function collectData(){
|
||||||
$this->collectExpandableDatas(self::URI . 'rss/news.xml', 10);
|
$feed = $this->getInput('feed');
|
||||||
|
if (empty($feed))
|
||||||
|
$feed = 'news';
|
||||||
|
$this->collectExpandableDatas(self::URI . 'rss/' . $feed . '.xml');
|
||||||
}
|
}
|
||||||
|
|
||||||
protected function parseItem($newsItem){
|
protected function parseItem($newsItem){
|
||||||
$item = parent::parseItem($newsItem);
|
$item = parent::parseItem($newsItem);
|
||||||
$item['content'] = $this->extractContent($item['uri']);
|
$item['content'] = $this->extractContent($item, $item['uri']);
|
||||||
|
if (is_null($item['content']))
|
||||||
|
return null; //Filtered article
|
||||||
return $item;
|
return $item;
|
||||||
}
|
}
|
||||||
|
|
||||||
private function extractContent($url){
|
private function extractContent($item, $url){
|
||||||
$html2 = getSimpleHTMLDOMCached($url);
|
$html = getSimpleHTMLDOMCached($url);
|
||||||
$text = '<p><em>'
|
if (!is_object($html))
|
||||||
. $html2->find('span.sub_title', 0)->innertext
|
return 'Failed to request NextInpact: ' . $url;
|
||||||
. '</em></p><p><img src="'
|
|
||||||
. $html2->find('div.container_main_image_article', 0)->find('img.dedicated', 0)->src
|
foreach(array(
|
||||||
. '" alt="-" /></p><div>'
|
'filter_premium' => 'h2.title_reserve_article',
|
||||||
. $html2->find('div[itemprop=articleBody]', 0)->innertext
|
'filter_brief' => 'div.brief-inner-content'
|
||||||
. '</div>';
|
) as $param_name => $selector) {
|
||||||
|
$param_val = intval($this->getInput($param_name));
|
||||||
|
if ($param_val != 0) {
|
||||||
|
$element_present = is_object($html->find($selector, 0));
|
||||||
|
$element_wanted = ($param_val == 2);
|
||||||
|
if ($element_present != $element_wanted) {
|
||||||
|
return null; //Filter article
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
if (is_object($html->find('div[itemprop=articleBody], div.brief-inner-content', 0))) {
|
||||||
|
|
||||||
|
$subtitle = trim($html->find('span.sub_title, div.brief-head', 0));
|
||||||
|
if(is_object($subtitle) && $subtitle->plaintext !== $item['title']) {
|
||||||
|
$subtitle = '<p><em>' . $subtitle->plaintext . '</em></p>';
|
||||||
|
} else {
|
||||||
|
$subtitle = '';
|
||||||
|
}
|
||||||
|
|
||||||
|
$postimg = $html->find(
|
||||||
|
'div.container_main_image_article, div.image-brief-container, div.image-brief-side-container', 0
|
||||||
|
);
|
||||||
|
if(is_object($postimg)) {
|
||||||
|
$postimg = '<p><img src="'
|
||||||
|
. $postimg->find('img.dedicated', 0)->src
|
||||||
|
. '" alt="-" /></p>';
|
||||||
|
} else {
|
||||||
|
$postimg = '';
|
||||||
|
}
|
||||||
|
|
||||||
|
$text = $subtitle
|
||||||
|
. $postimg
|
||||||
|
. $html->find('div[itemprop=articleBody], div.brief-inner-content', 0)->outertext;
|
||||||
|
|
||||||
|
} else {
|
||||||
|
$text = $item['content']
|
||||||
|
. '<p><em>Failed retrieve full article content</em></p>';
|
||||||
|
}
|
||||||
|
|
||||||
|
$premium_article = $html->find('h2.title_reserve_article', 0);
|
||||||
|
if (is_object($premium_article)) {
|
||||||
|
$text .= '<p><em>' . $premium_article->innertext . '</em></p>';
|
||||||
|
}
|
||||||
|
|
||||||
$premium_article = $html2->find('h2.title_reserve_article', 0);
|
|
||||||
if (is_object($premium_article))
|
|
||||||
$text = $text . '<p><em>' . $premium_article->innertext . '</em></p>';
|
|
||||||
return $text;
|
return $text;
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
|
||||||
|
|
@ -32,43 +32,39 @@ class NextgovBridge extends FeedExpander {
|
||||||
protected function parseItem($newsItem){
|
protected function parseItem($newsItem){
|
||||||
$item = parent::parseItem($newsItem);
|
$item = parent::parseItem($newsItem);
|
||||||
|
|
||||||
$item['content'] = '';
|
$article_thumbnail = 'https://cdn.nextgov.com/nextgov/images/logo.png';
|
||||||
|
$item['content'] = '<p><b>' . $item['content'] . '</b></p>';
|
||||||
|
|
||||||
$namespaces = $newsItem->getNamespaces(true);
|
$namespaces = $newsItem->getNamespaces(true);
|
||||||
if(isset($namespaces['media'])) {
|
if(isset($namespaces['media'])) {
|
||||||
$media = $newsItem->children($namespaces['media']);
|
$media = $newsItem->children($namespaces['media']);
|
||||||
if(isset($media->content)) {
|
if(isset($media->content)) {
|
||||||
$attributes = $media->content->attributes();
|
$attributes = $media->content->attributes();
|
||||||
$item['content'] = '<img src="' . $attributes['url'] . '">';
|
$item['content'] = '<p><img src="' . $attributes['url'] . '"></p>' . $item['content'];
|
||||||
|
$article_thumbnail = str_replace(
|
||||||
|
'large.jpg',
|
||||||
|
'small.jpg',
|
||||||
|
strval($attributes['url'])
|
||||||
|
);
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
$item['enclosures'] = array($article_thumbnail);
|
||||||
$item['content'] .= $this->extractContent($item['uri']);
|
$item['content'] .= $this->extractContent($item['uri']);
|
||||||
return $item;
|
return $item;
|
||||||
}
|
}
|
||||||
|
|
||||||
private function stripWithDelimiters($string, $start, $end){
|
|
||||||
while (strpos($string, $start) !== false) {
|
|
||||||
$section_to_remove = substr($string, strpos($string, $start));
|
|
||||||
$section_to_remove = substr($section_to_remove, 0, strpos($section_to_remove, $end) + strlen($end));
|
|
||||||
$string = str_replace($section_to_remove, '', $string);
|
|
||||||
}
|
|
||||||
|
|
||||||
return $string;
|
|
||||||
}
|
|
||||||
|
|
||||||
private function extractContent($url){
|
private function extractContent($url){
|
||||||
$article = getSimpleHTMLDOMCached($url)
|
$article = getSimpleHTMLDOMCached($url);
|
||||||
or returnServerError('Could not request Nextgov: ' . $url);
|
|
||||||
|
|
||||||
$contents = $article->find('div.wysiwyg', 0)->innertext;
|
if (!is_object($article))
|
||||||
$contents = $this->stripWithDelimiters($contents, '<div class="ad-container">', '</div>');
|
return 'Could not request Nextgov: ' . $url;
|
||||||
$contents = $this->stripWithDelimiters($contents, '<div', '</div>'); //ad outer div
|
|
||||||
return $this->stripWithDelimiters($contents, '<script', '</script>');
|
$contents = $article->find('div.wysiwyg', 0);
|
||||||
$contents = ($article_thumbnail == '' ? '' : '<p><img src="' . $article_thumbnail . '" /></p>')
|
$contents->find('svg.content-tombstone', 0)->outertext = '';
|
||||||
. '<p><b>'
|
$contents = $contents->innertext;
|
||||||
. $article_subtitle
|
$contents = stripWithDelimiters($contents, '<div class="ad-container">', '</div>');
|
||||||
. '</b></p>'
|
$contents = stripWithDelimiters($contents, '<div', '</div>'); //ad outer div
|
||||||
. trim($contents);
|
return trim(stripWithDelimiters($contents, '<script', '</script>'));
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
|
||||||
127
bridges/NyaaTorrentsBridge.php
Normal file
127
bridges/NyaaTorrentsBridge.php
Normal file
|
|
@ -0,0 +1,127 @@
|
||||||
|
<?php
|
||||||
|
class NyaaTorrentsBridge extends BridgeAbstract {
|
||||||
|
|
||||||
|
const MAINTAINER = 'ORelio';
|
||||||
|
const NAME = 'NyaaTorrents';
|
||||||
|
const URI = 'https://nyaa.si/';
|
||||||
|
const DESCRIPTION = 'Returns the newest torrents, with optional search criteria.';
|
||||||
|
const PARAMETERS = array(
|
||||||
|
array(
|
||||||
|
'f' => array(
|
||||||
|
'name' => 'Filter',
|
||||||
|
'type' => 'list',
|
||||||
|
'values' => array(
|
||||||
|
'No filter' => '0',
|
||||||
|
'No remakes' => '1',
|
||||||
|
'Trusted only' => '2'
|
||||||
|
)
|
||||||
|
),
|
||||||
|
'c' => array(
|
||||||
|
'name' => 'Category',
|
||||||
|
'type' => 'list',
|
||||||
|
'values' => array(
|
||||||
|
'All categories' => '0_0',
|
||||||
|
'Anime' => '1_0',
|
||||||
|
'Anime - AMV' => '1_1',
|
||||||
|
'Anime - English' => '1_2',
|
||||||
|
'Anime - Non-English' => '1_3',
|
||||||
|
'Anime - Raw' => '1_4',
|
||||||
|
'Audio' => '2_0',
|
||||||
|
'Audio - Lossless' => '2_1',
|
||||||
|
'Audio - Lossy' => '2_2',
|
||||||
|
'Literature' => '3_0',
|
||||||
|
'Literature - English' => '3_1',
|
||||||
|
'Literature - Non-English' => '3_2',
|
||||||
|
'Literature - Raw' => '3_3',
|
||||||
|
'Live Action' => '4_0',
|
||||||
|
'Live Action - English' => '4_1',
|
||||||
|
'Live Action - Idol/PV' => '4_2',
|
||||||
|
'Live Action - Non-English' => '4_3',
|
||||||
|
'Live Action - Raw' => '4_4',
|
||||||
|
'Pictures' => '5_0',
|
||||||
|
'Pictures - Graphics' => '5_1',
|
||||||
|
'Pictures - Photos' => '5_2',
|
||||||
|
'Software' => '6_0',
|
||||||
|
'Software - Apps' => '6_1',
|
||||||
|
'Software - Games' => '6_2',
|
||||||
|
)
|
||||||
|
),
|
||||||
|
'q' => array(
|
||||||
|
'name' => 'Keyword',
|
||||||
|
'description' => 'Keyword(s)',
|
||||||
|
'type' => 'text'
|
||||||
|
)
|
||||||
|
)
|
||||||
|
);
|
||||||
|
|
||||||
|
public function collectData() {
|
||||||
|
|
||||||
|
// Build Search URL from user-provided parameters
|
||||||
|
$search_url = self::URI . '?s=id&o=desc&'
|
||||||
|
. http_build_query(array(
|
||||||
|
'f' => $this->getInput('f'),
|
||||||
|
'c' => $this->getInput('c'),
|
||||||
|
'q' => $this->getInput('q')
|
||||||
|
));
|
||||||
|
|
||||||
|
// Retrieve torrent listing from search results, which does not contain torrent description
|
||||||
|
$html = getSimpleHTMLDOM($search_url)
|
||||||
|
or returnServerError('Could not request Nyaa: ' . $search_url);
|
||||||
|
$links = $html->find('a');
|
||||||
|
$results = array();
|
||||||
|
foreach ($links as $link)
|
||||||
|
if (strpos($link->href, '/view/') === 0 && !in_array($link->href, $results))
|
||||||
|
$results[] = $link->href;
|
||||||
|
if (empty($results) && empty($this->getInput('q')))
|
||||||
|
returnServerError('No results from Nyaa: ' . $url, 500);
|
||||||
|
|
||||||
|
//Process each item individually
|
||||||
|
foreach ($results as $element) {
|
||||||
|
|
||||||
|
//Limit total amount of requests
|
||||||
|
if(count($this->items) >= 20) {
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
|
||||||
|
$torrent_id = str_replace('/view/', '', $element);
|
||||||
|
|
||||||
|
//Ignore entries without valid torrent ID
|
||||||
|
if ($torrent_id != 0 && ctype_digit($torrent_id)) {
|
||||||
|
|
||||||
|
//Retrieve data for this torrent ID
|
||||||
|
$item_uri = self::URI . 'view/' . $torrent_id;
|
||||||
|
|
||||||
|
//Retrieve full description from torrent page
|
||||||
|
if ($item_html = getSimpleHTMLDOMCached($item_uri)) {
|
||||||
|
|
||||||
|
//Retrieve data from page contents
|
||||||
|
$item_title = str_replace(' :: Nyaa', '', $item_html->find('title', 0)->plaintext);
|
||||||
|
$item_desc = str_get_html(markdownToHtml($item_html->find('#torrent-description', 0)->innertext));
|
||||||
|
$item_author = extractFromDelimiters($item_html->outertext, 'href="/user/', '"');
|
||||||
|
$item_date = intval(extractFromDelimiters($item_html->outertext, 'data-timestamp="', '"'));
|
||||||
|
|
||||||
|
//Retrieve image for thumbnail or generic logo fallback
|
||||||
|
$item_image = $this->getURI() . 'static/img/avatar/default.png';
|
||||||
|
foreach ($item_desc->find('img') as $img) {
|
||||||
|
if (strpos($img->src, 'prez') === false) {
|
||||||
|
$item_image = $img->src;
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
//Build and add final item
|
||||||
|
$item = array();
|
||||||
|
$item['uri'] = $item_uri;
|
||||||
|
$item['title'] = $item_title;
|
||||||
|
$item['author'] = $item_author;
|
||||||
|
$item['timestamp'] = $item_date;
|
||||||
|
$item['enclosures'] = array($item_image);
|
||||||
|
$item['content'] = $item_desc;
|
||||||
|
$this->items[] = $item;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
$element = null;
|
||||||
|
}
|
||||||
|
$results = null;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
@ -9,16 +9,6 @@ class Releases3DSBridge extends BridgeAbstract {
|
||||||
|
|
||||||
public function collectData(){
|
public function collectData(){
|
||||||
|
|
||||||
function extractFromDelimiters($string, $start, $end){
|
|
||||||
if(strpos($string, $start) !== false) {
|
|
||||||
$section_retrieved = substr($string, strpos($string, $start) + strlen($start));
|
|
||||||
$section_retrieved = substr($section_retrieved, 0, strpos($section_retrieved, $end));
|
|
||||||
return $section_retrieved;
|
|
||||||
}
|
|
||||||
|
|
||||||
return false;
|
|
||||||
}
|
|
||||||
|
|
||||||
function typeToString($type){
|
function typeToString($type){
|
||||||
switch($type) {
|
switch($type) {
|
||||||
case 1: return '3DS Game';
|
case 1: return '3DS Game';
|
||||||
|
|
@ -76,8 +66,8 @@ class Releases3DSBridge extends BridgeAbstract {
|
||||||
$ignDate = time();
|
$ignDate = time();
|
||||||
$ignCoverArt = '';
|
$ignCoverArt = '';
|
||||||
|
|
||||||
$ignSearchUrl = 'http://www.ign.com/search?q=' . urlencode($name);
|
$ignSearchUrl = 'https://www.ign.com/search?q=' . urlencode($name);
|
||||||
if($ignResult = getSimpleHTMLDOM($ignSearchUrl)) {
|
if($ignResult = getSimpleHTMLDOMCached($ignSearchUrl)) {
|
||||||
$ignCoverArt = $ignResult->find('div.search-item-media', 0)->find('img', 0)->src;
|
$ignCoverArt = $ignResult->find('div.search-item-media', 0)->find('img', 0)->src;
|
||||||
$ignDesc = $ignResult->find('div.search-item-description', 0)->plaintext;
|
$ignDesc = $ignResult->find('div.search-item-description', 0)->plaintext;
|
||||||
$ignLink = $ignResult->find('div.search-item-sub-title', 0)->find('a', 1)->href;
|
$ignLink = $ignResult->find('div.search-item-sub-title', 0)->find('a', 1)->href;
|
||||||
|
|
@ -127,6 +117,7 @@ class Releases3DSBridge extends BridgeAbstract {
|
||||||
$item['title'] = $name;
|
$item['title'] = $name;
|
||||||
$item['author'] = $publisher;
|
$item['author'] = $publisher;
|
||||||
$item['timestamp'] = $ignDate;
|
$item['timestamp'] = $ignDate;
|
||||||
|
$item['enclosures'] = array($ignCoverArt);
|
||||||
$item['uri'] = empty($ignLink) ? $searchLinkDuckDuckGo : $ignLink;
|
$item['uri'] = empty($ignLink) ? $searchLinkDuckDuckGo : $ignLink;
|
||||||
$item['content'] = $ignDescription . $releaseDescription . $releaseSearchLinks;
|
$item['content'] = $ignDescription . $releaseDescription . $releaseSearchLinks;
|
||||||
$this->items[] = $item;
|
$this->items[] = $item;
|
||||||
|
|
|
||||||
|
|
@ -8,67 +8,66 @@ class TheHackerNewsBridge extends BridgeAbstract {
|
||||||
|
|
||||||
public function collectData(){
|
public function collectData(){
|
||||||
|
|
||||||
function stripWithDelimiters($string, $start, $end){
|
|
||||||
while(strpos($string, $start) !== false) {
|
|
||||||
$section_to_remove = substr($string, strpos($string, $start));
|
|
||||||
$section_to_remove = substr($section_to_remove, 0, strpos($section_to_remove, $end) + strlen($end));
|
|
||||||
$string = str_replace($section_to_remove, '', $string);
|
|
||||||
}
|
|
||||||
|
|
||||||
return $string;
|
|
||||||
}
|
|
||||||
|
|
||||||
function stripRecursiveHtmlSection($string, $tag_name, $tag_start){
|
|
||||||
$open_tag = '<' . $tag_name;
|
|
||||||
$close_tag = '</' . $tag_name . '>';
|
|
||||||
$close_tag_length = strlen($close_tag);
|
|
||||||
if(strpos($tag_start, $open_tag) === 0) {
|
|
||||||
while(strpos($string, $tag_start) !== false) {
|
|
||||||
$max_recursion = 100;
|
|
||||||
$section_to_remove = null;
|
|
||||||
$section_start = strpos($string, $tag_start);
|
|
||||||
$search_offset = $section_start;
|
|
||||||
do {
|
|
||||||
$max_recursion--;
|
|
||||||
$section_end = strpos($string, $close_tag, $search_offset);
|
|
||||||
$search_offset = $section_end + $close_tag_length;
|
|
||||||
$section_to_remove = substr(
|
|
||||||
$string,
|
|
||||||
$section_start,
|
|
||||||
$section_end - $section_start + $close_tag_length
|
|
||||||
);
|
|
||||||
|
|
||||||
$open_tag_count = substr_count($section_to_remove, $open_tag);
|
|
||||||
$close_tag_count = substr_count($section_to_remove, $close_tag);
|
|
||||||
} while($open_tag_count > $close_tag_count && $max_recursion > 0);
|
|
||||||
$string = str_replace($section_to_remove, '', $string);
|
|
||||||
}
|
|
||||||
}
|
|
||||||
return $string;
|
|
||||||
}
|
|
||||||
|
|
||||||
$html = getSimpleHTMLDOM($this->getURI())
|
$html = getSimpleHTMLDOM($this->getURI())
|
||||||
or returnServerError('Could not request TheHackerNews: ' . $this->getURI());
|
or returnServerError('Could not request TheHackerNews: ' . $this->getURI());
|
||||||
$limit = 0;
|
$limit = 0;
|
||||||
|
|
||||||
foreach($html->find('article') as $element) {
|
foreach($html->find('div.body-post') as $element) {
|
||||||
if($limit < 5) {
|
if($limit < 5) {
|
||||||
|
|
||||||
$article_url = $element->find('a.entry-title', 0)->href;
|
$article_url = $element->find('a.story-link', 0)->href;
|
||||||
$article_author = trim($element->find('span.vcard', 0)->plaintext);
|
$article_author = trim($element->find('i.fa-user', 0)->parent()->plaintext);
|
||||||
$article_title = $element->find('a.entry-title', 0)->plaintext;
|
$article_title = $element->find('h2.home-title', 0)->plaintext;
|
||||||
$article_timestamp = strtotime($element->find('span.updated', 0)->plaintext);
|
|
||||||
$article = getSimpleHTMLDOM($article_url)
|
|
||||||
or returnServerError('Could not request TheHackerNews: ' . $article_url);
|
|
||||||
|
|
||||||
$contents = $article->find('div.articlebodyonly', 0)->innertext;
|
//Date without time
|
||||||
$contents = stripRecursiveHtmlSection($contents, 'div', '<div class=\'clear\'');
|
$article_timestamp = strtotime(
|
||||||
$contents = stripWithDelimiters($contents, '<script', '</script>');
|
extractFromDelimiters(
|
||||||
|
$element->find('i.fa-calendar', 0)->parent()->outertext,
|
||||||
|
'</i>',
|
||||||
|
'<span>'
|
||||||
|
)
|
||||||
|
);
|
||||||
|
|
||||||
|
//Article thumbnail in lazy-loading image
|
||||||
|
if (is_object($element->find('img[data-echo]', 0))) {
|
||||||
|
$article_thumbnail = array(
|
||||||
|
extractFromDelimiters(
|
||||||
|
$element->find('img[data-echo]', 0)->outertext,
|
||||||
|
"data-echo='",
|
||||||
|
"'"
|
||||||
|
)
|
||||||
|
);
|
||||||
|
} else {
|
||||||
|
$article_thumbnail = array();
|
||||||
|
}
|
||||||
|
|
||||||
|
if ($article = getSimpleHTMLDOMCached($article_url)) {
|
||||||
|
|
||||||
|
//Article body
|
||||||
|
$contents = $article->find('div.articlebody', 0)->innertext;
|
||||||
|
$contents = stripRecursiveHtmlSection($contents, 'div', '<div class="ad_');
|
||||||
|
$contents = stripWithDelimiters($contents, 'id="google_ads', '</iframe>');
|
||||||
|
$contents = stripWithDelimiters($contents, '<script', '</script>');
|
||||||
|
|
||||||
|
//Date with time
|
||||||
|
if (is_object($article->find('meta[itemprop=dateModified]', 0))) {
|
||||||
|
$article_timestamp = strtotime(
|
||||||
|
extractFromDelimiters(
|
||||||
|
$article->find('meta[itemprop=dateModified]', 0)->outertext,
|
||||||
|
"content='",
|
||||||
|
"'"
|
||||||
|
)
|
||||||
|
);
|
||||||
|
}
|
||||||
|
} else {
|
||||||
|
$contents = 'Could not request TheHackerNews: ' . $article_url;
|
||||||
|
}
|
||||||
|
|
||||||
$item = array();
|
$item = array();
|
||||||
$item['uri'] = $article_url;
|
$item['uri'] = $article_url;
|
||||||
$item['title'] = $article_title;
|
$item['title'] = $article_title;
|
||||||
$item['author'] = $article_author;
|
$item['author'] = $article_author;
|
||||||
|
$item['enclosures'] = $article_thumbnail;
|
||||||
$item['timestamp'] = $article_timestamp;
|
$item['timestamp'] = $article_timestamp;
|
||||||
$item['content'] = trim($contents);
|
$item['content'] = trim($contents);
|
||||||
$this->items[] = $item;
|
$this->items[] = $item;
|
||||||
|
|
|
||||||
|
|
@ -3,37 +3,24 @@ class WeLiveSecurityBridge extends FeedExpander {
|
||||||
|
|
||||||
const MAINTAINER = 'ORelio';
|
const MAINTAINER = 'ORelio';
|
||||||
const NAME = 'We Live Security';
|
const NAME = 'We Live Security';
|
||||||
const URI = 'http://www.welivesecurity.com/';
|
const URI = 'https://www.welivesecurity.com/';
|
||||||
const DESCRIPTION = 'Returns the newest articles.';
|
const DESCRIPTION = 'Returns the newest articles.';
|
||||||
|
|
||||||
private function stripWithDelimiters($string, $start, $end){
|
|
||||||
while(strpos($string, $start) !== false) {
|
|
||||||
$section_to_remove = substr($string, strpos($string, $start));
|
|
||||||
$section_to_remove = substr($section_to_remove, 0, strpos($section_to_remove, $end) + strlen($end));
|
|
||||||
$string = str_replace($section_to_remove, '', $string);
|
|
||||||
}
|
|
||||||
|
|
||||||
return $string;
|
|
||||||
}
|
|
||||||
|
|
||||||
|
|
||||||
protected function parseItem($item){
|
protected function parseItem($item){
|
||||||
$item = parent::parseItem($item);
|
$item = parent::parseItem($item);
|
||||||
|
|
||||||
$article_html = getSimpleHTMLDOMCached($item['uri']);
|
$article_html = getSimpleHTMLDOMCached($item['uri']);
|
||||||
if(!$article_html) {
|
if(!$article_html) {
|
||||||
$item['content'] .= '<p>Could not request ' . $this->getName() . ': ' . $item['uri'] . '</p>';
|
$item['content'] .= '<p><em>Could not request ' . $this->getName() . ': ' . $item['uri'] . '</em></p>';
|
||||||
return $item;
|
return $item;
|
||||||
}
|
}
|
||||||
|
|
||||||
$article_content = $article_html->find('div.wlistingsingletext', 0)->innertext;
|
$article_content = $article_html->find('div.formatted', 0)->innertext;
|
||||||
$article_content = $this->stripWithDelimiters($article_content, '<script', '</script>');
|
$article_content = stripWithDelimiters($article_content, '<script', '</script>');
|
||||||
$article_content = '<p><b>'
|
$article_content = stripRecursiveHTMLSection($article_content, 'div', '<div class="comments');
|
||||||
. $item['content']
|
$article_content = stripRecursiveHTMLSection($article_content, 'div', '<div class="similar-articles');
|
||||||
. '</b></p>'
|
$article_content = stripRecursiveHTMLSection($article_content, 'span', '<span class="meta');
|
||||||
. trim($article_content);
|
$item['content'] = trim($article_content);
|
||||||
|
|
||||||
$item['content'] = $article_content;
|
|
||||||
|
|
||||||
return $item;
|
return $item;
|
||||||
}
|
}
|
||||||
|
|
|
||||||
|
|
@ -3,8 +3,7 @@ class WordPressBridge extends FeedExpander {
|
||||||
const MAINTAINER = 'aledeg';
|
const MAINTAINER = 'aledeg';
|
||||||
const NAME = 'Wordpress Bridge';
|
const NAME = 'Wordpress Bridge';
|
||||||
const URI = 'https://wordpress.org/';
|
const URI = 'https://wordpress.org/';
|
||||||
const CACHE_TIMEOUT = 10800; // 3h
|
const DESCRIPTION = 'Returns the newest full posts of a WordPress powered website';
|
||||||
const DESCRIPTION = 'Returns the newest full posts of a Wordpress powered website';
|
|
||||||
|
|
||||||
const PARAMETERS = array( array(
|
const PARAMETERS = array( array(
|
||||||
'url' => array(
|
'url' => array(
|
||||||
|
|
@ -13,8 +12,8 @@ class WordPressBridge extends FeedExpander {
|
||||||
)
|
)
|
||||||
));
|
));
|
||||||
|
|
||||||
private function clearContent($content){
|
private function cleanContent($content){
|
||||||
$content = preg_replace('/<script[^>]*>[^<]*<\/script>/', '', $content);
|
$content = stripWithDelimiters($content, '<script', '</script>');
|
||||||
$content = preg_replace('/<div class="wpa".*/', '', $content);
|
$content = preg_replace('/<div class="wpa".*/', '', $content);
|
||||||
$content = preg_replace('/<form.*\/form>/', '', $content);
|
$content = preg_replace('/<form.*\/form>/', '', $content);
|
||||||
return $content;
|
return $content;
|
||||||
|
|
@ -27,6 +26,10 @@ class WordPressBridge extends FeedExpander {
|
||||||
|
|
||||||
$article = null;
|
$article = null;
|
||||||
switch(true) {
|
switch(true) {
|
||||||
|
case !is_null($article_html->find('[itemprop=articleBody]', 0)):
|
||||||
|
// highest priority content div
|
||||||
|
$article = $article_html->find('[itemprop=articleBody]', 0);
|
||||||
|
break;
|
||||||
case !is_null($article_html->find('article', 0)):
|
case !is_null($article_html->find('article', 0)):
|
||||||
// most common content div
|
// most common content div
|
||||||
$article = $article_html->find('article', 0);
|
$article = $article_html->find('article', 0);
|
||||||
|
|
@ -39,15 +42,37 @@ class WordPressBridge extends FeedExpander {
|
||||||
// another common content div
|
// another common content div
|
||||||
$article = $article_html->find('.post-content', 0);
|
$article = $article_html->find('.post-content', 0);
|
||||||
break;
|
break;
|
||||||
|
|
||||||
case !is_null($article_html->find('.post', 0)):
|
case !is_null($article_html->find('.post', 0)):
|
||||||
// for old WordPress themes without HTML5
|
// for old WordPress themes without HTML5
|
||||||
$article = $article_html->find('.post', 0);
|
$article = $article_html->find('.post', 0);
|
||||||
break;
|
break;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
foreach ($article->find('h1.entry-title') as $title)
|
||||||
|
if ($title->plaintext == $item['title'])
|
||||||
|
$title->outertext = '';
|
||||||
|
|
||||||
|
$article_image = $article_html->find('img.wp-post-image', 0);
|
||||||
|
if(!empty($item['content']) && (!is_object($article_image) || empty($article_image->src))) {
|
||||||
|
$article_image = str_get_html($item['content'])->find('img.wp-post-image', 0);
|
||||||
|
}
|
||||||
|
if(is_object($article_image) && !empty($article_image->src)) {
|
||||||
|
if(empty($article_image->getAttribute('data-lazy-src'))) {
|
||||||
|
$article_image = $article_image->src;
|
||||||
|
} else {
|
||||||
|
$article_image = $article_image->getAttribute('data-lazy-src');
|
||||||
|
}
|
||||||
|
$mime_type = getMimeType($article_image);
|
||||||
|
if (strpos($mime_type, 'image') === false)
|
||||||
|
$article_image .= '#.image'; // force image
|
||||||
|
if (empty($item['enclosures']))
|
||||||
|
$item['enclosures'] = array($article_image);
|
||||||
|
else
|
||||||
|
$item['enclosures'] = array_merge($item['enclosures'], $article_image);
|
||||||
|
}
|
||||||
|
|
||||||
if(!is_null($article)) {
|
if(!is_null($article)) {
|
||||||
$item['content'] = $this->clearContent($article->innertext);
|
$item['content'] = $this->cleanContent($article->innertext);
|
||||||
}
|
}
|
||||||
|
|
||||||
return $item;
|
return $item;
|
||||||
|
|
|
||||||
|
|
@ -110,8 +110,8 @@ class YGGTorrentBridge extends BridgeAbstract {
|
||||||
|
|
||||||
foreach($results->find('tr') as $row) {
|
foreach($results->find('tr') as $row) {
|
||||||
$count++;
|
$count++;
|
||||||
if($count == 1) continue;
|
if($count == 1) continue; // Skip table header
|
||||||
if($count == 12) break;
|
if($count == 22) break; // Stop processing after 21 items (20 + 1 table header)
|
||||||
$item = array();
|
$item = array();
|
||||||
$item['timestamp'] = $row->find('.hidden', 1)->plaintext;
|
$item['timestamp'] = $row->find('.hidden', 1)->plaintext;
|
||||||
$item['title'] = $row->find('a', 1)->plaintext;
|
$item['title'] = $row->find('a', 1)->plaintext;
|
||||||
|
|
@ -135,7 +135,7 @@ class YGGTorrentBridge extends BridgeAbstract {
|
||||||
$url_full[5] = urlencode($url_full[5]);
|
$url_full[5] = urlencode($url_full[5]);
|
||||||
$url_full[6] = urlencode($url_full[6]);
|
$url_full[6] = urlencode($url_full[6]);
|
||||||
$url = implode('/', $url_full);
|
$url = implode('/', $url_full);
|
||||||
$page = getSimpleHTMLDOM($url) or returnServerError('Unable to query Yggtorrent page !');
|
$page = getSimpleHTMLDOMCached($url) or returnServerError('Unable to query Yggtorrent page !');
|
||||||
$author = $page->find('.informations', 0)->find('a', 4)->plaintext;
|
$author = $page->find('.informations', 0)->find('a', 4)->plaintext;
|
||||||
$content = $page->find('.default', 1);
|
$content = $page->find('.default', 1);
|
||||||
return array('author' => $author, 'content' => $content);
|
return array('author' => $author, 'content' => $content);
|
||||||
|
|
|
||||||
|
|
@ -1,9 +1,9 @@
|
||||||
<?php
|
<?php
|
||||||
class ZDNetBridge extends BridgeAbstract {
|
class ZDNetBridge extends FeedExpander {
|
||||||
|
|
||||||
const MAINTAINER = 'ORelio';
|
const MAINTAINER = 'ORelio';
|
||||||
const NAME = 'ZDNet Bridge';
|
const NAME = 'ZDNet Bridge';
|
||||||
const URI = 'http://www.zdnet.com/';
|
const URI = 'https://www.zdnet.com/';
|
||||||
const DESCRIPTION = 'Technology News, Analysis, Comments and Product Reviews for IT Professionals.';
|
const DESCRIPTION = 'Technology News, Analysis, Comments and Product Reviews for IT Professionals.';
|
||||||
|
|
||||||
//http://www.zdnet.com/zdnet.opml
|
//http://www.zdnet.com/zdnet.opml
|
||||||
|
|
@ -160,143 +160,42 @@ class ZDNetBridge extends BridgeAbstract {
|
||||||
));
|
));
|
||||||
|
|
||||||
public function collectData(){
|
public function collectData(){
|
||||||
|
$baseUri = static::URI;
|
||||||
function stripCdata($string){
|
|
||||||
$string = str_replace('<![CDATA[', '', $string);
|
|
||||||
$string = str_replace(']]>', '', $string);
|
|
||||||
return trim($string);
|
|
||||||
}
|
|
||||||
|
|
||||||
function extractFromDelimiters($string, $start, $end){
|
|
||||||
if(strpos($string, $start) !== false) {
|
|
||||||
$section_retrieved = substr($string, strpos($string, $start) + strlen($start));
|
|
||||||
$section_retrieved = substr($section_retrieved, 0, strpos($section_retrieved, $end));
|
|
||||||
return $section_retrieved;
|
|
||||||
}
|
|
||||||
|
|
||||||
return false;
|
|
||||||
}
|
|
||||||
|
|
||||||
function stripWithDelimiters($string, $start, $end){
|
|
||||||
while(strpos($string, $start) !== false) {
|
|
||||||
$section_to_remove = substr($string, strpos($string, $start));
|
|
||||||
$section_to_remove = substr($section_to_remove, 0, strpos($section_to_remove, $end) + strlen($end));
|
|
||||||
$string = str_replace($section_to_remove, '', $string);
|
|
||||||
}
|
|
||||||
|
|
||||||
return $string;
|
|
||||||
}
|
|
||||||
|
|
||||||
function stripRecursiveHtmlSection($string, $tag_name, $tag_start){
|
|
||||||
$open_tag = '<' . $tag_name;
|
|
||||||
$close_tag = '</' . $tag_name . '>';
|
|
||||||
$close_tag_length = strlen($close_tag);
|
|
||||||
if(strpos($tag_start, $open_tag) === 0) {
|
|
||||||
while(strpos($string, $tag_start) !== false) {
|
|
||||||
$max_recursion = 100;
|
|
||||||
$section_to_remove = null;
|
|
||||||
$section_start = strpos($string, $tag_start);
|
|
||||||
$search_offset = $section_start;
|
|
||||||
do {
|
|
||||||
$max_recursion--;
|
|
||||||
$section_end = strpos($string, $close_tag, $search_offset);
|
|
||||||
$search_offset = $section_end + $close_tag_length;
|
|
||||||
$section_to_remove = substr(
|
|
||||||
$string,
|
|
||||||
$section_start,
|
|
||||||
$section_end - $section_start + $close_tag_length
|
|
||||||
);
|
|
||||||
|
|
||||||
$open_tag_count = substr_count($section_to_remove, $open_tag);
|
|
||||||
$close_tag_count = substr_count($section_to_remove, $close_tag);
|
|
||||||
} while ($open_tag_count > $close_tag_count && $max_recursion > 0);
|
|
||||||
$string = str_replace($section_to_remove, '', $string);
|
|
||||||
}
|
|
||||||
}
|
|
||||||
return $string;
|
|
||||||
}
|
|
||||||
|
|
||||||
$baseUri = self::URI;
|
|
||||||
$feed = $this->getInput('feed');
|
$feed = $this->getInput('feed');
|
||||||
if(strpos($feed, 'downloads!') !== false) {
|
if(strpos($feed, 'downloads!') !== false) {
|
||||||
$feed = str_replace('downloads!', '', $feed);
|
$feed = str_replace('downloads!', '', $feed);
|
||||||
$baseUri = str_replace('www.', 'downloads.', $baseUri);
|
$baseUri = str_replace('www.', 'downloads.', $baseUri);
|
||||||
}
|
}
|
||||||
$url = $baseUri . trim($feed, '/') . '/rss.xml';
|
$url = $baseUri . trim($feed, '/') . '/rss.xml';
|
||||||
$html = getSimpleHTMLDOM($url)
|
$this->collectExpandableDatas($url);
|
||||||
or returnServerError('Could not request ZDNet: ' . $url);
|
}
|
||||||
$limit = 0;
|
|
||||||
|
|
||||||
foreach($html->find('item') as $element) {
|
protected function parseItem($item){
|
||||||
if($limit < 10) {
|
$item = parent::parseItem($item);
|
||||||
$article_url = preg_replace(
|
|
||||||
'/([^#]+)#ftag=.*/',
|
|
||||||
'$1',
|
|
||||||
stripCdata(extractFromDelimiters($element->innertext, '<link>', '</link>'))
|
|
||||||
);
|
|
||||||
|
|
||||||
$article_author = stripCdata(extractFromDelimiters($element->innertext, 'role="author">', '<'));
|
$article = getSimpleHTMLDOMCached($item['uri']);
|
||||||
$article_title = stripCdata($element->find('title', 0)->plaintext);
|
if(!$article)
|
||||||
$article_subtitle = stripCdata($element->find('description', 0)->plaintext);
|
returnServerError('Could not request ZDNet: ' . $url);
|
||||||
$article_timestamp = strtotime(stripCdata($element->find('pubDate', 0)->plaintext));
|
|
||||||
$article = getSimpleHTMLDOM($article_url)
|
|
||||||
or returnServerError('Could not request ZDNet: ' . $article_url);
|
|
||||||
|
|
||||||
if(!empty($article_author)) {
|
$contents = $article->find('article', 0)->innertext;
|
||||||
$author = $article_author;
|
foreach(array(
|
||||||
} else {
|
'<div class="shareBar"',
|
||||||
$author = $article->find('meta[name=author]', 0);
|
'<div class="shortcodeGalleryWrapper"',
|
||||||
if(is_object($author)) {
|
'<div class="relatedContent',
|
||||||
$author = $author->content;
|
'<div class="downloadNow',
|
||||||
} else {
|
'<div data-shortcode',
|
||||||
$author = 'ZDNet';
|
'<div id="sharethrough',
|
||||||
}
|
'<div id="inpage-video'
|
||||||
}
|
) as $div_start) {
|
||||||
|
$contents = stripRecursiveHtmlSection($contents, 'div', $div_start);
|
||||||
$thumbnail = $article->find('meta[itemprop=image]', 0);
|
|
||||||
if(is_object($thumbnail)) {
|
|
||||||
$thumbnail = $thumbnail->content;
|
|
||||||
} else {
|
|
||||||
$thumbnail = '';
|
|
||||||
}
|
|
||||||
|
|
||||||
$contents = $article->find('article', 0)->innertext;
|
|
||||||
foreach(array(
|
|
||||||
'<div class="shareBar"',
|
|
||||||
'<div class="shortcodeGalleryWrapper"',
|
|
||||||
'<div class="relatedContent',
|
|
||||||
'<div class="downloadNow',
|
|
||||||
'<div data-shortcode',
|
|
||||||
'<div id="sharethrough',
|
|
||||||
'<div id="inpage-video'
|
|
||||||
) as $div_start) {
|
|
||||||
$contents = stripRecursiveHtmlSection($contents, 'div', $div_start);
|
|
||||||
}
|
|
||||||
$contents = stripWithDelimiters($contents, '<script', '</script>');
|
|
||||||
$contents = stripWithDelimiters($contents, '<meta itemprop="image"', '>');
|
|
||||||
$contents = trim(stripWithDelimiters($contents, '<section class="sharethrough-top', '</section>'));
|
|
||||||
$content_img = strpos($contents, '<img'); //Look for first image
|
|
||||||
if (($content_img !== false && $content_img < 512) || $thumbnail == '') {
|
|
||||||
$content_img = ''; //Image already present on article beginning or no thumbnail
|
|
||||||
} else {
|
|
||||||
$content_img = '<p><img src="'.$thumbnail.'" /></p>'; //Include thumbnail
|
|
||||||
}
|
|
||||||
$contents = $content_img
|
|
||||||
. '<p><b>'
|
|
||||||
. $article_subtitle
|
|
||||||
. '</b></p>'
|
|
||||||
. $contents;
|
|
||||||
|
|
||||||
$item = array();
|
|
||||||
$item['author'] = $author;
|
|
||||||
$item['uri'] = $article_url;
|
|
||||||
$item['title'] = $article_title;
|
|
||||||
$item['timestamp'] = $article_timestamp;
|
|
||||||
$item['content'] = $contents;
|
|
||||||
$this->items[] = $item;
|
|
||||||
$limit++;
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
|
$contents = stripWithDelimiters($contents, '<script', '</script>');
|
||||||
|
$contents = stripWithDelimiters($contents, '<meta itemprop="image"', '>');
|
||||||
|
$contents = stripWithDelimiters($contents, '<svg class="svg-symbol', '</svg>');
|
||||||
|
$contents = trim(stripWithDelimiters($contents, '<section class="sharethrough-top', '</section>'));
|
||||||
|
$item['content'] = $contents;
|
||||||
|
|
||||||
|
return $item;
|
||||||
|
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
|
||||||
|
|
@ -39,7 +39,7 @@ class AtomFormat extends FormatAbstract{
|
||||||
foreach($item['enclosures'] as $enclosure) {
|
foreach($item['enclosures'] as $enclosure) {
|
||||||
$entryEnclosures .= '<link rel="enclosure" href="'
|
$entryEnclosures .= '<link rel="enclosure" href="'
|
||||||
. $this->xml_encode($enclosure)
|
. $this->xml_encode($enclosure)
|
||||||
. '"/>'
|
. '" type="' . getMimeType($enclosure) . '" />'
|
||||||
. PHP_EOL;
|
. PHP_EOL;
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
|
||||||
|
|
@ -37,7 +37,7 @@ class MrssFormat extends FormatAbstract {
|
||||||
if(isset($item['enclosures'])) {
|
if(isset($item['enclosures'])) {
|
||||||
$entryEnclosures .= '<enclosure url="'
|
$entryEnclosures .= '<enclosure url="'
|
||||||
. $this->xml_encode($item['enclosures'][0])
|
. $this->xml_encode($item['enclosures'][0])
|
||||||
. '"/>';
|
. '" type="' . getMimeType($item['enclosures'][0]) . '" />';
|
||||||
|
|
||||||
if(count($item['enclosures']) > 1) {
|
if(count($item['enclosures']) > 1) {
|
||||||
$entryEnclosures .= PHP_EOL;
|
$entryEnclosures .= PHP_EOL;
|
||||||
|
|
@ -45,7 +45,7 @@ class MrssFormat extends FormatAbstract {
|
||||||
Some media files might not be shown to you. Consider using the ATOM format instead!';
|
Some media files might not be shown to you. Consider using the ATOM format instead!';
|
||||||
foreach($item['enclosures'] as $enclosure) {
|
foreach($item['enclosures'] as $enclosure) {
|
||||||
$entryEnclosures .= '<atom:link rel="enclosure" href="'
|
$entryEnclosures .= '<atom:link rel="enclosure" href="'
|
||||||
. $enclosure . '" />'
|
. $enclosure . '" type="' . getMimeType($enclosure) . '" />'
|
||||||
. PHP_EOL;
|
. PHP_EOL;
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
|
||||||
17
index.php
17
index.php
|
|
@ -11,12 +11,18 @@ if(file_exists('DEBUG')) {
|
||||||
$debug_whitelist = trim(file_get_contents('DEBUG'));
|
$debug_whitelist = trim(file_get_contents('DEBUG'));
|
||||||
|
|
||||||
$debug_enabled = empty($debug_whitelist)
|
$debug_enabled = empty($debug_whitelist)
|
||||||
|| in_array($_SERVER['REMOTE_ADDR'], explode("\n", $debug_whitelist));
|
|| in_array($_SERVER['REMOTE_ADDR'],
|
||||||
|
explode("\n", str_replace("\r", '', $debug_whitelist)
|
||||||
|
)
|
||||||
|
);
|
||||||
|
|
||||||
if($debug_enabled) {
|
if($debug_enabled) {
|
||||||
ini_set('display_errors', '1');
|
ini_set('display_errors', '1');
|
||||||
error_reporting(E_ALL);
|
error_reporting(E_ALL);
|
||||||
define('DEBUG', true);
|
define('DEBUG', true);
|
||||||
|
if (empty($debug_whitelist)) {
|
||||||
|
define('DEBUG_INSECURE', true);
|
||||||
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
@ -36,14 +42,17 @@ Configuration::loadConfiguration();
|
||||||
Authentication::showPromptIfNeeded();
|
Authentication::showPromptIfNeeded();
|
||||||
|
|
||||||
date_default_timezone_set('UTC');
|
date_default_timezone_set('UTC');
|
||||||
error_reporting(0);
|
|
||||||
|
|
||||||
/*
|
/*
|
||||||
Move the CLI arguments to the $_GET array, in order to be able to use
|
Move the CLI arguments to the $_GET array, in order to be able to use
|
||||||
rss-bridge from the command line
|
rss-bridge from the command line
|
||||||
*/
|
*/
|
||||||
parse_str(implode('&', array_slice($argv, 1)), $cliArgs);
|
if (isset($argv)) {
|
||||||
$params = array_merge($_GET, $cliArgs);
|
parse_str(implode('&', array_slice($argv, 1)), $cliArgs);
|
||||||
|
$params = array_merge($_GET, $cliArgs);
|
||||||
|
} else {
|
||||||
|
$params = $_GET;
|
||||||
|
}
|
||||||
|
|
||||||
// FIXME : beta test UA spoofing, please report any blacklisting by PHP-fopen-unfriendly websites
|
// FIXME : beta test UA spoofing, please report any blacklisting by PHP-fopen-unfriendly websites
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -55,17 +55,27 @@ EOD;
|
||||||
}
|
}
|
||||||
|
|
||||||
private static function getHeader() {
|
private static function getHeader() {
|
||||||
$status = '';
|
$warning = '';
|
||||||
|
|
||||||
if(defined('DEBUG') && DEBUG === true) {
|
if(defined('DEBUG') && DEBUG === true) {
|
||||||
$status .= 'debug mode active';
|
if(defined('DEBUG_INSECURE') && DEBUG_INSECURE === true) {
|
||||||
|
$warning .= <<<EOD
|
||||||
|
<section class="critical-warning">Warning : Debug mode is active from any location,
|
||||||
|
make sure only you can access RSS-Bridge.</section>
|
||||||
|
EOD;
|
||||||
|
} else {
|
||||||
|
$warning .= <<<EOD
|
||||||
|
<section class="warning">Warning : Debug mode is active from your IP address,
|
||||||
|
your requests will bypass the cache.</section>
|
||||||
|
EOD;
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
return <<<EOD
|
return <<<EOD
|
||||||
<header>
|
<header>
|
||||||
<h1>RSS-Bridge</h1>
|
<h1>RSS-Bridge</h1>
|
||||||
<h2>·Reconnecting the Web·</h2>
|
<h2>Reconnecting the Web</h2>
|
||||||
<p class="status">{$status}</p>
|
{$warning}
|
||||||
</header>
|
</header>
|
||||||
EOD;
|
EOD;
|
||||||
}
|
}
|
||||||
|
|
|
||||||
|
|
@ -115,12 +115,27 @@ abstract class FeedExpander extends BridgeAbstract {
|
||||||
}
|
}
|
||||||
|
|
||||||
protected function parseATOMItem($feedItem){
|
protected function parseATOMItem($feedItem){
|
||||||
$item = array();
|
// Some ATOM entries also contain RSS 2.0 fields
|
||||||
|
$item = $this->parseRSS_2_0_Item($feedItem);
|
||||||
|
|
||||||
if(isset($feedItem->id)) $item['uri'] = (string)$feedItem->id;
|
if(isset($feedItem->id)) $item['uri'] = (string)$feedItem->id;
|
||||||
if(isset($feedItem->title)) $item['title'] = (string)$feedItem->title;
|
if(isset($feedItem->title)) $item['title'] = (string)$feedItem->title;
|
||||||
if(isset($feedItem->updated)) $item['timestamp'] = strtotime((string)$feedItem->updated);
|
if(isset($feedItem->updated)) $item['timestamp'] = strtotime((string)$feedItem->updated);
|
||||||
if(isset($feedItem->author)) $item['author'] = (string)$feedItem->author->name;
|
if(isset($feedItem->author)) $item['author'] = (string)$feedItem->author->name;
|
||||||
if(isset($feedItem->content)) $item['content'] = (string)$feedItem->content;
|
if(isset($feedItem->content)) $item['content'] = (string)$feedItem->content;
|
||||||
|
|
||||||
|
//When "link" field is present, URL is more reliable than "id" field
|
||||||
|
if (count($feedItem->link) === 1) {
|
||||||
|
$this->uri = (string)$feedItem->link[0]['href'];
|
||||||
|
} else {
|
||||||
|
foreach($feedItem->link as $link) {
|
||||||
|
if(strtolower($link['rel']) === 'alternate') {
|
||||||
|
$item['uri'] = (string)$link['href'];
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
return $item;
|
return $item;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
@ -130,6 +145,7 @@ abstract class FeedExpander extends BridgeAbstract {
|
||||||
if(isset($feedItem->title)) $item['title'] = (string)$feedItem->title;
|
if(isset($feedItem->title)) $item['title'] = (string)$feedItem->title;
|
||||||
// rss 0.91 doesn't support timestamps
|
// rss 0.91 doesn't support timestamps
|
||||||
// rss 0.91 doesn't support authors
|
// rss 0.91 doesn't support authors
|
||||||
|
// rss 0.91 doesn't support enclosures
|
||||||
if(isset($feedItem->description)) $item['content'] = (string)$feedItem->description;
|
if(isset($feedItem->description)) $item['content'] = (string)$feedItem->description;
|
||||||
return $item;
|
return $item;
|
||||||
}
|
}
|
||||||
|
|
@ -154,11 +170,17 @@ abstract class FeedExpander extends BridgeAbstract {
|
||||||
|
|
||||||
$namespaces = $feedItem->getNamespaces(true);
|
$namespaces = $feedItem->getNamespaces(true);
|
||||||
if(isset($namespaces['dc'])) $dc = $feedItem->children($namespaces['dc']);
|
if(isset($namespaces['dc'])) $dc = $feedItem->children($namespaces['dc']);
|
||||||
|
if(isset($namespaces['media'])) $media = $feedItem->children($namespaces['media']);
|
||||||
|
|
||||||
if(isset($feedItem->guid)) {
|
if(isset($feedItem->guid)) {
|
||||||
foreach($feedItem->guid->attributes() as $attribute => $value) {
|
foreach($feedItem->guid->attributes() as $attribute => $value) {
|
||||||
if($attribute === 'isPermaLink'
|
if($attribute === 'isPermaLink'
|
||||||
&& ($value === 'true' || filter_var($feedItem->guid, FILTER_VALIDATE_URL))) {
|
&& ($value === 'true' || (
|
||||||
|
filter_var($feedItem->guid, FILTER_VALIDATE_URL)
|
||||||
|
&& !filter_var($item['uri'], FILTER_VALIDATE_URL)
|
||||||
|
)
|
||||||
|
)
|
||||||
|
) {
|
||||||
$item['uri'] = (string)$feedItem->guid;
|
$item['uri'] = (string)$feedItem->guid;
|
||||||
break;
|
break;
|
||||||
}
|
}
|
||||||
|
|
@ -170,11 +192,21 @@ abstract class FeedExpander extends BridgeAbstract {
|
||||||
} elseif(isset($dc->date)) {
|
} elseif(isset($dc->date)) {
|
||||||
$item['timestamp'] = strtotime((string)$dc->date);
|
$item['timestamp'] = strtotime((string)$dc->date);
|
||||||
}
|
}
|
||||||
|
|
||||||
if(isset($feedItem->author)) {
|
if(isset($feedItem->author)) {
|
||||||
$item['author'] = (string)$feedItem->author;
|
$item['author'] = (string)$feedItem->author;
|
||||||
|
} elseif (isset($feedItem->creator)) {
|
||||||
|
$item['author'] = (string)$feedItem->creator;
|
||||||
} elseif(isset($dc->creator)) {
|
} elseif(isset($dc->creator)) {
|
||||||
$item['author'] = (string)$dc->creator;
|
$item['author'] = (string)$dc->creator;
|
||||||
|
} elseif(isset($media->credit)) {
|
||||||
|
$item['author'] = (string)$media->credit;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
if(isset($feedItem->enclosure) && !empty($feedItem->enclosure['url'])) {
|
||||||
|
$item['enclosures'] = array((string)$feedItem->enclosure['url']);
|
||||||
|
}
|
||||||
|
|
||||||
return $item;
|
return $item;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
@ -199,14 +231,14 @@ abstract class FeedExpander extends BridgeAbstract {
|
||||||
}
|
}
|
||||||
|
|
||||||
public function getURI(){
|
public function getURI(){
|
||||||
return $this->uri ?: parent::getURI();
|
return !empty($this->uri) ? $this->uri : parent::getURI();
|
||||||
}
|
}
|
||||||
|
|
||||||
public function getName(){
|
public function getName(){
|
||||||
return $this->name ?: parent::getName();
|
return !empty($this->name) ? $this->name : parent::getName();
|
||||||
}
|
}
|
||||||
|
|
||||||
public function getIcon(){
|
public function getIcon(){
|
||||||
return $this->icon ?: parent::getIcon();
|
return !empty($this->icon) ? $this->icon : parent::getIcon();
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
|
||||||
|
|
@ -21,7 +21,7 @@ function getContents($url, $header = array(), $opts = array()){
|
||||||
curl_setopt($ch, CURLOPT_PROXY, PROXY_URL);
|
curl_setopt($ch, CURLOPT_PROXY, PROXY_URL);
|
||||||
}
|
}
|
||||||
|
|
||||||
// We always want the resonse header as part of the data!
|
// We always want the response header as part of the data!
|
||||||
curl_setopt($ch, CURLOPT_HEADER, true);
|
curl_setopt($ch, CURLOPT_HEADER, true);
|
||||||
|
|
||||||
$data = curl_exec($ch);
|
$data = curl_exec($ch);
|
||||||
|
|
@ -152,3 +152,53 @@ function parseResponseHeader($header) {
|
||||||
return $headers;
|
return $headers;
|
||||||
|
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Determine MIME type from URL/Path file extension
|
||||||
|
* Remark: Built-in functions mime_content_type or fileinfo requires fetching remote content
|
||||||
|
* Remark: A bridge can hint for a MIME type by appending #.ext to a URL, e.g. #.image
|
||||||
|
* Based on https://stackoverflow.com/a/1147952
|
||||||
|
*/
|
||||||
|
function getMimeType($url) {
|
||||||
|
static $mime = null;
|
||||||
|
|
||||||
|
if (is_null($mime)) {
|
||||||
|
$mime = array(
|
||||||
|
'jpg' => 'image/jpeg',
|
||||||
|
'gif' => 'image/gif',
|
||||||
|
'png' => 'image/png',
|
||||||
|
'image' => 'image/*'
|
||||||
|
);
|
||||||
|
if (is_file('/etc/mime.types')) {
|
||||||
|
$file = fopen('/etc/mime.types', 'r');
|
||||||
|
while(($line = fgets($file)) !== false) {
|
||||||
|
$line = trim(preg_replace('/#.*/', '', $line));
|
||||||
|
if(!$line)
|
||||||
|
continue;
|
||||||
|
$parts = preg_split('/\s+/', $line);
|
||||||
|
if(count($parts) == 1)
|
||||||
|
continue;
|
||||||
|
$type = array_shift($parts);
|
||||||
|
foreach($parts as $part)
|
||||||
|
$mime[$part] = $type;
|
||||||
|
}
|
||||||
|
fclose($file);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
if (strpos($url, '?') !== false) {
|
||||||
|
$url_temp = substr($url, 0, strpos($url, '?'));
|
||||||
|
if (strpos($url, '#') !== false) {
|
||||||
|
$anchor = substr($url, strpos($url, '#'));
|
||||||
|
$url_temp .= $anchor;
|
||||||
|
}
|
||||||
|
$url = $url_temp;
|
||||||
|
}
|
||||||
|
|
||||||
|
$ext = strtolower(pathinfo($url, PATHINFO_EXTENSION));
|
||||||
|
if (!empty($mime[$ext])) {
|
||||||
|
return $mime[$ext];
|
||||||
|
}
|
||||||
|
|
||||||
|
return 'application/octet-stream';
|
||||||
|
}
|
||||||
|
|
|
||||||
|
|
@ -20,7 +20,7 @@ function debugMessage($text){
|
||||||
$calling = $backtrace[2];
|
$calling = $backtrace[2];
|
||||||
$message = $calling['file'] . ':'
|
$message = $calling['file'] . ':'
|
||||||
. $calling['line'] . ' class '
|
. $calling['line'] . ' class '
|
||||||
. $calling['class'] . '->'
|
. (isset($calling['class']) ? $calling['class'] : '<no-class>') . '->'
|
||||||
. $calling['function'] . ' - '
|
. $calling['function'] . ' - '
|
||||||
. $text;
|
. $text;
|
||||||
|
|
||||||
|
|
|
||||||
123
lib/html.php
123
lib/html.php
|
|
@ -40,7 +40,19 @@ function backgroundToImg($htmlContent) {
|
||||||
|
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Convert relative links in HTML into absolute links
|
||||||
|
* @param $content HTML content to fix. Supports HTML objects or string objects
|
||||||
|
* @param $server full URL to the page containing relative links
|
||||||
|
* @return content with fixed URLs, as HTML object or string depending on input type
|
||||||
|
*/
|
||||||
function defaultLinkTo($content, $server){
|
function defaultLinkTo($content, $server){
|
||||||
|
$string_convert = false;
|
||||||
|
if (is_string($content)) {
|
||||||
|
$string_convert = true;
|
||||||
|
$content = str_get_html($content);
|
||||||
|
}
|
||||||
|
|
||||||
foreach($content->find('img') as $image) {
|
foreach($content->find('img') as $image) {
|
||||||
$image->src = urljoin($server, $image->src);
|
$image->src = urljoin($server, $image->src);
|
||||||
}
|
}
|
||||||
|
|
@ -49,5 +61,116 @@ function defaultLinkTo($content, $server){
|
||||||
$anchor->href = urljoin($server, $anchor->href);
|
$anchor->href = urljoin($server, $anchor->href);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
if ($string_convert) {
|
||||||
|
$content = $content->outertext;
|
||||||
|
}
|
||||||
|
|
||||||
return $content;
|
return $content;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Extract the first part of a string matching the specified start and end delimiters
|
||||||
|
* @param $string input string, e.g. '<div>Post author: John Doe</div>'
|
||||||
|
* @param $start start delimiter, e.g. 'author: '
|
||||||
|
* @param $end end delimiter, e.g. '<'
|
||||||
|
* @return extracted string, e.g. 'John Doe', or false if the delimiters were not found.
|
||||||
|
*/
|
||||||
|
function extractFromDelimiters($string, $start, $end) {
|
||||||
|
if (strpos($string, $start) !== false) {
|
||||||
|
$section_retrieved = substr($string, strpos($string, $start) + strlen($start));
|
||||||
|
$section_retrieved = substr($section_retrieved, 0, strpos($section_retrieved, $end));
|
||||||
|
return $section_retrieved;
|
||||||
|
} return false;
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Remove one or more part(s) of a string using a start and end delmiters
|
||||||
|
* @param $string input string, e.g. 'foo<script>superscript()</script>bar'
|
||||||
|
* @param $start start delimiter, e.g. '<script'
|
||||||
|
* @param $end end delimiter, e.g. '</script>'
|
||||||
|
* @return cleaned string, e.g. 'foobar'
|
||||||
|
*/
|
||||||
|
function stripWithDelimiters($string, $start, $end) {
|
||||||
|
while(strpos($string, $start) !== false) {
|
||||||
|
$section_to_remove = substr($string, strpos($string, $start));
|
||||||
|
$section_to_remove = substr($section_to_remove, 0, strpos($section_to_remove, $end) + strlen($end));
|
||||||
|
$string = str_replace($section_to_remove, '', $string);
|
||||||
|
}
|
||||||
|
return $string;
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Remove HTML sections containing one or more sections using the same HTML tag
|
||||||
|
* @param $string input string, e.g. 'foo<div class="ads"><div>ads</div>ads</div>bar'
|
||||||
|
* @param $tag_name name of the HTML tag, e.g. 'div'
|
||||||
|
* @param $tag_start start of the HTML tag to remove, e.g. '<div class="ads">'
|
||||||
|
* @return cleaned string, e.g. 'foobar'
|
||||||
|
*/
|
||||||
|
function stripRecursiveHTMLSection($string, $tag_name, $tag_start){
|
||||||
|
$open_tag = '<' . $tag_name;
|
||||||
|
$close_tag = '</' . $tag_name . '>';
|
||||||
|
$close_tag_length = strlen($close_tag);
|
||||||
|
if(strpos($tag_start, $open_tag) === 0) {
|
||||||
|
while(strpos($string, $tag_start) !== false) {
|
||||||
|
$max_recursion = 100;
|
||||||
|
$section_to_remove = null;
|
||||||
|
$section_start = strpos($string, $tag_start);
|
||||||
|
$search_offset = $section_start;
|
||||||
|
do {
|
||||||
|
$max_recursion--;
|
||||||
|
$section_end = strpos($string, $close_tag, $search_offset);
|
||||||
|
$search_offset = $section_end + $close_tag_length;
|
||||||
|
$section_to_remove = substr($string, $section_start, $section_end - $section_start + $close_tag_length);
|
||||||
|
$open_tag_count = substr_count($section_to_remove, $open_tag);
|
||||||
|
$close_tag_count = substr_count($section_to_remove, $close_tag);
|
||||||
|
} while ($open_tag_count > $close_tag_count && $max_recursion > 0);
|
||||||
|
$string = str_replace($section_to_remove, '', $string);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return $string;
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Convert Markdown tags into HTML tags. Only a subset of the Markdown syntax is implemented.
|
||||||
|
* @param $string input string in Markdown format
|
||||||
|
* @return output string in HTML format
|
||||||
|
*/
|
||||||
|
function markdownToHtml($string) {
|
||||||
|
|
||||||
|
//For more details about how these regex work:
|
||||||
|
// https://github.com/RSS-Bridge/rss-bridge/pull/802#discussion_r216138702
|
||||||
|
// Images: https://regex101.com/r/JW9Evr/1
|
||||||
|
// Links: https://regex101.com/r/eRGVe7/1
|
||||||
|
// Bold: https://regex101.com/r/2p40Y0/1
|
||||||
|
// Italic: https://regex101.com/r/xJkET9/1
|
||||||
|
// Separator: https://regex101.com/r/ZBEqFP/1
|
||||||
|
// Plain URL: https://regex101.com/r/2JHYwb/1
|
||||||
|
// Site name: https://regex101.com/r/qIuKYE/1
|
||||||
|
|
||||||
|
$string = preg_replace('/\!\[([^\]]+)\]\(([^\) ]+)(?: [^\)]+)?\)/', '<img src="$2" alt="$1" />', $string);
|
||||||
|
$string = preg_replace('/\[([^\]]+)\]\(([^\)]+)\)/', '<a href="$2">$1</a>', $string);
|
||||||
|
$string = preg_replace('/\*\*(.*)\*\*/U', '<b>$1</b>', $string);
|
||||||
|
$string = preg_replace('/\*(.*)\*/U', '<i>$1</i>', $string);
|
||||||
|
$string = preg_replace('/__(.*)__/U', '<b>$1</b>', $string);
|
||||||
|
$string = preg_replace('/_(.*)_/U', '<i>$1</i>', $string);
|
||||||
|
$string = preg_replace('/[-]{6,99}/', '<hr />', $string);
|
||||||
|
$string = str_replace(' ', '<br />', $string);
|
||||||
|
$string = preg_replace('/([^"])(https?:\/\/[^ "<]+)([^"])/', '$1<a href="$2">$2</a>$3', $string.' ');
|
||||||
|
$string = preg_replace('/([^"\/])(www\.[^ "<]+)([^"])/', '$1<a href="http://$2">$2</a>$3', $string.' ');
|
||||||
|
|
||||||
|
//As the regex are not perfect, we need to fix <i> and </i> that are introduced in URLs
|
||||||
|
// Fixup regex <i>: https://regex101.com/r/NTRPf6/1
|
||||||
|
// Fixup regex </i>: https://regex101.com/r/aNklRp/1
|
||||||
|
|
||||||
|
$count = 1;
|
||||||
|
while($count > 0) {
|
||||||
|
$string = preg_replace('/ (src|href)="([^"]+)<i>([^"]+)"/U', ' $1="$2_$3"', $string, -1, $count);
|
||||||
|
}
|
||||||
|
|
||||||
|
$count = 1;
|
||||||
|
while($count > 0) {
|
||||||
|
$string = preg_replace('/ (src|href)="([^"]+)<\/i>([^"]+)"/U', ' $1="$2_$3"', $string, -1, $count);
|
||||||
|
}
|
||||||
|
|
||||||
|
return '<div>' . trim($string) . '</div>';
|
||||||
|
}
|
||||||
|
|
|
||||||
|
|
@ -40,11 +40,17 @@ a:hover {
|
||||||
margin-left: 1em;
|
margin-left: 1em;
|
||||||
font-size: 200%;
|
font-size: 200%;
|
||||||
}
|
}
|
||||||
header > p.status {
|
header > section.warning {
|
||||||
font-weight: bold;
|
width: 40%;
|
||||||
margin: 1em;
|
background-color: #ffc600;
|
||||||
color: red;
|
color: #5f5f5f;
|
||||||
}
|
}
|
||||||
|
header > section.critical-warning {
|
||||||
|
width: 40%;
|
||||||
|
background-color: #cf3e3e;
|
||||||
|
font-weight: bold;
|
||||||
|
color: white;
|
||||||
|
}
|
||||||
|
|
||||||
/* Input boxes */
|
/* Input boxes */
|
||||||
input[type="text"] {
|
input[type="text"] {
|
||||||
|
|
|
||||||
Loading…
Reference in a new issue