Retrieve Data from a Remote Webpage - PHP

Testy · 发表于 2013-2-21 14:00:00

PHP’s file() functions are great for opening, reading, writing to, and doing other dirty tricks with files. Any html or php page is of course a text file so we can open them and extract data in many different ways. Here are just a couple.

Retrieving meta tags from a remote webpage
PHP has a useful function called get_meta_tags that allows you to read meta data as an array and then extract certain elements.
As an example, the following code will snatch the meta data from www.waypoints.ws.

$tags = get_meta_tags('http://www.waypoints.ws/');
$keywords = $tags['keywords'];
$description = $tags['description'];
$author = $tags['author'];
echo "<b>Description:</b> $description
";
echo "<b>Keywords:</b> $keywords
";
echo "<b>Author:</b> $author
";

复制代码

Will return this:

I use a variant of the get_meta_tags function to display page information as part of a preview function for fat.ly short URL service.

Retrieving title tags from a remote webpage
The get_meta_tags function will only parse data above the closing head tag in your page so it’s relatively quick. The function won’t read the title tag of your page – so to do that we open up the entire page and extract the desired text. This process is far more time consuming.
First, we open the page for reading. I’m using fopen() but you could just as easily use file_get_contents().

<?php
$url = "http://www.google.com.au/";
$page = fopen($url, 'r');
$content = "";
while( !feof( $page ) ) {
$buffer = trim( fgets( $page, 4096 ) );
$content .= $buffer;
}
?>

复制代码

Second, we extract the text between the title tags.

eregi('',$content,$tmp);
$result = ereg_replace('[[:blank:]]',' ',$tmp[1]);
echo "$result";

复制代码

Retrieving any text data from a remote webpage
Using the same code above, we could extract any text element of a page between two defined and unique strings of text. This means you that you can effectively snatch portions of remote web pages for inclusion into your own site. Before you did anything remotely resembling this you would should ensure you have permission to do so. Not doing so is theft.

<?php
$url = "http://www.your-URL-in-here.com/";
$page = fopen($url, 'r');
$content = "";
while( !feof( $page ) ) {
$buffer = trim( fgets( $page, 4096 ) );
$content .= $buffer;
}
$start = "<text1>";
$end = "<\/text2>";
preg_match( "/$start(.*)$end/s", $content, $match);
$mytext = $match[1];
echo "$mytext";
?>

复制代码

Retrieving header information from a remote webpage
A developer can extract header information from a remote page using the following code:

<?php
$fp = fopen('http://www.google.com', 'r');
// Creates variable $http_response_header
print_r($http_response_header);
// or
$meta_data = stream_get_meta_data($fp);
print_r($meta_data);
?>

复制代码

internoetics
January 8, 2010 By Marty

demo · 发表于 2013-2-22 11:18:16

本帖最后由 demo 于 2013-2-23 03:21 编辑

Some codes you might need while get data from webpage:

<?php
$url = "http://www.google.com.au/";
$page = fopen($url, 'r');
$content = "";
while( !feof( $page ) ) {
$buffer = trim( fgets( $page, 4096 ) );
$content .= $buffer;
}
echo "$content";
echo "\r\n==========Remove script============\r\n";
$tmp = preg_replace('%<script.*?>.*?</script>%is','|',$content);
$tmp1 = preg_replace('%<style>.*?</style>%is',':',$tmp);
//eregi('',$content,$tmp);
$result = ereg_replace('[[:blank:]]',' ',$tmp);
echo "$tmp1";
echo "\r\n==========Use strip_tags to remove html tags with some exceptions============\r\n";
$tmp = strip_tags($content,'<a><head>');
echo "$tmp";
echo "\r\n==========Text only============\r\n";
// This echoes correctly all the text that is not inside HTML tags
$html_reg = '/<+\s*\/*\s*([A-Z][A-Z0-9]*)\b[^>]*\/*\s*>+/i';
$tmp = preg_replace( $html_reg, '', $content );
//echo htmlentities( preg_replace( $html_reg, '', $html ) );
echo "$tmp";
echo "\r\n==========Get text============\r\n";
/* $tmp = preg_replace('/<script[^>]*?>[^<]*?<\/script>/i','***',$content);
eregi('',$content,$tmp);
* used for bb replace .....
$patterns = array(
"/\[link\](.*?)\[\/link\]/",
"/\[url\](.*?)\[\/url\]/",
"/\[img\](.*?)\[\/img\]/",
"/\[b\](.*?)\[\/b\]/",
"/\[u\](.*?)\[\/u\]/",
"/\[i\](.*?)\[\/i\]/"
);
$replacements = array(
"<a href="\\1">\\1</a>",
"<a href="\\1">\\1</a>",
"<img src="\\1">",
"<b>\\1</b>",
"<u>\\1</u>",
"<i>\\1</i>"
);
*/
$patterns = array(
"/<a href="(.*?)"(.*?)>(.*?)<\/a>/",
"/<img src="(.*?)"(.*?)\/>/"
);
$replacements = array(
"\r\n\\3 -->http://www.demo.com\\1\r\n",
"\r\n Img --> http://www.demo.com\\1\r\n"
);
$result = preg_replace($patterns,$replacements, $tmp1);
//$result = ereg_replace('[[:blank:]]',' ',$tmp);
echo "$result";
?>

复制代码

More flexible code to get link or picture as:

<?php
$url = "http://news.msn.com/";
$page = fopen($url, 'r');
$content = "";
while( !feof( $page ) ) {
$buffer = trim( fgets( $page, 4096 ) );
$content .= $buffer;
}
$start = "<section id="featured_classic fill > cluster" class="featured " data-aop="featured_classic fill > cluster">";
$end = "<section id="clusters">";
preg_match( "/$start(.*)$end/s", $content, $match);
$mytext = $match[1];
echo "$mytext\r\nData List \r\n";
$patterns = array(
"/<a href="(.*?)"(.*?)>(.*?)<\/a>/",
"/<img(.*?)src="(.*?)"(.*?)\/>/"
);
$replacements = array(
"\r\n...\\3 -->http://www.demo.com\\1\r\n",
"\r\n...Img --> http://www.demo.com\\2\r\n"
);
$result = preg_replace($patterns,$replacements, $mytext);
//$result = ereg_replace('[[:blank:]]',' ',$tmp);
echo "$result";
?>

复制代码

And this will give you information of the header:

<?php
$fp = fopen('http://www.google.com', 'r');
// Creates variable $http_response_header
print_r($http_response_header);
// or
$meta_data = stream_get_meta_data($fp);
print_r($meta_data);
?>

复制代码

Pretty cool.

账号		自动登录	找回密码
密码			注册