Using External Input Data in Off-the-shelf Web Scrapers

There is a question I’ve wanted to shed some light upon for a long time already: “What if I need to scrape several URL’s based on data in some external database?“.

For example, recently one of our visitors asked a very good question (thanks, Ed):

    “I have a large list of amazon.com asin. I would like to scrape 10 or so fields for each asin. Is there any web scraping software available that can read each asin from a database and form the destination url to be scraped like http://www.amazon.com/gp/product/{asin} and scrape the data?”

This question impelled me to investigate this matter. I contacted several web scraper developers, and they kindly provided me with detailed answers that allowed me to bring the following summary to your attention:
Visual Web Ripper

An input data source can be used to provide a list of input values to a data extraction project. A data extraction project will be run once for each row of input values. You can find the additional information here.
Web Content Extractor

You can use the -at”filename” command line option to add new URLs from TXT or CSV file:

    WCExtractor.exe projectfile -at”filename” -s

projectfile: the file name of the project (*.wcepr) to open.
filename – the file name of the CSV or TXT file that contains URLs separated by newlines.
-s – starts the extraction process

You can find some options and examples here.
Mozenda

Since Mozenda is cloud-based, the external data needs to be loaded up into the user’s Mozenda account. That data can then be easily used as part of the data extracting process. You can construct URLs, search for strings that match your inputs, or carry through several data fields from an input collection and add data to it as part of your output. The easiest way to get input data from an external source is to use the API to populate data into a Mozenda collection (in the user’s account). You can also input data in the Mozenda web console by importing a .csv file or importing one through our agent building tool.

Once the data is loaded into the cloud, you simply initiate building a Mozenda web agent and refer to that Data list. By using the Load page action and the variable from the inputs, you can construct a URL like http://www.amazon.com/gp/product/%asin%.
Helium Scraper

Here is a video showing how to do this with Helium Scraper:


The video shows how to use the input data as URLs and as search terms. There are many other ways you could use this data, way too many to fit in a video. Also, if you know SQL, you could run a query to get the data directly from an external MS Access database like
SELECT * FROM [MyTable] IN "C:\MyDatabase.mdb"

Note that the database needs to be a “.mdb” file.
WebSundew Data Extractor
Basically this allows using input data from external data sources. This may be CSV, Excel file or a Database (MySQL, MSSQL, etc). Here you can see how to do this in the case of an external file, but you can do it with a database in a similar way (you just need to write an SQL script that returns the necessary data).
In addition to passing URLs from the external sources you can pass other input parameters as well (input fields, for example).
Screen Scraper

Screen Scraper is really designed to be interoperable with all sorts of databases. We have composed a separate article where you can find a tutorial and a sample project about scraping Amazon products based on a list of their ASINs.


Source: http://extract-web-data.com/using-external-input-data-in-off-the-shelf-web-scrapers/

Selenium IDE and Web Scraping

Selenium is a browser automation framework that includes IDE, Remote Control server and bindings of various flavors including Java, .Net, Ruby, Python and other. In this post we touch on the basic structure of the framework and its application to  Web Scraping.
What is Selenium IDE


Selenium IDE is an integrated development environment for Selenium scripts. It is implemented as a Firefox plugin, and it allows recording browsers’ interactions in order to edit them. This works well for software tests, composing and debugging. The Selenium Remote Control is a server specific for a particular environment; it causes custom scripts to be implemented for controlled browsers. Selenium deploys on Windows, Linux, and iOS. How various Selenium components are supported with major browsers read here.
What does Selenium do and Web Scraping

Basically Selenium automates browsers. This ability is no doubt to be applied to web scraping. Since browsers (and Selenium) support JavaScript, jQuery and other methods working with dynamic content why not use this mix for benefit in web scraping, rather than to try to catch Ajax events with plain code? The second reason for this kind of scrape automation is browser-fasion data access (though today this is emulated with most libraries).

Yes, Selenium works to automate browsers, but how to control Selenium from a custom script to automate a browser for web scraping? There are Selenium PHP and other language libraries (bindings) providing for scripts to call and use Selenium. It is possible to write Selenium clients (using the libraries) in almost any language we prefer, for example Perl, Python, Java, PHP etc. Those libraries (API), along with a server, the Java written server that invokes browsers for actions, constitute the Selenum RC (Remote Control). Remote Control automatically loads the Selenium Core into the browser to control it. For more details in Selenium components refer to here.


A tough scrape task for programmer

“…cURL is good, but it is very basic.  I need to handle everything manually; I am creating HTTP requests by hand.
This gets difficult – I need to do a lot of work to make sure that the requests that I send are exactly the same as the requests that a browser would
send, both for my sake and for the website’s sake. (For my sake
because I want to get the right data, and for the website’s sake
because I don’t want to cause error messages or other problems on their site because I sent a bad request that messed with their web application).  And if there is any important javascript, I need to imitate it with PHP.
It would be a great benefit to me to be able to control a browser like Firefox with my code. It would solve all my problems regarding the emulation of a real browser…
it seems that Selenium will allow me to do this…” -Ryan S

Yes, that’s what we will consider below.
Scrape with Selenium

In order to create scripts that interact with the Selenium Server (Selenium RC, Selenium Remote Webdriver) or create local Selenium WebDriver script, there is the need to make use of language-specific client drivers (also called Formatters, they are included in the selenium-ide-1.10.0.xpi package). The Selenium servers, drivers and bindings are available at Selenium download page.
The basic recipe for scrape with Selenium:

    Use Chrome or Firefox browsers
    Get Firebug or Chrome Dev Tools (Cntl+Shift+I) in action.
    Install requirements (Remote control or WebDriver, libraries and other)
    Selenium IDE : Record a ‘test’ run thru a site, adding some assertions.
    Export as a Python (other language) script.
    Edit it (loops, data extraction, db input/output)
    Run script for the Remote Control

The short intro Slides for the scraping of tough websites with Python & Selenium are here (as Google Docs slides) and here (Slide Share).
Selenium components for Firefox installation guide

For how to install the Selenium IDE to Firefox see  here starting at slide 21. The Selenium Core and Remote Control installation instructions are there too.
Extracting for dynamic content using jQuery/JavaScript with Selenium

One programmer is doing a similar thing …

1. launch a selenium RC (remote control) server
2. load a page
3. inject the jQuery script
4. select the interested contents using jQuery/JavaScript
5. send back to the PHP client using JSON.

He particularly finds it quite easy and convenient to use jQuery for
screen scraping, rather than using PHP/XPath.
Conclusion

The Selenium IDE is the popular tool for browser automation, mostly for its software testing application, yet also in that Web Scraping techniques for tough dynamic websites may be implemented with IDE along with the Selenium Remote Control server. These are the basic steps for it:

    Record the ‘test‘ browser behavior in IDE and export it as the custom programming language script
    Formatted language script runs on the Remote Control server that forces browser to send HTTP requests and then script catches the Ajax powered responses to extract content.

Selenium based Web Scraping is an easy task for small scale projects, but it consumes a lot of memory resources, since for each request it will launch a new browser instance.



Source: http://extract-web-data.com/selenium-ide-and-web-scraping/

Beneficial Data Collection Services

Internet is becoming the biggest source for information gathering. Varieties of search engines are available over the World Wide Web which helps in searching any kind of information easily and quickly. Every business needs relevant data for their decision making for which market research plays a crucial role. One of the services booming very fast is the data collection services. This data mining service helps in gathering relevant data which is hugely needed for your business or personal use.

Traditionally, data collection has been done manually which is not very feasible in case of bulk data requirement. Although people still use manual copying and pasting of data from Web pages or download a complete Web site which is shear wastage of time and effort. Instead, a more reliable and convenient method is automated data collection technique. There is a web scraping techniques that crawls through thousands of web pages for the specified topic and simultaneously incorporates this information into a database, XML file, CSV file, or other custom format for future reference. Few of the most commonly used web data extraction processes are websites which provide you information about the competitor's pricing and featured data; spider is a government portal that helps in extracting the names of citizens for an investigation; websites which have variety of downloadable images.

Aside, there is a more sophisticated method of automated data collection service. Here, you can easily scrape the web site information on daily basis automatically. This method greatly helps you in discovering the latest market trends, customer behavior and the future trends. Few of the major examples of automated data collection solutions are price monitoring information; collection of data of various financial institutions on a daily basis; verification of different reports on a constant basis and use them for taking better and progressive business decisions.

While using these service make sure you use the right procedure. Like when you are retrieving data download it in a spreadsheet so that the analysts can do the comparison and analysis properly. This will also help in getting accurate results in a faster and more refined manner.




Source: http://ezinearticles.com/?Beneficial-Data-Collection-Services&id=5879822

Text Data Mining Can Be Profitable

There are billions of search terms performed on the internet every year,and the companies which make use of this vast amount of information are the ones who will be able to market effectively in the future. It is here that text data mining comes into its own, a technique which enables researchers to find patterns within groups of text which will enable them to make predictions as to how customers or other groups of people will act in the future. This article will take a look at text data mining and how we can help various groups of people to find the best things in the data analysis.

It is always a good idea to do some study of the text mining techniques before going on to text mining implementation, and this can be said to be especially true of the insurance industry where not only text mining but also generic data mining using in statistics can be a great help in determining profitability and also showing actuaries how to make future calculations.

Consultancy is an important part of text data mining, and the text mining consultant can bring a huge amount of knowledge to a company whatever the service or services that are providing, particularly if he has an extensive knowledge of text data mining technology and can help to build a system around it.

Of course it is not only commercial applications that can use text mining, because it also has used in security, in that it can help to track criminal intent on the Internet. There are also applications in the biomedical world, in order to help find clusters of data in the right way. But it is in the online world and in the field of marketing that text mining is being used extensively, particularly in customer relationship management [CRM] techniques, where the tools are among some of the most advanced.

Knowing how text mining algorithms work is essential for any consultant who works in this field, because it is an important tool in the marketing technique possibilities. By understanding how text data mining can help an organization a consultant or marketer can make great strides in profitability and this is something that most organizations would be glad for.




Source: http://ezinearticles.com/?Text-Data-Mining-Can-Be-Profitable&id=2314536

Various Data Mining Techniques

Also called Knowledge Discover in Databases (KDD), data mining is the process of automatically sifting through large volumes of data for patterns, using tools such as clustering, classification, association rule mining, and many more. There are several major data mining techniques developed and known today, and this article will briefly tackle them, along with tools for increased efficiency, including phone look up services.

Classification is a classic data mining technique. Based on machine learning, it is used to classify each item on a data set into one of predefined set of groups or classes. This method uses mathematical techniques, like linear programming, decision trees, neural network, and statistics. For instance, you can apply this technique in an application that predicts which current employees will most probably leave in the future, based on the past records of those who have resigned or left the company.

Association is one of the most used techniques, and it is where a pattern is discovered basing on a relationship of a specific item on other items within the same transaction. Market basket analysis, for example, uses association to figure out what products or services are purchased together by clients. Businesses use the data produced to devise their marketing campaign.

Sequential patterns, too, aim to discover similar patterns in data transaction over a given business phase or period. These findings are used for business analysis to see relationships among data.

Clustering makes useful cluster of objects that maintain similar characteristics using an automatic method. While classification assigns objects into predefined classes, clustering defines the classes and puts objects in them. Predication, on the other hand, is a technique that digs into the relationship between independent variables and between dependent and independent variables. It can be used to predict profits in the future - a fitted regression curve used for profit prediction can be drawn from historical sale and profit data.

Of course, it is highly important to have high-quality data in all these data mining techniques. A multi-database web service, for instance, can be incorporated to provide the most accurate telephone number lookup. It delivers real-time access to a range of public, private, and proprietary telephone data. This type of phone look up service is fast-becoming a defacto standard for cleaning data and it communicates directly with telco data sources as well.

Phone number look up web services - just like lead, name, and address validation services - help make sure that information is always fresh, up-to-date, and in the best shape for data mining techniques to be applied.

Equip your business with better leads and get better conversion rates by using phone look up and similar real-time web services.




Source: http://ezinearticles.com/?Various-Data-Mining-Techniques&id=6985662

Benefits of Predictive Analytics and Data Mining Services

Predictive Analytics is the process of dealing with variety of data and apply various mathematical formulas to discover the best decision for a given situation. Predictive analytics gives your company a competitive edge and can be used to improve ROI substantially. It is the decision science that removes guesswork out of the decision-making process and applies proven scientific guidelines to find right solution in the shortest time possible.

Predictive analytics can be helpful in answering questions like:

    Who are most likely to respond to your offer?
    Who are most likely to ignore?
    Who are most likely to discontinue your service?
    How much a consumer will spend on your product?
    Which transaction is a fraud?
    Which insurance claim is a fraudulent?
    What resource should I dedicate at a given time?

Benefits of Data mining include:

    Better understanding of customer behavior propels better decision
    Profitable customers can be spotted fast and served accordingly
    Generate more business by reaching hidden markets
    Target your Marketing message more effectively
    Helps in minimizing risk and improves ROI.
    Improve profitability by detecting abnormal patterns in sales, claims, transactions etc
    Improved customer service and confidence
    Significant reduction in Direct Marketing expenses

Basic steps of Predictive Analytics are as follows:

    Spot the business problem or goal
    Explore various data sources such as transaction history, user demography, catalog details, etc)
    Extract different data patterns from the above data
    Build a sample model based on data & problem
    Classify data, find valuable factors, generate new variables
    Construct a Predictive model using sample
    Validate and Deploy this Model

Standard techniques used for it are:

    Decision Tree
    Multi-purpose Scaling
    Linear Regressions
    Logistic Regressions
    Factor Analytics
    Genetic Algorithms
    Cluster Analytics
    Product Association

Should you have any queries regarding Data Mining or Predictive Analytics applications, please feel free to contact us. We would be pleased to answer each of your queries in detail. Email us at info@outsourcingwebresearch.com




Source: http://ezinearticles.com/?Benefits-of-Predictive-Analytics-and-Data-Mining-Services&id=4766989

What is Data Mining?

Data mining is the process in which there is analysis of data forming different angles and perspectives and summarizing the same data into the relevant information. This kind of information could be utilized to increase the revenue, cutting the costs or both.

Software is mainly used for analyzing data and also assists in accumulation of data for the different sources and categorize and summarize the given data into some useful form.

Though the data mining is new term, the software used for mining the data was previously used. With the constant upgradations of the software and the processing power, the market tools, data mining software has increased in its accuracy. Formerly, this data mining was widely used by the businessmen for the market research and the analysis. There were few companies that used the computers to examine through the column of the supermarket data.

The data mining is the technique of running the data through the sophisticated algorithms for discovering the meaningful correlations and patterns that would have otherwise remained hidden. It is very helpful, since it aids in understanding the techniques and methods of business and you can accordingly apply your own intelligence fitting in the current market trend. Even the future performances get enhanced by the predictive analysis.

Business Intelligence operations occur in the background. Users of the mining operation can just see the end result. The users are in apposition to get the results through the mails and can also go through the recommendation through web pages and emails.

The data mining process indicates the invention of trends and tactics. The moment you discover and understand the market trends, you have the knowledge of which article is sold more and which article is sold with the other one. This kind of tend has an enormous impact on business organization. In this manner, the business gets enhanced as the market gets analyzed in a perfect manner. Due to these correlations, the performance of business organization increases to a lot of extent.

Mining gives a chance or opportunity to enhance the future performance of the business organization. There is a common philosophical phrase that, 'he who does not learn from the history is destined to repeat the same'. Therefore, if these predictions are done with the help and assistance of the historical information (data), then you can get sufficient data for improvising the products of the business organization.

Mining enables the embedding of the recommendations in the applications. Simple summary statements and the proposals can be displayed within the operational applications. Data mining also needs powerful machines. The algorithms might be applied to a Java or a Dataset code for using the same. Data mining is very useful for knowing the trends and making future predictions based on the predictive analysis. It also helps in cost cutting and increase in the revenue of the business organization

This article is part of Expertstown. You can visit Experts Town's Business Intelligence Blog for more information.




Source: http://ezinearticles.com/?What-is-Data-Mining?&id=3816784