A simple way to turn a website into JSON

Recently, while surfing the web I stumbled upon an simple web scraping service named Web Scrape Master. It is a kind of RESTful web service that extracts data from a specified web site and returns it to you in JSON format.
How it works

Though I don’t know what this service may be useful for, I still like its simplicity: all you need to do is to make an HTTP GET request, passing all necessary parameters in the query string:
http://webscrapemaster.com/api/?url={url}&xpath={xpath}&attr={attr}&callback={callback}

    url  - the URL of the website you want to scrape
    xpath – xpath determining the data you need to extract
    attr - attribute the name you need to get the value of (optional)
    callback - JSON callback function (optional)

For example, for the following request to our testing ground:

http://webscrapemaster.com/api/?url=http://testing-ground.extract-web-data.com/blocks&xpath=//div[@id=case1]/div[1]/span[1]/div

You will get the following response:

[{"text":"<div class='name'>Dell Latitude D610-1.73 Laptop Wireless Computer</div>","attrs":{"class":"name"}}]
Visual Web Scraper

Also, this service offers a special visual tool for building such requests. All you need to do is to enter the URL of the website and click to the element you need to scrape:
Visual Web Scraper
Conclusion

Though I understand that the developer of this service is attempting to create a simple web scraping service, it is still hard to imagine where it can be useful. The task that the service does can be easily accomplished by means of any language.

Probably if you already have software receiving JSON from the web, and you want to feed it with data from some website, then you may find this service useful. The other possible application is to hide your IP when you do web scraping. If you have other ideas, it would be great if you shared them with us.



Source: http://extract-web-data.com/a-simple-way-to-turn-a-website-into-json/

Selenium IDE and Web Scraping

Selenium is a browser automation framework that includes IDE, Remote Control server and bindings of various flavors including Java, .Net, Ruby, Python and other. In this post we touch on the basic structure of the framework and its application to  Web Scraping.
What is Selenium IDE


Selenium IDE is an integrated development environment for Selenium scripts. It is implemented as a Firefox plugin, and it allows recording browsers’ interactions in order to edit them. This works well for software tests, composing and debugging. The Selenium Remote Control is a server specific for a particular environment; it causes custom scripts to be implemented for controlled browsers. Selenium deploys on Windows, Linux, and iOS. How various Selenium components are supported with major browsers read here.
What does Selenium do and Web Scraping

Basically Selenium automates browsers. This ability is no doubt to be applied to web scraping. Since browsers (and Selenium) support JavaScript, jQuery and other methods working with dynamic content why not use this mix for benefit in web scraping, rather than to try to catch Ajax events with plain code? The second reason for this kind of scrape automation is browser-fasion data access (though today this is emulated with most libraries).

Yes, Selenium works to automate browsers, but how to control Selenium from a custom script to automate a browser for web scraping? There are Selenium PHP and other language libraries (bindings) providing for scripts to call and use Selenium. It is possible to write Selenium clients (using the libraries) in almost any language we prefer, for example Perl, Python, Java, PHP etc. Those libraries (API), along with a server, the Java written server that invokes browsers for actions, constitute the Selenum RC (Remote Control). Remote Control automatically loads the Selenium Core into the browser to control it. For more details in Selenium components refer to here.



A tough scrape task for programmer

“…cURL is good, but it is very basic.  I need to handle everything manually; I am creating HTTP requests by hand.
This gets difficult – I need to do a lot of work to make sure that the requests that I send are exactly the same as the requests that a browser would
send, both for my sake and for the website’s sake. (For my sake
because I want to get the right data, and for the website’s sake
because I don’t want to cause error messages or other problems on their site because I sent a bad request that messed with their web application).  And if there is any important javascript, I need to imitate it with PHP.
It would be a great benefit to me to be able to control a browser like Firefox with my code. It would solve all my problems regarding the emulation of a real browser…
it seems that Selenium will allow me to do this…” -Ryan S

Yes, that’s what we will consider below.
Scrape with Selenium

In order to create scripts that interact with the Selenium Server (Selenium RC, Selenium Remote Webdriver) or create local Selenium WebDriver script, there is the need to make use of language-specific client drivers (also called Formatters, they are included in the selenium-ide-1.10.0.xpi package). The Selenium servers, drivers and bindings are available at Selenium download page.
The basic recipe for scrape with Selenium:

    Use Chrome or Firefox browsers
    Get Firebug or Chrome Dev Tools (Cntl+Shift+I) in action.
    Install requirements (Remote control or WebDriver, libraries and other)
    Selenium IDE : Record a ‘test’ run thru a site, adding some assertions.
    Export as a Python (other language) script.
    Edit it (loops, data extraction, db input/output)
    Run script for the Remote Control

The short intro Slides for the scraping of tough websites with Python & Selenium are here (as Google Docs slides) and here (Slide Share).
Selenium components for Firefox installation guide

For how to install the Selenium IDE to Firefox see  here starting at slide 21. The Selenium Core and Remote Control installation instructions are there too.
Extracting for dynamic content using jQuery/JavaScript with Selenium

One programmer is doing a similar thing …

1. launch a selenium RC (remote control) server
2. load a page
3. inject the jQuery script
4. select the interested contents using jQuery/JavaScript
5. send back to the PHP client using JSON.

He particularly finds it quite easy and convenient to use jQuery for
screen scraping, rather than using PHP/XPath.
Conclusion

The Selenium IDE is the popular tool for browser automation, mostly for its software testing application, yet also in that Web Scraping techniques for tough dynamic websites may be implemented with IDE along with the Selenium Remote Control server. These are the basic steps for it:

    Record the ‘test‘ browser behavior in IDE and export it as the custom programming language script
    Formatted language script runs on the Remote Control server that forces browser to send HTTP requests and then script catches the Ajax powered responses to extract content.

Selenium based Web Scraping is an easy task for small scale projects, but it consumes a lot of memory resources, since for each request it will launch a new browser instance.



Source: http://extract-web-data.com/selenium-ide-and-web-scraping/

Online Data Entry Positions From Home - Know What Is Required

Some of the hardened data entry workers are earning a lot of money each and every day, the opportunity to earn hundreds of dollars per day are not a myth. Many people started out just entering data in a part time role. Once they got into the swing of things they soon realised that it was well within their power to take this part time job to the next level and earn some serious money.

As an online data entry worker you no longer have to worry about financial issues, you can quite easily give up your day jobs and stay at home to earn money. The time is right for considering such a change of life style, increases in interest rates and the cost of living means that a standard salary just isn't enough any more. People with steady 9-5 jobs are barely scraping by every month. The reason I started this home was to be able to spend time with my young children and to earn a little bit of extra cash. I never realised that the potential was available to actually make a full time living from it, let alone a very good living.

There are no free data entry work from home jobs which are even worth signing up for, save your time and keep looking for companies which actually give you all the information you need to get started. I signed up for a few free schemes and they weren't worth the time it took to fill in a subscription form. Good data entry from home positions are available, just be careful while researching. Many companies promise you the earth but far from deliver, unfortunately this means that genuine people have to search a lot harder to earn money online.

If you want to know how I earn money online then keep reading.

Any job can have its ups and downs, just try to keep focusing on the reason you started and the goals you wish to achieve. If you start to feel stressed or under pressure take a walk or do something else which will clear your head for a couple of hours.

Only you can decide how much or how little you can achieve in one day, so you don't have to push yourself to the limit every day or it will soon become hard work, which isn't why you want to start data entry from home




Source: http://ezinearticles.com/?Online-Data-Entry-Positions-From-Home---Know-What-Is-Required&id=797035

Business Intelligence Data Mining

Data mining can be technically defined as the automated extraction of hidden information from large databases for predictive analysis. In other words, it is the retrieval of useful information from large masses of data, which is also presented in an analyzed form for specific decision-making.

Data mining requires the use of mathematical algorithms and statistical techniques integrated with software tools. The final product is an easy-to-use software package that can be used even by non-mathematicians to effectively analyze the data they have. Data Mining is used in several applications like market research, consumer behavior, direct marketing, bioinformatics, genetics, text analysis, fraud detection, web site personalization, e-commerce, healthcare, customer relationship management, financial services and telecommunications.

Business intelligence data mining is used in market research, industry research, and for competitor analysis. It has applications in major industries like direct marketing, e-commerce, customer relationship management, healthcare, the oil and gas industry, scientific tests, genetics, telecommunications, financial services and utilities. BI uses various technologies like data mining, scorecarding, data warehouses, text mining, decision support systems, executive information systems, management information systems and geographic information systems for analyzing useful information for business decision making.

Business intelligence is a broader arena of decision-making that uses data mining as one of the tools. In fact, the use of data mining in BI makes the data more relevant in application. There are several kinds of data mining: text mining, web mining, social networks data mining, relational databases, pictorial data mining, audio data mining and video data mining, that are all used in business intelligence applications.

Some data mining tools used in BI are: decision trees, information gain, probability, probability density functions, Gaussians, maximum likelihood estimation, Gaussian Baves classification, cross-validation, neural networks, instance-based learning /case-based/ memory-based/non-parametric, regression algorithms, Bayesian networks, Gaussian mixture models, K-means and hierarchical clustering, Markov models and so on.




Source: http://ezinearticles.com/?Business-Intelligence-Data-Mining&id=196648

Basics of Web Data Mining and Challenges in Web Data Mining Process

Today World Wide Web is flooded with billions of static and dynamic web pages created with programming languages such as HTML, PHP and ASP. Web is great source of information offering a lush playground for data mining. Since the data stored on web is in various formats and are dynamic in nature, it's a significant challenge to search, process and present the unstructured information available on the web.

Complexity of a Web page far exceeds the complexity of any conventional text document. Web pages on the internet lack uniformity and standardization while traditional books and text documents are much simpler in their consistency. Further, search engines with their limited capacity can not index all the web pages which makes data mining extremely inefficient.

Moreover, Internet is a highly dynamic knowledge resource and grows at a rapid pace. Sports, News, Finance and Corporate sites update their websites on hourly or daily basis. Today Web reaches to millions of users having different profiles, interests and usage purposes. Every one of these requires good information but don't know how to retrieve relevant data efficiently and with least efforts.

It is important to note that only a small section of the web possesses really useful information. There are three usual methods that a user adopts when accessing information stored on the internet:

• Random surfing i.e. following large numbers of hyperlinks available on the web page.
• Query based search on Search Engines - use Google or Yahoo to find relevant documents (entering specific keywords queries of interest in search box)
• Deep query searches i.e. fetching searchable database from eBay.com's product search engines or Business.com's service directory, etc.

To use the web as an effective resource and knowledge discovery researchers have developed efficient data mining techniques to extract relevant data easily, smoothly and cost-effectively.

Should you have any queries regarding Web Data mining processes, please feel free to contact us at info@outsourcingwebresearch.com





Source: http://ezinearticles.com/?Basics-of-Web-Data-Mining-and-Challenges-in-Web-Data-Mining-Process&id=4937441

The Benefits of Data Mining

Data mining can truly help a business reach its fullest potential. It is a way to assess how business is being affected by certain characteristics, and can help business owners increase their profits and avoid making business mistakes down the line. Essentially, through this process, a business is analyzing certain data from different perspectives in order to get a full rounded view of how their company is doing. Business owners can get a broad perspective on things such as customer trending, where they are losing money and where they are making money. The information can also reveal ways that can help a business cut unneeded costs and can help them increase their overall income.

Data mining software is one tool that can help a company assess and analyze their data in more efficient terms. It can be extremely user friendly and allow people to delve into their data from a variety of different angles and points of view. In more technical terms, data mining software allows you to see the correlations and patterns of one's own data compared with those across many other regional databases.

People have been using data mining for many years in different formats. Only since the technology has become available has data software been used. But there have been many ways in the past for companies to assess their data and use it to their advantage. By taking polls, or using store scanners, product codes and bar codes, people have been able to gather data, analyze it and use it to their advantage. But it cannot be denied that the availability of greater technology has greatly increased the ability to store or gather data, make predictions about outcomes and use customer trend reports to greater advantages. The ability to store infinite amounts of data has given business owners a great advantage and truly has helped increase sales and lower costs. This data mining has actually led to data being stored in data warehouses. In data warehouses, various organizations will integrate their mined data into one large data warehouse. The information accessible in data warehouses is available to further help companies reduce risk taking and integrate proper selling techniques to improve business.

Data mining also can allow companies to see where their best selling points are and give them the opportunity to take advantage of this information. For example, if a pharmacy places a display of lip balm at the cashier counter, data mining can detect how many people bought lip balm from the cashier counter rather people who bought the lip balm when it was placed at another point in the store. Data mining can determine where the most effective points of sale are throughout a store or if a certain promotion went well one time of the month, but did not go well at another time of the month. Companies can make offers based on the buying habits of their customers as well.

Data mining can truly help businesses reach their highest profitability by paying attention to customer trending.

Improving your overall business performance is never easy. However, new innovations in data mining software can increase your information forecasting capabilities and enhance your profit drivers as well!




Source: http://ezinearticles.com/?The-Benefits-of-Data-Mining&id=4565509

Data Entry Services From India - Definitely Boon-Rich

Whatever the opinion of the US presidency concerning outsourcing to India, the practice is sure to continue for a long, long time. The monetary benefits from outsourcing become crystal clear when you look at this statistic for the pay of entry level accountants. For a US worker, the hourly wage is $23 but for a company in India, it is only $11.22. The scope of data entry services from India is really wide with centers in the country offering their assistance for finance, academic, insurance, healthcare, website and legal related applications of these services. Read more of this article and understand why outsourcing of the back office task to the world's largest democracy is definitely boon rich.

Data Entry Services from India - the Benefits

The first important benefit as discussed in the first paragraph is the monetary benefit. US companies can cut their operating expenses by at least 30 percent when they outsource to India. This is because of the reduced operating expenses of companies in India, which in turn is because of the lower costs for leases, rents, utility rates, taxes and so on. More benefits are discussed below:

• Experts in the English language - India excels many other countries in English language competence. Thus use of language in the output and communication is not a problem. A lot of companies in India are giving their staff training in US accent, so that too is not a problem.

• Advantageous Time Difference - Some people may see the time difference between the US and India as disadvantageous because they wouldn't be working at the same time. However, the time difference is actually advantageous because US companies can function for more number of hours. When the employees of the US companies sleep, the employees of the Indian companies would be hard at work to finish the back office task by the specified deadline.

• Quality Output - You can expect to get practically error-free output from the country because of the employees being highly educated. These employees (a lot of them in the 16 to 25 age group) also have exceptional knowledge of software and other Information Technology. Quality levels are consistently maintained through the process of double entry which is very beneficial in identifying and correcting arbitrary miskeyed strokes that even experienced data entry staff would miss.

• Reduce Risks - Indian providers of data entry services are known for their expertise. So while you're busy taking care of your everyday core business responsibilities, they're giving you error-free output that can reduce error-related risks for your business.

• Favorable Turnaround - This is a benefit which many Indian business process outsourcing companies highlight on their websites. Whatever the volume of your project, it would be within a favorable turnaround without compromising quality.

More Advantageous Services

It's not just data entry services but also medical transcription, medical billing and coding, website design, search engine optimization, call center operations and other services from India which are boon rich. Companies across the globe know this and continue to benefit from the fact.

As a leading outsourcing company, Managed Outsource Solutions (MOS) can help you streamline your data entry processes and improve the overall functioning of your organization.




Source: http://ezinearticles.com/?Data-Entry-Services-From-India---Definitely-Boon-Rich&id=7242367