How to use Social Media Scraping to be your Competitors’ Nightmare

How to use Social Media Scraping to be your Competitors’ Nightmare

Big data and competitive intelligence have been in the limelight for quite some time now. The almost magical power of big data to help a company make just the right decisions have been talked about a lot. When it comes to big data, the kind of benefits that a business can get totally depends upon the sources they acquire it from. Social media is one of the best sources from where you can get data that helps your business in a multitude of ways. Now that every business is deep rooted on the internet, social media data becomes all the more relevant and crucial. Here is how you can use data scraped from social media sites to get an edge in the competition.

Keeping watch on your competitors

Social media is the best place to watch your competitors’ activity and take counter initiatives to keep up or take over them. If you want to know what your competitors are up to, a social media scraping setup for scraping the posts that mention your competitors’ brand/product names can do the trick. This can also be used to learn a thing or two from their activities on social media so that you can take respective measures to stay ahead of them. For example, you could know if your competitor is running a special promotional offer at the moment and come up with something better than theirs to keep up. This can do wonders if you are in a highly competitive industry like Ecommerce where the competition is intense. If you are not using some help from web scraping technology to keep a close watch on your competitors, you could easily get left over in this fast-paced business scene.

Solving customer issues at the earliest

Customers are vocal about their experience with different products and services on social media sites these days. If you have a customer whose issue was left unsolved, there is a good chance that he/she will take it to the social media to vent the frustration. Watching out for such instances and giving them prompt support should be something you should do if you want to retain these customers and stop them from ruining your brand’s image. By scraping social media sites for posts that mention your product/service, you can easily find out if there are such grievances from customers. This can make sure to an extent that you don’t let unhappy customers stay that way, which eventually hurts your business in the long run. Customers can make or break your company, so using social media scraping to serve the customers better can help you succeed eventually.

Sentiment analysis

Social media data can play a good job at helping you understand user sentiments. With the help of social media scraping, a business can get the big picture about general perception of their brand by their users. This can go a long way since this level of feedback can help you fix unnoticed issues with your company and service quickly. By rectifying them, you can make your brand more appealing to the customers. Sentiment analysis will provide you with the opportunity to transform your business into how customers want it to be. Social media scraping is the one and only way to have access to this user sentiment data which can help you optimize your business for the customers.

Web crawling for social media data

When social media data possess so much value to businesses, it makes sense to look for efficient ways to gather and use this data. Manually scrolling through millions of tweets doesn’t make sense, this is why you should use social media scraping to aggregate the relevant data for your business. Besides, web scraping technologies make it possible to handle huge amounts of data with ease. Since the size of data is huge when it comes to business related requirements, web scraping is the only scalable solution worth considering. To make things even simpler, there are reliable web scraping solutions that offer social media scraping services for brand monitoring.

Bottom line

Since social media has become an integral part of online businesses, the data available on these sites possess immense value to companies in every industry. Social media scraping can be used for brand monitoring and gaining competitive intelligence that can be used to optimize your business model for maximum effectiveness. This will in turn make your company stand out from the competition and the added advantage of insights gained from social media data will help you to take over your competitors.

Source: https://www.promptcloud.com/blog/social-media-scraping-for-competitive-intelligence

Business Intelligence & Data Warehousing in a Business Perspective

Business Intelligence & Data Warehousing in a Business Perspective

Business Intelligence

Business Intelligence has become a very important activity in the business arena irrespective of the domain due to the fact that managers need to analyze comprehensively in order to face the challenges.

Data sourcing, data analysing, extracting the correct information for a given criteria, assessing the risks and finally supporting the decision making process are the main components of BI.

In a business perspective, core stakeholders need to be well aware of all the above stages and be crystal clear on expectations. The person, who is being assigned with the role of Business Analyst (BA) for the BI initiative either from the BI solution providers' side or the company itself, needs to take the full responsibility on assuring that all the above steps are correctly being carried out, in a way that it would ultimately give the business the expected leverage. The management, who will be the users of the BI solution, and the business stakeholders, need to communicate with the BA correctly and elaborately on their expectations and help him throughout the process.

Data sourcing is an initial yet crucial step that would have a direct impact on the system where extracting information from multiple sources of data has to be carried out. The data may be on text documents such as memos, reports, email messages, and it may be on the formats such as photographs, images, sounds, and they can be on more computer oriented sources like databases, formatted tables, web pages and URL lists. The key to data sourcing is to obtain the information in electronic form. Therefore, typically scanners, digital cameras, database queries, web searches, computer file access etc, would play significant roles. In a business perspective, emphasis should be placed on the identification of the correct relevant data sources, the granularity of the data to be extracted, possibility of data being extracted from identified sources and the confirmation that only correct and accurate data is extracted and passed on to the data analysis stage of the BI process.

Business oriented stake holders guided by the BA need to put in lot of thought during the analyzing stage as well, which is the second phase. Synthesizing useful knowledge from collections of data should be done in an analytical way using the in-depth business knowledge whilst estimating current trends, integrating and summarizing disparate information, validating models of understanding, and predicting missing information or future trends. This process of data analysis is also called data mining or knowledge discovery. Probability theory, statistical analysis methods, operational research and artificial intelligence are the tools to be used within this stage. It is not expected that business oriented stake holders (including the BA) are experts of all the above theoretical concepts and application methodologies, but they need to be able to guide the relevant resources in order to achieve the ultimate expectations of BI, which they know best.

Identifying relevant criteria, conditions and parameters of report generation is solely based on business requirements, which need to be well communicated by the users and correctly captured by the BA. Ultimately, correct decision support will be facilitated through the BI initiative and it aims to provide warnings on important events, such as takeovers, market changes, and poor staff performance, so that preventative steps could be taken. It seeks to help analyze and make better business decisions, to improve sales or customer satisfaction or staff morale. It presents the information that manager's need, as and when they need it.

In a business sense, BI should go several steps forward bypassing the mere conventional reporting, which should explain "what has happened?" through baseline metrics. The value addition will be higher if it can produce descriptive metrics, which will explain "why has it happened?" and the value added to the business will be much higher if predictive metrics could be provided to explain "what will happen?" Therefore, when providing a BI solution, it is important to think in these additional value adding lines.

Data warehousing

In the context of BI, data warehousing (DW) is also a critical resource to be implemented to maximize the effectiveness of the BI process. BI and DW are two terminologies that go in line. It has come to a level where a true BI system is ineffective without a powerful DW, in order to understand the reality behind this statement, it's important to have an insight in to what DW really is.

A data warehouse is one large data store for the business in concern which has integrated, time variant, non volatile collection of data in support of management's decision making process. It will mainly have transactional data which would facilitate effective querying, analyzing and report generation, which in turn would give the management the required level of information for the decision making.

The reasons to have BI together with DW

At this point, it should be made clear why a BI tool is more effective with a powerful DW. To query, analyze and generate worthy reports, the systems should have information available. Importantly, transactional information such as sales data, human resources data etc. are available normally in different applications of the enterprise, which would obviously be physically held in different databases. Therefore, data is not at one particular place, hence making it very difficult to generate intelligent information.

The level of reports expected today, are not merely independent for each department, but managers today want to analyze data and relationships across the enterprise so that their BI process is effective. Therefore, having data coming from all the sources to one location in the form of a data warehouse is crucial for the success of the BI initiative. In a business viewpoint, this message should be passed and sold to the managements of enterprises so that they understand the value of the investment. Once invested, its gains could be achieved over several years, in turn marking a high ROI.

Investment costs for a DW in the short term may look quite high, but it's important to re-iterate that the gains are much higher and it will span over many years to come. It also reduces future development cost since with the DW any requested report or view could be easily facilitated. However, it is important to find the right business sponsor for the project. He or she needs to communicate regularly with executives to ensure that they understand the value of what's being built. Business sponsors need to be decisive, take an enterprise-wide perspective and have the authority to enforce their decisions.

Process

Implementation of a DW itself overlaps with some phases of the above explained BI process and it's important to note that in a process standpoint, DW falls in to the first few phases of the entire BI initiative. Gaining highly valuable information out of DW is the latter part of the BI process. This can be done in many ways. DW can be used as the data repository of application servers that run decision support systems, management Information Systems, Expert systems etc., through them, intelligent information could be achieved.

But one of the latest strategies is to build cubes out of the DW and allow users to analyze data in multiple dimensions, and also provide with powerful analytical supporting such as drill down information in to granular levels. Cube is a concept that is different to the traditional relational 2-dimensional tabular view, and it has multiple dimensions, allowing a manager to analyze data based on multiple factors, and not just two factors. On the other hand, it allows the user to select whatever the dimension he wish to choose for analyzing purposes and not be limited by one fixed view of data, which is called as slice & dice in DW terminology.

BI for a serious enterprise is not just a phase of a computerization process, but it is one of the major strategies behind the entire organizational drivers. Therefore management should sit down and build up a BI strategy for the company and identify the information they require in each business direction within the enterprise. Given this, BA needs to analyze the organizational data sources in order to build up the most effective DW which would help the strategized BI process.

High level Ideas on Implementation

At the heart of the data warehousing process is the extract, transform, and load (ETL) process. Implementation of this merely is a technical concern but it's a business concern to make sure it is designed in such a way that it ultimately helps to satisfy the business requirements. This process is responsible for connecting to and extracting data from one or more transactional systems (source systems), transforming it according to the business rules defined through the business objectives, and loading it into the all important data model. It is at this point where data quality should be gained. Of the many responsibilities of the data warehouse, the ETL process represents a significant portion of all the moving parts of the warehousing process.

Creation of a powerful DW depends on the correctness of data modeling, which is the responsibility of the database architect of the project, but BA needs to play a pivotal role providing him with correct data sources, data requirements and most importantly business dimensions. Business Dimensional modeling is a special method used for DW projects and this normally should be carried out by the BA and from there onwards technical experts should take up the work. Dimensions are perspectives specific to a business that could be used for analysis purposes. As an example, for a sales database, the dimensions could include Product, Time, Store, etc. Obviously these dimensions differ from one business to another and hence for each DW initiative those dimensions should be correctly identified and that could be very well done by a person who has experience in the DW domain and understands the business as well, making it apparent that DW BA is the person responsible.

Each of the identified dimensions would be turned in to a dimension table at the implementation phase, and the objective of the above explained ETL process is to fill up these dimension tables, which in turn will be taken to the level of the DW after performing some more database activities based on a strong underlying data model. Implementation details are not important for a business stakeholder but being aware of high level process to this level is important so that they are also on the same pitch as that of the developers and can confirm that developers are actually doing what they are supposed to do and would ultimately deliver what they are supposed to deliver.

Security is also vital in this regard, since this entire effort deals with highly sensitive information and identification of access right to specific people to specific information should be correctly identified and captured at the requirements analysis stage.

Advantages

There are so many advantages of BI system. More presentation of analytics directly to the customer or supply chain partner will be possible. Customer scores, customer campaigns and new product bundles can all be produced from analytic structures resulting in high customer retention and creation of unique products. More collaboration within information can be achieved from effective BI. Rather than middle managers getting great reports and making their own areas look good, information will be conveyed into other functions and rapidly shared to create collaborative decisions increasing the efficiency and accuracy. The return on human capital will be greatly increased.

Managers at all levels will save their time on data analysis, and hence saving money for the enterprise, as the time of managers is equal to money in a financial perspective. Since powerful BI would enable monitoring internal processes of the enterprises more closely and allow making them more efficient, the overall success of the organization would automatically grow. All these would help to derive a high ROI on BI together with a strong DW. It is a common experience to notice very high ROI figures on such implementations, and it is also important to note that there are many non-measurable gains whilst we consider most of the measurable gains for the ROI calculation. However, at a stage where it is intended to take the management buy-in for the BI initiative, it's important to convert all the non measurable gains in to monitory values as much as possible, for example, saving of managers time can be converted in to a monitory value using his compensation.

The author has knowledge in both Business and IT. Started career as a Software Engineer and moved to work in the business analysis area of a premier US based software company.

Source: http://ezinearticles.com/?Business-Intelligence-and-Data-Warehousing-in-a-Business-Perspective&id=35640

Difference between Data Mining and KDD

Difference between Data Mining and KDD

Data, in its raw form, is just a collection of things, where little information might be derived. Together with the development of information discovery methods(Data Mining and KDD), the value of the info is significantly improved.

Data mining is one among the steps of Knowledge Discovery in Databases(KDD) as can be shown by the image below.KDD is a multi-step process that encourages the conversion of data to useful information. Data mining is the pattern extraction phase of KDD. Data mining can take on several types, the option influenced by the desired outcomes.

Knowledge Discovery in Databases Steps
Data Selection

KDD isn’t prepared without human interaction. The choice of subset and the data set requires knowledge of the domain from which the data is to be taken. Removing non-related information elements from the dataset reduces the search space during the data mining phase of KDD. The sample size and structure are established during this point, if the dataset can be assessed employing a testing of the info.
Pre-processing

Databases do contain incorrect or missing data. During the pre-processing phase, the information is cleaned. This warrants the removal of “outliers”, if appropriate; choosing approaches for handling missing data fields; accounting for time sequence information, and applicable normalization of data.
Transformation

Within the transformation phase attempts to reduce the variety of data elements can be assessed while preserving the quality of the info. During this stage, information is organized, changed in one type to some other (i.e. changing nominal to numeric) and new or “derived” attributes are defined.
Data mining

Now the info is subjected to one or several data-mining methods such as regression, group, or clustering. The information mining part of KDD usually requires repeated iterative application of particular data mining methods. Different data-mining techniques or models can be used depending on the expected outcome.
Evaluation

The final step is documentation and interpretation of the outcomes from the previous steps. Steps during this period might consist of returning to a previous step up the KDD approach to help refine the acquired knowledge, or converting the knowledge in to a form clear for the user.In this stage the extracted data patterns are visualized for further reviews.
Conclusion

Data mining is a very crucial step of the KDD process.

For further reading aboud KDD and data mining ,please check this link.

Source: http://nocodewebscraping.com/difference-data-mining-kdd/

Invest in Data Extraction to Grow Your Business

Invest in Data Extraction to Grow Your Business

Automating your employees’ processes can help you increase productivity while keeping the cost of used resources at a minimum. This can help you focus your time and money in much needed areas of your company so that you can thrive in your industry. Data extraction can help you achieve automation by targeting online data sources (websites, blogs, forums, etc) for information and data that can be useful to your business. By using software rather than your employees, you can oftentimes get more accurate data and more thorough information that people may miss. The software can handle the volume that you need and will deliver the results that you desire to help your company.
See the Power of Data Extraction Online

To see all of the ways that data extraction tools and software can benefit your business, visit Connotate.com. There you can read about the features of the software, practical uses for businesses and also schedule a demo before you buy.

Source: http://www.connotate.com/invest-in-data-extraction-to-grow-your-business/

Best Alternative For Linkedin Data Scraping

Best Alternative For Linkedin Data Scraping

When I started my career in sales, one of the things that my VP of sales told me is that ” In sales, assumptions are the mother of all f**k ups “. I know the F word sounds a bit inappropriate, but that is the exact word he used. He was trying to convey the simple point that every prospect is different, so don’t guess, use data to come up with decisions.

I joined Datahut and we are working on a product that helps sales people. I thought I should discuss it with you guys and take your feedback.

Let me tell you how the idea evolved itself. At Datahut, we get to hear a lot of problems customers want to solve. Almost 30 percent of all the inbound leads ask us to help them with lead generation.

Most of them simply ask, “Can you scrape Linkedin for me”?

Every time, we politely refused.

But not anymore, we figured out a way to solve their problem without scraping Linkedin.

This should raise some questions in your mind.

1) What problem is he trying to solve?– Most of the time their sales team does not have the accurate data about the prospects. This leads to a total chaos. It will end up in a waste of both time and money by selling the leads that are not sales qualified.

2) Why do they need data specifically from Linkedin? – LinkedIn is the world’s largest business network. In his view, there is no better place to find leads for his business than Linkedin. It is right in a way.

3) Ok, then what is wrong in scraping Linkedin? – Scraping Linkedin is against its terms and it can lead to legal issues. Linkedin has an excellent anti-scraping mechanism which can make the scraping costly.

4) How severe is the problem? – The problem has a direct impact on the revenues as the productivity of the sales team is too low. Without enough sales, the company is a joke.

5) Is there a better way? – Of course yes. The people with profiles in LinkedIn are in other sites too. eg. Google plus, CrunchBase etc. If we can mine and correlate the data, we can generate leads with rich information. It will have better quality than scraping LinkedIn.

6) What to do when the machine intelligence fails? – We have to use human intelligence. Period!

Datahut is working on a platform that can help you get leads that match your ideal buyer persona. It will be a complete Business intelligence platform powered by machine and human intelligence for an efficient lead research & discovery.We named it Leadintel. We’ve also established some partnerships that help to enrich the data and saves the trouble of lawsuits.

We are opening our platform for beta users. You can request an invitation using the contact form. What do you think about this? What are your suggestions?

Thanks for reading this blog post. Datahut offers affordable data extraction services (DaaS) . If you need help with your web scraping projects let us know and we will be glad to help.

Source:http://blog.datahut.co/best-alternative-for-linkedin-data-scraping/

Python 3 web-scraping examples with public data

Someone on the NICAR-L listserv asked for advice on the best Python libraries for web scraping. My advice below includes what I did for last spring’s Computational Journalism class, specifically, the Search-Script-Scrape project, which involved 101-web-scraping exercises in Python.

Best Python libraries for web scraping

For the remainder of this post, I assume you’re using Python 3.x, though the code examples will be virtually the same for 2.x. For my class last year, I had everyone install the Anaconda Python distribution, which comes with all the libraries needed to complete the Search-Script-Scrape exercises, including the ones mentioned specifically below:
The best package for general web requests, such as downloading a file or submitting a POST request to a form, is the simply-named requests library (“HTTP for Humans”).

Here’s an overly verbose example:

import requests
base_url = 'http://maps.googleapis.com/maps/api/geocode/json'
my_params = {'address': '100 Broadway, New York, NY, U.S.A',
             'language': 'ca'}
response = requests.get(base_url, params = my_params)
results = response.json()['results']
x_geo = results[0]['geometry']['location']
print(x_geo['lng'], x_geo['lat'])
# -74.01110299999999 40.7079445

For the parsing of HTML and XML, Beautiful Soup 4 seems to be the most frequently recommended. I never got around to using it because it was malfunctioning on my particular installation of Anaconda on OS X.
But I’ve found lxml to be perfectly fine. I believe both lxml and bs4 have similar capabilities – you can even specify lxml to be the parser for bs4. I think bs4 might have a friendlier syntax, but again, I don’t know, as I’ve gotten by with lxml just fine:

import requests
from lxml import html
page = requests.get("http://www.example.com").text
doc = html.fromstring(page)
link = doc.cssselect("a")[0]
print(link.text_content())
# More information...
print(link.attrib['href'])
# http://www.iana.org/domains/example

The standard urllib package also has a lot of useful utilities – I frequently use the methods from urllib.parse. Python 2 also has urllib but the methods are arranged differently.

Here’s an example of using the urljoin method to resolve the relative links on the California state data for high school test scores. The use of os.path.basename is simply for saving the each spreadsheet to your local hard drive:

from os.path import basename
from urllib.parse import urljoin
from lxml import html
import requests
base_url = 'http://www.cde.ca.gov/ds/sp/ai/'
page = requests.get(base_url).text
doc = html.fromstring(page)
hrefs = [a.attrib['href'] for a in doc.cssselect('a')]
xls_hrefs = [href for href in hrefs if 'xls' in href]
for href in xls_hrefs:
  print(href) # e.g. documents/sat02.xls
  url = urljoin(base_url, href)
  with open("/tmp/" + basename(url), 'wb') as f:
    print("Downloading", url)
    # Downloading http://www.cde.ca.gov/ds/sp/ai/documents/sat02.xls
    data = requests.get(url).content
    f.write(data)

And that’s about all you need for the majority of web-scraping work – at least the part that involves reading HTML and downloading files.
Examples of sites to scrape

The 101 scraping exercises didn’t go so great, as I didn’t give enough specifics about what the exact answers should be (e.g. round the numbers? Use complete sentences?) or even where the data files actually were – as it so happens, not everyone Googles things the same way I do. And I should’ve made them do it on a weekly basis, rather than waiting till the end of the quarter to try to cram them in before finals week.

The Github repo lists each exercise with the solution code, the relevant URL, and the number of lines in the solution code.

The exercises run the gamut of simple parsing of static HTML, to inspecting AJAX-heavy sites in which knowledge of the network panel is required to discover the JSON files to grab. In many of these exercises, the HTML-parsing is the trivial part – just a few lines to parse the HTML to dynamically find the URL for the zip or Excel file to download (via requests)…and then 40 to 50 lines of unzipping/reading/filtering to get the answer. That part is beyond what typically considered “web-scraping” and falls more into “data wrangling”.

I didn’t sort the exercises on the list by difficulty, and many of the solutions are not particulary great code. Sometimes I wrote the solution as if I were teaching it to a beginner. But other times I solved the problem using the style in the most randomly bizarre way relative to how I would normally solve it – hey, writing 100+ scrapers gets boring.

But here are a few representative exercises with some explanation:
1. Number of datasets currently listed on data.gov

I think data.gov actually has an API, but this script relies on finding the easiest tag to grab from the front page and extracting the text, i.e. the 186,569 from the text string, "186,569 datasets found". This is obviously not a very robust script, as it will break when data.gov is redesigned. But it serves as a quick and easy HTML-parsing example.
29. Number of days until Texas’s next scheduled execution

Texas’s death penalty site is probably one of the best places to practice web scraping, as the HTML is pretty straightforward on the main landing pages (there are several, for scheduled and past executions, and current inmate roster), which have enough interesting tabular data to collect. But you can make it more complex by traversing the links to collect inmate data, mugshots, and final words. This script just finds the first person on the scheduled list and does some math to print the number of days until the execution (I probably made the datetime handling more convoluted than it needs to be in the provided solution)
3. The number of people who visited a U.S. government website using Internet Explorer 6.0 in the last 90 days

The analytics.usa.gov site is a great place to practice AJAX-data scraping. It’s a very simple and robust site, but either you are aware of AJAX and know how to use the network panel (and in this case, locate ie.json, or you will have no clue how to scrape even a single number on this webpage. I think the difference between static HTML and AJAX sites is one of the tougher things to teach novices. But they pretty much have to learn the difference given how many of today’s websites use both static and dynamically-rendered pages.
6. From 2010 to 2013, the change in median cost of health, dental, and vision coverage for California city employees

There’s actually no HTML parsing if you assume the URLs for the data files can be hard coded. So besides the nominal use of the requests library, this ends up being a data-wrangling exercise: download two specific zip files, unzip them, read the CSV files, filter the dictionaries, then do some math.
90. The currently serving U.S. congressmember with the most Twitter followers

Another example with no HTML parsing, but probably the most complicated example. You have to download and parse Sunlight Foundation’s CSV of Congressmember data to get all the Twitter usernames. Then authenticate with Twitter’s API, then perform mulitple batch lookups to get the data for all 500+ of the Congressional Twitter usernames. Then join the sorted result with the actual Congressmember identity. I probably shouldn’t have assigned this one.
HTML is not necessary

I included no-HTML exercises because there are plenty of data programming exercises that don’t have to deal with the specific nitty-gritty of the Web, such as understanding HTTP and/or HTML. It’s not just that a lot of public data has moved to JSON (e.g. the FEC API) – but that much of the best public data is found in bulk CSV and database files. These files can be programmatically fetched with simple usage of the requests library.

It’s not that parsing HTML isn’t a whole boatload of fun – and being able to do so is a useful skill if you want to build websites. But I believe novices have more than enough to learn from in sorting/filtering dictionaries and lists without worrying about learning how a website works.

Besides analytics.usa.gov, the data.usajobs.gov API, which lists federal job openings, is a great one to explore, because its data structure is simple and the site is robust. Here’s a Python exercise with the USAJobs API; and here’s one in Bash.

There’s also the Google Maps geocoding API, which can be hit up for a bit before you run into rate limits, and you get the bonus of teaching geocoding concepts. The NYTimes API requires creating an account, but you not only get good APIs for some political data, but for content data (i.e. articles, bestselling books) that is interesting fodder for journalism-related analysis.

But if you want to scrape HTML, then the Texas death penalty pages are the way to go, because of the simplicity of the HTML and the numerous ways you can traverse the pages and collect interesting data points. Besides the previously mentioned Texas Python scraping exercise, here’s one for Florida’s list of executions. And here’s a Bash exercise that scrapes data from Texas, Florida, and California and does a simple demographic analysis.

If you want more interesting public datasets – most of which require only a minimal of HTML-parsing to fetch – check out the list I talked about in last week’s info session on Stanford’s Computational Journalism Lab.

Source URL :  http://blog.danwin.com/examples-of-web-scraping-in-python-3-x-for-data-journalists/

Python 3 web-scraping examples with public data

Someone on the NICAR-L listserv asked for advice on the best Python libraries for web scraping. My advice below includes what I did for last spring’s Computational Journalism class, specifically, the Search-Script-Scrape project, which involved 101-web-scraping exercises in Python.

Best Python libraries for web scraping

For the remainder of this post, I assume you’re using Python 3.x, though the code examples will be virtually the same for 2.x. For my class last year, I had everyone install the Anaconda Python distribution, which comes with all the libraries needed to complete the Search-Script-Scrape exercises, including the ones mentioned specifically below:
The best package for general web requests, such as downloading a file or submitting a POST request to a form, is the simply-named requests library (“HTTP for Humans”).

Here’s an overly verbose example:

import requests
base_url = 'http://maps.googleapis.com/maps/api/geocode/json'
my_params = {'address': '100 Broadway, New York, NY, U.S.A',
             'language': 'ca'}
response = requests.get(base_url, params = my_params)
results = response.json()['results']
x_geo = results[0]['geometry']['location']
print(x_geo['lng'], x_geo['lat'])
# -74.01110299999999 40.7079445

For the parsing of HTML and XML, Beautiful Soup 4 seems to be the most frequently recommended. I never got around to using it because it was malfunctioning on my particular installation of Anaconda on OS X.
But I’ve found lxml to be perfectly fine. I believe both lxml and bs4 have similar capabilities – you can even specify lxml to be the parser for bs4. I think bs4 might have a friendlier syntax, but again, I don’t know, as I’ve gotten by with lxml just fine:

import requests
from lxml import html
page = requests.get("http://www.example.com").text
doc = html.fromstring(page)
link = doc.cssselect("a")[0]
print(link.text_content())
# More information...
print(link.attrib['href'])
# http://www.iana.org/domains/example

The standard urllib package also has a lot of useful utilities – I frequently use the methods from urllib.parse. Python 2 also has urllib but the methods are arranged differently.

Here’s an example of using the urljoin method to resolve the relative links on the California state data for high school test scores. The use of os.path.basename is simply for saving the each spreadsheet to your local hard drive:

from os.path import basename
from urllib.parse import urljoin
from lxml import html
import requests
base_url = 'http://www.cde.ca.gov/ds/sp/ai/'
page = requests.get(base_url).text
doc = html.fromstring(page)
hrefs = [a.attrib['href'] for a in doc.cssselect('a')]
xls_hrefs = [href for href in hrefs if 'xls' in href]
for href in xls_hrefs:
  print(href) # e.g. documents/sat02.xls
  url = urljoin(base_url, href)
  with open("/tmp/" + basename(url), 'wb') as f:
    print("Downloading", url)
    # Downloading http://www.cde.ca.gov/ds/sp/ai/documents/sat02.xls
    data = requests.get(url).content
    f.write(data)

And that’s about all you need for the majority of web-scraping work – at least the part that involves reading HTML and downloading files.
Examples of sites to scrape

The 101 scraping exercises didn’t go so great, as I didn’t give enough specifics about what the exact answers should be (e.g. round the numbers? Use complete sentences?) or even where the data files actually were – as it so happens, not everyone Googles things the same way I do. And I should’ve made them do it on a weekly basis, rather than waiting till the end of the quarter to try to cram them in before finals week.

The Github repo lists each exercise with the solution code, the relevant URL, and the number of lines in the solution code.

The exercises run the gamut of simple parsing of static HTML, to inspecting AJAX-heavy sites in which knowledge of the network panel is required to discover the JSON files to grab. In many of these exercises, the HTML-parsing is the trivial part – just a few lines to parse the HTML to dynamically find the URL for the zip or Excel file to download (via requests)…and then 40 to 50 lines of unzipping/reading/filtering to get the answer. That part is beyond what typically considered “web-scraping” and falls more into “data wrangling”.

I didn’t sort the exercises on the list by difficulty, and many of the solutions are not particulary great code. Sometimes I wrote the solution as if I were teaching it to a beginner. But other times I solved the problem using the style in the most randomly bizarre way relative to how I would normally solve it – hey, writing 100+ scrapers gets boring.

But here are a few representative exercises with some explanation:
1. Number of datasets currently listed on data.gov

I think data.gov actually has an API, but this script relies on finding the easiest tag to grab from the front page and extracting the text, i.e. the 186,569 from the text string, "186,569 datasets found". This is obviously not a very robust script, as it will break when data.gov is redesigned. But it serves as a quick and easy HTML-parsing example.
29. Number of days until Texas’s next scheduled execution

Texas’s death penalty site is probably one of the best places to practice web scraping, as the HTML is pretty straightforward on the main landing pages (there are several, for scheduled and past executions, and current inmate roster), which have enough interesting tabular data to collect. But you can make it more complex by traversing the links to collect inmate data, mugshots, and final words. This script just finds the first person on the scheduled list and does some math to print the number of days until the execution (I probably made the datetime handling more convoluted than it needs to be in the provided solution)
3. The number of people who visited a U.S. government website using Internet Explorer 6.0 in the last 90 days

The analytics.usa.gov site is a great place to practice AJAX-data scraping. It’s a very simple and robust site, but either you are aware of AJAX and know how to use the network panel (and in this case, locate ie.json, or you will have no clue how to scrape even a single number on this webpage. I think the difference between static HTML and AJAX sites is one of the tougher things to teach novices. But they pretty much have to learn the difference given how many of today’s websites use both static and dynamically-rendered pages.
6. From 2010 to 2013, the change in median cost of health, dental, and vision coverage for California city employees

There’s actually no HTML parsing if you assume the URLs for the data files can be hard coded. So besides the nominal use of the requests library, this ends up being a data-wrangling exercise: download two specific zip files, unzip them, read the CSV files, filter the dictionaries, then do some math.
90. The currently serving U.S. congressmember with the most Twitter followers

Another example with no HTML parsing, but probably the most complicated example. You have to download and parse Sunlight Foundation’s CSV of Congressmember data to get all the Twitter usernames. Then authenticate with Twitter’s API, then perform mulitple batch lookups to get the data for all 500+ of the Congressional Twitter usernames. Then join the sorted result with the actual Congressmember identity. I probably shouldn’t have assigned this one.
HTML is not necessary

I included no-HTML exercises because there are plenty of data programming exercises that don’t have to deal with the specific nitty-gritty of the Web, such as understanding HTTP and/or HTML. It’s not just that a lot of public data has moved to JSON (e.g. the FEC API) – but that much of the best public data is found in bulk CSV and database files. These files can be programmatically fetched with simple usage of the requests library.

It’s not that parsing HTML isn’t a whole boatload of fun – and being able to do so is a useful skill if you want to build websites. But I believe novices have more than enough to learn from in sorting/filtering dictionaries and lists without worrying about learning how a website works.

Besides analytics.usa.gov, the data.usajobs.gov API, which lists federal job openings, is a great one to explore, because its data structure is simple and the site is robust. Here’s a Python exercise with the USAJobs API; and here’s one in Bash.

There’s also the Google Maps geocoding API, which can be hit up for a bit before you run into rate limits, and you get the bonus of teaching geocoding concepts. The NYTimes API requires creating an account, but you not only get good APIs for some political data, but for content data (i.e. articles, bestselling books) that is interesting fodder for journalism-related analysis.

But if you want to scrape HTML, then the Texas death penalty pages are the way to go, because of the simplicity of the HTML and the numerous ways you can traverse the pages and collect interesting data points. Besides the previously mentioned Texas Python scraping exercise, here’s one for Florida’s list of executions. And here’s a Bash exercise that scrapes data from Texas, Florida, and California and does a simple demographic analysis.

If you want more interesting public datasets – most of which require only a minimal of HTML-parsing to fetch – check out the list I talked about in last week’s info session on Stanford’s Computational Journalism Lab.

Source URL :  http://blog.danwin.com/examples-of-web-scraping-in-python-3-x-for-data-journalists/