Newspaper

A python package for article scraping and curation

The newspaper package for python makes it extremely easy to download article content. Not only that, it has built in functionality to perform some natural language processing on the content downloaded.

As a quick intro to this package lets see what we can do with this article from Fox Sports as an example:

Once the newspaper package is installed to your machine we can follow along with the introductory documentation for the package.

The following code snippet will extract some basic data about the Fox Sports article:

from newspaper import Article

url = 'https://www.foxsports.com.au/nrl/nrl-premiership/teams/sharks/james-maloneys-shock-confession-over-relationship-with-shane-flanagan/news-story/7134a366ebb93358cb23a2291ff78409'
article = Article(url)

article.download()
article.parse()

# see who the authors are
article.authors

# when was the publish date
article.publish_date

# the article title
article.title

# the actual text of the article
article.text

# the url of the main image of the article
article.top_img

The authors it was able to extract:

Simon Brunsdon, Staff Writers, Source, Fox Sports

The publish date:

datetime.datetime(2018, 3, 14, 9, 16, tzinfo=tzutc())

Article Title:

James Maloney Shane Flanagan, Penrith Panthers signing, Cronulla Sharks NRL

Text of the article:

“JAMES Maloney has made the shock admission he hasn’t spoken to Cronulla coach Shane Flanagan since being traded to Penrith during the off-season.\n\nMaloney was caught up in the biggest movement of a crazy player market circus which dominated headlines during October and November.\n\nIt involved a big-money swap deal that saw him punted to the Panthers and Matt Moylan land in the Shire.\n\nLIVE stream every 2018 NRL Telstra Premiership game on FOX SPORTS. Get your free 2-week trial now >. If you’re overseas, you can still stream it LIVE on Watch NRL …

The URL of the main image in the article:

https://cdn.newsapi.com.au/image/v1/d38eb7c8b9e88aa0b501cbecf760dddc

This is very useful information, and can form the basis of a content collection project.

Newspaper also has some Natural Language Processing capabilities and can be called as follows:

import nltk
#nltk.download('punkt') #this may be necessary if it is your first time running this

article.nlp()

# Extract keywords
article.keywords

# Summary of the article
article.summary

Article keywords extracted:

shane, signing, nrl, watch, swap, penrith, flanagan, world, james…

Summary of the article:

‘JAMES Maloney has made the shock admission he hasn’t spoken to Cronulla coach Shane Flanagan since being traded to Penrith during the off-season.\nMaloney was caught up in the biggest movement of a crazy player market circus which dominated headlines during October and November.\nIt involved a big-money swap deal that saw him punted to the Panthers and Matt Moylan land in the Shire.\n“It was a bit out of the blue and it just happened, so that’s how it went.”Sharks coach Shane Flanagan and James Maloney celebrate the 2016 premiership.\nI’m comfortable with the decision and I’m sure Cronulla is happy with the decision.”Maloney admits the sudden swap deal surprised him.’

So with less than 20 lines of code it is possible to download and extract key pieces of information about an article as well extract some meaning.