Top applications across the top players

Amazon

• Frequently Bought Together
• Customers Who Bought This Item Also Bought

Netflix

• Movie Recommendation

Facebook

• People You Might Know (aka friend suggestions)
• Face detection

Bing

• Entity extraction from web page and queries, like names, addresses. It was running inside IE toolbar, Bing index generation and query processing.

Google

• Click Fraud Detection

Yelp

• Dataset Challenges

NLP API Services

• Alchemy API
• Google NLP API
• Thomson Reuters Open Calais
• Keyword IO – good keyword tool

Deep Learning Datasets

• DeepLearning.net Dataset

Interesting Public Datasets

There are quite a bit of ML competitions in Kaggle. And each of these competitions, a good amount of dataset are released in public. Here are the list of datasets that I found interesting.

Entertainment Datasets

A set of celebrity, images and movie data below. It is about 1000 to 2000 celebrities. You can cross check People.com for its completeness.
* Celebrity Face on Web from Microsoft
* Celebrity Twitter Accounts – over 1000+ celebrity twitter accounts there.
* Cross-Age Celebrity Dataset (CACD)

• MovieLens Latest Dataset
• OMDB Movie Dataset
• Human curated Movie list from IMDB
• Million of Songs (500GB)

Proper Name

• CMU female and male first names
• People names in group

eCommerce

• Otto Group product classification dataset – For this competition, we have provided a dataset with 93 features for more than 200,000 products. The objective is to build a predictive model which is able to distinguish between our main product categories. The winning models will be open sourced. Competition held at Kaggle.
• eCommerce Search Relevance – This set contains image URLs, rank on page, description for each product, search query that lead to each result, and more, each from five major English-language ecommerce sites.
• Sentiment Training Data Towards Products or Brands
• Top 500 brands
• Apparel Top Brands
• 2500+ Popular Brands and 550+ Popular Categories
• Google Product Taxonomy

Internet Marketing

• Wikipedia Traffic Statistics
• Common Crawl Corpus (541TB) – Have you ever wanted to get your hands on crawl data for billions of web pages with trillions of links? Here’s your chance. The Common Crawl Corpus provides a rich set of tools, examples, and projects you can jump into today.
• Internet Advertisements dataset – Given the details of images on web pages predict whether an image is an advertisement or not.
• Human activity recognition using smart phones dataset – From smart phone movement data predict the type of activity performed by the person holding the smart phone.
• Facebook Public Page Post dataset – Data scraper for Facebook Pages, and also code accompanying the blog post How to Scrape Data From Facebook Page Posts for Statistical Analysis.
• All Subreddits from ScrapingHub – This dataset contains a list with all the subreddits from reddit.com.

Finance

• Zillow Home Value Dataset
• Springleaf marketing dataset – Given features of customers predict whether they are a marketing target or not. And Springleaf is a lending company. Competition held in Kaggle.
• Lending Club Loan Data

Other Interesting DataSets

• OpenStreetMap database
• Project Gutenberg (742GB) – Project Gutenberg makes over 46,000 books available for analysis. These books are now on the public domain because their copyrights have expired.
• NOAA National Climatic Data Center (3.3 TB) – This dataset contains data on over 150 years of weather from many sources ranging from weather stations to airport readings to satellite data.
• Amazon.com Employee Access Challenge – Given historical resource access changes for employees predict the resources required by employees.
• WordSimilarity-353 Test Collection contains two sets of English word pairs along with human-assigned similarity judgements. The collection can be used to train and/or test computer algorithms implementing semantic similarity measures (i.e., algorithms that numerically estimate similarity of natural language words).
• Google Book Ngram

Reference

• UCI datasets
• DMOZ dataset directory
• Amazon datasets
• Awesome datasets
• MRAN datasets
• Crowdflower Public Datasets
• Reddit datasets