Location via proxy:   [ UP ]  
[Report a bug]   [Manage cookies]                
Scribd
Upload
1K views

Introduction To Data Science

This document provides an outline for the CS109 Data Science course, including an overview of key concepts and principles covered in each section. The course aims to teach data munging, exploratory analysis, predictive modeling, and communicating results through three acts focused on predictions, recommendations, and network analysis. Students will apply these skills to real-world datasets in homework and a final project using tools like Python, IPython, and cloud computing platforms. Prerequisites include programming experience and basic statistics knowledge.

Uploaded by

Michael Wee
Copyright
© Attribution Non-Commercial (BY-NC)
Available Formats
Download as PDF, TXT or read online on Scribd
1K views

Introduction To Data Science

This document provides an outline for the CS109 Data Science course, including an overview of key concepts and principles covered in each section. The course aims to teach data munging, exploratory analysis, predictive modeling, and communicating results through three acts focused on predictions, recommendations, and network analysis. Students will apply these skills to real-world datasets in homework and a final project using tools like Python, IPython, and cloud computing platforms. Prerequisites include programming experience and basic statistics knowledge.

Uploaded by

Michael Wee
Copyright
© Attribution Non-Commercial (BY-NC)
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 74

STAT121 / AC209 / E-109

CS109 Data Science


Hanspeter Pster pster@seas.harvard.edu Joe Blitzstein blitzstein@stat.harvard.edu

Outline
What? Why? Who? How?

Outline
What? Why? Who? How?

Data Science
To gain insights into data through computation, statistics, and visualization

A Data Scientist Is...


A data scientist is someone who knows more statistics than a computer scientist and more computer science than a statistician. - Josh Blumenstock Data Scientist = statistician + programmer + coach + storyteller + artist - Shlomo Aragmon

Nate Silver

Nate Silver won the election Harvard Business Review

#natesilverfacts

http://techcrunch.com/2012/11/07/nate-silver-as-software/

Nate Silver on Pundits


Silver: Pundits are no better than a coin toss. Stewart: Do you foresee a coin getting its own show? The coin toss show?
http://www.thedailyshow.com/watch/wed-october-17-2012/nate-silver

Some Key Principles


use many data sources (the plural of anecdote is not data) understand how the data were collected (sampling is essential) weight the data thoughtfully (not all polls are equally good) use statistical models (not just hacking around in Excel) understand correlations (e.g., states that trend similarly) think like a Bayesian, check like a frequentist (reconciliation) have good communicationskills (What does a 60% probability even mean? How can we visualize, validate, and understand the conclusions?)

Human Genome

Microarrays

Afmetrix Chip

[wikipedia]

Sequencing

Sequencing Cost

Genome Data

Genome Visualization

[Krzywinski+2009]+

[Thorvaldsd,r-2013]-

[Meyer&2009]&

Personalized Therapy
...10 years from now, each cancer patient is going to want to get a genomic analysis of their cancer and will expect customized therapy based on that information. Director, The Cancer Genome Atlas (TCGA), Time Magazine, 6/13/11

Netix Prize

Some Challenges

massive data (500k users, 20k movies, 100m ratings) curse of dimensionality (very high-dimensional problem) missing data (99% of data missing; not missing at random) extremely complicated set of factors that affect peoples ratings of movies (actors, directors, genre, ...) need to avoid overtting (test data vs. training data)

Netix Prize Progress

http://blogs.hbr.org/cs/2012/10/big_data_hype_and_reality.html

Connectome
What is the connectivity of large brain circuits?

Ramn y Cajal, 1905

Connectome Workow

Ultra-Thin Section EM

Automatic Reconstruction

2D Segmentation

Combine Multiple 2D Segmentations with Fusion

Globally Consistent 3D Segmentation


[Kaynig et al., CVPR 10] [Vazquez et al., ICCV 2011]

2012

Data Science
Computer Science Statistics

Domain Science

Drew Conway

Machine Data Management Data Mining Machine Learning Visualization

Human Human Cognition Perception Story Telling Decision Making Theory

Business Intelligence Statistics

Data Science

Inspired by Daniel Keim, Visual Analytics: Denition, Process, and Challenges

Outline

What? Why? Who? How?

The Age of Big Data

BBC, 2013

Crime Prevention

Boston Globe, Sunday, Aug 4, 2013

Big Data
2.5 exabytes

daily data

years

2012
[IBMbigdata]

[Domo]

Between the dawn of civilization and 2003, we only created ve exabytes of information; now were creating that amount every two days. Eric Schmidt, Google (and others)

http://onesecond.designly.com/

Smarter Devices

Michael Franklin, UC Berkeley

Commodity Computing

Michael Franklin, UC Berkeley

Ubiquitous Connectivity

Michael Franklin, UC Berkeley

travers808,Visual.ly

1 Zetabyte = 1 Billion Terabytes

Jim Gray, Microsoft

By 2018, the US could face a shortage of up to 190,000 workers with analytical skills McKinsey Global Institute The sexy job in the next 10 years will be statisticians. Data Scientists? Hal Varian, Prof. Emeritus UC Berkeley Chief Economist, Google

Hal Varian Explains...


The ability to take data to be able to understand it, to process it, to extract value from it, to visualize it, to communicate it's going to be a hugely important skill in the next decades, not only at the professional level but even at the educational level for elementary school kids, for high school kids, for college kids. Because now we really do have essentially free and ubiquitous data. Hal Varian

Ask an interesting question.

What is the scientic goal? What would you do if you had all the data? What do you want to predict or estimate?

Get the data.

How were the data sampled? Which data are relevant? Are there privacy issues?

Explore the data.

Plot the data. Are there anomalies? Are there patterns?

Model the data.

Build a model. Fit the model. Validate the model.

Communicate and visualize the results.

What did we learn? Do the results make sense? Can we tell a story?

Outline

What? Why? Who? How?

Hanspeter Pster

An Wang

My Background

Grew up in Switzerland M.Sc. in EE from ETH Zurich Ph.D. in CS from SUNY Stony Brook 11 years in industry (MERL) At Harvard since 2007, Visual Computing Group (4 Ph.D., 7 PD) Teach CS109 / CS171, taught CS175 / CS264 / CS205 Director of the Institute of Applied Computational Science (IACS) Two daughters, Lilly (10) and Audrey (7)

Professor of the Practice in Statistics, Co-Director of Undergraduate Studies in Statistics blitz@fas.harvard.edu, twitter @stat110, SC 714

Joe Blitzstein

CS109 Staff

Chris Beaumont, Head TF Johanna Beyer Nicolas Bonneel Alex DAmour Rahul Dave Brandon Haynes

Ray Jones Steffen Kirchhoff Seymour Knowles-Barley Alexander Lex Deqing Sun Tim Brenner, A/V

About You

Outline
What? Why? Who? How?

CS109 Key Facets



data munging/scraping/sampling/cleaningin order to get an informative, manageable data set; data storage and management in order to be able to access data - especially big data - quickly and reliably during subsequent analysis; exploratory data analysisto generate hypotheses and intuition about the data; predictionbased on statistical tools such as regression, classication, and clustering; and communicationof results through visualization, stories, and interpretable summaries.

Act I: Predictions
Data Science Process Data Types and Data Munging Probability Review Classication & Regression Cross Validation, Clustering Visualization & Story Telling

Act II: Recommendations


Bayesian Thinking & Computation Monte Carlo Methods Machine Learning Methods MapReduce and Amazons EC2 Databases (Margo Seltzer)

Act III: Network Analysis


Network Visualization Network Sampling Community Detection Guest Lecture

Abstractions...

...and Tools

xkcd

Homework
Real-World focus Scrape and wrangle messy data Apply sophisticated statistical analysis Visualize and communicate results Election data, movie reviews,Yelp! data, etc.

Final Project
Pick a project of your choosing Teams of up to 2 students Process books, web sites, screencasts IPython (exceptions possible) Best project prizes!

cs109.org

Is this course for me ???

Prerequisites
Programming experience

C, C++, Java, Python, etc.


Basic statistical knowledge

STAT100, ideally STAT110


Willingness to learn new software & tools

This can be time consuming You will need to read online documentation

Be Patient Be Flexible Be Constructive

http://davidzinger.wordpress.com/2007/05/page/2/

Next Steps

HW 0

Good test of your basic skills Installation of several Python frameworks Not graded, do it as soon as possible

Read syllabus carefully Do readings


Post comments to Piazza using #readings

You might also like