# Social Science Statistics Blog

January 2009
Sun | Mon | Tue | Wed | Thu | Fri | Sat
--- | --- | --- | --- | --- | --- | ---
|  |  |  | 1 | 2 | 3
4 | 5 | 6 | 7 | 8 | 9 | 10
11 | 12 | 13 | 14 | 15 | 16 | 17
18 | 19 | 20 | 21 | 22 | 23 | 24
25 | 26 | 27 | 28 | 29 | 30 | 31

### Authors' Committee

#### Chair:

Matt Blackwell (Gov)

#### Members:

Martin Andersen (HealthPol)
Kevin Bartz (Stats)
Deirdre Bloome (Social Policy)
John Graves (HealthPol)
Rich Nielsen (Gov)
Maya Sen (Gov)
Gary King (Gov)

### Weekly Research Workshop Sponsors

Alberto Abadie, Lee Fleming, Adam Glynn, Guido Imbens, Gary King, Arthur Spirling, Jamie Robins, Don Rubin, Chris Winship

Weekly Workshop Schedule

Search this site:

### Recent Comments

- Re: App Stats: Glynn on "Using Post-Treatment Variables to Establish Upper Bounds on Causal Effects: Assessing Executive Selection Procedures in New Democracies"
  By: Office Furniture Warehouse
- Re: App Stats: Bahar on "International Knowledge Diffusion and the Comparative Advantage of Nations"
  By: Christa Herzog
- Re: App Stats: Bahar on "International Knowledge Diffusion and the Comparative Advantage of Nations"
  By: Frigler.de
- Re: App Stats: Bahar on "International Knowledge Diffusion and the Comparative Advantage of Nations"
  By: sri
- Re: App Stats: Bahar on "International Knowledge Diffusion and the Comparative Advantage of Nations"
  By: sri
- Re: App Stats: Bahar on "International Knowledge Diffusion and the Comparative Advantage of Nations"
  By: W7midia
- Re: App Stats: Titiunik on "Using Regression Discontinuity to Uncover the Personal Incumbency Advantage"
  By: SEO Services Canada
- Re: App Stats: Yamamoto on "A Multinomial Response Model for Varying Choice Sets, with Application to Partially Contested Multiparty Elections"
  By: Robert Rowshan
- Re: App Stats: Yamamoto on "A Multinomial Response Model for Varying Choice Sets, with Application to Partially Contested Multiparty Elections"
  By: KLLD
- Re: App Stats: Pfister on "Visual Computing in Biology"
  By: Biology education

### Recent Entries

- Workshop on Random Network Models
- Studying the 2008 primaries with prediction markets: Malhotra and Snowberg
- Amazon Mechanical Turk for Data Entry Tasks
- Multiple comparisons and the "Axe" effect
- NYT pays tribute to R

### Categories

- Regular Post

### Blogroll

SMR Blog
Brad DeLong
Cognitive Daily
Complexity & Social Networks
Developing Intelligence
EconLog
The Education Wonks
Empirical Legal Studies
Free Exchange
Freakonomics
Health Care Economist
Junk Charts
Language Log
Law & Econ Prof Blog
Machine Learning (Theory)
Marginal Revolution
Mixing Memory
Mystery Pollster
New Economist
Political Arithmetik
Political Science Methods
Pure Pedantry
Science & Law Blog
Simon Jackman
Social Science++
Statistical modeling, causal inference, and social science

### Archives

- May 2012
- April 2012
- March 2012
- February 2012
- January 2012
- November 2011
- October 2011
- September 2011
- July 2011
- June 2011
- May 2011
- April 2011
- March 2011
- February 2011
- January 2011
- December 2010
- November 2010
- October 2010
- September 2010
- August 2010
- May 2010
- April 2010
- March 2010
- February 2010
- January 2010
- December 2009
- November 2009
- October 2009
- September 2009
- August 2009
- June 2009
- May 2009
- April 2009
- March 2009
- February 2009
- January 2009
- December 2008
- November 2008
- October 2008
- September 2008
- August 2008
- June 2008
- May 2008
- April 2008
- March 2008
- February 2008
- January 2008
- December 2007
- November 2007
- October 2007
- September 2007
- June 2007
- May 2007
- April 2007
- March 2007
- February 2007
- January 2007
- December 2006
- November 2006
- October 2006
- September 2006
- August 2006
- July 2006
- June 2006
- May 2006
- April 2006
- March 2006
- February 2006
- January 2006
- December 2005
- November 2005
- October 2005
- September 2005

### Notification

Enter e-mail to [un]subscribe:
Powered by
Movable Type 4.24-en
Syndicate this site (XML)
[an error occurred while processing this directive]

« December 2008 | Main | February 2009 »

30 January 2009

### Workshop on Random Network Models

Professor Joseph Blitzstein will open a course this spring, "Statistics 340. Random Network Models". Those who are interested in the booming network industry should definitely come have a look. The course will be reading and discussion based and involves no exams. It will meet regularly from 1 to 2:30 on Fridays at science center 706. I took a probability course from Joe and found he was a hilarious, encouraging and patient teacher. Some kids from 110 posted some clips of his teaching on the youtube. You will like the musical interludes and the Markov ball game.

http://www.youtube.com/watch?v=iAwS7vzvLnY

http://www.youtube.com/watch?v=TQvVLhWOiis

If you want to watch Joe's performance live, better come today.

Posted by Weihua An at 10:48 AM

22 January 2009

### Studying the 2008 primaries with prediction markets: Malhotra and Snowberg

With Obama now in office the rest of the country may be about ready to move on from the 2008 election, but political scientists are of course still finding plenty to write about. Neil Malhotra and Erik Snowberg recently circulated a working paper in which they use data from political prediction markets in 2008 to examine two key questions about presidential primaries: whether primaries constrain politicians from appealing to the middle of the electorate and whether states with early primaries play a disproportionately large role in choosing the nominee. It's a very short and preliminary working paper that applies some novel methods to interesting data. Ultimately the paper can't say all that much about these big questions, not just because 2008 was an unusual year but also because of the limitations of prediction market data and the usual problems of confounding. But there is some interesting stuff in the paper and I expect it will improve in revision -- I hope these comments can help.

The most clever insight in the paper is that you can combine data from different prediction markets to estimate an interesting conditional probability -- the probability that a primary candidate will win the general election conditional on winning the nomination. (If p(G) is the probability of winning the general election and p(N) is the probability of winning the nomination (both of which are evident in prediction market contract prices), p(G|N) -- the probability of winning the general election if nominated -- can be calculated as p(G)/p(N).) In the first part of the paper, the authors focus on how individual primaries in the 2008 election affected this conditional probability for each candidate. This is interesting because classic theories in political science posit that primary elections force candidates to take positions that satisfy their partisans but hurt their general election prospects by making it harder for them to appeal to the electoral middle. If that is the case, then ceteris paribus one would expect that the conditional election probabilities would have gone down for Obama and Clinton each time it looked like the primary season would become more drawn out -- which is what happened as results of several of the primaries rolled in.

As it turns out, p(G|N) didn't move much in most primaries; if anything, it went up when the primary season seemed likely to extend longer (e.g. for Obama in New Hampshire). Perhaps this was because of the much talked about positive countervailing factors -- i.e. the extended primary season actually sharpened each candidate's electoral machines and increased their free media exposure. Of course, Malhotra and Snowberg have no way of knowing whether the binding effect of primaries exists and was almost perfectly counterbalanced by these positive factors, or whether none of these factors really mattered very much.

There is yet another possibility, which is that conditional probabilities did not move much for most primaries because most primaries did not change the market's view of how long the primary season would be. Knowing how the conditional probability changed during a particular primary only tells us something about whether having more primaries helps or hurts candidates' general election prospects if that primary changed people's expectations about how long the primary season would be. There were certainly primaries where this was the case (New Hampshire and Ohio/Texas come to mind) but for most of the primaries there was very little new information about how many more primaries would follow. Malhotra and Snowberg proceed as if they were looking for an average effect of a primary taking place on a candidate's conditional general election prospects, but if they want to talk about how having more primaries affects candidates' electability in the general election, they need to focus more squarely on cases where expectations about the length of the primary season actually changed (and, ideally, not much else changed). I would say the March Ohio/Texas primary was the best case of that, and at that time Barack Obama's p(G|N) dropped by 3 points -- a good indication that the market assumed that the net effect of a longer season on general election prospects was negative. (Although of course that primary also presumably revealed new information about whether Obama would be able to carry Ohio in the general election -- it's hard to disentangle these things.)

The second part of the paper explicitly considers the problem of assessing how "surprised" the prediction markets were in particular primaries (without explaining why this was not an issue in the first part), and employs a pretty ad hoc means of upweighting effect estimates for the relatively unsurprising contests. Some kind of correction makes sense but it seemed to me that the correction was so important in producing their results that it should be explained more fully in further revisions of the paper.

So to sum up, I liked the use of prediction markets to estimate the conditional general election probability for a candidate at a point in time, and I think it's worth getting some estimates of how particular events moved this probability. I think at this stage the conclusions are a bit underdeveloped and oversold, considering how many factors are at play and how unclear it is what information each primary introduced. But I look forward to future revisions.

Posted by Andy Eggers at 10:18 AM

16 January 2009

### Amazon Mechanical Turk for Data Entry Tasks

Yesterday I tried using Amazon's Mechanical Turk service for the first time to save myself from some data collection drudgery. I found it fascinating. For the right kind of task, and with a little bit of setup effort, it can drastically reduce the cost and hassle of getting good data compared to other methods (such as using RAs).

Quick background on Mechanical Turk (MTurk): The service acts as a marketplace for jobs that can be done quickly over a web interface. "Requesters" (like me) submit tasks and specify how much they will pay for an acceptable response; "Workers" (known commonly as "Turkers") browse submitted tasks and choose ones to complete. A Requester could ask for all sorts of things (e.g. write me a publishable paper), but because you can't do much to filter the Turkers and they aren't paid for unacceptable work, the system works best for tasks that can be done quickly and in a fairly objective way. The canonical tasks described in the documentation are discrete, bite-sized tasks that could almost be done by a computer -- indicating whether a person appears in a photo, for example. Amazon bills the service as "Artificial Artificial Intelligence," because to the Requester it seems as if a very smart computer were solving the problem for you (while in fact it's really a person). This is also the idea behind the name of the service, a reference to an 18th century chess-playing automaton that actually had a person inside (known as The Turk).

The task I had was to find the full text of a bunch of proposals from meeting agendas that were posted online. I had the urls of the agendas and a brief description of each proposal, and I faced the task of looking up each one. I could almost automate the task (and was sorely tempted), but it would require coding time and manual error checking. I decided to try MTurk.

The ideal data collection task on MTurk is the common situation where you have a spreadsheet with a bunch of columns and you need someone to go through and do something pretty rote to fill out another column. That was my situation: for every proposal I have a column with the url and a summary of what was proposed, and I wanted someone to fill in the "full text" column. To do a task like this, you need to design a template that applies to each row in the spreadsheet, indicating how the data from the existing columns should appear and where the Turker should enter the data for the missing column. Then you upload the spreadsheet and a separate task is created for each row in the spreadsheet. If everything looks good you post the tasks and watch the data roll in.

To provide a little more detail: Once you sign up to be a Requester at the MTurk website, you start the process of designing your "HIT" (Human Intelligence Task). MTurk provides a number of templates to get you started. The easiest approach is to pick the "Blank Template," which is very poorly named, because the "Blank Template" is in fact full of various elements you might need in your HIT; just cut out the stuff you don't need and edit the rest. (Here it helps to know some html, but for most tasks you can probably get by without knowing much.) The key thing is that when you place a variable in the template (e.g. ${party_id}), it will be filled by an entry from your spreadsheet, based on the spreadsheet's column names. So a very simple HIT would be a template that says

> Is this sentence offensive? ${sentence}

followed by buttons for "yes" and "no" (which you can get right from the "Blank Template"). If you then upload a CSV with a column entitled "sentence" and 100 rows, you will generate 100 HITs, one for each sentence.

It was pretty quick for me to set up my HIT template, upload a CSV, and post my HITs.

Then the real fun begins. Within two minutes the first responses started coming in; I think the whole job (26 searches -- just a pilot) was done in about 20 minutes. (And prices are low on MTurk -- it cost me $3.80.) I had each task done by two different Turkers as a check for quality, and there was perfect agreement.

One big question people have is, "Who are these people who do rote work for so little?" You might think it was all people in developing countries, but it turns out that a large majority are bored Americans. There's some pretty interesting information out there about Turkers, largely from Panos Ipeirotis's blog (a good source on all things MTurk in fact). Most relvenat for understanding Turkers is survey of Turkers he conducted via (of course) MTurk. For $.10, Turkers were asked to write why they complete tasks on MTurk. The responses are here. My takeaway was that people do MTurk HITs to make a little money when they're bored, as an alternative to watching TV or playing games. One man's drudgery is another man's entertainment -- beautiful.

Posted by Andy Eggers at 9:49 AM

13 January 2009

### Multiple comparisons and the "Axe" effect

Like many of us, I'm always on the lookout for good examples to use in undergraduate methods courses. My high school chemistry teacher (a former nun) said that the best teaching examples involved sex, food, or money, and that seems like reasonable advice for statistics as well. In that vein, I noted a recent article on the "Axe effect" in Metro:

> 'Axe effect' really works, a new study swears

> Researchers in the U.K. asked women to rate the attractiveness of men wearing Axe's British counterpart, Lynx, against those who were wearing an odorless placebo.

> On a 7-point scale, men wearing Lynx scored a 4.2, 0.4 point higher than those wearing the placebo.

> But here's the catch: The women did not meet the men face-to-face. They watched them on video.

> So what explains the discrepancy in ratings? Men wearing Lynx reported feeling more confident about themselves. So the difference in attitude appears more responsible for getting you lucky than the scent itself.

This story was not just reported in a subway tabloid; a long article appeared in the Economist. (Although at least the Metro story reported an effect size, unlike the Economist).

Is there an Axe effect? The news stories are reporting on a study in the International Journal of Cosmetic Science, "Manipulation of body odour alters men's self-confidence and judgements of their visual attractiveness by women". The researchers recruited male students and staff members from the University of Liverpool, randomly assigned some of them to use deodorant or a placebo. They then took photographs of the men as well as videos of them pretending to chat up an attractive woman. The photos and videos of the men were evaluated by "a panel of eight independent female raters" for attractiveness and self-confidence.

Medium | Attractiveness | Confidence
--- | --- | ---
Photo | Not significant | (not asked)
Video, no sound | Significant! | Not significant
Video w/ sound | Not significant | Not significant

There may be an Axe effect on women's perception of men's attractiveness (but not self-confidence) if they see them on video if they can't hear them. Or it might be a fluke. This seems like a classic multiple comparison problem. With five tests, it is not that unlikely that one of them would be (barely) statistically significant. The proposed mechanism for the one "effect" (which attracted all of the media attention) was increased self-confidence on the part of the male subjects, so it seems a little odd that an effect would be found on perceived attractiveness and not on self-confidence. We might be more confident that something is going on if the effect sizes were reported for the non-significant results, but they don't appear in the paper. So, the Axe effect may be for real, but only if you keep your mouth shut.

Posted by Mike Kellermann at 8:19 PM

6 January 2009

### NYT pays tribute to R

Today's New York Times has an article about the increasing popularity of R and what it means for commercial packages. See here for ``Data Analysts Captivated by Power of R''.

Posted by Sebastian Bauhoff at 11:09 PM