I’ve been making too many systems posts and I’d like to better balance those things with more researchy things. Also, I meant to type “truth” instead of “spock” but I couldn’t resist and it’s close enough anyway.
I came across a blog post over at The Chronicle for Higher Education a week or so ago and I really didn’t think about it much. I don’t doubt that many people with a bachelor’s degree end up in jobs that either don’t need the degree or else much of the education isn’t related to their job.
A suggestion in Google Reader highlighted something I skipped over while reading:
there are 5,057 janitors in the U.S. with Ph.D.’s, other doctorates, or professional degrees
The post I read only included the PhD part, not the note about other doctorates or professional degrees. That got me thinking. Maybe the survey group isn’t applicable to me (i.e., none have PhDs in computer science). Maybe people reported their professions incorrectly (e.g., disgruntled PhDs). Maybe people reported their education incorrectly for laughs. Maybe you can get mail-order PhDs. Maybe “professional degrees” is a vague group for some people. Or maybe it’s completely accurate and applicable to my situation.
There are all sorts of complications in designing studies; I’ve done a couple studies and helped with several others. Obtaining objective information is genuinely complicated – has someone ever asked you “Do you seriously like X?” Aside from the content and distribution of how many people like X or not, the question pushes you to say no, which can lead the results to show exactly what they wanted. (Or the reverse is true – someone with an aversion may want to say “yes” just to be contrary.) Admittedly my example is uncommon in scientific studies, but more subtle problems can happen (e.g., not providing enough information to say yes, but providing enough information to say no, unclear categorizations).
The real question in my mind is: How much information would it take for me to believe something as true or false? I’m completely willing to believe that there are some PhDs in janitor positions. But what would it take for me to believe that 5000 PhD/doctorate/prof degrees are janitors? Probably I’d have to see the original publication and it would need to include detailed experimental design (or a cite/link to the methodology).
The problem also applies to other evaluations (e.g., a system that decreases perplexity by 50%) – you need to know the details of the evaluation to know whether it’s good (and a reasonable baseline). 50% reduction on unseen words could be trivial (just come up with a different estimate of N_0) or if it’s measured properly on all words then it’s amazing.
It wasn’t my point to criticize a blog post or anything; rather, the details of an evaluation (be it a survey or otherwise) are essential to interpreting data but are sometimes missing or difficult to find. At the same time, it’s important to avoid bias (e.g., do I require full information only for things that disagree with my prior beliefs?).
Really it’s food for thought. Earlier in life, there were things I accepted at face value because I didn’t take the effort to understand the details of the evaluation. And there continue to be situations where I may inadvertently accept figures without thinking.