Tag Archives: methodology

Sometimes Methodology Isn’t Everything

Posted by

1

Brad Delong points us to a study published in the British Medical Association jounal BMJ and quotes from it:

Smith and Pell: Parachute use to prevent death and major trauma related to gravitational challenge: systematic review of randomised controlled trials 327 (7429): 1459 — bmj.com:

No randomised controlled trials of parachute use have been undertaken

The basis for parachute use is purely observational, and its apparent efficacy could potentially be explained by a “healthy cohort” effect”

The full journal article is well worth following at the link he quotes.  Besides the laugh (warning: the positive health effects of laughing haven’t been proved by randomized controlled trials either), the authors suggest by parody an excellent point.  Sometimes rigid adherence to one single methodology in science is sometimes not only uncalled for and useless, it can also be immoral.  Some Austrian and New Classical economists might want to take note.

Employment News, A Muddle – Part 2

Posted by

6

In this post, I’m going to look at the methodology that produces our monthly employment reports because with today’s report, it’s particularly timely. As mentioned in the first post, the employment situation report for January 2011 is a real muddle, full of apparent contradiction. The obvious contradiction is 36,000 new jobs (an incredibly weak number) vs. a 0.4 point drop in unemployment rate from 9.4 to 9.0% (an encouraging number if we can believe it).  But once you read the full report, there’s even more confusion to be had. The report says that only 36,000 jobs were created but it also elsewhere says 117,000 more people were employed. Huh? Well it turns out there’s reasons. First, words matter – a job is not the same thing as an employed person (people can have two jobs). Second, while all the data is published each month as a single “employment situation report”, in fact the data being reported are estimates, not exact counts. This is not a monthly employment census, it’s a survey-based estimate/analysis. And the estimates depend crucially on five inputs:

  1. A sample survey of households (people) done by BLS.
  2. A sample survey of businesses that employ people, also done by BLS.
  3. Monthly estimates of the demographics and population of the nation provided by the Census Bureau.
  4. Econometric models that the BLS uses that forecasts the number and size of new businesses created each month and the number that failed. A model is used since reliable actual data isn’t available for a long time after the month it happens.
  5. The seasonal adjustment method.  In trying to seasonally adjust the data, this month’s SA data is in effect dependent upon the seasonal pattern exhibited in the past. If this year is different from the past, a temporary distortion enters.

This month we’ve got a near perfect storm of noise and confusion coming from at least 4 of these inputs.

First, the two surveys sharply disagreed. This could be a fluke occurence, an outlier of sampling, or it could indicate beginnings of a trend such as the number of jobs held person changing (normally stable), or it could be..who knows?  Garth Brazelton at Reviving Economics tries to explain through his exasperation:

…Typically always true from November – January imop.
But this one is particularly f***ed up.

News is reporting that according to BLS only 36,000 jobs were added in January (about 1/4 of what was expected).

Meanwhile, BLS reports the unemployment rate falls by another 0.4% – 2 months in a row – to 9.0%. …

In any case, it is important to note that BLS reports 2 surveys:
The first is the household survey (which surveys people, duh) – that is where we get the unemployment rate.
The other survey is for businesses and the government – that is where news organizations usually report jobs gained or lost.  (36,000 more employed in Jan. compared to Dec. 2010)

The unemployment rate simply cannot be compared to establishment based employment payroll changes because the two numbers come from 2 completely different surveys, measuring slightly different things:

Reported change in civilian employment from Dec. 2010 to Jan. 2011 (private): 117,000  (HOUSEHOLD SURVEY)

Reported change in total private non-farm employment from Dec. 2010 to Jan. 2011: 36,000  (ESTABLISHMENT SURVEY)

The unemployment rate comes from the survey of the former statistic, not the latter.  Media does everyone a dis-service by not pointing this out.

So, you know what you should do with all these news reports and the January statistics?  Throw them in the garbage, and wait for February and March.

Unlike Garth, I don’t think we need to throw the report in the garbage (see my post here), but we need to think critically about it.  Calculated Risk explains:

This data comes from two separate surveys. The unemployment rate comes from the Current Population Survey (CPS: commonly called the household survey), a monthly survey of about 60,000 households. The payroll jobs number comes from Current Employment Statistics (CES: establishment survey), a sample of approximately “140,000 businesses and government agencies representing approximately 410,000 worksites”.

These are very different surveys: the CPS gives the total number of employed (and unemployed including the alternative measures), and the CES gives the total number of positions (excluding some categories like the self-employed, and a person working two jobs counts as two positions).

A couple of key concepts (from the BLS):

The CES employment series are estimates of nonfarm wage and salary jobs, not an estimate of employed persons; an individual with two jobs is counted twice by the payroll survey. The CES employment series excludes employees in agriculture and private households and the self-employed.

And the CPS:

The CPS estimate of employment is for the total number of employed persons. Included are categories of workers that are not covered by the Current Employment Statistics (CES) survey: self-employed persons, private household workers, agriculture workers, unpaid family workers, and workers on leave without pay during the reference period. Multiple jobholders are counted once in the estimate of total employed.

Unemployed persons include those who did not have a job during the reference week, had actively looked for work in the prior 4 weeks, and were available for work. Actively looking for work includes activities such as contacting a possible employer, contacting an employment agency or employment center, having a job interview, sending out resumes, filling out job applications, placing or answering job advertisements, and checking union or professional registers.

In January the headline CES number showed a gain of 36,000 non-farm jobs (by the definitions above). The CPS showed an increase of 117,000 employed people.

These two surveys are almost always different, and both are useful for understanding the employment situation. 

Second, another problem confounding things this month is what’s called the “benchmark adjustment”.  Put very simply, the survey data is used to establish some proportions and ratios among those sampled. Then these ratios and proportions are scaled up to national numbers by multiplying by the latest population estimates from the Census bureau. But the Census bureau monthly estimates of population and demographics are themselves estimates that get revised yearly and of course, definitively every ten years. So once each year, the BLS has to adjust it’s data series on employment and jobs to keep it in line with the annual revision of population from Census. This year, these “benchmark adjustments” were very large (around 500,000).  Again I turn to Calculated Risk:

On Feb 4th, with the release of the January employment report, the BLS will make the following three changes / revisions:

1) Annual Benchmark revision to the Establishment Survey Data

With the release of January 2011 data on February 4, 2011, the Current Employment Statistics survey will introduce revisions to nonfarm payroll employment, hours, and earnings data to reflect the annual benchmark adjustments for March 2010 and updated seasonal adjustment factors. Not seasonally adjusted data beginning with April 2009 and seasonally adjusted data beginning with January 2006 are subject to revision.

Last October the BLS released the preliminary annual benchmark revision of minus 366,000 payroll jobs. Usually the preliminary estimate is pretty close to the final benchmark estimate. 

Finally, we have the issue of seasonality and winter. The data series on employment and labor force must be seasonally adjusted to be meaningful and informative since there’s a huge month-to-month swing in jobs that’s very predictable: summer hiring for kids, Christmas season hiring, July shut-downs in some industries, etc. The raw, non-seasonally adjusted data (NSA) just changes too wildly each month to be easily interpreted. So seasonal adjustment (SA) is necessary. But SA can distort numbers if this year’s seasonal change happens at a slightly different time than it normally has in the past.  This year, Christmas and New Years were on a weekend. More importantly, much of the nation (fortunately not Southeast Michigan, IMO) was wracked by snow and winter storms that were out-of-normal. So the severe winter and snow may have distorted the numbers. After all, snow doesn’t just keep kids home from school. It also keeps job hunters from being active. It keeps people out of stores and causes retailers to temporarily lay-off workers (i.e. tell’em not to come to work). It keeps businesses from hiring (temporarily) since you can’t hire somebody when both interviewer and interviewee are stuck in a snow drift. Old man winter could have had an impact on the numbers, but we have no certain way of knowing. This is particularly true since the surveys do not happen during the entire month. Instead the two surveys (CPS and CES) are done during a few particular days during mid-month. If that’s during a blizzard, well, you can guess the impact. Right now a lot of commentators (see  are willing to write-off the bad numbers in this report to winter snow. They assume the numbers will bounce back next month.  Maybe. Maybe not. We’ll see.  Meanwhile, let’s see what conclusions we can take away from report in the next post.

 

The History of Modern Macroeconomics

Posted by

1

A good piece from Brad Delong.  At the core, modern macroeconomic theory is relatively empty and vacuous when it comes to the major crises: last year’s melt-down, the Great Depression, the many bubbles, etc.  I have to agree with Brad’s conclusion that macro theory must re-connect with economic history ( I would suggest micro theory needs too, also).  Otherwise it becomes an imaginary trip through a fantasy world of math models (what Deidre McCloskey once called a “hyperspace of assumptions”).

Economic History and Modern Macro: What Happened?

Economic History and the Recession

If you ask a modern economic historian—like, say, me—if I know why the world is currently in the grips of a financial crisis and a deep downturn, I will say that I do know and I will give you this answer:

This is the latest episode in a long history of similar episodes of bubble—crash—crisis—recession, episodes that date back at least to the canal bubble of the early 1820s, the 1825-6 failure of Pole, Thornton, and company, and the subsequent first industrial recession in Britain. We have seen this process at work in many other historical episodes as well—1870, 1890, 1929, and 2000, for example. For some reason asset prices get way out of whack and rise to unsustainable levels. Sometimes the culprit is lousy internal controls in financial firms that overreward subordinates for taking risk; sometimes it is government guarantees; sometimes it is the selection of the market as a long run of good fortune leaves the financial market dominated by cockeyed unrealistic overoptimists.

Then the crash comes. And when the crash comes the risk tolerance of the market collapses: everybody knows that there are immense unrealized losses in financial assets and nobody is sure that they know where they are. The crash is followed by a flight to safety. The flight to safety is followed by a steep fall in the velocity of money as investors everywhere hoard cash. And the fall in monetary velocity brings on a recession.

I will not say that this is the pattern of all recessions: it isn’t. But I will say that this is the pattern of this recessions—that we have been here before.

Macroeconomic Theory and the Recession

If you ask the same question of a modern macroeconomist—like, say, the extremely sharp Narayana Kocherlakota of the University of Minnesota—you will find that he says that he does not know: Continue reading →

We need a better measure than GDP

Posted by

0

Nobel winner Joseph Stiglitz explains why GDP is NOT a good measure of society’s well-being and offers ideas on better measures.  It’s a good critique of GDP.

National income statistics such as GDP and gross national product were originally intended as a measure of market economic activity, including the public sector. But they have increasingly been thought of as measures of societal well-being, which they are not. Of course, good statisticians have warned against this error. Much economic activity occurs within the home – and this can contribute to individual well-being as much as, or more than, market production.

….

What we measure affects what we do. If we have the wrong metrics, we will strive for the wrong things. In the quest to increase GDP, we may end up with a society in which most citizens have become worse off. We care, moreover, not just for how well off we are today but how well off we will be in the future. If we are borrowing unsustainably from this future, we should want to know.

I strongly recommend reading the entire article here:   FT.com / Comment / Opinion – Towards a better measure of well-being.

Krugman on How Did Economists Get It So Wrong? – Excellent

Posted by

1

Long, but excellent reading on the recent (last few decades) of the history of macro thinking.  I think Krugman understates some issues, but much of it is good.  I recommend.

It’s hard to believe now, but not long ago economists were congratulating themselves over the success of their field. Those successes — or so they believed — were both theoretical and practical, leading to a golden era for the profession. On the theoretical side, they thought that they had resolved their internal disputes. Thus, in a 2008 paper titled “The State of Macro” (that is, macroeconomics, the study of big-picture issues like recessions), Olivier Blanchard of M.I.T., now the chief economist at the International Monetary Fund, declared that “the state of macro is good.” The battles of yesteryear, he said, were over, and there had been a “broad convergence of vision.” And in the real world, economists believed they had things under control: the “central problem of depression-prevention has been solved,” declared Robert Lucas of the University of Chicago in his 2003 presidential address to the American Economic Association. In 2004, Ben Bernanke, a former Princeton professor who is now the chairman of the Federal Reserve Board, celebrated the Great Moderation in economic performance over the previous two decades, which he attributed in part to improved economic policy making.

Last year, everything came apart.

For the complete article, see:  http://www.nytimes.com/2009/09/06/magazine/06Economic-t.html?_r=1&pagewanted=all

Krugman’s article caused quite a stir in academic and professional economics.  Many have agreed with him, such as Jeff Frankel who posted this:

The Queen of England during the summer asked economists why no one had predicted the credit crunch and recession.   Paul Krugman points out that, inasmuch as economists can almost never predict the timing of recessions (and don’t claim to be able to), the real questions are worse.  The real questions are, rather how macroeconomists (most of us) could have gotten it so wrong as to believe that: (i) a severe recession like this was not even looming ahead as a danger, and (ii) a breakdown of many of the world’s most liquid financial markets, in New York and London, was not possible.To anyone wondering about these questions, I recommend Krugman’s essay in the New York Times Sunday magazine, September 6:  “How Did Economists Get it So Wrong?” . I think he has it exactly right.

I would only add that he is modest in skipping over one point:  during Japan’s lost decade of growth in the 1990s Paul forcefully drew from the Japanese experience the implication that a severe economic breakdown was, after all, possible in a modern industrialized economy – a breakdown that both was reminiscent of the Great Depression and was outside the ken of modern macroeconomic theory.   But macroeconomics went on as before.   (Likewise with the stock market correction of 1987, the LTCM crisis of 1998, and the dotcom bust of 2000-01.   I do think, however, that our field did a better job with the emerging market crises of 1994-2001, in part because it was considered permissible to argue that financial markets in this case were highly imperfect.)

Even the cartoons in the NYT article are good…  except that I have never seen Olivier Blanchard in a double-breasted suit.    But Robert Lucas definitely merits a place there:   when given one page to defend orthodox economists regarding the crisis in a recent  Economist essay, he actually thought it was a useful rebuttal to point out that critics are repeating arguments they have made before.  And he also thought it was useful to explain: “The term “efficient” as used here means that individuals use information in their own private interest. It has nothing to do with socially desirable pricing; people often confuse the two.”  — as if it is not the latter question that the public is wondering about.

I am pleased that at least some in the profession are waking to the serious methodological and intellectual problems that have crept into economics (not just macro) in recent decades.  There’s a long way to go though before we return to math only being a tool to check/explore the logic instead of the centerpiece.  We need to put real-world economies back in focus as the topic of our theorizing and research.

Macro: an awful mess today

Posted by

0

I was not alone.  Apparently Tim Harford of FT.com was also confused and disappointed while learning the modern macro models and theories.  I however figured it was a bunch of nonsense.  The assumptions made by modern macro models, particularly the Rational Expectations stuff of New Classical and New Keynesian theory are the problem.  By making assumptions that enable them to use their elegant math, they assume away any possibility of the model being useful.  The assumptions are not just “simplifying”, they are contra-reality.  It is as if I were to create a theory of flying in physics by first assuming that gravity cannot exist.  There’s a lot of work to be done in macro now to repair this failed research program.

I am struck by the soul-searching that has gripped the profession in the face of the economic crisis. The worry is not so much that macroeconomists did not forecast the problem – bad forecasts are more a sign of a complex world than intellectual bankruptcy – but that macroeconomics seems unable to provide answers. Sometimes it cannot even ask the right questions.

Willem Buiter, a former member of the UK’s Monetary Policy Committee who blogs for the FT, complains that macroeconomists have simply discarded the difficult stuff to make their models more elegant: “They took these non-linear stochastic dynamic general equilibrium models into the basement and beat them with a rubber hose until they behaved.”

Mark Thoma offers additional insight at Economist’s View.

‘Goodbye, homo economicus’: Apparently economics is thin-skinned

Posted by

0

Anatole Kaletsky indicts the modern economist profession for the current crisis  in the Prospect:

Was Adam Smith an economist? Was Keynes, Ricardo or Schumpeter? By the standards of today’s academic economists, the answer is no. Smith, Ricardo and Keynes produced no mathematical models. Their work lacked the “analytical rigour” and precise deductive logic demanded by modern economics…. If any of these giants of economics applied for a university job today, they would be rejected. As for their written work, it would not have a chance of acceptance in the Economic Journal or American Economic Review….

….The truth is even worse than this rhetorical question suggests: not only have economists, as a profession, failed to guide the world out of the crisis, they were also primarily responsible for leading us into it.

via Features: ‘Goodbye, homo economicus’ by Anatole Kaletsky | Prospect Magazine April 2009 issue 157.

I have long shared held the views expressed by Kaletsky.  Perhaps that’s why I’m not in some research school and writing in the major Econ journals.  Alas, it appears that economists (the current reigning crop of “leading mainstream academic economists” are rather thin-skinned.  Kaletsky’s critique was generally not well received by the PTB in the profession.  Witness Mark Thoma and more Thoma and yet more Thoma and  Krugman. Methinks they doth protest too much.  They take great offense at the Kaletsky’s supposed attack on “math”, when in fact Kaletsky is attacking the methodology and thinking that limits itself only to math, that will not question assumptions, and then raises the math outcome to the status of “truth”.  Economics left the study of the real economy and real people behind in the era 1950-1980.  It fell into the grip of MMMM ( modern math model mania).

Some of my own thoughts and reactions to Thoma’s reaction to Kaletsky below the fold:

Continue reading →

Systemic failure of economics methodology

Posted by

0

What I’ve been thinking for some time. Economics, particularly the mainstream analyses, has lost it’s way. The failures of the current crisis point out failures of economic advice and policy making. Those policies were based on models & theories that have a flawed methodology. The “positivist” methodology of economics and it’s accompanying physics envy dating from the mid-2oth century has led us astray.

Assumptions do matter and the realism of those assumptions matters, too. Highly recommend reading:
“The Financial crisis and systemic failure of economics” – Mark Thoma.

The Financial Crisis and the Systemic Failure of Academic Economics, by David Colander, Hans Föllmer, Armin Haas, Michael Goldberg, Katarina Juselius, Alan Kirman, and Thomas Lux: [From the conclusion] …”We believe that economics has been trapped in a sub-optimal equilibrium in which much of its research efforts are not directed towards the most prevalent needs of society. Paradoxically self-reinforcing feedback effects within the profession may have led to the dominance of a paradigm that has no solid methodological basis and whose empirical performance is, to say the least, modest. Defining away the most prevalent economic problems of modern economies and failing to communicate the limitations and assumptions of its popular models, the economics profession bears some responsibility for the current crisis. It has failed in its duty to society to provide as much insight as possible into the workings of the economy and in providing warnings about the tools it created. It has also been reluctant to emphasize the limitations of its analysis. We believe that the failure to even envisage the current problems of the worldwide financial system and the inability of standard macro and finance models to provide any insight into ongoing events make a strong case for a major reorientation in these areas and a reconsideration of their basic premises.”

The Austrian and Chicago Schools

Posted by

0

via Economist’s View: The Austrian and Chicago Schools.

This is from History of Economic Thought: A Critical Perspective, by E.K. Hunt, a long out of print textbook I had when I was an undergraduate at CSU Chico [update: it is has been published again by M E Sharpe]. It explains how the “Austrian and Chicago schools reduce all human behavior to rational maximizing exchanges and hence are able to prove that in every respect, economic and non-economic, a free market, capitalist system is the best of all possible worlds,” and gives some of the critical reactions to that point of view:

The Austrian and Chicago Schools

The school of neoclassical economists that advocates extreme laissez-faire Capitalism represents the contemporary counterparts of Senior and Bastiat. In a sense this group really represents two separate but similar schools – the Austrian School and the Chicago School. The Austrian School traces its lineage directly back to Carl Menger (Chapter Eleven), Menger’s extreme methodological individualism is the basis of the social philosophy of the Austrian School.

While Menger’s first generation of disciples included both social reformers and conservatives, the ultraconservative nature of the Austrian School is more properly thought of as the product of two of Menger’s second-generation disciples, Ludwig von Mises and Friedrich A. Hayek. Both von Mises and Hayek taught at various times at the University of Chicago. Together with Frank H. Knight, who taught for many years at the University of Chicago, they were the most important influences in the formation of the Chicago School. For the past generation, Milton Friedman has been the most influential member of the Chicago School. In 1976, Friedman was awarded the Nobel Prize in economics. Continue reading →