What is the Statistics Department 25 Years From Now?
A long time ago, Peter Bickel said to me -- speak at the graduation. Now Peter is smart. He knows if you ask people to speak so far in advance that the actuality seems remote, they may say yes. And I did. But as time got short, the actuality impinged. Friends said "tell some long jokes". Besides the fact that I'm dismal at telling jokes, it just didn't seem right. Instead I decided to talk about something that we share together -- the future of this odd and funny field that we're all a part of.
Imagine that you have been away for 25 years and out of touch with what has been happening here. Perhaps you have been working in the middle of the Sahara for an oil exploration company, or taking surveys in Patagonia, or counting rare bird species in the middle of a vast game preserve.
But now you have decided to take a vacation and visit Berkeley and look up old professors and friends in the Statistics Department. Of course, you anticipate some changes. Your old professors will be older. The young ones will be bald. And what else??
So here is a crystal ball, and this is what I see.
If I asked graduates in other fields, say physics, or mathematics, or engineering, how things would look 25 years from now, the answers would be easier -- simply more of the same. After all, Archimedes was doing calculus, engineering and physics 2000 years ago, so what's 25 more years. But Statistics is a field in rapid change -- what will happen in the next 25 years is a hard reading.
Partly it's hard because Statistics is an odd area. If you think about it, you may be likely to say to yourself "What a strange field I'm in". It is strange. And it's this strangeness, this misshapenness, that will be a major part of the force toward its change. So let me talk a little about why I find it strange...
There are a vast army of people who call themselves statisticians. Many thousands are in all crevices of government and industry. They design and oversee surveys, industrial experiments, quality control, data analysis, and a myriad of other practical tasks.
The uses of statistics pervade our society. They are used and terribly misused all through the social sciences and health fields. Statistics show that there is a link between A and B or that X causes Y. A few weeks ago, for an elementary statistics project, I began clipping articles out of the New York Times that involved the use of statistics. Now my desk is littered.
There were always at least two or three articles a day -- another cholesterol study, an opinion survey about crime, people who worked in a nuclear plant have higher cancer rates, and so on. It is surprising how much the world around us depends on the use of statistics.
But have you ever tried this experiment -- ask a non-technical friend what a doctor does, or an engineer, chemist or physicist. Answers are usually sensible. Then ask "What is it that you think a statistician does?" Strange answers come out. Probably, the most common is that a statistician is something like an actuary. They sit in musty old offices and collect numbers which get published in large tables.
It's odd that even though the articles involving statistics in the newspapers far outnumber those involving say, physics or chemistry, people in general know very little about what we do. Of course, we also may know very little about what it is the great unwashed mass of statisticians do.
Because of its ill-defined nature, people become statisticians through various detours. Who goes into statistics? Who becomes a statistician? Certainly all of you, but the interesting question is -- how did you get here? My impression is that most statisticians are accidental tourists.
In my case, for example, I got tired of doing probability theory, quit the university, and tried to make a living as a consultant. It turned out that there weren't many openings for consultants in probability theory, but there were good pickings in statistics. By force of wanting to eat, I gradually got converted, and wound up having great fun.
But the truth of the matter is -- which I have never admitted publicly before, that I have never taken a course in statistics in my life. Neither has my friend and colleague Jerry Friedman, who just finished three years as chair of the Stanford Statistics Department. He was an experimental physicist who got his start in statistics by analyzing tracks of high-energy particle collisions in the Stanford Linear Accelerator. John Tukey was a pure mathematician, George Box a chemist. Many other distinguished statisticians somehow drifted onboard.
Other fields have dedicated travelers. Sometimes one can hear statements like, "Ever since I was 14 years old, I wanted to be a mathematician, or maybe a physicist or a doctor." I've never heard anyone saying, "Ever since I was 14 years old, I wanted to be a statistician."
Another facet of its strangeness is that in no other field is the theory so removed from practice. For instance, there have probably been about 1000 papers published over this last ten years on the asymptotics of one-dimensional density estimation. The analog might be like ten papers on the Newtonian two-body problem coming out each month in the Physical Reviews. A large part of theoretical statistics occupies a world of different dimensionality than the world practicing statisticians live on.
The root of the problem is that statistics is going through a fundamental identity change. Universally, statistics departments started as parts of mathematics departments. Gifted young mathematicians who were drafted in the 2nd World War were put to work doing statistics in order to be useful. In short order they put statistics on a firm postulational basis, rang in decision theory, and statistics became theorems and proofs.
As statistics departments separated out from mathematics, they moved away from the idea of statistics as pure mathematics and the possibility of using data crept in. Now statistics is in a flux, it has not found its own orbit yet, but powerful forces are moving it on.
Money is a big force. Money talks to our department in two ways. The first is that the National Science Foundation has begun saying that they will put the bulk of their grants in statistics into applied work. Faculty that want grant money are going to have to leave the realm of pure mathematics. Second, is that the universities are filling up and it's harder to find jobs there. More of our graduates will have to find jobs in government and industry -- places that are singularly unconcerned about theorems and proofs.
But an equally powerful force is that statisticians are finding out the real fun in interesting applied problems. Finding out something about DNA sequencing by using a Hidden Markov Model is pretty rewarding, so is participating in setting up a large AIDS experiment, or in doing image reconstruction.
Problems are getting larger and more interesting. The data and difficulties in problems such as speech recognition, written character recognition, robotic control, are large and complex. These could be our problems. Developing methods to use the information flowing from the sensors of a robot to recognize obstacles or grasp objects is a statistical problem. So is the problem of using the data in an electrical current from a microphone to recognize words and sentences. Most of the work in these areas is currently being done by computer scientists, engineers and physical scientists, but statisticians are beginning to nibble around the edges.
To know where we are going we have to decide what are we really good at. What is at the core of statistics? Is it that we are first-rate mathematicians? Hardly. What then? At best we are wizards in figuring out how to gather good information, analyze information, and draw conclusions. This is what we are good at. And I think this is where we will be when our identity crisis is resolved.
Does that mean there won't be any theory or mathematics going on? A good model for us is in computer science departments. They, too, often started as parts of a mathematics department and then separated. They are diverse departments. They have chip and network designers. They also have strong theorists -- consider the nice developments of NP completeness and probabilistic analysis of algorithms. But the theorists live in the same world as the chip designers.
Well, so here you are, 25 years later walking onto the Berkeley campus to see what has happened. The first thing that you realize, coming in from the Euclid entrance, is that Evans Hall is no longer there. You stop a young student and ask, "What happened to Evans Hall?" He thinks for a while and says, "Here is what I remember reading about it -- they had some student riots in early 2000 demanding that Evans be torn down because it was an architectural eyesore, and the administration capitulated."
So where is the Statistics Department now? Statistics Department? I don't know of any Statistics Department, but try that building over there. You enter a lovely little building covered on the outside with purple tile and see above the door a sign saying Information Sciences. As you wander around, things seem a bit more familiar -- isn't that Professor Nolan passing by? While a few of the faces seem familiar even after the passage of 25 years, the activities are different.
As you wander from room to room listening at each open door (yes the doors are now wide open and inviting, and the halls cozy), in one room people are looking at Fourier transforms of speech waves pinned up on walls and discussing the grammatical structure of the English sentence. In this room, three MDs and two young statisticians are working over the details of a ten year study of brain cancer treatments. In another, some astronomers and Professor Stark are arguing about how strongly the data points to evidence of a Big Bang origin for the universe. Whoops, here comes a little robot cruising down the hall chased by Professor Evans and two computer scientists.
So, this is what I am uniquely privileged to see and so are you. Call me and let me know how it looks in 25 years. Be sure you get in on your share of the fun. May the force be with you.