Day Two, Panel Three: The Scientific Investigation of War Crimes

Moderator: Eric Stover, Director, Human Rights Center, University of California, Berkeley, and Vice President, Crimes of War Project Introduction

b) Making it Count: Journalists, Statistics, and Human Rights
Patrick Ball, Deputy Director, Science and Human Rights Program, American Association for the Advancement of Science

PATRICK BALL: I hope that people don't get to napping in the back. But I'll talk really loudly and try to say some startling things. It may seem implausible that I'm going to say startling things, given that I'm going to talk about statistics, but give me a shot.

The talk is titled "Making it Count - Journalists' Statistics in Human Rights." And what I'll talk about is how statistics can provide a mechanism for doing a moral accounting of a large-scale atrocity. And I'll talk throughout the next 20 minutes or so about what I mean by an accounting.

Perhaps the first question about an accounting is why we would want to do one. What's in it for anyone to figure out how many people have been killed, how many have been forcibly displaced or how many in what patterns of any atrocity. And I'll suggest that there are three reasons of greatest relevance today for why we would want to do an accounting. First, because the victims demand it.

Eric alluded to some of this earlier when he talked about individuals demanding to know the outcome of particular events that affected them. Where is my son who disappeared five years ago? What happened to this group of my neighbors who were seized some time ago?

In Guatemala at the Truth Commission one victim who came to us brought a plastic bag. And in the plastic bag were the bones of his son. And he said, "I've been carrying these bones since we recovered, them because I can't have my son rest until I get a piece of paper from the Commission saying my son was innocent and he committed no crime. And only then will I bury these bones. Only then will he rest. And only then will I be able to rest."

So the truth has a fundamental importance to people who are victims. And I'll suggest today that the truth has both, the component of knowing that this particular individual was innocent. It also has a very broad macro component which addresses the second point.

This morning we heard a little bit about what happens when a particular individual testimony turns out to be false, when it turns out to be a lie. It's a very important story. The question that I wanted to ask but I realized I was going to get to talk later so I seized that other moment, was what happened if there were a thousand stories just like that little girl's that were all true? Would that change our reaction at finding out that one of them was a lie?

And I'll suggest to you today that I think it's very different if it turns out we collect a thousand stories and one or two or five or ten or fifty of them are lies. And, in fact, the systems that I designed that I'll talk about this afternoon are all designed with precisely the idea that some fraction of them are lies. But that our findings and the conclusions we reach are robust to the idea that some fraction of them are lies. And, in fact, they should be robust to a very large fraction of those stories being lies.

So a third reason we do moral accounting is to avoid the trap of moral equivalence. You know, it's a terrible situation in this country we're looking at. There have been violations committed on all sides. That kind of reporting used to be common. I see it much less now. It was very common in Central America, I remember. People who didn't want to look hard at policy said there are violations on all sides, we can't really make a call here.

But after the Truth Commissions in El Salvador and Guatemala we know that although, in fact, literally there were violations on all sides, the ratio of violations from one side to the other is on the order of 20 or 30 to 1.

And I would suggest to you that although literally it may be true that there are violations on all sides, it requires a very broad statistical view to understand whether or not that's a disinformation, that's a distraction or, in fact, we may need to think about a situation in terms of moral equivalence. But statistics offers a way out of that trap.

So with that, I'll go on to the first several examples. I'm going to talk about four projects briefly today. One are NGOs and truth commissions and the Truth Commission in El Salvador, then the United Nations Commission for Historical Clarification in Guatemala, also a truth commission of sorts, then a book I've just published called Policy or Panic: the Flight of Ethic Albanians from Kosovo." I have copies of that, a few copies here. But if people would like copies they can drop a card with me and I'll be happy to send you one.

And then finally I'd like to address the question of how many Kosovo Albanians died in the first five and a half months of 1999, a question around which I think there is some controversy.

I will not talk about the National Commission for Truth and Justice in Haiti nor a book on Guatemala which is outside nor the Truth and Reconciliation Commission in South Africa. However, I put these slides up to indicate that the use of quantitative methods in large-scale human rights work is actually increasingly common. There are good statistics out there. And in the next example I would suggest that there have been good human rights statistics available for over 20 years. That does not mean that the statistics that have been reported are good statistics. And what I will urge at the end of this talk is let's go and find the good statistics and report them. And I'll give you some pointers about what good statistics typically look like.

There was a small NGO in El Salvador called the Non-governmental Human Rights Commission. And after their finding in 1977 they collected over the course of the next 12 or 13 years about 9,000 testimonies from individuals who had witnessed or suffered large-scale human rights atrocities. I worked with them from 1991 until 1993 and we coded those testimonies into a database including over 17,000 victims who suffered more than 29,000 violations.

Now, the reason I use this particular NGO as this example is not only because I work there but because with the number of journalists that I've talked to in the last three or four years people who have worked in Central America in the middle '80s said, "You know, we really had a lot of doubts about those guys. They weren't really the most plausible source we thought." Well okay, that's an important point of view. Of course, that's an imperical question. It's a testable question whether or not what they reported in the middle of the 1980s was true or not. And I'll test it because I'm a statistician not case by case but rather by comparing the statistical patterns that they reported with statistical patterns of a source that has later been found to be very credible which is the Truth Commission in El Salvador.

So the next slide shows a graph in which the number of violations reported by the Non-government Human Rights Commission, which are the dash line and the line with little triangles here, is compared to the line that patterns reported by the Truth Commission.

So what do we find? Well, first we find that for most periods the Truth Commission found many more killings. Well, they're both partial samples and the Truth Commission had a lot more resources so that should be unsurprising to us. What's perhaps more important is that the patterns match each other in time. They're heavily spewed to 1980 and 1981 when most of the killings occurred.

And we see similar small rises in '86 and '89. In fact, these two series correlate at -- let me get the statistic right -- at .84 the Truth Commission and the Non-government Human Rights Commission data correlate at .84. They track each other very closely. So what that means is that in retrospect we can say that the Non-government Human Rights Commission of El Salvador reported in real time reliable statistics.

Now, were some of them lies? I don't know. Probably they were. And it's probably the case that we could look at a few of the cases they reported and find them to be non-credible for one reason or another. And it's also the case that we could point to particular elements of the directorship of the organization or whatnot. It would give us some reason to think that perhaps this was not an organization that we had perhaps the greatest faith in.

But at the end of the day their numbers were right. And this is very important because it's a lesson for us to look into the next set of places that undergo crises and think about what their NGOs are telling us and say, "Hey, what do we know about what NGOs do?

Now, I'm in the process of refining this analysis to great detail, take it down to the month, look at different parts of the country, look at different political actors and see if the relationships between what the Truth Commission reported and what the NGO reported are as tight as I think.

So far the evidence I have suggests that they are very close relationships. But it will be interesting to continue this as we look at other NGOs in other part so the world. I hope this finding will generalize. And from my experience I believe that it will. Certainly in Guatemala it was very similar. And so we'll go to Guatemala. And one of the questions that we asked in Guatemala is, "Hey, a lot of people were killed here. But how many at the end of the day? How many people in total were killed?"

We had the benefit of two large-scale non-governmental projects, one conducted by the Catholic Church and another conducted by a small NGO. And then, of course, the Truth Commission. They're up there as their acronyms and I won't go into detail.

But the first question we can ask is how many people were killed in Guatemala? How many people were documented as having been killed? Now, people may remember that there was something of a controversy last year around statistics coming out of Kosovo in which after having only documented 2000-odd killings somebody then concluded that that's how many people had been killed.

And that was absurd. That was a bad reading of how data works. You never document everything that happens. That's why we take samples. That's why we've developed methods to correct for these kinds of things. In one of the wealthiest countries in the world with one of the most elaborate in the world the U.S. Census Bureau cannot even just count all the living people in the United States.

And this should give us some idea of how hard it is to count people who have been dead for a few months or 15 years in a place where there are no information systems and very little registry available to make those kinds of estimates.

So I'm going to show you very briefly how we can do that. And the question that we ask when we have three projects like that is, how do the projects relate to each other? Are the projects completely independent? Did they go out there and document completely separate universes as the top example suggests? Did the projects go out there and just talk to the same people in every single case and, thereby, document universes that are just subsets of one another? Or did the projects have partial complex overlap between them? And, in fact, as we would all suspect the case is the latter.

Now, let me ask you all to make an intuitive but quantitative leap here. Imagine the first situation. We send interviewers out into the field from three independent projects. They go out and talk to a lot of people. They come back and they didn't encounter the same story. Now, maybe they were tens of thousands of people lying. I think that's implausible. What's much more likely is that the universe of possible stories that could be heard is vast. It's so vast that sending thousands of people out there they never encountered the same events.

So if we had found the first possibility we would infer that the universe of violations is vast. In the second case people went out there, did all these interviews and all the story spaced was compressed into a very small area. And our inference correspondingly would be that the universe of violations is actually not much bigger than what we found.

In the third case we can use the overlap between the three projects to make an inference about the overall size of the universe. If people are interested in mathematics I'm happy to do the calculation. I suspect people are not interested so I'll go straight to the punch line. This is what it looks like in terms of the overlap. We measured the cases by matching them between the data sets and derived these numbers, which by themselves are not important.

What is important is that we were able to make the inference about the undocumented killings. And the total turns out to be between 119,500 and 142,000 in the space we wrote the document our total estimate is that more than 200,000 people were killed.

Now, let's go back to my earlier question. Were good statistics available in real time? Journalists can't wait 15 years for us to come up with an estimate obviously, right? Journalists typically don't wait 15 minutes if they ask me a question.

So were the numbers available in real time? Not only were the numbers available in real time, they were all underestimates. The number of killings that people estimated occurred in Guatemala before we did this work were on the order of 150,000, okay? So the NGOs again were being conservative, which is not what people who worked with Guatemalan NGOs thought, that they were being conservative. We're accumulating data points here and I think it's worth looking at. So let's move on to Kosovo because as Eric has pointed out, time is moving on.

What I wanted to do is when we began this project we were very aware of the concern which has now been substantiated as we heard this morning that journalists were concerned that Kosovo Albanians were not telling them the truth. So I thought, and we thought in this project, how can we build a project that will come up with conclusions that are not dependent upon individual refugee stories? How can we find ways to look at the patterns of ethnic cleansing or actually to test the hypothesis about whether mass migration was ethnic cleansing or whether it was a reaction to some other phenomenon? How can we test that without reference to refugees' stories? So the questions were, how can we establish where refugees came from across time village by village day by day? Can we link those flows to NATO bombing which people may remember was one of the hypothesis, people were fleeing NATO bombing. I heard a very amusing sound byte lately. If bombing and air strikes caused people to flee then everyone in Belgrade would now live in Hungary. Well, probably they would. Anyway, let's leave that aside because a sound byte doesn't usually work as scientific evidence, usually. So instead we thought let's conclude about the plausibility of competing explanations based on our analysis of the refugee flows.

So people may remember that the border was a very chaotic place. People were flowing across. On one day there were more than 40,000 people that crossed across a little tiny thin road. And so we were suspicious. But when we looked at the records collected by the border guards, the Albanian border guards, it turned out that they documented more than 272,000 people. Now, that's more than half of the people who crossed the border there. I estimate that about 404,000 people crossed the border.

So given that we've been able to fill in the holes. I work for a small NGO. We didn't have enormous power and resources so we had to fill in the holes by making inferences using other kinds of data, using survey data, using UNHCR reports that were pretty broad estimates by day from the border from Macedonia, from Bosnia, and from Montenegro.

Using all those pieces we could triangulate and estimate how many people were leaving their homes and entering Albania on each day. Remember, these are different things. Not everybody who leaves their home on a given day is able to leave the country. Some of them are able and others of them get hung up. They get hung up at check points. They run to the forest to try to hang out for a while. Others of them have a route that's long enough that it takes them sometime to leave.

And that process was accounted for. If people are interested I would recommend that they have a look at the report itself. I won't have time to detail that today. But the idea is what was immediately striking looking at these graphs is there are three waves.

There's one very early wave between the 24th of March and the 6th of April, another wave between the 7th of April and the 23rd and then another from the 24th to the 11th. In fact, 97 percent of all the refugees who left Kosovo left before May 11. After that there were just small numbers.

So the first thing we said as we looked at that was, "My goodness, that's very striking how those waves are so distinct in time. And so let's look at it regionally." And this is by municipality. And Pristina is actually anomalous, unlike all the other municipalities in its region. It's very much front-loaded.

All the people who left Pristina left, pretty much left in the first phase. But the rest of the places that were front loaded in which most of the people left in the first phase are in the south and southwest.

Here is Zuberaka , Pec, and Prizren. If I show the others you'll see the same pattern whereby people are leaving, almost everybody is leaving in this first phase. And in the middle phase almost nobody is moving at all. And there's little bumps in the final phase.

Across the top with asterisks we've marked times in which NATO air strikes hit each of those municipalities. The first hypothesis we're able to rule out right away is that NATO air strikes could not possibly be motivating movement. The patterns in time just don't make sense.

Here in Zuberaka, for example, everybody leaves by the 6th of April and the first air strikes don't occur until early May. Other places you see similarly arbitrary relationships between huge amounts of flow and when an air strike occurs. So without going into detail I can say that was easy to rule out.

What was more interesting is where were people leaving from? You'll remember, that was our original question, where were people leaving from? Well, we've looked here and the first thing that occurs is that in the south and southwest people are leaving primarily in phase one, hardly at all in phase two, and then a little bit in phase three.

But if we go to the north central region, Mitrovica, Istok, Lookyan and then Krusevac, Ferashak is a little different. I presented it here because it's a weird case for a couple of reasons. We see a different pattern. It's not front loaded at all. Instead we get a big spike in phase two. What is the spike in phase two saying? It says different process. It says completely different things going on in the north central region relative to the areas of the south and southwest.

In fact, if we look at these across a map as I'll do in a second you'll see phase one, boom, the corridor along the mountains in the south and southwest is completely clearl. Then there's a pause. There's a four-day pause in which almost nobody moves anywhere. Well, the four-day pause coincides interestingly with a cease-fire announced to celebrate Orthodox Easter. It also could perhaps be a moment in which forces that were conducting ethnic cleansing redeployed. I don't know, the statistics don't tell me that. But it would be suggestive because they redeploy perhaps to areas of the north central region in which they then conducted this second phase of cleansing.

So what we see is the focus is that looking at the proportion of people from areas of the country, first south and southwest then not south and southwest and then south and southwest, same time periods. Same time periods is the magnitude which leads me to the conclusion that there had to have been some sort of central coordination. This is plotted on a series of maps that are available in the report. And areas showing the density of displacement. These are using the kinds of GIS systems that Christopher mentioned in the previous talk. And it leads us to the conclusions that the patterns from three places originating in three ways and originating in different places suggests a cause other than local or tactical reactions. There has to be central coordination. Second, the timing and departures could not have motivated by NATO air strikes. And, third, and perhaps most interestingly, refugee flows were not coincident with mass killings. And we can look at that in Q&A if there's interest in it. This is the Web site where this stuff is all available on the Internet. Since I don't have time, I want to do the next slide.

Unfortunately, my last piece was going to talk about why when in April 1999, NATO suggested that 100,000 were missing, every editor in this room should have said, No way. That's completely absurd. I think Eric's going to cut me off. So I'll just leave that hanging for why that was completely absurd. That would have, by the way, had that been true that would have indicated a killing rate in excess of 8 times greater than ever happened in Cambodia. Okay? That should give some people pause. The only killing rate higher than that in history would have been Rwanda. Okay? So people should have ruled that out as completely absurd. Right away that should have rung a bell. I'll leave that perhaps for Q&A. In any case, thank you very much for your patience.

Patrick Ball,Bio.
Ph.D., Deputy Director, American Association for the Advancement of Science (AAAS) Science and Human Rights Program

Back to Top