Sampling Error

As a result of the presidential elections, the Ukrainian sociological community has suffered losses that amount to a credibility gap and a crisis within arbitration entities. The conflict that began during the last election campaign recently got a new lease on life. After a number of associates of the Institute of Sociology issued a statement, The Day tried to contact some sociologists for a round table in order to set the record straight on the exit poll scandals. But the contacted experts, who initially agreed to take part in this discussion, later politely refused, saying that they were not yet ready to sit around the same table with the very people who had “betrayed both science and their own colleagues.” In their view, there is no sense in publicizing these mutual reproaches and accusations in the press or delving into complex discussions of “weight coefficients,” and effectiveness and impartiality of methods, samplings, and errors. Specialists must first examine this problem by themselves, all the more so as the exit poll scandals of the last elections undermined the very structure of the sociological community, including the Sociological Association of Ukraine (SAU), Ukraine’s most authoritative organization of professionals, which encompasses all rival sociological centers, among them the Academy of Science’s Institute of Sociology. Before the last elections, the association’s judgment was a yardstick of impartiality and professionalism in the sociological community, a litmus test for all disputes about “contracted ratings” and “bogus structures.” “If a survey center has not been accredited by the SAU, they are non-professionals, and you’d better stay clear of them,” SAU-accredited sociologists kept saying to journalists, which is what we have been doing for the last eight years.

So when sociologists from the five most influential SAU-accredited structures announced the creation of a consortium for the 2004 exit polls, nobody thought that precisely this institution would become the hub of the nascent “war of the exit polls.” We expected the same old practice of invalidating results of polls conducted by non-existent firms or, at most, pointing out trivial differences between the results obtained by authoritative structures, which would be soberly analyzed by experts and explained to the public in a civilized way. We were expecting attempts to discredit certain poling companies and the exit poll as a whole but not that these attempts would be successful (through the efforts of sociologists themselves).

Let us briefly recall the sad story of the reversal of circumstances surrounding the exit polls during the last presidential elections. A few months before the elections, five well-known sociological institutions- SOCIS Center, Kyiv International Institute of Sociology (KIIS), Social Monitoring Center, Razumkov Center, and Democratic Initiatives Foundation, the latter playing the role of initiator-announced the creation of the Exit Poll-2004 Consortium. A little later, another group, Ukrainian Exit Poll, comprising the Ukrainian Institute of Social Studies, a number of Ukraine’s national universities, and the Sociological Association of Ukraine, announced that it would be conducting a parallel exit poll. The very fact of the association’s participation instantly laid the grounds for a conflict, making an impartial assessment of events impossible because the members of the consortium were simultaneously members of the SAU. Moreover, one of the consortium members, KIIS president Valeriy Khmelko, was also one of the 12 members of the SAU board. We will get back to this a bit later; now about the exit polls.

A week before the voting in the first round of the presidential elections, the managers of two consortium members, KIIS and the Razumkov Center, said that, owing to the complicated political situation, it was advisable to apply an entirely new method of data collection-questionnaires. Their partners, SOCIS and Social Monitoring, rejected the experiment because they doubted the reliability of this method. As a result, on the evening of October 31 the consortium announced two essentially differing results. According to KIIS and the Razumkov Center, Viktor Yushchenko was leading by a margin of 6.4%. According to the results of the “secret” exit poll, Viktor Yanukovych was leading by a margin of 4.4%.

The latter was sort of a “buffed up” result because immediately after the polling stations closed, all the consortium members, except for Social Monitoring, which showed an extremely narrow margin of 0.3% in favor of Yanukovych, announced that Yushchenko was in the lead. However, two and a half hours later, these two centers amended their results. What happened is still unclear. Announcing the results, the sociologists warned that these were preliminary conclusions subject to further changes.

Many people feared that this discrepancy, which emerged for the very first time in the sociological community precisely during these elections (this applied to the data collected in pre- and post-election surveys), would not be resolved just by revising the results. Four days later a press conference was held to announce the updated information, which inevitably sparked a scandal. The sociologists accused the managers of SOCIS and Social Monitoring of rigging the data. The managers revealed that they had been pressured by their survey sponsors (eight embassies and four international foundations) into using only questionnaires and were thus withdrawing from the consortium.

The Ukrainian Exit-Poll also announced the results of its survey: Yanukovych had won the first round by almost 4%. Incidentally, Ukrainian Exit Poll project manager Oleksandr Yaremenko admitted during the second round that his survey had flopped because a large number of voters refused to take part in the poll.

It’s worth taking a look at the exit poll that was conducted by the Russian Public Opinion Foundation (POF), which has no corresponding network in Ukraine. The general public is still in the dark about which national sociological firm conducted this survey. As the POF announced at a press conference, its first-round exit poll failed because of the “unprecedented number of refusals.” However, information leaked to the press indicates that an exit poll did take place but gave the “wrong” result, showing that Yushchenko was leading. In its turn, the Ukrainian Marketing Group, which supposedly funded the POF’s field studies, refused to rig the data, so Gleb Pavlovsky, the POF’s spokesman during the elections, could do nothing but announce the poll’s failure. Nevertheless, the Russian firm contributed a lot to creating mistrust toward exit polls on the whole. This has highlighted the necessity of setting clear-cut rules for sociological research to ensure maximum transparency for foreign research organizations that collect information on the attitudes of Ukrainian citizens. As a matter of fact, foreign firms often do not care about their reputation in a foreign country. On November 21 the exit poll results that were announced on television, just like after the first round, did not give the public a clear answer about the winner. According to data collected by National Exit Poll (the remaining part of the consortium, comprising the Razumkov Center and KIIS), Yushchenko won in the runoff with 54% of the vote against Yanukovych’s 43%. Like they had done on October 31, these companies had conducted a poll using questionnaires. The other group, People’s Choice, which was represented by former consortium members SOCIS and Social Monitoring, announced different results: SOCIS showed that Yanukovych had defeated Yushchenko (48.2% vs. 46.8%) while the results of Social Monitoring’s poll indicated that Yushchenko had won (49.4% vs. Yanukovych’s 45.9%). What is more, SOCIS manager Mykola Churilov said his exit poll results should not be taken into account because the survey had in fact failed.

Nevertheless, in spite of the ambiguous results and the tarnished reputation of this kind of survey, the exit polls still managed to accomplish their mission. Encouraged by the exit poll data showing Yushchenko’s win in the runoff, his followers began to celebrate a victory. During the night, when the notorious Central Electoral Commission, headed by Kivalov, announced entirely different figures and everyone understood that there was no victory in sight, the crowd’s jubilation turned into a protest. Then came the Orange Revolution, the Supreme Court appeal, the rerun, the inauguration, ministerial appointments — and all the exit polls and sociologists faded into oblivion.

The public, which had been closely following the exit poll scandals and disputes, never received an answer to the question: had some public-opinion centers become tainted by political games or did the problem lie in certain mysterious sociological instruments that broke down in the extremely electrified political atmosphere? Unfortunately, no “debriefing” by international and national sociology specialists, which had been promised on the eve of the election campaign, ever took place. Furthermore, who can act as arbiter if the Sociological Association, until recently the most prestigious sociological institution in Ukraine, was drawn into the conflict because of someone’s failure to duly assess the situation and which is now on the verge of breaking apart?

Last week, a few Kyiv-based SAU members publicly expressed their concern about the situation that emerged from the ranks of the sociological community during the presidential elections. They noted that on the eve of the first round SAU’s managers decided to take part in the Ukrainian Exit Poll, thus opposing it to the National Exit Poll conducted by the Consortium. What also puzzles the Kyiv sociologists is the mess surrounding the results that were announced by SOCIS and Social Monitoring. “After the first round,” the statement says, “O. Balakireva, head of SAU’s Kyiv branch (Social Monitoring) and M. Churilov (SOCIS), without explaining their own manipulation of the primary exit poll information (the so-called ‘re-weighing of data’), publicly accused their Consortium counterparts of pressuring them and, later, of non-professionalism.” The statement also cites instances when A. Yaremenko, vice-president of SAU and director of the Ukrainian Institute of Social Studies (UISS), criticized some of his colleagues and sociological services, calling them incompetent.

“The SAU managers’ actions and inaction severely damaged the reputation of sociology, which the public now views as an unprofessional, over-politicized, and conflict-prone field of study. Rather than being an association of researchers chiefly engaged in examining the reliability of information collected by others, its members are ready at the drop of a hat to serve political interests,” the statement’s final item states. The 17 sociologists who signed the statement believe that, under these circumstances, it is crucial to express non-confidence in the SAU management, convene an extraordinary SAU congress in the next few months, and launch a debate on the relationship between sociology and government in the light of the 2004 presidential election experience. Should the SAU management refuse to convene an extraordinary congress of the association, the group of Kyiv sociologists, mostly research associates at the Institute of Sociology, reserves the right to withdraw from the SAU.

Meanwhile, Valeriy Khmelko, president of the Kyiv International Institute of Sociology (a Consortium member), quit the SAU’s 12-member board on February 11. The researcher explained his decision in an open letter to SAU members. We are not going to cite this letter at length because it is primarily intended for Khmelko’s colleagues. On October 8, 2004, the SAU management (minus Khmelko) decided to support the Ukrainian Exit Poll, thereby allegedly violating the principle of fair competition among equivalent sociological centers accredited with the association. By showing this support, the management obviously turned a blind eye to a number of points in the statement of UISS director Yaremenko, in which he discredited his colleagues from the Consortium, thus violating the sociological Code of Professional Ethics approved at the last SAU congress. All of Khmelko’s attempts to establish the circumstances that had led his colleagues to make such a decision have been futile. However, as he writes in his letter, “Before February 11 this year, when the SAU board gathered for its first meeting after October 8, I refrained from making any public comments, hoping every day to convince my colleagues of the need to recognize the flawed nature of a policy whereby the SAU’s governing body, in the presence of competition between several research projects, with SAU collective members in each of them, proclaims itself the organizer and participant of one of these projects in order to ‘raise its status.’ This is why I suggested at the February 11 meeting that the board should cancel the earlier decision to support only the exit polls organized by Yaremenko because those decisions were wrong and made without seriously discussing Yaremenko’s own arguments about the polls he had organized and without taking into consideration the interests of other SAU board-accredited collective members that were conducting other exit polls.” The rejection of his suggestion caused Khmelko to take a serious step: “I consider it impossible to remain a member of the board that is deliberately continuing to act the way it did during the election campaign.” Khmelko also urged his colleagues to support the statement of the 17 sociologists who expressed non- confidence in the SAU board and suggested convening an extraordinary SAU congress as soon as possible, considering there is a year or so to go before the next regular congress. The Day’s correspondent discussed this matter with Mr. Yaremenko whose actions the dissenting sociologists consider one of the causative factors of the conflict. He noted that the SAU statute does not forbid the association to conduct independent sociological surveys or to make decisions to support any kind of exit polls. He also claimed that the association’s board had not denied support to any of its members to conduct a poll for the simple reason that none of them had requested this kind of support. “That’s why it is unfair to make retroactive accusations that the board did something wrong, engaged in some kind of discrimination, or violated the principles of competition,” Yaremenko added. “We have a statute, and if people observe the statute, their actions are legitimate.” The sociologist also pointed out that one of the main arguments in favor of conducting the Ukrainian Exit Poll was that this project would demonstrate that Ukrainian organizations are capable of conducting a quality survey on their own. “I promoted the state, whether someone likes this or not,” the UISS director said, adding that he was ready to disclose the whole array of data for any kind of audit.

Commenting on the current situation in the SAU, Yaremenko said there was no split so far. “I would say there is an attitude-not a very constructive or well-balanced one, to my mind- which can lead to certain problems,” he said, “but I think the SAU is still strong enough to avert a split.” At the same time, the sociologist agreed that some members might quit the association. Yet he does not consider this a serious problem “because the SAU is a non-governmental organization, and everyone has the right to defend his point of view.” Yaremenko sees a way out of the overall situation: initiate an open- minded and profound debate (first in private, then in public). Then sociologists should organize a number of intensive public activities, such as research, round tables, conferences, and media appearances.

I will take the liberty of mentioning another crucial prerequisite. All the above-mentioned sociological organizations are competitors in the dynamic market of sociological studies. As the last election campaign showed, it is in the interests of all participants that this market be transparent and governed by clear-cut rules.

The Day will continue monitoring developments in the sociological community and allow each of the parties to the exit poll conflict to present their side. In our opinion, far more interesting than tracking past developments is monitoring the future, i.e., carrying out concrete research on Ukrainian attitudes and society’s requirements and assessments of the new leadership.