C ONCLUSIONS : I NCONSISTENCY RESOLUTION WITH THE HELP OF CYCLIC RE - EVALUATION AND

In Section 8, we raised the Paradox of Problem-Solving Efficacy (PPSE). Now, we are in a position to propose two different resolutions to it.

In Section 10, we applied a meta-scientific model around the concept of ‘experimental complex’. From this, we get the first resolution to (PPSE):

Resolution to (PPSE) based on cyclic re-evaluation Non-exact replications and methodological variants (a) are effective tools of problem-solving if

– they are progressive,

– a limit of the experimental complex can be reached (temporarily, and relative to the informational state), and

– the experimental complex has only one limit, or conflicts with other limits can be resolved;

– conflicts with methodological variants which are limits of other experimental com-plexes can be resolved;

(b) are ineffective tools of problem-solving if – they are not progressive, or

– the chain of non-exact replications is not capable of reaching a limit of the experi-mental complex, or

– conflicts among different limits of the same experimental complex cannot be re-solved, or

– conflicts with methodological variants which are limits of other experimental com-plexes cannot be resolved.

(c) The cyclic process of re-evaluation provides us valuable starting points for the elabora-tion of new, more refined non-exact replicaelabora-tions which might lead to a limit of the ex-perimental complex (that is, it is an open process); and

(d) is guided by problem-solving strategies (Contrastive Strategy, Combinative Strategy).

That is, while progressivity is a local characteristic of non-exact replications, effectiveness is a global feature. This means that progressivity is relative to an experiment and its non-exact replication, while effectiveness can be judged only relative to an experimental complex. During the re-evaluation process, we are usually “underway” in the sense that we take progressive steps (solve a problem) but we are not in a position to decide about the effectiveness of the re-evaluation process yet. That is, in most cases we cannot claim that we have reached a limit nor state that no limit can be reached, etc. but are in the middle of the problem-solving process, where several possible further steps may present themselves.

Nonetheless, there are still two caveats. First, new pieces of information can overrule ear-lier decisions. Thus, a non-exact replication can turn out to be problematic and lose its limit-status. From this it follows that effectiveness can be judged only in the long run, and decisions are not final but only provisional. Second, a limit of an experimental complex may be

incon-sistent with a limit of another experimental complex. Therefore, besides intra-complex rela-tions, inter-complex relations have to be reconstructed and evaluated, too.

The metascientific model we applied supposes that experiments and experimental com-plexes alike are open processes in the sense that, in possession of new pieces of information, they may be continued, modified, or even discarded. Therefore, there are no experiments whose results were unquestionable (both practically and theoretically), nor immune to any improve-ment, refineimprove-ment, or criticism. A second key feature of our model is that experimental com-plexes are supposed to be not linear but cyclic. This means that a given step of the re-evaluation process does not necessarily lead to better results.⁸⁸ It may turn back to earlier stages and tinue the revisions with an experiment for which a non-exact replication has already been con-ducted. Thirdly, conflicts among experiments cannot be resolved in a simple way, for example, by a mechanical comparison of the plausibility of their results. Instead, strategies of incon-sistency resolution as described in Section 10.2 have to be applied.

In Section 11, we took another route and made use of statistical meta-analysis as a tool of conflict resolution. Our considerations pave the way for another possible resolution of the Par-adox of Problem-Solving Efficacy (PPSE):

Resolution of (PPSE) based on statistical meta-analysis Non-exact replications and methodological variants (a) are effective tools of problem-solving if

– the number of exact and non-exact replications as well as methodological variants is high enough, so that data points processed by meta-analysis are available in a high number, resulting in more stable and well-founded estimates (that is, more plausible data);

– heterogeneity analyses either indicate consistency among the outcome of the exper-iments, or there is heterogeneity but the factors which lead to methodological or other kinds of differences, or subgroups among participants which behave differ-ently can be identified;

(b) are ineffective tools of problem-solving if

– the number of exact and non-exact replications as well as and methodological vari-ants is low, or

– heterogeneity analyses and the resulting subgroupings do not produce useable results but indicate the presence of systematic errors which cannot be identified and ruled out on the basis of the information available. In such cases we have a set of wildly varying results without any plausible explanation for the causes of the divergencies.

(c) The application of statistical meta-analysis motivates new directions of research insofar as

88 This motivated the distinction between the progressivity and effectiveness of non-exact replications in Sec-tion 3.1.

– the use of effect sizes instead of significance testing is more informative about the relationship of the variables at issue. Therefore, it requires more sophisticated expla-nations, and as a consequence, a refinement of theories;

– the calculation of the summary effect size provides a more reliable and more precise estimation of the strength of the relation between the variables investigated. This necessitates the revision of the empirical basis of theories, and a rethinking of the strength of empirical support for these theories;

– the heterogeneity analyses and the resulting subgroupings might initiate a search for further factors which might influence the results.⁸⁹

(d) corresponds to the use of the Combinative Strategy insofar as it synthetises the results of all available experiments.

As with the model based on cyclic re-evaluation, the application of statistical meta-analysis is an open-process, too. Therefore, adding new experiments may lead to different results, and earlier decisions about the effectivity might be in need of revision, too.

89 Cf. “It would be interesting and vital for further research to examine which of the properties for topic-vehicle pair – interpretive diversity, similarity, aptness, conventionality, relationality – dominates metaphor-simile dis-tinction and how these properties interact with each other.” (Utsumi & Kuwabara 2005: 6)

III.THE EVALUATION OF THEORIES WITH RESPECT TO EXPERIMENTAL RESULTS