- 7 - 2.2 Semantically Implied Modalities
When investigating modality-specific effects in language processing, it is important to mention that there are also modality-specific influences resulting from semantic processing when we understand the meaning of the processed concepts. In the last decades, research in cognitive science has more and more moved away from the assumption that concepts are represented in an amodal way (see Murphy, 2000, for a review of traditional models). Rather, it is assumed that conceptual knowledge is grounded in the sensory-motor system. According to embodied cognition theories, it is supposed that the neural system involved in semantic processing is the same as for perceiving sensory input and producing motor output (e.g., Barsalou, 1999; Barsalou, 2008; Barsalou, Simmons, Barbey, & Wilson, 2003; Fischer & Zwaan, 2008; Glenberg, 1997; Glenberg & Kaschak, 2002; Kiefer & Pulvermüller, 2012; Zwaan, 2003). This assumption has been supported by a large number of functional imaging studies. For example, Kiefer, Sim, Herrnberger, Grothe, and Hoenig (2008) have shown that processing audition-related words like “telephone” evokes an activation of superior temporal auditory areas, even if they are presented visually as written words. Hauk, Johnsrude, and Pulvermüller (2004) demonstrated that silent reading of single action words (i.e., words that are semantically linked to motor programs) like “kick”, “lick”, or “pick” evokes a similar somatotopic pattern as the actual performance of the represented action. Tettamanti and colleagues (2005) provided similar results for listening to action words embedded in sentences. They found neural activation of sectors of the premotor cortex, where the described actions are motorically coded. Likewise, evidence exists that understanding tool words like “hammer”, for ex- ample, results in activations of the motor cortex, which are specific for hand-movements representing the actions by which the tools are used for (e.g., Chao, Haxby, & Martin, 1999; Pulvermüller, Mohr, & Schleichert, 1999). Thus, understanding the meaning of a concept seems to evoke an internal simulation of the sensory-motor experiences associated with this concept.
In this paper we have argued that sign languages employ exactly the same grammaticalization paths as do spoken languages. That is, the pathways proposed in the literature are modality-independent. However, we also discussed two modality-specific properties of grammaticalization. First, as opposed to spoken languages, sign languages only have very few (if any) instances of type 2-grammaticalization (i.e. from free to bound grammatical morpheme). This may either be due to the fact that sign languages are comparably young languages or to the general scarcity of affixational morphology in sign languages. Second, the grammaticalization of gestures is different in signed and spoken languages. Since the source gestures must belong to the same articulatory and perceptual domain as the target language, sign languages have the unique possibility of grammaticalizing manual and non-manual visual gestures. By contrast, spoken languages do not grammaticalize visual but acoustic gestures. Moreover, grammaticalized gestures seem to be more frequent in sign languages than in spoken languages. Hence, although grammaticalized gestures can be found in both modalities, the grammaticalization of gestures clearly has modality-specific properties.
2.3 Models considering modality-specific influences
While the majority of dual-task accounts focused on central, amodal mechanisms of cognitive control, some accounts did consider modalities. Opposing the idea of one central amodal resource, multiple resource theories assume that dual-task costs arise because tasks compete for multiple domain-specific resources (Navon & Gopher, 1979; Wickens, 1980, 2002). Wickens (1984) proposed a three-dimensional taxonomy of these resources differentiating between: processing stage (encoding, central processing, response-execution), central codes (verbal vs. spatial), and modalities (visual, auditory, manual, vocal). Dual-task costs are defined by the degree to which two tasks draw on the same resources along these three dimensions. Hence, no conflict would arise when the tasks involve different modalities (i.e., different stimulus and response modalities) and central codes. In addition, Wickens, Sandry, and Vidulich (1983) proposed specific compatibility relations between the modalities and central codes along the three dimensions (i.e., between visual input - spatial task - manual response and between auditory input - verbal task - vocal output). Even though this idea considers modality-specific influences on dual-task performance, the compatibility relation is restricted to an overlap across all three dimensions. Moreover, the concept of resources was defined circularly, by assuming that resources were shared because there were dual-task costs and dual task costs arise because of shared resources (Navon, 1984). Furthermore, the rather sparse empirical evidence tested on primarily complex tasks, even though it has been pointed out that simple stimulus-response tasks are better suited to investigate the underlying mechanisms in dual-task performance (e.g., Koch, 2008; Meyer & Kieras, 1997a; Pashler, 1994a).
One reason for doing the interview on Majorca was the multilingual situation I encountered there. I was curious to work with students from there and get an insight into their culture. The majority of inhabitants there speak Catalan and Spanish equally well and because of the high tourist rate (approx. 7 Million Germans spend their holiday each year on Majorca) many Majorcan people are working in tourism. Having a job in this sector also assumes to speak German and English well. So, people living there are literally facing a multilingual environment. In the oral interview my test subjects stated that there are areas and places on Majorca where everyone only speaks German or English. In those specific areas even locals are not fully able to communicate without knowing German or English.
The excellent temporal resolution of the ERP method allows us to identify the point in time at which evaluative features influence auditory processing. In the auditory domain, the N1 component is most often reported to be modulated by the participant’s attentional state in the relatively early time range (i.e., within 100-150 ms following tone onset) during the sensory encoding of the auditory stimulus (e.g., Folyi, Fehér, & Horváth, 2012; Herrmann & Knight, 2001; Woldorff & Hillyard, 1991). The auditory N1 is a negative-going waveform that peaks maximally at fron- tocentral electrodes approximately 80-120 ms following a tone onset or other transi- ent change. It results from the activity of various neural sources, presumably includ- ing primary and secondary auditory cortices and non-modalityspecific brain regions (Giard et al., 1994; Näätänen & Picton, 1987; Woods, 1995). The N1 is considered as an example of “exogenous” auditory ERP components, as it reacts sensitively to changes in acoustic features and stimulus presentation characteristics (e.g., Barry, Cocker, Anderson, Gordon, & Rennie, 1992; Budd, Barry, Gordon, Rennie, & Michie, 1998; Crottaz-Herbette & Ragot, 2000; Dimitrijevic, Michalewski, Zeng, Pratt, & Starr, 2008; Jacobson, Lombardi, Gibbens, Ahmad, & Newman, 1992; Weise, Schröger, Fehér, Folyi, & Horváth, 2012). In functional terms, it reflects tran- sient detection as it can be elicited by stimulus onsets, offsets, and changes in contin- uous stimulation (e.g., Dimitrijevic et al., 2008; Näätänen & Winkler, 1999; Weise et al., 2012).
information of the physical nature of the sensed materials, using a general cross-modality learning (CML) framework . Intuitively, most existing multi-modality feature learning methods basically follow the concatenation-based fusion strat- egy . However, either early fusion or latter one might be incapable of effectively addressing the aforementioned chal- lenge, as there is a lack of completely-paired multi-modality samples in the whole dataset. The problem setting naturally motivates us to find a latent shared feature space by learning modality-specific projections from the training samples.
Modality compatibility denotes the match between sensory stimulus modality and the sensory modality of the anticipated response effect (for example, vocal responses usually lead to auditory effects, so that auditory–vocal stimulus–response mappings are modality-compatible, whereas visual–vocal mappings are modality incompatible). In task switching studies, it has been found that switching between two modality-incompatible mappings (auditory-manual and visual–vocal) resulted in higher switch costs than switching between two modality-compatible mappings (auditory–vocal and visual-manual). This finding suggests that with modality-incompatible mappings, the anticipation of the effect of each response primes the stimulus modality linked to the competing task, creating task confusion. In Experiment 1, we examined whether modality- compatibility effects in task switching are increased by strengthening the auditory–vocal coupling using spatial-verbal stimuli relative to spatial-location stimuli. In Experiment 2, we aimed at achieving the same goal by requiring temporal stimulus discrimination relative to spatial stimulus localisation. Results suggest that both spatial-verbal stimuli and temporal discrimi- nation can increase modality-specific task interference through a variation of the strength of anticipation in the response-effect coupling. This provides further support for modality specificity of cognitive control processes in task switching.
Despite significant advances in emotion recognition from one in- dividual modality, previous studies fail to take advantage of other modalities to train models in mono-modal scenarios. In this work, we propose a novel joint training model which implicitly fuses audio and visual information in the training procedure for either speech or facial emotion recognition. Specifically, the model consists of one modality-specific network per individual modality and one shared network to map both audio and visual cues into final predictions. In the training process, we additionally take the loss from one auxil- iary modality into account besides the main modality. To evaluate the effectiveness of the implicit fusion model, we conduct exten- sive experiments for mono-modal emotion classification and regres- sion, and find that the implicit fusion models outperform the standard mono-modal training process.
4 Theoretical contribution and open questions of the present findings As outlined above, the empirical findings of the present dissertation contribute to a better understanding of modality-compatibility effects on cognitive processes. One of my introductory points was that central processes have modality-specific interactions. Taking a set of stimuli and responses and manipulating only the mapping between them –in this case the S- R modality mapping– is a traditional approach for isolating the central processes. In contrast to a simple S-R association view, modality-compatibility effects indicate a bidirectional route between sensory and motor processes (see e.g., Hazeltine & Schumacher, 2016; Miller, 1998; Pashler, 1984, for a review), and specifically, according to the ideomotor approach, it is assumed that response selection is guided by the anticipated sensory-response effect that the response produces (Badets, et al., 2016; Földes et al., 2017; Greenwald, 1972; Greenwald & Shulman, 1973; Herwig & Waszak, 2009; Shin et al, 2010; Stephan & Koch, 2010, 2011). I also stated that humans behave according to their goals, such as holding a coffee cup when they want to drink coffee or talking to someone when they want to share some information. It might be seen that in order to be able to behave according to our goal, top-down and bottom-up processes have to be coupled, that is, cognitive control has to organize and activate sensory- motor processes; otherwise simple S-R associations would guide the behavior (Cohen, Dunbar, & McClelland, 1990; Norman & Shallice, 1986; Ridderinkhof, 2002). Therefore, while we assume a modality-specificity of central processes, it also has to be interacted with cognitive control processes, but it still needs to be clarified how the modality-compatibility effects interact with cognitive control processes or in other words, how modality-compatibility effects contribute to goal-based behavior.
In Section 5.2.4 the total task duration of three specific tasks is predicted with two different CogTool models and one ACT-R model. Figure 5.13 shows the Cog- Tool script window for demonstrating a task with the multimodal RBA design. On the left of the window the current frame is displayed. Tasks can be demonstrated by clicking on the interactive (orange) elements of the frame. A click on a button causes a transition to a subsequent frame. The current view of the script window shows a click on the microphone that opens a context menu as several utterances have been defined causing different transitions. By selecting one of these utterances the respective transition is triggered. On the right of the window the script gener- ated by CogTool is displayed. By graphical or speech input in a frame steps are added to the script. The example shows the completed script for task 15 includ- ing the sub tasks: city=”Rostock”, culinary-category=”mediterran”, time=”elf uhr”, persons=”neunzehn personen”. The steps of the scripts are listed sequentially line by line from top to bottom. In the columns the used frame, the executed action, and the used device are displayed. The Think-mod-sel actions are manually integrated as modality selection steps before each input via a device. Once an input via a device is made CogTool automatically integrates another ”think” step into the script before the step for the actual input. In order to be able to better distinguish the ”think” steps, the names are changed. Each ”think” step for modality selection is followed by a modalityspecific ”think” step. In the current view line four including the sec- ond Think-mod-sel step is selected. The duration of the Think-mod-sel step is set to 0.1 seconds. The durations of the modalityspecific steps will be changed later in the ACT-R model. All the three tasks that were used for the performance prediction were demonstrated in the same way as depicted here for touch screen, for speech and for multimodal input. The CogTool script for touch screen input and for speech input provide the basis for the multimodal ACT-R models.
positive outcome on the birch pollen-related apple allergy after 1 year of birch pollen-SCIT 239 . They reported a 10-fold decrease of the visual analogue scale scores used for evaluating the intensity of the oral allergy during a double-blind placebo-controlled food challenge with meals containing Golden Delicious in 69 % of the patients. An interesting feature of this study is the use of recombinant Mal d 1 in skin-prick tests for the evaluation of the sensitivity of the skin related to the apple allergen. This method allowed a standardized and comparable evaluation of this parameter at different time points. The results accorded with the previous clinical data, as a 20-fold decrease was observed after 1 year of treatment. In addition, a trend for increase was seen for birch pollen- and apple-specific IgE levels, whereas the corresponding IgG4 antibodies were significantly induced for both specificities. These similar changes for birch pollen- and apple-specific antibodies indicated in this study that birch pollen-SCIT induced cross-reactive antibodies beneficial for the apple allergy.
Domain-Specific Modeling has become increasingly popular in the past decade. These languages allow raising the level of abstraction away from the solution domain (code) to the problem domain with obvious benefits such as improved development productivity and product quality. While typically the domains- specific modeling languages are built for a narrow area within a company the next obvious step is to “globalize” languages so that coordinated use of domain- specific languages becomes possible. We identify some key challenges for research in three contexts: organization, language and technology. In the organizational context often already a single DSL may change organizational tasks, roles and structures. How multiple coordinated ones while influence to organizations and to development processes. In the language context, the coordination must be specified at the level of language specification but it is not clear how currently used metamodeling languages allow to do that. For example OMG’s MOF does not even identify “language” so it is questionable how it can then integrate a number of them? Finally, and within the technology context, it is not realistic to expect that all languages can be coordinated within a single tool so what kind of tool integration approach would work among a set of tools?
Table 1 – Variance decompositions: contributions of uncertainty components to the volatility of real economic activity growth
Notes: Contributions of the global, region-specific, country-specific and idiosyncratic components to the variance of real economic activity growth (average of contributions to real GDP growth, real private consumption growth, real gross fixed capital formation, employment growth, unemployment rate, industrial production growth, retail sales growth, real export growth and real import growth) over the whole sample period 1971Q1-2016Q4. * Idiosyncratic contribution derived as residual.
In order to investigate the route of uptake of Bet v 1-Protein, Hybrid-Peptide and Bet v 1-Peptide, A549 cells and BM-DCs were preincubated with blocking reagents which inhibit different routes of antigen uptake. Monensin, an agent that inhibits receptor mediated endocytosis, blocked the uptake of Bet v 1-Protein and Hybrid- Peptide by ~40% leading to the suggestion that an active internalisation process is likely to contribute with these two antigens (Figure 9). This blocking strategy shows one major limitation as it only demonstrates that antigens are internalised via receptor mediated endocytosis, but does not give any information about the binding receptors which are involved in antigen detection. In order to identify these binding receptors it would be necessary to use specific blocking antibodies directed against certain PRRs (e.g. Toll-like receptors, NOD-like receptors, C-type lectin receptors). Furthermore blocking macropinocytosis by incubating the A549 cells and BM-DCs with Cytochalasin D showed only a minor effect on the internalisation of Bet v 1- Protein and Hybrid-Peptide in A549 cells, but not in BM-DCs (Figure 9). Our data demonstrate that receptor mediated endocytosis and macropinocytosis are partially involved in the internalisation of Bet v 1-Protein and Hybrid-Peptide. While using the same blocking strategies, the internalisation of Bet v 1-Peptide was not affected in A549 cells or BM-DCs. However, there is evidence indicating that antigens are able to penetrate cell-membranes independent of endocytosis, of energy or of specific receptors (Hammad et al., 2008; Heitz et al., 2009). Based on the fact that internalisation of Bet v 1-Peptide could not be blocked with any used reagents, it is very likely that it is internalised via passive leakage. With these experiments we tried to get an overview of the internalisation mechanisms involved in the uptake of antigens which differ in size and structure. However, these mechanisms can only be part of a much larger concept and need further investigations.
We develop the workflow design framework RCE [ 1 ] that allows tool integrators to integrate discipline-specific tools and make them available to other users via a peer-to-peer network of RCE instances. Workflow designers can then build complex automated workflows comprising discipline-specific tools and standard tools provided by RCE. By abstracting from technical details, RCE allows users to focus on those aspects that are relevant to their specific work.
modality. A synthetic modal claim is a claim of the type ‘Necessarily/possibly p’, meant not as a claim about whether p is logically necessary (or ‘analytic’) but as a modal claim as I have described it. By saying that a proposition p is synthetically necessary I mean that p is not logically necessary (analytic) and that ‘Necessarily p’ is true. By saying that a proposition p is synthetically possible I mean that p is not logically necessary and that ‘Possibly p’ is true. Logicists claim that logical necessity is necessity in the strongest sense. Necessity in any other sense is called ‘natural’ necessity and is taken to be necessity in a weaker sense (e.g. Chalmers 1996, 41). The argument for this is that ‘the class of natural possibilities is [...] a subset of the class of logical possibilities’ (Chalmers 1996, 37) and the class of logical necessities is a subset of the class of natural necessities. This is derived from a definition of ‘natural necessity’ which includes both logical and synthetic necessity. I object that such a concept of ‘natural necessity’ is too much a mixed bag. Nothing that is logically necessary is synthetically necessary, and nothing that is synthetically necessary is logically necessary, and logical necessity and synthetic necessity have so little in common that it is not useful to subsume them under one concept of ‘necessity’. To say that logical necessity is stronger than synthetic necessity is as uninformative as saying that redness is stronger than coherence because the class of statements that claim that something is red is a subset of the class of coherent statements and the class of incoherent statements is a subset of the class of statements that do not claim that something is red. Modal questions that arise in philosophy, such as whether backward causation is possible or whether the mental supervenes on the physical are questions, not about what is coherent but about, what is possible; that is, they are to be understood in terms of synthetic necessity. And there is no reason for saying that this is not necessity in the strongest sense.
• Word hypothesis graphs: No embedded speech recognizers are known by the author to date to return word hypothesis graphs for recognized spoken input. This is due to the em- bedded speech recognizers often being streamlined to be more efficient with respect to pro- cessing and memory requirements. This has the effect that properties such as timestamp and confidence value can not be retrieved on a per word basis, but rather only on an utterance basis from which the expected temporal order of words and an overall utterance confidence value can be derived. This limitation is common in other mobile demonstrators (e.g. (Kumar et al., 2004)), but could be overcome by incorporating server-side recognizers, although not even all server-sided speech recognizers provide such functionality in their returned results. • Timing information: Functionality in the underlying operating system only provides access to timing information that is exact to the second. It is however often desirable to have timestamp information that is exact to the millisecond, for example in identifying the time that individual semantic constituents in an utterance (e.g. demonstrative pronouns like ‘this’ and ‘that’) are provided by a user in a particular modality. Timing in milliseconds for the MSA and BPN applications was achieved on the PDA through the use of system timers, but was found to be too resource intensive for the CPU when used for long periods of time. Despite the difficulties arising from device restrictions and dynamically changing environ- ment conditions, users are gradually breaking free from the traditional stationary desktop com- puting paradigm and entering the realms of mobile, ubiquitous, and pervasive computing. This is confirmed by a recent report by the market analysis company Gartner 6 , which shows that 816.6
By noting how the “characteristic forms of a tool’s or a medium’s distortion, of its weakness and limitations, become sources of emotional meaning,” Eno in- itiates a groundbreaking insight about medium-specific noise. He reminds us that there is such a thing as a medium’s “characteristic forms …[of] distortion,” and that these forms can be used as means of expression. By observing this, Eno also implicitly comes to evoke the simple but important question: what are the means by which we can represent one medium within another? The compelling answer to this question is that medium-specific noise offers a crucial means to represent one medium inside another: it is the means by which we can represent the me- diality of the vinyl record and the super 8, respectively, within a CD and a 35mm movie. 3 Once these evocative powers of medium-specific noise are established, we must go on to ask: what does medium-specific noise afford, and how does it operate in various circumstances? In two groundbreaking articles, musicologists Joseph Auner and Steven Link have started to address these questions in the realm of music. As is made clear by their contributions, medium-specific noise provides a highly plastic tool for sculpting space in the aural realm. It also pro- vides tools for articulating complex emotional worlds.