Die vorliegende Dissertation ist das Resultat eines langen Weges bestehend aus Schule, Studium, Promotion und praktischer Aktivitäten. Auf diesem Weg haben mich eine Vielzahl von Personen begleitet ohne die ich diesen nicht hätte beschreiten können und denen ich nun danken möchte:

Zuallererst und mit Nachdruck möchte ich mich bei Torsten Zesch bedanken. Seit Torsten mich im Studium in die schwarze Magie des machine learnings eingeführt hat, war er mir stets ein Mentor, der mich immer ermutigt hat, sich nicht mit oberflächlichen Antworten zufrieden zu geben. Während meiner Zeit als studentische Hilfskraft und als Doktorand habe ich von Torsten unglaublich viel über wissenschaftliches Arbeiten, Informatik, NLP aber auch darüber gelernt, das große Ganze nicht aus den Augen zu verlieren. Ich bedanke mich bei Chris Biemann, der mir wertvolles Feedback gegeben hat und mit dem ich ein spannendes GermEval 2017 erleben durfte. Mein Dank gilt der DFG, dem Graduiertenkolleg UCSM und der Universität Duisburg-Essen, die Sachmittel und ein produktives Arbeitsumfeld zur Verfügung gestellt haben.

Ausdrücklich möchte ich mich bei meinen Kollegen am LTL Andrea Horbach, Darina Gold, Huangpan Zhang, Osama Hamed und Tobias Horsmann bedanken, die immer für Diskussionen und Gedankenexperimente zur Verfügung standen. Andrea und Darina wurden nie müde jemandem mit weniger tiefem linguistischen Verständnis Konzepte wie named entities zu verdeutlichen. Besonderer Dank gilt Tobias, der mich währende meiner gesamten Promotion mit Rat und Tat begleitet hat und mit dem ich mich aufgemacht habe die Tiefen des deep learnings zu erforschen. Tobias stand dabei für endlose Diskussionen über shapes und layers zur Verfügung und ich habe viel von seiner pragmatischen Herangehensweise gelernt. Ich danke auch allen SHKs am LTL, die mich bei Annotationen oder bei der Entwicklung von Prototypen unterstützt haben.

Ich bedanke mich bei Andre Körner, Christoph Schmidt, Dimitar Denev und Heiko Beier, die mich bei Ausflügen in die Praxis gefördert haben und von denen ich viel über Kundenorientierung und workarounds gelernt habe. Mein Dank gilt auch allen UCSM Doktoranden, Postdocs und PIs, die mir interessante Einblicke in andere Forschungsdisziplinen und -kulturen ermöglicht haben. Hervorheben möchte ich die gesamte IWG hate speech, in der ich besonders viel über interdisziplinäres Arbeiten gelernt habe.

Ich bedanke mich auch bei allen Freunden und Kommilitonen, die mir während des Studiums zur Seite standen. Hervorheben möchte ich hier Gerrit Stöckigt, Benjamin Räthel und Jonas Braier, die mich in besonders vielen Kursen und Prüfungen begleitet haben. Bedanken möchte ich mich auch bei allen meinen Freunden aus meiner Schulzeit, die mich immer durch ihre spezielle Art unterstützt haben. Weiterhin bedanke ich mich bei meiner gesamten Familie.

Der größte Dank geht an meine Eltern, die auf so vielfältige Weise zu dieser Dissertation beigetragen haben, dass man diese Beiträge nicht einzeln würdigen kann. Zuletzt möchte ich mich von ganzem Herzen bei Julia bedanken. Für Alles.

I thank the numerous anonymous reviewers that helped to improve my research projects and my submissions. Furthermore, I thank the NLP community for constructive feedback and many interesting conversations. I would also like to thank my international collaborators Nicolás E. Díaz Ferreyra, Oren Melamud, Saif M. Mohammad, and Svetlana Kiritchenko with whom I have carried out research projects and excursions that have been deeply rewarding for me. My special thanks goes to Saif, who made my visit to Ottawa a fascinating experience and from whom I learned a lot about how to do research that is compelling to a large number of people.

Stance can be defined as positively or negatively evaluating persons, things, or ideas (Du Bois, 2007). Understanding the stance that people express through social media has several applications: It allows governments, companies, or other information seekers to gain insights into how people evaluate their ideas or products. Being aware of the stance of others also enables social media users to engage in discussions more efficiently, which may ultimately lead to better collective decisions.

Since the volume of social media posts is too large to be analyzed manually, computeraided methods for understanding stance are necessary. In this thesis, we study three major aspects of such computer-aided methods: (i) abstract formalizations of stance which we can quantify across multiple social media posts, (ii) the creation of suitable datasets that correspond to a certain formalization, and (iii) stance detection systems that can automatically assign stance labels to social media posts.

We examine four different formalizations that differ in how specific the insights and supported use-cases are: Stance on Single Targets defines stance as a tuple consisting of a single target (e.g. Atheism) and a polarity (e.g. being in favor of the target), Stance on Multiple Targets models a polarity expressed towards an overall target and several logically linked targets, and Stance on Nuanced Targets is defined as a stance towards all texts in a given dataset. Moreover, we study Hateful Stance, which models whether a post expresses hatefulness towards a single target (e.g. women or refugees).

Machine learning-based systems require training data that is annotated with stance labels. Since annotated data is not readily available for every formalization, we create our own datasets. On these datasets, we perform quantitative analyses, which provide insights into how reliable the data is, and into how social media users express stance. Our analysis shows that the reliability of datasets is affected by subjective interpretations and by the frequency with which targets occur. Additionally, we show that the perception of hatefulness correlates with the personal stance of the annotators. We conclude that stance annotations are, to a certain extent, subjective and that future attempts on data creation should account for this subjectivity. We present a novel process for creating datasets that contain subjective stances towards nuanced assertions and which provide comprehensive insights into debates on controversial issues.

To investigate the state-of-the-art of stance detection methods, we organized and participated in relevant shared tasks, and conducted experiments on our own datasets. Across all datasets, we find that comparatively simple methods yield a competitive performance. Furthermore, we find that neuronal approaches are competitive, but not clearly superior to more traditional approaches on text classification. We show that approaches based on judgment similarity – the degree to which texts are judged similarly by a large number of people – outperform reference approaches by a large margin. We conclude that judgment similarity is a promising direction to achieve improvements beyond the state-of-the-art in automatic stance detection and related tasks such as sentiment analysis or argument mining.

Stance (dt: Haltung, Position oder Standpunkt) bezeichnet die positive oder negative Evaluation von Personen, Dingen oder Ideen (Du Bois, 2007). Versteht man den Stance, den Menschen in den sozialen Medien zum Ausdruck bringen, eröffnen sich vielfältige Anwendungsmöglichkeiten: Auf der einen Seiten können Regierungen, Unternehmen oder andere Informationssuchende Einblicke darüber gewinnen, wie Menschen ihre Entscheidungen, Ideen oder Produkte bewerten. Auf der anderen Seite können Social Media Nutzer, denen der Stance anderer Nutzer bekannt ist, effizientere Diskussionen führen und letztendlich bessere kollektive Entscheidungen treffen.

Da die Anzahl der in sozialen Medien getätigter Beiträge zu hoch für eine manuelle Analyse ist, sind computergestützte Methoden zum Verständnis von Stance notwendig. In dieser Arbeit untersuchen wir drei Hauptaspekte solcher computergestützten Methoden: (i) abstrakte Stance Formalisierungen, die sich über mehrere Social Media Beiträge hinweg quantifizieren lassen, (ii) die Erstellung geeigneter Datensätze, die einer bestimmten Formalisierung entsprechen, und (iii) automatische Systeme zur Erkennung von Stance, die Social Media Beiträgen ein Stance Label zuordnen können. Wir untersuchen vier verschiedene Formalisierungen, die sich darin unterscheiden, wie spezifisch die Erkenntnisse sind, welche sie bei der Analyse von Social Media Debatten liefern: Stance gegenüber einzelnen Targets*Im Kontext dieser Arbeit lässt sich target mit Zielobjekt oder Gegenstand der Evaluation übersetzen definiert Stance als ein Tupel, welches aus einem einzigen Target (z.B. Atheismus) und einer Polarität (z.B. für oder gegen das Target sein) besteht. Stance gegenüber mehreren Targets modelliert eine Polarität, die gegenüber einem übergeordneten Target und mehreren logisch verknüpften Targets ausgedrückt wird. Stance gegenüber nuancierten Targets, modelliert Stance als eine Polarität gegenüber allen Texten in einem bestimmten Datensatz. Darüber hinaus untersuchen wir hasserfüllten Stance als eine Formalisierung, die modelliert, ob ein Text Hass gegenüber einem einzelnen Target (z.B. Frauen oder Flüchtlingen) ausdrückt.

Systeme, die auf Methoden des maschinellen Lernens basieren, benötigen eine ausreichende Menge von mit Labeln versehenen Trainingsdaten. Da solche Daten nicht für jede Formalisierung verfügbar sind, wurden im Rahmen dieser Arbeit eigene Datensätze erstellt. Auf der Basis dieser Datensätze führen wir quantitative Analysen durch, welche Aufschluss darüber geben, wie zuverlässig die Annotation der Daten ist und in welcher Weise Social Media-Nutzer Stance kommunizieren. Unsere Analyse zeigt, dass die Zuverlässigkeit unserer Daten durch subjektive Interpretationen der Annotatoren und durch die Häufigkeit, mit der bestimmte Targets auftreten, beeinflusst wird. Unsere Studien zeigen weiterhin, dass die Wahrnehmung von Hass mit dem persönlichen Stance der Annotatoren korreliert, woraus wir folgern, dass Stance Annotationen bis zu einem gewissen Grad subjektiv sind und dass diese Subjektivität bei der Datenerstellung zukünftig berücksichtigt werden sollte. Darüber hinaus schlagen wir einen neuartigen Prozess für die Erstellung von Datensätzen vor, die subjektive Annotationen beinhalten, die der Formalisierung Stance gegenüber nuancierten Targets entsprechen und damit umfassende Einblicke in die zugrundeliegende Social Media Debatte liefert.

Um den Stand der Technik der automatischen Stance Erkennung zu untersuchen, haben wir relevante shared tasks organisiert und an ihnen teilgenommen, sowie Experimente an eigenen Datensätzen durchgeführt. Unsere Untersuchungen zeigen über alle Experimente und Datensätze hinweg, dass vergleichsweise einfache Methoden eine äußerst wettbewerbsfähige Leistung erbringen. Des Weiteren zeigen unsere Betrachtungen, dass neuronale Ansätze zwar wettbewerbsfähig, aber nicht deutlich besser als herkömmliche Ansätze zur Textklassifizierung sind. Wir zeigen, dass Ansätze, die auf der Beurteilungsähnlichkeit basieren – definiert als das Ausmaß mit dem Texte von einer großen Anzahl von Menschen ähnlich beurteilt werden – die Leistung von Referenzansätzen deutlich übertreffen. Daraus schließen wir, dass diese Beurteilungsähnlichkeit eine vielversprechende Richtung ist, um weitere Verbesserungen in den Bereichen automatischen Erkennung von Stance und verwandten Aufgaben wie Sentimentanalyse oder Argument Mining zu erzielen.

Stance refers to an evaluation, either positive or negative, which is directed towards a person, thing, or idea and is one of the most basic human cognitions (Du Bois, 2007). Our stance drives us to vote for a certain party or candidate, to buy a certain product, or to avoid or approach people. Knowing a group's stance on an issue is a key prerequisite for meaningful decisions. For example, if a government has to decide, within the democratic decision-making process, whether to invest in wind energy, it is – besides other factors – crucial to know the citizens' stance towards this target.

Commonly, one estimates the stance of groups by conducting interviews, focussed surveys, or voting procedures. However, as these methods are cost- and time-intensive, they do not scale to an arbitrarily high number of topics and people. In addition to the limits posed by their costs, there are several well-known biases, such as the sampling-bias (i.e. if the selected sample is not representative for the whole group), that question the validity of these methods (Rosenthal, 1965; Smart, 1966).

Given that nowadays people are expressing their stances in large quantities on social media sites, social media can provide an alternative source for estimating the stance of groups or the whole society. This source also has the advantage that social media users express their stances out of their own motivation and therefore their statements are not affected by an interviewer or a setting in which a survey takes place. Although the enormous availability of social media allows a wider overview than traditional methods on collecting stance, it also has the drawback that the data volumes are too large to be analyzed manually. Hence, in this thesis we study computer-assisted analysis processes, which can be automated to high degrees and, therefore, can be applied to almost unlimited amounts of data.

Besides the difference in the volumes of data, social media data differs from survey data in the extent to which the data is structured. While survey data consists of multiple choice or at least semi-structured free-text answers, people express themselves freely in social media. This freedom means that our analysis processes need to cope with the variance and the ambiguities inherent to natural language. Below, we illustrate this characteristic of natural language with three statements which express a stance towards wind energy:

• (A) Wind is the least expensive source for generating energy!

• (B) Windmills are cheaper than alternatives.

• (C) Wind is the most expensive source for generating energy!

Examples A) and B) demonstrate that statements can convey a stance in favor of wind energy, while having highly different surface forms (i.e. only the word Wind occurs in both examples). In contrast, Examples A) and C) have highly similar surface forms,*The examples are identical with the exception of the word least in Example A) being changed to most in Example C). but express opposing stances. Furthermore, Example B) can be interpreted in different ways and thus is somewhat ambiguous. This means that the term alternatives could either refer to other energy sources (e.g. coal or solar) or to other forms of harvesting wind energy (e.g. by using a kite).

In order to cope with these characteristics of natural language, we rely on stance detection systems that are based on techniques from the fields of natural language processing and supervised machine learning. These systems have to be trained on a sufficiently large amount of samples to learn a function that maps regularities in the surface forms to a predefined stance formalization. We define stance formalization as an abstract representation of the expressed stance. For instance, in examples A) and B) we could define the symbol {WIND ENERGY, $\oplus$} to represent that both texts express a stance in favor of wind energy. More specifically, we formally define stance as a tuple consisting of (i) a target (e.g. wind energy) and (ii) a stance polarity (e.g. being in favor of ($\oplus$) or being against ($\ominus$) the target).

As our definition leaves the exact nature of target and polarity open, it can be used as a blueprint for more concrete formalizations. In the present thesis, we consider four formalizations that differ in how complex they are and thus in what use-cases they support:

1. Stance on Single Targets: In the least complex formalization, we formalize a stance tuple consisting of a single, predefined target and a three-way polarity (i.e. $\oplus$, $\ominus$, and NONE). For instance, this formalization supports a use-case in which a government is interested in people's overall stance towards issues such as the independence of Catalonia, feminism, or abortion. The formalization also supports decision-makers in a company who want to know if their product is being evaluated positively, negatively or neutrally.

2. Stance on Multiple Targets: In certain use-cases, we are interested in differentiating between more finely grained stances. For instance, in an energy debate, we might be interested in what stance people have towards solar, towards wind, and towards nuclear energy. Hence, in the formalization Stance on Multiple Targets, we simultaneously model a $\oplus$, $\ominus$, or NONE stance towards an overall target and a set of logically linked targets. The relationship between the different, predefined targets can be described by classical semantic relation types such as meronymy, hyperonymy, or hyponymy.

3. Stance on Nuanced Targets: If our use-case is to obtain a comprehensive understanding of all nuances of stance towards a topic or if we do not have knowledge of the most relevant aspects of a topic, it is not viable to predefine a set of logically linked targets. Hence, for the formalization Stance on Nuanced Targets, we formalize $\oplus$ or $\ominus$ stance towards all texts in the dataset we want to analyze.

4. Hateful Stance: Stance formalizations cannot only be distinguished in terms of the granularity of the targets, but also in terms of different polarities. For the fourth formalization, we study Hateful Stance as an especially unpleasant form of stance taking. We define hateful stance analogously to Stance on Single Targets, but define a Hatefulness Polarity instead of the regular polarity. We argue that this formalization can help to contain hateful stance and the negative consequences it has on its targets.

We provide an abstract example for each of the three formalizations Stance on Single Targets, Stance on Multiple Targets, and Stance on Nuanced Targets in Subfigure formalization of Figure1.1. Note that we do not show an example for Hateful Stance as this formalization differs from the formalization Stance on Single Targets only by the use of a different polarity.

Before we can train systems that automatically assigns the described formalisms to social media text, we need a sufficient amount of training examples. This training data needs exactly those labels that correspond to the formalization our systems should recognize, when applied to new data. Since data is not available for all of our formalizations, we manually created datasets ourselves. While our individual efforts on data creation differ in several details, they all incorporate the three following steps: We (i) first gather raw data relevant to the issue we want to explore, (ii) let several annotators independently annotate a stance formalization, and (iii) consolidate the different annotations into a final gold label. This process is visualized in the Subfigure data creation of Figure 1.1.

The data creation process allows us to perform a number of quantitative analyses that help us to understand our stance formalizations in more detail: More specifically, we analyze the degree to which multiple annotators agree on the annotation of a formalism (inter-rater reliability), which indicates how subjective the annotations are (Artstein and Poesio, 2008). Subjective annotations pose a serious problem for automated systems that are trained on this data, as the resulting predictions will correspond to the subjective annotations. A third person may not agree with such a subjective prediction, which renders such a system useless.

We develop various metrics that provide us with empirical insights in how social media users express stances. These include scores that quantify how polarizing statements or issues are, or scores that describe the degree to which a text is similarly judged by a large number of people. Furthermore, we study the frequency with which individual elements of the formalizations occur, or the frequency with which several elements co-occur.

After the creation of datasets, we can attempt to train stance detection systems. All of our stance detection systems correspond to prototypical text classification systems. Thus, as shown in Subfigure detection systems in Figure 1.1, they all involve the three steps preprocessing, machine learning, and evaluation. First, while preprocessing, the data must be translated into a form that a computer can digest. Preprocessing includes a segmentation of the data (e.g. splitting a text into words) and a vectorization of it. For vectorization, we represent raw data by a set of measurable properties. For instance, we represent text as a sequence of word vectors (cf. Mikolov et al. (2013b)) or by averaging the polarity scores of all words (i.e. how positive or negative a word is perceived (Mohammad and Turney, 2010)) that are contained in the text.

Next, we use machine learning algorithms to learn a function that maps the vectorized data to a stance label. To investigate which type of machine learning is best suited for stance detection, we experiment with a set of algorithms that can be roughly categorized into (a) neural networks (often referred to as deep learning (Glorot and Bengio, 2010)) and (b) more traditional learning algorithms such as SVMs.

Finally, we perform evaluations that measure the quality of the entire systems and their individual components. For these evaluations, we test the performance of our predictions on data, which was not part of the training process, but which is annotated with the same stance labels. In order to compare systems under fair conditions, we participated in and organized shared task initiatives. Shared tasks refer to comparative evaluation set-ups in which organizers make sure that all participating systems are evaluated under the same conditions (e.g all systems are evaluated using the same performance metrics). Furthermore, we conduct controlled experiments in which we evaluate the effectiveness of individual components. For instance, we test how the overall systems' performance changes if we assume a perfect performance of upstream components.

Now, we describe the main contributions of this thesis. We group these contributions into our three main research areas Formalization, Data Creation, and Detection Systems:

Formalization

We provide a comprehensive overview of stance formalisms and compare stance with related NLP concepts such as sentiment and arguments. Thereby, we carve out the conceptual similarities of the NLP concepts target-based sentiment, topic-based sentiment, aspect-based sentiment, and stance. We argue that these theoretical considerations pave the way for combining future efforts in these areas.

Our theoretical considerations indicate that existing stance formalisms may be too coarse-grained to provide a comprehensive overview of all nuances of a debate.To close this gap, we propose a new stance formalization, which we call stance on nuanced targets and which models stance towards all texts in a dataset.

In addition, we describe how our formalization of stance can be adapted to model hatefulness that is expressed towards targets (e.g. refugees or women). In this way, we contribute to consolidating the research of new NLP tasks such as aggression detection or hate speech detection into a unified field.

Data Creation

For studying the formalization stance on multiple targets, we created two novel datasets and made them available to the research community – one contains tweets about atheism and the other contains Youtube comments about the death penalty. We show that people do not always agree on their interpretation of which stance a text expresses towards multiple targets. As a major reason for disagreement, we identify the ambiguity and distribution of the predefined targets. Our study suggests that good targets should be selected according to these criteria.

Moreover, we propose a novel process of collecting data to study the formalization stance on nuanced targets. Our process consists of two steps which can be carried out via crowdsourcing: First, we engage people to generate a comprehensive dataset of assertions relevant to certain issues. Second, we ask people what stance they have towards the collected data assertions (i.e. whether they personally agree or disagree with the assertions). The resulting dataset covers sixteen different issues (e.g. vegetarianism and veganism, legalization of Marijuana, or mandatory vaccination), contains over 2000 assertions and about 70,000 stance annotations. We show how this data can be analyzed in order to obtain novel insights into the nature of the corresponding debate.

We also examined our formalization of hateful stance using two datasets. Specifically, we were involved in the creation of one of the first German datasets that contains hatefulness labels. In addition, we created a dataset which is annotated with both stance on nuanced targets and hateful stance towards women. Our research shows that reliably annotating hateful stance is even more challenging than the annotation of stance on multiple targets. Furthermore, we show that whether an annotator perceives a text as hateful is influenced by whether the text is phrased implicitly and by what stance the annotator has towards the text. More specifically, we show that annotators that agree with a text are unlikely to perceive the text as hateful, or vice versa, annotators that perceive a text as hateful are unlikely to agree with it.

Detection Systems

We participated in the first shared tasks on stance detection (i.e. SemEval 2016 Task 6 and StanceCat@IberEval 2017). In addition, we co-organized a shared task that focussed on customer reviews about Deutsche Bahn - the German public train operator. With our involvement in these initiatives, we have actively contributed to determining the current state of the art in the area of single-target stance detection. Across different initiatives, we find that comparatively simple systems yield highly competitive results, but that the usage of word polarity lists, dense word vectors, and ensemble learning consistently leads to improvements over baseline systems. In addition, we could demonstrate that the state of the art in calculating word-relatedness does not sufficiently model the lexical semantics that are required for stance detection.

In our datasets that are annotated with stance on multiple targets, we show that overall stance classifiers already model stance on explicit targets to a large extent. Furthermore, we demonstrate that the overall stance can be quite reliably predicted if specific targets are given. This means that future attempts on stance detection could potentially reuse models that have been trained to predict stance on targets that correspond to the more specific targets.

In the context of stance on nuanced targets, we propose two novel NLP tasks: predicting the stance of individuals and predicting the stance of groups. For solving these tasks, we propose to rely on an automatically estimated judgment similarity – the degree to which two assertions are judged similarly by a large number of people. We show that judgment similarity can be quite reliably estimated from texts and is a useful means for solving the proposed tasks. For the prediction of stance of groups our approach based on judgment similarity outperforms reference approaches by a wide margin. Furthermore, we demonstrate that approaches that are based on judgment similarity substantially outperform competitive baselines in the task of predicting hateful stance.

The research presented in this thesis was partly published in peer-reviewed conference and workshop proceedings and, thereby, made accessible to the research community. We will now report and briefly summarize these publications.

We have contributed to determine the current state of the art in detecting Stance on Single Targets by participating in and organizating shared task initiatives on the topic. Our participation in the first shared task on stance detection (i.e. SemEval 2016 Task 6 by Mohammad et al.) is described in the publication:

Michael Wojatzki and Torsten Zesch. 2016a. ltl.uni-due at SemEval-2016 Task 6: Stance Detection in Social Media Using Stacked Classifiers. In Proceedings of the International Workshop on Semantic Evaluation (SemEval), San Diego, USA, pages 428–433.

In our submission, we experiment with a combination of neural and non-neural classifiers. While our system does not outperform competitive baselines, we could demonstrate that the approach has the potential for significant performance gains. Furthermore, we co-organized a shared task that focussed on social media customer feedback which targets Deutsche Bahn. We report the findings of this shared task in the publication:

Michael Wojatzki and Torsten Zesch. 2017. Neural, Non-neural and Hybrid Stance Detection in Tweets on Catalan Independence. In Proceedings of the Workshop on Evaluation of Human Language Technologies for Iberian Languages (IBEREVAL), Murcia, Spain, pages 178–184.

Our analysis shows that comparatively simple systems yield competitive results, but we also identify methods that consistently lead to improvements over these baselines.

We investigate Stance on Multiple Targets by analyzing two newly created datasets: First, we examine the relationship between overall and specific stances in a dataset about atheism. We find that overall stance can be quite reliably predicted from more specific stances. The creation and analysis of this dataset is described in the publication:

Michael Wojatzki and Torsten Zesch. 2016b. Stance-based Argument Mining: Modeling Implicit Argumentation Using Stance. In Proceedings of the Conference on Natural Language Processing (KONVENS), Bochum, Germany, pages 313–322.

Subsequently, we compared two different sets of specific targets in a dataset about the death penalty. We could not conclude that one set is clearly preferable over another, but that they complement each other well. The dataset along with the experiments are described in the publication:

Michael Wojatzki and Torsten Zesch. 2018. Comparing Target Sets for Stance Detection: A Case Study on YouTube Comments on Death Penalty. In Proceedings of the Conference on Natural Language Processing (KONVENS), Vienna, Austria, pages 69–79.

In certain use-cases, existing stance formlizations may be too coarse-grained to obtain a comprehensive overview of all nuances of a debate. Hence, we propose a new formalization of stance (Stance on Nuanced Targets), which models stance on an even more fine-grained level. To obtain data that is annotated with corresponding labels, we propose a new process for collecting data. We described this process, the resulting data, as well as the analysis of the data in the publication:

Michael Wojatzki, Saif M. Mohammad, Torsten Zesch, and Svetlana Kiritchenko. 2018b. Quantifying Qualitative Data for Understanding Controversial Issues. In Proceedings of the Language Resources and Evaluation Conference (LREC), Miyazaki, Japan, pages 1405–1418.

Next, we describe two novel NLP tasks that we can try to solve on the basis of the collected datasets. In addition, we describe how judgment similarity can be used to solve these tasks. The two tasks, as well as our approach based on judgment similarity, are described in the publication:

Michael Wojatzki, Torsten Zesch, Saif M. Mohammad, and Svetlana Kiritchenko. 2018c. Agree or Disagree: Predicting Judgments on Nuanced Assertions. In Proceedings of the Conference on Lexical and Computational Semantics (*SEM 2018), New Orleans, USA, pages 214–224.

Subsequently, we examine the formalization Hateful Stance. To that end, we investigate the reliability of hatefulness annotation in a dataset about the European refugee crisis. We find that the annotations are indeed influenced by subjective interpretations. This examination is described in the publication:

Björn Ross, Michael Rist, Guillermo Carbonell, Benjamin Cabrera, Nils Kurowsky, and Michael Wojatzki. 2016. Measuring the reliability of hate speech annotations: the case of the European refugee crisis. In Proceedings of the Workshop on Natural Language Processing for Computer-Mediated Communication (NLP4CMC), Bochum, Germany, pages 6–9. (All authors of this publication were equally involved in the research.)

Next, we investigate whether the implicitness of texts has an influence on how hateful they are perceived to be. The experiment indicates that implicitness has an influence on the perception of hatefulness but that this influence is moderated by other factors (e.g. by whether a text contains a threat). We report his experiment in the publication:

Darina Benikova, Michael Wojatzki, and Torsten Zesch. 2017. What Does This Imply?: Examining the Impact of Implicitness on the Perception of Hate Speech. In Proceedings of the International Conference of the German Society for Computational Linguistics and Language Technology (GSCL), Berlin, Germany, pages 171–179. (The author of this thesis developed the research design, performed the quantitative analyses, and implemented and evaluated the machine learning approaches).

Finally, we create and analyze a dataset that is annotated with both stance on nuanced targets and hateful stance. We also find a strong relationship between hateful stance and stance on nuanced targets, and demonstrate how this relationship can be used for predicting hatefulness. These investigations, as well as approaches on automatically detecting hateful stance, are described in the publication:

Michael Wojatzki, Tobias Horsmann, Darina Benikova, and Torsten Zesch. 2018a. Do Women Perceive Hate Differently: Examining the Relationship Between Hate Speech, Gender, and Agreement Judgments. In Proceedings of the Conference on Natural Language Processing (KONVENS), Vienna, Austria, pages 110–120.

In the following, we provide an overview of the structure of this thesis. In Chapter 2, we first give an overview of our formalization of stance and related NLP concepts such as sentiment or arguments. Hereby, we show that all stance formalizations can be defined as tuples consisting of targets and a polarity that is expressed towards that target.

Stance on Single Targets

In Chapter3, we examine the simplest formalization of stance in which we define stance towards a single target. First, we give a brief overview of NLP and machine learning systems, which represent the state of the art in this area. Subsequently, we describe our participation and organization in three shared task initiatives on the topic. We also show that the targets used in these tasks require different lexical semantics as they are measured by state of the art methods.

Stance on Multiple Targets

In Chapter 4, we study a more fine-grained formalization that models stance towards multiple, logically-linked targets. Since there was no suitable data available for our investigations, we created our own dataset. To that end, we first introduce the basics of annotating and evaluating NLP datasets in this chapter. Subsequently, we describe how we create and analyze two different datasets: one about atheism and one about the death penalty.

Stance on Nuanced Targets

In Chapter 5, we propose a new formalization of stance, which models stance on an even more fine-grained level. We introduce a novel process of collecting data that corresponds to this new formalization of stance. Subsequently, we show how the data can be analyzed to obtain novel insights into controversial issues. In addition, we define wo novel tasks that can be evaluated on the collected data, and show how judgment similarity can be used to solve these tasks.

Hateful Stance

In Chapter 6, we adapt our stance formalization stance on single targets to capture hateful expressions which target, for instance, refugees or women. First, we describe how this formalization relates to concepts such as hate speech and provide an overview of the current state of the art in detecting hateful stance. Subsequently, we describe two annotation experiments to determine whether experts differ from laypersons in the annotation of hatefulness and to determine the influence of implicitness. Next, we describe how we create a dataset that contains annotations that correspond to hateful stance and to stance on nuanced targets. Finally, we analyze this dataset and demonstrate how judgment similarity can be utilized to predict hateful stance.

There are multiple ways to express stance using natural language. For example, if one favors wind energy in a debate on how the society should generate electricity, one could express the exact same stance in various ways:

• (A) We should use wind energy to produce electricity!

• (B) I prefer wind energy over other options.

• (C) Wind energy is the only sane option.

If we face several statements and our task is to generate quantitative insights about the expressed stances, we must be able to determine whether the different statements express the same stance or not. Therefore, we need abstract formalizations of stance that can be measured and quantified across all utterances. For the utterances above, we could – for example – introduce a symbol {WIND ENERGY, $\oplus$} and use it to represent the expressed stances.

This process of enhancing (textual) data with such formalizations is called annotation (Bird and Liberman, 2001; Ide and Romary, 2004). In this chapter, we will shed light on different approaches on formalizing stance, and how these formalizations relate to similar concepts such as sentiment or arguments.

Stance as a Target–Polarity Tuple

The meaning of a message expressed in natural language can change dramatically due to the context.For instance, in a debate on healthy food with the expression Apples are awesome!, the author communicates that she likes the fruit called apple. However, in a debate that targets computer brands, the same utterance may express that the author favors products of the brand Apple. If the debate is specifically concerned with the pros and cons of PCs, one could even conclude that the author dislikes PCs. To capture this ambiguity, we rely on a formalization of stance that is defined towards a target (cf. Mohammad et al. (2016)). Hence, we define stance as a tuple consisting of:

A) a polarity of the stance that is being expressed. The polarity expresses how the target is evaluated. For instance, one can be in favor of $\oplus$ or against $\ominus$ a target.

B) a target towards which the stance is expressed. A target can theoretically be anything that one can favor or oppose, or what can be the subject of a debate. Examples for targets include controversial social issues (e.g. atheism or gay rights), persons of public interest (e.g. politicians), or commercial products and services (e.g. electronic devices). A target can also be a facet of an issue or even a single statement.

Figure 2.1 exemplifies these two basic building blocks of stance formalisms. The figure also shows how the same message communicates two different stances depending on two targets.

We argue that systems that are able to automatically detect stance are useful in a variety of applications such as sorting and filtering transcribed speeches according to the taken stance or identifying newspaper articles that support or oppose a certain position. An area where such systems could be especially useful is the social media domain as it is especially rich in opinionated and thus stance-bearing texts. We here understand social media data as any kind of user-generated content which is created and shared via internet-based applications (Kaplan and Haenlein, 2010). These applications include websites for networking (e.g. facebook.com or researchgate.net), micro-blogging (e.g. twitter.com, tumblr.com), as well as platforms for sharing media (e.g. youtube.com, instagram.com and reddit.com) (Barbier et al., 2013, p.1). Besides user-generated content, social media data also comprises the attributes of users and informations on user-user, and user-content connections (Barbier et al., 2013, p.5).

The need for automatizing social media stance detection is a consequence of the quickly growing amount of social media content. While contained stances may provide valuable insights into public opinion, social dynamics, or customer groups, the amount of social media data makes manual processing hardly possible. Social media users share different types of content. However, when expressing a stance, they still mostly rely on text.

Thus, in this work we cast stance detection as inferring stance from text. The computer science branch that is concerned with automatically processing texts is called natural language processing (NLP).*Other common names for the field are computational linguistics, linguistic computing or language technology. Although each name may emphasize a different focus of research, there is no universally recognized distinction between them and they are often used interchangeable. We here decided to use the term NLP to emphasizes the empirical nature of the conducted research.

We will now describe how the task of stance detection is affected by almost all levels of NLP and discuss the challenges that the social media domain poses along these levels. The purpose of this short description is, on the one hand, to more precisely outline the problem of detecting stance from language data and, on the other hand, to define the terminology for the following chapters. Subsequent to this description, we will introduce the concept of annotation complexity, which will be used to classify and discuss related work on stance detection.

Traditionally, the challenges of natural language processing are described using a hierarchy of levels. These levels are phonology and orthography, morphology, syntax, semantic, pragmatic and discourse (cf. (Jurafsky and Martin, 2000, p.38)) We will now briefly outline these levels, describe how they relate to the automatic detection of stance, and discuss the challenges that arise when working on social media data.

Phonologic and Orthographical Level

The first level deals with translating a language input – written or spoken – into a form that is processable by a computer. On the phonological level, systems process the purely acoustic signal of language (i.e. sound waves). Typical tasks are automatic speech recognition, in which one tries to map sound waves to text, or speech synthesis which is targeted towards mapping text to acoustic units. As the goal of this work is to analyze social media – in which the vast mass of data is present in written form – we neglect the phonological level. On the orthographical level, systems automatically transcribe the text that can be found on scans, photographies or other pictures into digital text (so-called optical character recognition). Again, we rely on the fact that most of the language in social media is already in a machine-readable form and neglect this step.

Morphological Level

The morphological level is concerned with analyzing meaningful units within the input. These units include words or morphemes – the smallest meaningful or grammatically functional units of linguistics. A word may be composed of several morphemes (e.g. windmill is composed of wind and mill. Tasks on the morphological level include tokenization of input strings into morphemes or words, or normalizations of words to their base forms (called lemmas). The morphological level is important for stance detection as computers need to be able to recognize words in order to decode their semantics in downstream NLP steps, or to handle inflections. For instance, for stance detection it may be necessary to recognize that plural and singular forms (mills vs. mill) point to similar concepts. Spelling mistakes (e.g. omission of whitespaces) and non-standard use of text (e.g. usage of emoticons or urls) often make the tasks on the morphological level difficult on social media data – our main field of application. For instance, Derczynski et al. (2013) shows that the performance of off-the-shelf tokenizers dramatically decreases when applied to Twitter data.

Syntactical Level

Syntax refers to the rule system with which elementary units (e.g words or phrases) are combined (e.g. into sentences) in natural language. Consequently, the syntactical level is concerned with matching these rules to the extracted stream of tokens. Typical tasks include the tagging of tokens with word classes (so-called part-of-speech tags), the detection of grammatical dependencies. Syntax is important for understanding stance of messages as one, for instance, needs to distinguish between a statement (Apples are awesome) and its negated form (Apples are not awesome). In addition, in an statement such as I hate that bloated, significantly-worse-than-Clinton, Trump. syntactic knowledge is required to understand that the stance (indicated by the term hate) is expressed towards Trump and not towards Hilary Clinton. The non-standard nature of social media text may negatively affect the performance of tools that are designed to solve syntactic tasks. Examples for NLP tasks that can be solved less well on social media data than on standard texts are part-of-speech tagging (Gimpel et al., 2011; Ritter et al., 2011; Horsmann and Zesch, 2016) and dependency parsing (Foster et al., 2011).

Semantic Level

The semantic level is concerned with the meaning of words, morphemes, sentences or other previously extracted structures. Typically, semantic tasks try to measure the relationship (e.g. similarity) between the meaning representation of these units. In order to solve these task, it is often necessary to first resolve the ambiguity inherent in language (e.g. given the context, does the word apples refer to the fruit or the brand?). Stance detection can therefore be regarded as a highly semantic task. Similar to the underlying levels, the performance of tools that solve semantic tasks is often significantly lower on social media data compared to standard text (Ritter et al., 2011; Li et al., 2012).

Discourse Level

The discourse level is concerned with understanding the sequence of multiple messages of several speakers or authors. The discourse level can have a great impact on the perception and understanding of stance.Often, the meaning of utterances can only be understood in the context of utterances of other persons to which they refer. For instance, for understanding the stance that is communicated with I hate them! it is necessary to know if the utterance refers to a preceding one. A characteristic of social media is that different users can interact with each other. As a consequence, social media data contains plenty of especially complex discourses (Forsythand and Martell, 2007).

Pragmatic Level

The pragmatic level incorporates the situational context in which an utterance is made. This context can be of personal or social nature, or or shaped by the context of the current debate. Since stance is defined with respect to a given target, stance can be considered as a highly pragmatic task. Depending on the target, the semantic orientation (whether a word evokes good or bad associations) of the words may also change (Turney, 2002). For instance, the word unpredictable may convey a positive association if the target is a certain book or movie. However, if the target is a car, the word unpredictable induces a negative association. We also need pragmatic knowledge to know that the phrase maximally bloated expresses a negative stance towards Trump, but a positive stance if used to express a stance towards a inflatable boat.

The resolution of implicit expressions is also typically assigned to the pragmatic level. As shown in Figure 2.1, stance can be implicitly expressed by arguing for an entity that is negatively correlated with the target (e.g. arguing for windmills to express a $\ominus$ stance towards nuclear power). Hence, the resolution of implicit expressions und thus the pragmatic level is crucial for stance detection. Since social media data usually comes with rich context, automatic text processing can rely on such signals to improve performance (González-Ibánez et al., 2011).

In the following, we discuss different ways of formalizing stance and how these formalizations differ them from formalizms used in related NLP tasks. However, in the present thesis we do not consider all possible theoretical formalizations, but only those which have been applied to data in the form of concrete annotation schemes.

We examine stance and related formalizations according to the building blocks shown above (i.e. polarity, and targets). To further characterize them within these blocks, we analyze the complexity of the different annotation schemes. According to the international standard for linguistic annotation (Ide and Romary, 2004)*ISO TC37 SC WG1-1 every linguistic annotation consists of the two steps (i) segmentation and (ii) annotation of these segments with linguistic or otherwise informative classes. Thus, we consider the complexity of an annotation as a function of how complex the process of identifying the elementary units of the annotation scheme is (e.g. finding words is less complex than finding syntactic dependencies) and how complex the assignment of classes is (e.g. choosing whether a tweet is positive or negative is less complex than choosing if it is very positive, positive, neutral, negative, or very neutral). The segmentation complexity includes whether or not we can reliably utilize heuristics such as a separation according to white spaces (Fort et al., 2012). The complexity of assignment classes includes how large the set of classes is and how ambiguous the assignments are (Fort et al., 2012). If an annotation scheme is more complex one can distinguish a larger number of different cases. Thus, an annotation scheme's complexity is directly related to its expressiveness. However, complexity is also negatively correlated with inter-annotator agreement (Bayerl and Paul, 2011), as e.g. a more complex scheme results in more errors and thus a lower reliability. The intuition behind this negative correlation is that the more possibilities one has, the more errors are possible. Of course reliability is also effected various other factors. The factors include training of the annotators, properties of the data (e.g. social media vs. newswire text), how implicit the communication is and how much interpretation is required. Note, that we do not compare the reliability of annotations directly, as it is difficult to compare different reliability scores with each other and reliability scores do not account for whether the annotators are trained.

Of course, the complexity is also influenced by the syntactic, semantic and pragmatic context in which the annotation is done (Fort et al., 2012; Joshi et al., 2014). The complexity of annotations can be be quantified by relying on behavioural cues such as those which are provided from eye-tracking (Joshi et al., 2014; Tomanek et al., 2010). Since we have neither behavioral cues for the annotations nor the full context of all approaches, we will here analytically compare the different annotation schemes based on numeric properties such as the number of classes to assign. Our analytical approach to determining complexity neglects the complexity of the thought processes that are needed for both segmentation and class assignment (e.g. it is easily perceivable that annotating whether a sentence is intelligently formulated is much harder than annotating whether a sentence contains question marks). Nevertheless, in order to be able to compare approaches on measurable criteria, we will build our analysis on the number of possible choices.

As a first building block, we discuss stance polarity. As described above, stance polarity refers to the judgment the stance expresses towards the given target. A stance annotation scheme has to define which values (or classes of values) the polarity can adopt. The number of possible values directly influences the complexity of an annotation scheme. For instance, if we only allow for the two classes $\oplus$ and $\ominus$, unarguably, the scheme is less complex compared to a model that also uses a third NONE class. We will now describe the polarity classes that are used in stance research and then describe the greater variety of polarities, which can be found in the field of text polarity classification.

With respect to polarity, we can distinguish stance models based on whether they assume that stance is binary (FOR or AGAINST) or if they also model that messages sometimes do not contain any stance. This distinction also corresponds to whether stance is modelled in data originating from dedicated debating scenarios – in which all participants consciously participate – or in unconstrained debates as they are conducted in social networks. The former works typically formalize stance that is expressed on debate websites that try to enforce structured debates by asking users to label their message as having a FOR or AGAINST stance on the target of the debate (Somasundaran and Wiebe, 2010; Anand et al., 2011; Walker et al., 2012a; Hasan and Ng, 2013; Sridhar et al., 2014). In such constrained debates, there are hardly any contributions that do not relate to the target. Consequently, these works model stance as a binary phenomenon.

Binary annotation schemes that model stance expressed towards another text (which also can be seen as a target, cf. Section 2.3) commonly rely on the classes AGREE and DISAGREE. For instance, Menini and Tonelli (2016) model whether two argumentative texts agree or disagree with each other. Wang and Cardie (2014) and Celli et al. (2016) annotate whether a reply in a discussion forum agrees or disagrees with the post it replies to.

Binary formalisms can also be found in data that comes from constraint debates that occur in the offline world. For instance, Thomas et al. (2006) annotated whether the speeches in congressional floor debates SUPPORT or OPPOSE a given piece of legislation. Similarly, Faulkner (2014) labeled essays with the classes FOR and AGAINST a given target.

In contrast to these works on structured debates, there are approaches that model stance in unconstrained social media debates. The goal of these approaches is to structure a collection of e.g. posts on Twitter. By nature, such a collection of tweets contains data that does not carry any stance or where the stance cannot be inferred. Thus, these works consider an additional NONE class, which allows us to filter out these tweets that do not contribute to the debate. NONE class was first introduced by the SemEval 2016 task 6 Detecting Stance in Tweets (Mohammad et al., 2016). This model was adopted by following shared tasks, namely the NLPCC Task 4 on stance detection in Chinese microblogs (Xu et al., 2016)and the IberEval shared tasks on stance in tweets on Catalan independence (Taulé et al., 2017, 2018). The same polarity classes are also used in the annotations by Sobhani et al. (2017). Vychegzhanin and Kotelnikov (2017) additionally introduce a class CONFLICT depicting if a social media message contains both $\oplus$ and $\ominus$ stances.

To the best of our knowledge, there is only a limited amount of work on more fine grained stance classification. An example for a more finely grained polarities is the work by Persing and Ng (2016b) that uses the classes AGGREE STRONGLY, AGREE SOMEWHAT, NEUTRAL, DISAGREE SOMEWHAT, DISAGREE STRONGLY, and NEVER ADRESSED for labelling student essays. For consecutive texts, SemEval 2017 Task 8 A models a special notion of stance that aims to detect fake news by using the classes SUPPORT, DENY, QUERY, COMMENT (Derczynski et al., 2017). Such finely grained polarities, however, can be frequently found in research on text polarity.

Text polarity detection refers to approaches that try to measure the polarity – the emotional tone or overall sentiment – that is conveyed by a text. The used polarity distinctions range from binary (POSITIVE vs. NEGATIVE) labels (Hu and Liu, 2004), over psychologically-motivated sets such as the six basic emotions*anger, sadness, happiness, surprise, disgust, and fear (Ekman et al., 1969)(Mohammad and Turney, 2010), to complex linguistically motivated sets as modelled in WordNet (Strapparava et al., 2004).

Whether a whole text conveys a positive, negative, or a neutral sentiment is intensively studied by the Sentiment Analysis in Twitter series of shared tasks that have been conducted within the SemEval framework. So far, the format generated several challenges that all used the same set of polarity classes (positive, negative or neutral): SemEval 2013 task 2b (Nakov et al., 2013), SemEval 2014 task 9b Rosenthal et al. (2014), SemEval 2015 task 10b (Rosenthal and McKeown, 2012), SemEval 2016 task 4 a (Nakov et al., 2016), and SemEval-2017 task 4 a (Rosenthal et al., 2017). The same set of polarity classes (POSITIVE, NEGATIVE or NEUTRAL) is used by the TASS 2017 initiative for Spanish tweets (Martınez-Cámara et al., 2017) or by the SAIL initiative for tweets written in the Indian Languages Bengali, Hindi, and Tamil (Martınez-Cámara et al., 2017). Other shared tasks such as the Italien SENTIPOLC (Valerio et al., 2014; Barbieri et al., 2016) or the French DEFT (Benamara et al., 2017) do not require that the labels POSITIVE, NEGATIVE or NEUTRAL occur exclusively by allowing a MIXED category.

A more fine-grained sentiment score and thus a more complex annotation scheme is used in the shared tasks TASS 2016, 2015, 2014 and 2013 on detection sentiment on Spanish tweets (Garcıa-Cumbreras et al., 2016; Villena-Román et al., 2015, 2014a,b). In detail, they used a six point scale (STRONG POSITIVE, POSITIVE, NEUTRAL, NEGATIVE, STRONG NEGATIVE and one additional NO SENTIMENT class). Similarly, Semeval 2015 task 11 uses an eleven-point scale ranging from -5 (disgust) to +5 for (extreme pleasure) (Ghosh et al., 2015). A special set of emotions has been used by Pestian et al. (2012) to study the emotional tone of suicide notes (i.e. ABUSE, ANGER, BLAME, FEAR, GUILT, HOPELESSNESS, SORROW, FORGIVENESS, HAPPINESS, PEACEFULNESS, HOPEFULNESS, LOVE, PRIDE, THANKFULNESS, INSTRUCTIONS, and INFORMATION).

In addition to a fixed number of classes, there are also formalizations that express the intensity of emotions through real valued scores. A continuous scale allows for more fine grained gradations and thus introduces a higher complexity than a categorical scale. Examples for text polarity annotations on continuous scales include SemEval 2007 task 14 (Strapparava and Mihalcea, 2007), WASSA-2017 Shared Task on Emotion Intensity (Mohammad and Bravo-Marquez, 2017), and SemEval 2018 task 1 (Mohammad et al., 2018), which quantify the strength of expressed emotions (e.g. JOY, FEAR, SURPRISE) and polarity orientation (from MOST POSITIVE to MOST NEGATIVE).

In addition to the polarity of cohesive texts, there is also a large body of research on the polarity of single, isolated words. Studying the sentiment or emotional tone of words is closely related to the task of creating and analyzing sentiment lexicons. Efforts on lexicon creation often consider the words in isolation. Practical guidelines often explicitly state that annotators should try to avoid their personal context by imagining how people at large would judge the sentiment of a word (Mohammad, 2016). However, there are approaches on annotating words for being positive, negative, both, or neutral in the context of the sentence in which they occur. For example, the MPQA Subjectivity lexicon is created from the annotations of words in the MPQA corpus (Wilson et al., 2005).

Binary and therefore less complex distinctions between positive and negative opinion words are compiled by Stone et al. (1962) and Liu (2004). The same binary complexity is used by approaches that try to automatically increase the amount of hand-labeled words. These approaches try to infer the sentiment of a word from the context in which it occurs in a corpus. For instance, Turney (2002) considers the collocations to seed words such as EXCELLENT and POOR. Similarly, Hatzivassiloglou and McKeown (1997) compile a list of positive or negative adjectives and then expanded this list by assuming that adjectives that are in conjunction (e.g. and) have the same polarity. There are also approaches on creating binary lexicons that reflect domain-specific sentiment (e.g. bullish or bearish} in the financial domain (Das and Chen, 2007)).

Furthermore, there are approaches on creating lexicons that try to model a more fine-grained polarity. These approaches often rely on interviewing groups through offline or online surveys. For instance, researchers have collected fine-grained ratings for words for the dimensions POSITIVE or NEGATIVE (Mohammad et al., 2013), VALENCE, AROUSAL and DOMINANCE (Bradley and Lang, 1999), as well as PLEASANTNESS, ACTIVATION, and IMAGERY (how well a word brings an image to one's mind)(Whissell, 1989).

Finally, there are approaches on estimating fine-grained word polarity that make use of ontologies that model the meaning of words. Lexicons that can be generated from these ontologies map words to polarity scores between -1 and 1 (Cambria et al., 2010; Gatti et al., 2015) or to numerical scores describing how OBJECTIVE, POSITIVE, and NEGATIVE the words are (Esuli and Sebastiani, 2006).

Targets are the subject of the expressed stance. Accordingly, a target can be anything that can be evaluated or judged. Targets can be political, ethical, or social issues (e.g. gay rights), people or groups (e.g. Hillary Clinton or a political party), or concrete objects (e.g. a city, a product, or a brand). In addition, one can also formalize a stance towards any kind of statement (e.g. one can be against or in favor of a statement such as We should use windmills to generate electricity). In fact, if one compares the stance tuples { windmills,$\oplus$} and {We should use windmills to generate electricity,$\oplus$}, the distinction between statements and issues is blurry, since in a debate on electricity sources these stances are highly correlated.

If we have a formalism that includes a NONE polarity class, one can theoretically derive a stance for every possible target from every possible statement (even if the stance will mostly have a NONE polarity). Since one has to repeat the same annotation process for each given target, the complexity of the schema increases with each additional target. As there is an infinite number of possible targets (given by the infinite number of topics and statements), stance would have an infinitely high complexity. Therefore, due to practical reasons, approaches are limited to fixed amounts of targets.

In this section, we will look at stance detection, target-based and topic-based sentiment analysis, aspect-based sentiment analysis, and argument mining. These formalizations all have in common that they formalize evaluations based on a fixed set of targets. We will discuss the complexity of these approaches based on the size, structure, and origin of these sets. Subsequently, we will take a look at the topical domains of these targets and argue that the discussed approaches are all highly similar and that their different names rather reflect different research traditions.

We distinguish between the three stages of set complexity: (i) single targets (2) multiple targets, and (iii) nuanced targets.

The least complex form of target sets is that only one stance towards one target is annotated per data instance (i.e. the set contains exactly one annotation per element). In many applications, this low complexity is sufficient. Modelling stance towards a single target corresponds to voting situations (e.g. referendums on social issues), in which one also expresses a pro, contra, or neutral stance towards a single target.

Most previously discussed approaches on stance detection fall into this category (e.g. the shared tasks by Mohammad et al. (2016), Taulé et al. (2017), or Xu et al. (2016)). Therefore, it is irrelevant that most of the datasets cover several targets (e.g. atheism, abortion}, climate change, Hillary Clinton, and feminism in the dataset by Mohammad et al. (2016)), since each instance is annotated with stance towards only one target. We argue that these datasets are actually a collection of several sub-datasets (i.e. one atheism stance dataset, one abortion stance dataset, etc.). Approaches that formalize a binary stance in constraint debates (e.g. Anand et al. (2011) or Walker et al. (2012a)) can also be clearly classified as single target approaches, as their target is also defined by the constraints of the debate.

Target and Topic-based Sentiment

There are other annotation schemes than stance that measure the polarity of text expressed towards targets. One group of approaches that also formalize whether a text expresses a polar evaluation (i.e. positive, negative or neutral) towards a target is called target-based sentiment analysis. Examples of approaches that refer to themselves as target-based or target-dependent sentiment analysis are: Dong et al. (2014) who annotated tweets with sentiment towards one given target.

Their dataset contains evaluations of the targets bill gates, taylor swift, xbox, windows 7, and google. Similarly, Jiang et al. (2011) collected tweets about five targets (Obama, Google, iPad, Lakers, and Lady Gaga) and manually labeled each tweet with a sentiment towards the given target. Another group of approaches that formalizes evaluations of targets refer to themselves as topic-based sentiment analysis. However, the building blocks are again (i) a text that conveys (iii) a polarity (e.g. positive or negative sentiment) towards a topic (e.g. a person or a movie). Representatives of this group are O’Hare et al. (2009), SemEval 2016 task 4c (Nakov et al., 2016), and SemEval 2017 task 4b,c (Rosenthal et al., 2017).

Another set of approaches that formalize polar evaluations of targets is obtained directly from (web) platforms on which users can rate products, services or movies. These formalizations mostly consist of ratings (e.g. on a scale, number of stars) of these targets in conjunction with a written review. Since each review in this scheme rates only one movie, they are clearly single-target approaches. Popular examples for this include the Movielens dataset (Harper and Konstan, 2015) (star ratings of movies), the Yelp dataset (star ratings on a scale from 1 to 5 on various businesses including restaurants and shopping)*available from www.yelp.com/dataset/challenge; last accessed November 5th 2018, and the Netflix dataset (star ratings on a scale from 1 to 5 on movies) (Bennett and Lanning, 2007).

The discussion above shows that our formalization of stance (a tuple consisting of polarity and target) can be referred to with several names. While the individual projects may also have a different focuses, we argue that these names are basically interchangeable.

In addition to the approaches that formalize stance towards exactly one target, there are approaches that annotate stance towards multiple targets for each instance. These approaches are more complex and thus more expressive than single-target approaches. This complexity may be necessary if we face a problem in which there are more than two sides (e.g. if there are several candidates or parties for a election) or if we are interested in a more finely grained differentiation of positions (e.g. if we want to further differentiate the people with a $\oplus$ stance on nuclear power into those that think safety needs to be improved and those who think that the plants are already safe). Naturally, the increased complexity also has a negative influence on how easily the schemas can be annotated and thus on their reliability.

One of the least complex multiple target stance approaches is to annotate stance towards two targets. Such a dataset is created by Sobhani et al. (2017), who annotate each tweet of their collection with stance towards a pair of two targets. These target pairs are Donald Trump and Hillary Clinton, Donald Trump and Ted Cruz, or Hillary Clinton and Bernie Sanders.

Approaches at the intersection of argument mining and stance detection, annotate the overall stance and a set of argument tags. These approaches only differentiate between whether the argument is supported ($\oplus$ stance) or whether the argument does not occur (NONE stance). Examples of such the efforts are Conrad et al. (2012) and Sobhani et al. (2015) who both manually create a set of 15-20 argument tags based on domain knowledge (varying according to the overall target). By grouping reoccurring phrases in their dataset, Hasan and Ng (2013) annotate a set of a similar size.

Other approaches model both the stance that is expressed towards one overall target and stance towards a set of subordinated or otherwise related targets, which are extracted from collaboratively constructed resources. For instance, Boltužić and Šnajder (2015) first annotate forum posts with their overall stance on gay marriage, abortion, Obama or Marijuana and with stance towards pro and contra arguments (e.g. Abortion is a woman's right). The set of pro and contra arguments was manually extracted from the debate website iDebate.org. For each target, up to nine arguments are extracted. One of the largest target sets so far has been used by Ruder et al. (2018) who labele news articles with their stance towards 366 targets. In this study, the authors group the targets, which are derived from DBPedia and frequent Twitter mentions, into the types popular (e.g. Arsenal F.C., Russia), controversial (e.g. Abortion, Polygamy) and political (e.g. Ted Cruz, Xi Jinping).

Aspect-based Sentiment

A similar formalism is used by approaches of aspect-based sentiment analysis, which model what kind of polarity (positive, negative or neutral sentiment) an author expresses towards an entity (e.g. a camera) and its aspects (e.g. lens, prize). While the expression having a stance towards a camera may be a non-standard expression, a sentiment towards an entity is clearly a polar (e.g. $\oplus$ or $\ominus$Hu and Liu (2004)) evaluation of it. Thus, the sentiment on the entity can be seen as overall stance and the polarity expressed on aspects can be seen as stance towards these aspects.

An early aspect-based sentiment dataset is the one created by Hu and Liu (2004), which consists of reviews of five electronic products in which the sentences have been labeled with a sentiment on aspects from a manually pre-defined list of up to 96 aspects (called product features). Ganu et al. (2009) created a corpus of restaurant reviews in which they annotated about 3400 sentences with sentiment towards the restaurant and an aspect category (food, service, price, ambience, anectdotes, and miscellaneous). This dataset was incrementally expanded, and lead to a series of SemEval challenges on aspect-based sentiment analysis (Pontiki et al., 2014, 2015, 2016).*SemEval 2014 Task 4, SemEval 2015 Task 12, and SemEval 2016 Task 5. Note that these task had several sub-tasks which explored various facets of aspect-based sentiment. The final SemEval dataset contains reviews and tweets in eight languages (Arabic, Chinese, Dutch, English, French, Russian, Spanish, and Turkish) and seven domains (e.g. restaurants, laptops, mobile phones, digital cameras, hotels, museums and telecommunication). In each domain each review is annotated with sentiment towards up to 22 entities (e.g. laptop, display, keyboard), and which of the up to nine attributes of the entity is evaluated (e.g. its price, its usability).

Another recurring initiative on aspect-based sentiment analysis, was conducted within the TASS framework (Villena-Román et al., 2014b, 2015; Garcıa-Cumbreras et al., 2016; Martınez-Cámara et al., 2017). These challenges repeatedly made use of a collection of tweets on aspects of the political landscape of Spain and tweets collected during a football game between the clubs Real Madrid and F.C. Barcelona. The collection of football tweets (called the social TV corpus) was annotated with sentiment towards about 40 aspects (e.g. the different players or the referee). Each of the tweets about politics (the STOMPOL corpus) was annotated with a sentiment towards one or more of the six aspects economy, health system, education, political party (e.g. speeches, electoral programme, corruption, criticism), and other. In addition, these aspects are linked to one or more entities (the major political parties of Spain).

With SentiRuEval, a similar initiative was carried out for Russian restaurant and car reviews (Loukachevitch et al., 2015). In SentiRuEval the aspects are general for both domains, food, service, interior, and price for restaurants, driveability, reliability, safety, appearance, comfort, and costs. Blinov and Kotelnikov (2014) use a similar formalization by annotating the sentiment towards the aspects food, service, and interior in Russian restaurant reviews.

Other approaches on aspect-based sentiment include: Thet et al. (2010) who label clauses in movie reviews with sentiment that is expressed towards the movie (overall) and towards five additional aspects cast, director, story, scene, music. The dataset of McAuley et al. (2012) contains user reviews on beer, pubs, toys and audio books, an overall rating, and a rating according to a small number of aspects (e.g. feel, look, smell, taste).

In addition, there are related approaches that describe themselves as opinion analysis. For instance the NTCIR shared tasks OPINION and MOAT on opinion analysis also annotate a polarity towards sets of up to 32 topics to newspaper articles (Seki et al., 2007, 2008, 2010).

Comparing Stance with Aspect-, Topic- and Target-based Sentiment

Sobhani et al. (2017) and Wei et al. (2018) suggest that the difference between stance and aspect-based sentiment is the amount of implicitness that is allowed by the annotation schemes. For instance, the annotation guidelines of Pontiki et al. (2014) strictly exclude inferences and implicitness, while most stance annotation guidelines explicitly include them (Mohammad et al., 2016). However, there is also research on implicitly expressed polarity towards targets that is labels as sentiment analysis (Greene and Resnik, 2009; Russo et al., 2015; Van de Kauter et al., 2015a,b). Furthermore, to distinguish between implicitly and explicitly expressing a polarity is often a non-trivial decision. While it is clear to most people that we should build windmills contains a more explicit stance towards windmills than wind energy should be promoted more often, it is hard to tell if both utterances are fully explicit.

Another possible decision criterion between stance and related formalisms is the domain (e.g. politics or products) to which the formalizations are applied. To shed light on this criterion, we compared the domains used in works that refer to themselves as either stance, aspect-based sentiment, or target respectively topic-based sentiment. We show this comparison in 2.1. The distribution shows that aspect-based sentiment is more frequently used in commercial domain (products, services and brands), and that stance is more common for political or ethical questions. However, as shown by – for instance – Xu et al. (2016) which include the target phone SE in their stance dataset, there are several outliers to this pattern. Furthermore, we do not find a clear pattern for target respectivly topic-based sentiment. Note also, that because persons or groups are often closely associated with ethical and political issues or products and services, they are sometimes placeholders for the associated domain. For example, a stance towards Charles Darwin, often communicates a stance towards Evolution – as the topic is closely linked to his person. Similarly, expressing a stance towards Steve Jobs highly correlates with expressing a stance towards the brand Apple. Similar correlations are also conceivable between the other domains.

This analysis shows that domains are also not a sufficient decision criterion for distinguishing the formalizations from each other. We therefore conclude that stance, aspect-based sentiment, or target- and topic-based sentiment are virtually equivalent. Of course, the individual annotation projects have a different focus and may differ from each other.

The majority of the above described multi-target approaches relies on manually-defined sets of targets that are derived from domain knowledge or from external sources. With some notable exceptions such as Ruder et al. (2018), target sets that are created in this manner contain less than 25 targets.

Depending on the use case, these target sets that resemble small ontologies*in most cases the relationship between the targets within the sets of multi-target approaches can be described using classical semantic relation types such as meronymy, hyperonymy, and hyponymy. For instance, in aspect-based sentiment the aspects are often physical parts or attributes of the target entity (e.g. the relation between a laptop and its screen can be described as a meronymy), or in Sobhani et al. (2017) all targets are candidates for the U.S. presidential elections (and thus all candidates share the same hypernym). may not be complex enough to model all nuances that are expressed by people. For instance, in a debate on the legalization of Marijuana, one could face the utterances Marijuana should be legal for adults, Marijuana should be legal for chronic patients, Recreational use of Marijuana should be legal, The same legal regulations as for alcohol and tabac should apply for Marijuana. While for most people these utterances represent pro-legalization stances, the utterances all stress different nuances of the legalization of Marijuana and all express a slightly different stance. However, being able to determine the stance of people towards these nuances can be of importance when one, for instance, tries to find a compromise between opposing positions.

Yet, it is a difficult or even impossible task to manually model all nuances of targets that are discussed within a collection of utterances. Therefore, in this section, we discuss approaches that infer target sets from the data to which they are applied. We refer to these approaches as nuanced target stance detection. Among the nuanced target approaches, we distinguish between approaches that model stance towards spans within the texts and stance towards other utterances.

The first group of approaches formalizes stance as agreement or disagreement between several utterances. These utterances may relate to each other within a post-reply structures such as in forum threads, or are texts that are uttered in the same context. Consequently, the number of targets and hence the complexity of the annotation scheme is correlated with the size and structure of the dataset to annotate. Abbott et al. (2011) annotate whether posts from 4forum.com agree or disagree with the post to which they reply. Their corpus contains posts about ten controversial issues such as evolution, death penalty and climate change. The CorEA corpus created by Celli et al. (2014) contains user comments on blog posts that are annotated for agreement, disagreement and neutrality to the parent posts. Menini and Tonelli (2016) first extract excerpts from transcriptions of discourses and declarations of the U.S. presidents Kennedy and Nixon. Next, they annotate whether these excerpts agree or disagree with each other. In the dataset by Thomas et al. (2006)y Walker et al. (2012b) transcribed speeches from the US congress are labeled based on whether they approve (YEA) or disapprove (NAY) legislation initiatives. The Internet Argument Corpus by Walker et al. (2012b) contains questions and replies that were posted in a forum. The replies are annotated on an eleven point scale where minus five indicates full disagreement with the question and five indicates full agreement with the question.

Finally, there are approaches that combine agreement or disagreement between utterances with span-based annotations. In a dataset on Android, iPad, Twitter, and Layoffs (in technology companies) Ghosh et al. (2014) annotate spans of a post as the targets of a stance expressed by a responding post. A comparable scheme was used by Andreas et al. (2012) for discussion threads on LiveJournal and Wikipedia.

The second group of approaches base their annotations on other annotations such as named entities or the output of syntactic parsers. This pre-annotation significantly contributes to the complexity of the annotation. Thus, compared to approaches that use unprocessed utterances as targets, we consider these approaches – that require pre-annotation – to be more complex. Mitchell et al. (2013) rely on the Spanish and English Twitter dataset of Etter et al. (2013), which is labeled for named entities. They label all sentences in the corpus for the sentiment expressed towards these entities (They refer to their work as target-dependent sentiment). Kobayashi et al. (2006) manually annotate certain named entities (products or companies) and aspects (e.g. ENGINE, SIZE) as spans in a corpus consisting of Japanese weblog posts. Next, they extract opinion tuples consisting of subject, aspects, and evaluation (i.e. polarity). Similarly, the MPQA corpus contains sentiments (POSITIVE, NEGATIVE, and NEUTRAL) towards entities that are annotated as spans (Wiebe et al., 2005; Deng and Wiebe, 2015).

Span-based approaches are also related to approaches that formalize the text polarity of spans. These approaches annotate the polarity of phrases in the context of the sentence in which they occur. The datasets that have been used for SemEval 2013 task 2 (A) Nakov et al. (2013), SemEval 2014 task 9(A) Rosenthal et al. (2014), SemEval 2015 task 10(A)) Rosenthal et al. (2015) contain phrases*phrase are groups of words that form a syntactical unit within a sentence. Phrases can be single words that are annotated based on whether they convey a positive, negative or neutral sentiment given their context. Similarly, in the Stanford Sentiment Treebank Socher et al. (2013) phrases are annotated with sentiment scores ranging from one (VERY NEGATIVE) to 25 (VERY POSITIVE). Other approaches that formalize the polarity of phrases such as IJCNLP-2017 Task 2 Yu et al. (2017) or SemEval-2016 Task 7 Kiritchenko et al. (2016) do not provide context of the phrases and thus cannot be assigned to the span-based approaches.

Argument Mining

Another group of approaches that is more distantly related to span-based stance detection is called argument mining. The goal of argument mining is to automatically determine argumentative structures within texts (Green et al., 2014). The main difference between argument mining and stance detection is that argument mining not only formalizes if one positively ($\oplus$) or negatively ($\ominus$) evaluates a target, but also how this evaluation is expressed. To quantify this how, argument mining approaches typically perform two subtasks: (i) determining argumentative microstructures such as claims or premises within texts (called argument components or argumentative discourse units) and (ii) determining the relationships of these microstructures (Palau and Moens, 2009; Peldszus and Stede, 2013; Lippi and Torroni, 2016).

What kind of microstructures are modelled and how complex their relationships are depends on which argumentation theory a particular argument mining approach follows. A popular theory is the Toulmin model of argumentation (Toulmin, 1958), which postulates that an argument may contain the following components:*the simplified description of Toulmins argument components is adopted from Habernal et al. (2014)

• Claim: Claim refers to the assertion for which one argues that it is correct.

• Data (Grounds): The evidence that is used to support the claim that is made.

• Warrant The warrant refers to the logic connection of data and claim. This means that the warrant explains why the presented data supports the claim.

• Backing Backing presents information that support the trustworthiness of the warrant.

• Qualifier The qualifier refers to a statement that weakens or conditions (i.e. qualifies) the argument.

• Rebuttal A rebuttal is a statement which presents a counter-argument or which indicates situations when the claim is invalid.

Instead of the rather complex Toulmin model, other argument mining approaches rely on simpler models (e.g. based onFreeman (1991)) which postulate that arguments consist of at least one claim and any number (including zero) of premises. Approaches that have implemented a claim–premise scheme are the works of Palau and Moens (2009) and Peldszus and Stede (2013). Persing and Ng (2016a) and Stab and Gurevych (2014) additionally annotate major claims which express the author's stance on the overall topic of the debate. Rinott et al. (2015) model claims and context dependent evidence which are spans in the text that support the claim in the context of a given topic. The AraucariaDB dataset also contains premise annotations, but uses conclusions (statements that can be concluded from premises) instead of claims (Reed and Rowe, 2004). There are also works that do not formalize argumentation components but rather the quality of arguments (Wachsmuth et al., 2017). For instance, researchers have annotated arguments for their convincingness (Habernal and Gurevych, 2016) and persuasiveness (Persing and Ng, 2017), or differentiated between claims that are subjective or objective (Rosenthal and McKeown, 2012), or qualified or bald (Arora et al., 2009).

Due to the more complex goal of argument mining of formalizing how arguments are expressed, naturally, the annotation schemes are more complex than stance annotations. For example, even in a comparably simple claim–premise scheme, we first need to determine the spans of discourse units, then we need to assign the labels claim and premise to the units, and finally we need to link premises to the claims to which they correspond. Although argument mining as a whole and stance detection are not directly comparable due to different objectives, claims – a component which is contained in most argument mining models – are directly related to (span-based) stance.

As a claim is an assertion or statement for which one argues, we can conclude that the person that utters the whole argument considers the claim to be correct and hence has a $\oplus$-stance towards the claim. Similarly, for counter-arguing we can conclude a $\oplus$-stance towards the claim against which one argues. The other way around, any {$\oplus$|$\ominus$|NONE}-stance on a target may be transformed into a claim of the form I am {IN FAVOR OF|AGAINST|NEUTRAL TOWARDS} target. This intersection of argument mining and stance detection has only been addressed by a few approaches: For instance, some approaches model both a major claim, which expresses the author's stance on the overall topic, and a claim (Persing and Ng, 2016a; Stab and Gurevych, 2014). In addition, Kwon et al. (2007) differentiates between claims that support ($\oplus$ stance) and claims that oppose ($\ominus$ stance) the debate topic.

In this chapter, we provided an overview of the various formalizations that NLP researchers have used to quantify stance in texts. We identified the two components polarity and target as being central to all stance formalizations. Next, we reviewed related work and discussed the complexity of the underlying annotation schemes with respect to these components. For stance polarity we find that one can group existing formalizations into binary ($\oplus$ vs. $\ominus$) and three-way ($\oplus$ vs. $\ominus$ vs. NONE) approaches. We conclude that three-way approaches are better suited for social media data, as the NONE class can be used to filter unavoidable bycatch. Formalizations with more complex polarity inventory – such as those found in text-polarity research – have hardly been used to formalize stance.

According to the complexity of the used target sets, we distinguish between the (i) least complex single target formalizations, the (ii) more complex multiple target formalizations, and the (iii) most complex nuanced target formalizations. For single target approaches we find that stance is highly similar to target and topic-based sentiment. For multi target approaches we compare stance with aspect-based sentiment. We find that these formalizations are used interchangeably, but that aspect-based sentiment is used more frequently in the commercial domain, while stance is more common for political or ethical questions. For the most complex target sets, we identify approaches which formalize stance that is expressed towards other utterances and stance that is expressed towards text spans. The latter is related to \textbf{argument mining}, which often also includes the identifications of claims – statement for which one argues to be correct (and therefore expresses a $\oplus$ stance). We also conclude that span-based stance formalizations are more complex and may potentially suffer from unreliable social media preprocessing.

In the following three chapters, we will examine (i) single target stance, (ii) multi target stance, and (iii) stance towards nuanced target in more detail. In the next chapter, we will start with the least complex formalization – stance towards single targets.

In this chapter, we will look at the simplest case of stance detection, which involves predicting the stance towards single, predefined targets. As a concrete use-case we could, for instance, attempt to predict which stance towards wind energy is expressed by a set of tweets we have collected by searching for an appropriate hashtag (e.g. #windEnergy). If one then compares the number of $\oplus$ and $\ominus$ tweets, one can get an impression of whether the majority is for or against wind energy and how clear this vote is. We visualize this use-case in 3.1.

To study single target stance detection, we will first introduce supervised machine learning – the nowadays dominant paradigm in text classification tasks – and show how supervised machine learning can be used for automatically detecting stance. Next, we will describe our contributions to two of the first shared tasks on stance detection*SemEval 2016 Task 6 and StanceCat@IberEval 2017 as well as our organization of GermEval 2017, a shared task on aspect-based sentiment classification. As we have argued in the previous chapter, in this work, we treat stance and aspect-based sentiment as synonymous. In this chapter, we conduct a series of quantitative experiments to better understand the developed systems and the state of the art of single target stance detection.

Formally, the process of automatically detecting stance on single targets can be seen as the function $F(x,t)=y$ where $t$ is our given target (e.g. nuclear power), $x$ is any input text and $y$ is the predicted stance ($\in${$\oplus$,$\ominus$, NONE} resp. $\in${$\oplus$, $\ominus$ }). However, the complexity of stance detection (see Chapter 2) makes it impractical, or even impossible*given that human language is capable of infinitely producing new words and sentences, and hence inputs $x$., for humans to manually craft the function $F(x,t)=y$. Thus, in this work, we focus on machine learning to automatically determine the function $F(x,t)=y$ from large amounts of sample data. In 3.2, we visualize the basic workflow of machine learning approaches on stance detection: We start with labeled input data, which is first preprocessed to be usable for a machine learning algorithm. This preprocessing typically involves segmentation of the data (e.g. into words) and vectorization (e.g. translating the instances into sequence of word vectors). Subsequently, we can use a machine learning algorithm to train a model which can predict labels for unlabelled data. The quality of this prediction can finally be evaluated on other (gold) data.

Machine learning approaches can be distinguished based on whether they are supervised or unsupervised. Supervised means that the approaches have access to labeled training instances, while unsupervised means that the approaches solely rely on large amounts of relevant, but unlabelled data. In the case of stance detection, labeled training instances would be a tuple consisting of a text $x$, a stance label $y$ and the target $t$. While a supervised approaches learn functions that assign labels (i.e. $\in${$\oplus$, $\ominus$, NONE}) to input instances, unsupervised approaches try to detect inherent similarities or dissimilarities within the data and to use this to group the instances. These groups can be seen as generic labels, but unsupervised approaches usually cannot describe what these labels mean except how similar or dissimilar they are to the other labels (or groups). Since in our case we know which classes we are looking for, we focus here on supervised approaches. Supervised machine learning approaches can be further distinguished based on whether they are trained to predict discrete labels (i.e. $\in${$\oplus$, $\ominus$, NONE}) or continuous scores (e.g. real values). If we want to predict discrete labels (or classes), the approaches are called classifications, while if we want to predict continuous scores the approaches are called regressions.

Due to the complexity of language and stance, we rely on machine learning for creating models that can infer stance from language. However, before we can create such a model, we need to represent data in a way that is both expressive for an tweet's stance and processable for computers. We do this by representing instances through a set of measurable or countable properties – called features. The process of deriving and assigning these features is called feature extraction (Bishop, 2006, p.2). In the following, we describe typical feature extraction procedures used to represent social media data for stance detection. We discuss these steps along the hierarchy of NLP levels introduced in Section 2.1.1. Specifically, we will discuss human-defined approaches on representing stance-bearing texts in a meaningful way. In the subsequent Chapter 3.2.2, we will discuss methods that try to automatically extract meaningful representations.

In general, a good set of features should discriminate between instances with different class labels (Zheng and Casari, 2018, p.2). This means that, based on the values of the features, $\oplus$-instances should be more similar to other $\oplus$-instances than to $\ominus$-instances.

Ngram Features

One of the simplest ways to represent text is to count whether a particular word occurs or not. However, before we can do this, we first need to identify words in our input. As we have seen in the previous chapter, this can be a non-trivial task in social media. Therefore, researchers typically use social media-specific segmenters that are robust to the usage or emoticons and urls, or platform specific phenomena such as @-mentions in Twitter. A popular example for a social media specific tokenizer is the Twokenizer} which is optimized for segmenting tweets (Gimpel et al., 2011). Instead of using all words to represent instances, several researchers suggested to rely on tokens which are especially indicative of stance (Anand et al., 2011; Lai et al., 2017b; Somasundaran and Wiebe, 2010). These tokens may be platform specific (e.g. @-mentions or hashtags on Twitter), or punctuation symbols such as exclamation- and question marks.

In the case of stance detection, it seems promising to not just use the occurrence of single words as features, but also the sequences of words. For example, in utterance keep the noisy windmills away!, the sequence noisy windmills seems to be highly indicative of the author's stance towards windmills. These sequences of $n$ consecutive words are called ngrams (Jurafsky and Martin, 2000, p.117). Certain ngrams are commonly named after the size of $n$ (i.e. $n=1$: unigrams, $n=2$: bigrams, $n=3$: trigrams). Usually, ngrams are formed within the boundaries of sentences. For instance, in the example above, we can form the follwoing uni-, bi-, and trigrams:

• unigrams: $\{$keep, the, noisy, windmills, away, !$\}$

• bigrams: $\{$keep the, the noisy, noisy windmills, windmills away, away !$\}$

• trigrams: $\{$keep the noisy, the noisy windmills, noisy windmills away, windmills away !$\}$

If one orders ngrams according to their frequency, the distribution follows an inverse power law over most collections of text (Zipf, 1935; Smith and Devine, 1985; Ha et al.,2002).*This law is called Zipf's law after the linguist George Kingsley Zipf (1902-1950) In other words, the frequency of the $i$th most frequent ngram ($tf(ngram_{i})$) is proportional to $1/i$:

While the exponent of the inverse power law is close to one for unigrams (Zipf, 1935), it gets increasingly smaller for larger $n$-values (Smith and Devine, 1985; Ha et al., 2002). Hence, longer ngrams are even more sparsely distributed. In practice, ngrams with $n$>4 are rarely used for stance detection (Mohammad et al., 2016; Xu et al., 2016; Taulé et al., 2017).

Researchers and practitioners often assume that not all ngrams have the same importance for all tasks. For instance, in the tweet I hate windmills for being so loud the trigram for being so is probably less indicative of the author's stance than the trigram I hate windmills. To account for this varying importance of ngrams, words that appear in predefined lists (so-called stop-word lists) are often excluded from consideration or one conducts a relevance weighting of terms or ngrams. A widely-used weighting scheme is the term frequency inverse document frequency (tf-idf) approach, which assumes that words that occur in only a small amount of tweets (or other documents) are espcially discriminative (Jurafsky and Martin, 2000, p.805). tf-idf of an ngram $i$ within a document $d$ can be defined as:

In addition, the term ngram is often used for sequences of characters. Character ngrams are formed in exactly the same way as word ngrams (of course with characters instead of words as base units), however, their number is limited by the underlying alphabet. If we use character ngrams in this work, we will explicitly point this out.

Syntactic Features

To represent the stance expressed by a texts, it may be advantageous to make use of syntactic knowledge. For instance, in the utterance It's the windmills that are – due to maintenance costs – too expenisve!, we will only find very long (and thus sparse) ngrams that contain both windmills and expensive – a word which clearly communicates stance towards windmills. However, syntactic knowledge may tell us that expensive is an adjective that describes the word windmills. Using such stance-indicative tuples (i.e. (windmills, expensive)) can be automatically constructed from a dependency parse (Anand et al., 2011; Faulkner, 2014). In addition, part-of-speech (POS) information has been part of the feature repertoire since the beginning of word polarity research (Pang et al., 2002; Turney, 2002). Other syntactic features for representing stance include the occurrence of modal verbs (e.g. can, could, or may), conditional sentences, or negations (Mohammad et al., 2016; Taulé et al., 2017).

Semantic Features

As stance detection involves understanding the meaning of a text, it seems reasonable to incorporate semantic knowledge into our data representation. Usually, semantic knowledge is incorporated by enriching the representations with lexical semantics (the meaning of the contained words) and assigning text polarity scores to the instances.

Among the several ways to include word meaning into the data representations, the usage of word vectors became most popular in recent years (Collobert et al., 2011). The underlying assumption of these vectors is that words that share similar contexts (and thus are equally distributed within a text collection) are semantically similar.*One of the first to describe this principle was the philosopher Gotlob Frege one has to ask for the meaning of words within context, not in isolation | orig.: nach der Bedeutung der Wörter muss im Satzzusammenhange, nicht in ihrer Vereinzelung gefragt werden) (Frege, 1884, p.9). The hypothesis was also famously summarized with the quote: You shall know a word by the company it keeps (Firth, 1957, p.11). For instance, the words electricity and power are highly similar and also often occur in similar contexts (e.g. [...] you should save gas and $\_\_\_$ [...], [...] the quantity of $\_\_\_$ produced by the generator [...]). The word vectors are created in a way that words that share a large number of contexts also have similar vectors.

Traditionally, these vectors are created by filling a matrix (with words as rows and the contexts as columns) with the number of times the word occurs in the contexts (Gabrilovich and Markovitch, 2007; Turney and Pantel, 2010; Baroni et al., 2014). A context can be defined by single words, larger ngrams, but also by the entire document (Turney and Pantel, 2010). The vector of a word is then simply the row vector of the matrix. To quantify the similarity between two words, one can now compare their vectors. While many metrics have been suggested (e.g. Jaccard, Euclidean distance), one of the most prominent ones is cosine similarity (i.e. the normalized dot product between the vectors):

The vectors that are obtained using the described counting procedures are often referred to as sparse vectors since most of their entries are zero. These zeros mean that they contain a large amount of redundant information. In contrast, there are also so-called dense vectors, which almost exclusively have non-zero entries, and which are usually shorter (most approaches use 25-500 dimensions). The entries in dense vectors are usually within a continuous range of values, which means that their entries are real-valued and not integers as in most counting-based approaches.

Approaches for generating dense vectors can be grouped into dimensionality reduction methods and prediction-based methods (Levy and Goldberg, 2014). The goal of dimensionality reduction is to find dimensions within the word-context matrix that have the largest variance. Many dimensionality reduction problems are not exactly solvable and are therefore approximated. A prominent example of dimensionality reduction is Latent semantic analysis (LSA) (Deerwester et al., 1990). A more recent approach is GloVe that relies on co-occurrence ratios of words within an word-context matrix to create dense word vectors (Pennington et al., 2014).

Prediction-based methods for generating dense vectors train neural networks that solve word prediction tasks. After the training they use an inner state of the neural nets to derive the vectors. One of the first prediction-based approachs is word2vec, that uses two different shallow neural network architectures (skipgram: predict words from context and CBOW: predict context from words) to generate dense vectors (Mikolov et al., 2013a). Charagram (Wieting et al., 2016) and FastText (Bojanowski et al., 2017) are prediction-based methods that incorporated character ngram features in their training. There is also work on creating sense-specific word vectors (Neelakantan et al., 2014; Guo et al., 2014) or fitting them to knowledge sources (Faruqui et al., 2015).

To aggregate word vectors for representing whole instances, one usually concatenates or averages the vectors (Mitchell and Lapata, 2010; Le and Mikolov, 2014). There are also approaches that weigh each vector (e.g. using tf.idf) or prediction-based methods to create the vectors for whole sentences or paragraphs (Le and Mikolov, 2014).

Another popular choice for representing data for stance detection tasks are text polarity features. Specifically, almost the half of the submissions to the first SemEval shared task on stance detection relied on established word polarity lexicons (Mohammad et al., 2016). There are also approaches that create task- or data-specific sentiment lexicons (Somasundaran and Wiebe, 2010). A simple strategy to derive features for whole instance is to average the polarity scores of all words that are contained in the utterance. There are also more complex aggregation strategies that use context (e.g. negations) to adjust the polarity scores of words (Polanyi and Zaenen, 2006; Choi and Cardie, 2009). Other approaches use readily available sentiment tools (e.g. the tools of Socher et al. (2013) or Kiritchenko et al. (2014)) to directly assign sentiment scores to the whole utterance (Walker et al., 2012a; Habernal and Gurevych, 2016).

Pragmatic Features

As shown in Chapter 2, the formalization of stance depends on a given, predefined target. To model this dependence, several researchers have explored the idea of creating features that represent targets. For instance, Augenstein et al. (2016) concatenate word vector(s) that represent the target with the word vectors that represent the instances (e.g. if the target is atheism they add the word vector of atheism to the representation of the instances). Similarly, Du et al. (2017) and Zhou et al. (2017) translate input instances into a sequence of word vectors and an additional target representation. Lai et al. (2017b) used a set of features that model knowledge on the targets Hillary Clinton and Donald Trump. For instance, they create a list to capture the political party or party colleagues of Trump and Clinton.

Other approaches make use of situational context of social media. Walker et al. (2012a) and Sridhar et al. (2014) derive features from the structure of the debates. For instance, they model whether consecutive speakers agree or disagree with each other. Other features that are commonly used to describe the social media context include the number of replies a post has attracted, whether or not a post is the final post in a thread, the number of followers an author has, whether the author has a verified profile, or what time the posting is made (Artzi et al., 2012; Romero et al., 2013; Wei et al., 2016b).

Now that the instances have a computationally processable form, we can try to determine a function that maps labels to the vectorized data. One of the easiest ways to come up with such a function is to define rules that map certain feature manifestations to labels. For instance, one could first sum up all tokens that are tagged with a positive sentiment into a variable # positive and all tokens that are tagged with a negative sentiment #negative. Next one could define a rule that assigns the class $\oplus$ if # positive > # negative, $\ominus$ if # negative < # positive, and NONE if # positive = # negative. This simple rule models the intuition, that texts that are in favor of the target are more likely to have a positive tone and texts that are against the target are more likely to be written in a negative tone. However, this rule ignores the fact that the subject of the text does not have to be the target, but can even be a target that is associated with the other side of the debate. For example, it might be the case that our target is NUCLEAR POWER, but the text is about WINDMILLS. This may require us to reverse our rule (# positive > # negative: $\oplus$ and # negative < # positive: $\ominus$). Such a flip could be modelled as an additional rule. However, the amount of additional targets may be large (or infinite) and as these additional targets may have complex relationships to the target of interest, many other shifts other than a complete reversal are possible. Due to the productivity of language each target can also have a (theoretically infinite) large number of surface forms (i.e. target indicative words or phrases). Moreover, all kinds of language phenomena such as negations, comparisons, or irony and sarcasm potentially influence our simple rule. This example shows that even such a simple intuition quickly needs a set of rules that can be difficult to formulate manually. In the case of stance detection, the problem is even more fundamental, as there are hardly any general intuitions and the underlying linguistic, psychological, and philosophical mechanisms are not fully understood.

In contrast to this deductive approach, functions can also be induced from observations. That means, we infer such a function from labeled instances by using machine learning. In this work, we apply two families of machine learning approaches. These families can be distinguished based on which representation they work: (i) classical machine learning approaches that utilize features which have been (manually) engineered to separate the data regarding the label to be learned (cf. Section 3.1) and (ii) neural network-based approaches that make use of simpler data vectorizations (usually word vectors) and try to automatically derive suitable representations of the instances.

Curve Fitting As stated above, machine learning can abstractly be described as automatically determining a function that maps feature values onto labels (either classes in classification or real-valued scores in regression) based on a set of training examples. Once we have determined such a function, we can apply it to new data and automatically predict the label.

One of the simplest ways to determine such a function is to define a set of functions (e.g. all univariate, linear functions of the form $f (x) = ax + b$) and to select the one parametrization that has the best fit with the examples.*This family of algorithms is also called regression, but as we will see below they can be generalized to classication problems For linear functions, this means that we search for those parameters a and $b$that result in the lowest error on the given data. As error we here understand the difference between the scores that are predicted when feeding the trainings instances to $f(x)$ and the scores with which they are actually labeled (called the gold labels). As in practice most relationships (and feature spaces) are not univariate it may be necessary to generalize the equation to multi-variatelinear functions:

Many real-world problems are not linear in nature. However, the described fitting approach works with any base function. A popular choice is logistic regression, which is defined as:

If we want to use a fitted curve for a regression task, we can simply insert our variables (i.e. our features) and use the output as the prediction. However, if we want to apply it to a regression task, we need to discretize the space of output values. In the case of a logistic regression we can interpret the output values as the probability of one of the asymptotes. This means that in the simplest case of a logistic function, we have the asymptotes zero and one and thus can interpret the output as probability distribution for $y = 1$. Hence, an output value of $.7$ means a 70% probability of class $1$ and a 30% probability of class $1$. Analogously, an output value of $.3$ means a 30% probability of class $1$ and a 70% probability of class $1$. According to the maximum likelihood estimation,*Choose that estimation according to which the observed data seems most plausible. for classifying an instance, we can now simply predict $1$ for values $>= 50\%$ and $0$ otherwise. The probability estimation can be generalized to a multinomial logistic regression with more than two outcome classes using:

This transformation of logistic regression to a classification is ultimately based on a threshold (e.g. .5 in the example above). Such threshold-based approaches can also be used for other functions (e.g. linear, polynomial). However, the outlined probabilistic interpretation is hardly possible for functions that do not behave asymptotically.

Support Vector Machines

The basic idea of support vector machines (SVMs) is to determine a hyperplane that is able to optimally split the traininginstancesoftwo classes (Bishop, 2006, p.326). More specifically, SVMs perform this split by constructing that one hyperplane that has the maximum margin tothetraininginstances(Flach, 2012, p.211). They do not rely on all input vectors, but only on those closest to the hyperplane to be constructed (Flach, 2012, p.211). If the classes are not linearly separable in the given vector space, SVMs map the training vectors into a higher-dimensional space and try to construct the hyperplane there. Thereby, SVMs are able to learn complex non-linear functions, but are still interpretable as geometric decision boundaries in a high-dimensional feature space (Hearst et al., 1998). Furthermore, by using the socalled kernel-trick they do not rely on computations in that high-dimensional feature space and are thus computationally efficient (Bishop, 2006, p.326).

We will now describe the different ideas of the SVM algorithm in more detail: If we want to classify the classes $\oplus$ and $\ominus$, we are looking for a hyperplane that linearly separates instances, which are labeled accordingly and that are represented by in an n-dimensional feature space. This hyperplane can be specified using the following formula:

For classifying an instance (or the corresponding vector) as either $+1$ or $−1$, we can simply determine if the vector lies above or below the hyperplane:

Tackling Non-linearity

So far, we have shown how SVMs create a classifier by constructing a hyperplane that linearly separates our training instances. However, many real world problems are not simply linearly separable. See Figure 3.5(a) and 3.5(b) for classification problems that are not linearly separable.

There are two mechanisms that are used to deal with such data: (i) introducing a slack variable $\xi$ that allows that individual points may violate the margin and (ii) transforming the data into a higher-dimensional feature space, where the classes may be separable by a hyperplane.

If an SVM makes use of the slack variable $\xi$, it is called a soft margin SVM (as opposed to a hard margin SVMs that do not use $\xi$) (Flach, 2012, p.216). With a slack variable all instances that are labeled as $−1$ have to satisfy $w^T x + b ≤ −1 − ξ$ and all instances that are labeled as 1 have to satisfy $w^T x + b ≥ 1 − ξ$ with $ξ ≥ 0$. Hence, in the case of instances that violate the margin, the punishment is proportional to the extent of the misclassification. In Figure 3.5(c) we visualize how a soft margin SVM separates a non-linearly separable problem with a hyperplane. The resulting error is quantified by the slack variable $\xi$. Note that the slack helps also to learn a function that is not that strongly influenced by outliers and can thus potentially generalize better to unseen data (Flach, 2012, p. 216).

Even by using a slack variable, some problems remain difficult to separate (e.g. see Figure 3.5(b)). In these cases, we use a mapping function $Φ(x)$ to transform the present vector space into a higher-dimensional space, in which the problem may become linearly separable (Flach, 2012, p.225). This mechanism is visualized in Figure 3.5(d). However, performing both the mapping and the determination of the optimal hyperplane is computationally expensive. Thus, in practice, one uses the so-called kernel trick.

The kernel trick is based on describing the hyperplane as a series of dot products such as $x_1 \cdot x_2$. After a transformation into a higher-dimensional vector space, we would now have to calculate the dot products $\Phi(x_1) \cdot \Phi(x_2)$, which can be computationally expensive. Hence, instead of mapping function $Φ(x)$, we define a kernel $k(x, y)$, which behaves like the dot product in the space to be mapped (Flach, 2012, p.225). Using this trick, we do not need to expensively transform vectors into higher spaces and calculate their similarity, but we can directly use the kernel function. Common kernels are: (i) the polynomial kernels $k(x,y)=(x \cdot y)^d$, (ii) radial basis functions (RBF) kernels $k(x,y)=exp( - ||x - y||^2 - (2\sigma^2))$ and (iii) sigmoid kernels $k(x,y)=\sigma(x \cdot y)$.

There are several ways to generalize support vector machines to multi-class classification problems (Bishop, 2006, 338). The most common strategy is to train an individual classifier for each class which separates the class from the other classes. Having executed this one-vs-all classification for all classes, one can derive a final decision if one follows the prediction of that SVM that produced the largest margin.

The other family of learning algorithms we are using in this work can be classified as neural network approaches. The basic idea of these learning algorithms is to not be dependent on manual feature engineering, but to discover suitable representations of instances by themselves (Goodfellow et al., 2016, p.4).

Artificial Neuron

Neural networks comprise so-called artificial neurons (or perceptrons) (Rosenblatt, 1958). Artificial neurons are models of real, biological neurons – nerve cells which occur in almost all multicellular animals.*Although inspired by biological neural networks, artificial neural networks are also strongly influenced by best practices in engineering and machine learning. We do not want to give the impression that today’s artificial neural networks are learning or thinking in the same way as real beings.

Biological neurons receives a signal from receptors or other neurons. If this signal exceeds a certain threshold value the neuron fires – the neuron transmits a signal itself. Figure 3.6 shows how artificial neurons mimic this process: The neuron receives the input $x_1$...$x_n$ and computes a single output. As not all input is equally important, artificial neurons additionally introduce the parameters $w_1$...$w_n$ to weight the input. To increase or decrease the probability to fire (or to transmit a certain output) independently of the input, artificial neurons are equipped with a fixed bias input $b$, which is also weighted by a weight $w_0$.

The output is finally calculated by putting the weighted sum of all inputs and the biases through an activation function. Activation functions transform a continuous range of values (i.e. the weighted input of a neuron) into a different (usually smaller) number space. An exception to this is the linear activation function that returns the exact output value, and it is therefore often called identity function. Closest to the biological model is the heaviside step function*named after the British mathematician Oliver Heaviside which returns one if the input exceeds a certain threshold and zero otherwise. The activation function sigmoid transforms the input values into a space between zero and one:

Figure 3.7 shows the outputs of the described activation functions. While there are reasons for the usage of certain activation functions (e.g. the derivation of the rectifier is computationally less intensive than sigmoid), the choice of the activation function often remains an empirical question (Glorot and Bengio, 2010; Glorot et al., 2011). In practice, this means that one experiments with different activation functions and uses the one that yields the most satisfactory result.

Multilayer Perceptron and Deep Neural Networks

With artificial neurons as the basic building block of artificial neural networks, we now discuss how one can construct neural network architectures.

Network architectures are often described by a sequence of layers. Layers consist of (network) nodes, which are mostly neurons, but sometimes they perform other operations (e.g. fixed transformations). Unless otherwise stated, layers are always fully connected – i.e. all nodes of a layer are connected to all nodes of the layer’s neighbours. Fully connected layers are also often called dense layers (as they are densely-connected).

One of the simplest network architectures is the so-called multilayer perceptron. A multilayer perceptron has three different types of layers:*Further below we will describe some more advanced layers as well The input layer which builds the interface between the input instances and the network, the intermediate hidden layer, and finally the output layer that returns the result of the networks calculation. Except for the input layer, each layer’s nodes are neurons. We show an example of this architecture in Figure 3.8.

Hornik (1991) shows that a multilayer perceptron with only one input, one hidden and one output layer which uses a bounded activation function*the set of possible outcome values has a lower and upper bound can approximate any function. However, in practice, one often uses architectures with several hidden layers to learn more abstract representations. These hopefully deep representations learned using deep architectures are the reason why learning with neural networks is often referred to as deep learning (Glorot and Bengio, 2010).

Classification and Regression with Neural Networks

So far we have seen how neural networks convert an input into an output. Next, we discuss how neural networks can perform regressions or classifications.

In the case of regression we simply need an output layer consisting of a single node. If this layer is equipped with a linear activation, the network can produce every possible output value – given that the weights are set correctly. For classification, we use an output layer with the same number of nodes as we have classes. For instance, if we want to predict a three-way stance (classes: $\in\{\oplus, \ominus,$ NONE$\}$), we need an output layer with three nodes (as shown in Figure 3.8). Now, we can assign each class to one node and interpret the activation of this node as activation for this particular class. This activation can be converted into a concrete classification decision by a so-called softmax. Softmax maps an $n$-dimensional vector of real numbers $\vec{a}$ into an $n$-dimensional vector of real numbers $\vec{a'}$ in which each component is within the value range from zero to one. In this way, all the components of $\vec{a'}$ add up to 1. For each output neuron $o$ the softmax operation is defined as:

Training Neural Networks

The training of artificial neural networks is done using an algorithm called back propagation (Rumelhart et al., 1985). Back propagation consists of a forward pass in which we feed in the training data, and a backward pass in which we update the neural net based on the made error.

We will now outline the basic steps of this algorithm: First all parameters of the network (i.e. the weights and biases) are randomly initialized.

During the forward pass, we successively feed all training instances (i.e. input data + gold labels) to the network. This means that we present the training instances to the network as an input and calculate the output. Next, for each instance we compare the output which results from one instance with its gold label by, for instance, calculating the difference between a gold and a predicted score. Instead of simply calculating the difference between gold and prediction, several different error functions $E$ have been proposed. The underlying intuition for most of the more advanced error functions is not to weigh all errors equally, such as by squaring the errors. If we use a squared error function, we obtain a larger error for larger deviations and a comparatively small error for small deviations. Thereby, during training, large deviations between gold and prediction have a particularly strong impact on the weights of the network.

The overall idea of the backward pass is to update all trainable parameters (i.e. weights and biases) in a way that minmizes the error made. Therefore, we start backwards from the output layer and calculate how much each weight of each neuron contributes to the overall error. We can compute the contribution of a weight by estimating the partial derivative of the total error function with respect to this particular weight. The partial derivative can be computed using the chain rule.*The chain rule states that one can differentiate a function that consists of two concatenated functions by separately differentiating the two concatenated functions and multiplying them. If we, for instance, want to compute the contribution of the weight $w_1$ in Figure 3.6, we can compute:

Next, to decrease the total error, we can subtract $\frac{\delta E}{\delta w_1}$ from the current value of $w_1$. However, we do not simply subtract this value, but multiply it with a weight first. This weight is called learning rate. Intuitively, one could say that we follow a direction indicated by the derived function (upwards or downwards) by a defined step length.

Through repeating the described procedure with a lot of training instances, back propagation tries to find a global optimum for the parameters of the neural net. However, a known disadvantage of back propagation is that the algorithm can get stuck in local minima (Ruder, 2016). In addition, sometimes the weights fluctuate heavily (each instance strongly changes the weights) or that they converge only slowly to an optimum (each instance changes the weights only to a small extent). To overcome these problems, there are several algorithms that try to automatically increase or decrease the learning rate over time (Ruder, 2016). Examples for these algorithms include Adagrad (Duchi et al., 2011), Adadelta (Zeiler, 2012), and Adam (Kingma and Ba, 2014).

The function that is learnt on a training set might not necessarily generalize to a test set. Therefore, one often trains neural networks using regularization techniques such as early stopping or $L^2$ regularization (Goodfellow et al., 2016, p. 239–246). The idea behind early stopping, is to evaluate the performance of the net after each epoch on a separate test set. If the performance on this set starts to decrease, one can assume that the network is beginning to overfit the training data and thus one can stop training. $L^2$ regularization refers to penalizing parameters if they deviate from zero. Furthermore, a particularly popular method for preventing neural networks from overfitting is dropout, which refers to to randomly removing (or droping) neurons and their edges from the network (Srivastava et al., 2014).

Recurrent Neural Networks

So far, we looked at neural networks that are able to perform classification or regression of isolated instances. If we want to apply neural methods to words, we can simply use the dimensions of a word vector as input for a neural net. If we want to classify a sentence, we can concatenate the vectors of all words to a two dimensional matrix, and then use every dimension of this matrix as an input dimension. However, this modelling neglects that the sequence of the words of a sentence also contains meaningful information. For instance, if a sentence contains a negation, the position at which the negation occurs is important. If there is a don’t before the word love in the sentence I love global warming, the meaning of love is reversed. An architecture that is able to model sequences is the so-called recurrent neural networks (RNNs) (Rumelhart et al., 1986).

In addition to the flow of information from input layers to output layers (over several hidden layers) that is specific to one instance, RNNs introduce connections between the hidden layers over several instances. Figure 3.9 exemplifies these additional connections for a sequence of two instances. RNNs, however, are not limited to connecting two instances but can connect all subsequent elements of a sequence. Therefore, for the output of instance $n$ within a sequence of $k$ instances, these recurring connections enable the network to incorporate both the input instance $n$ and all previous instances $1$…$n-1$. Sometimes not only previous instances but also subsequent instances have an influence on an instance. For example, in the sentence The president pushes the country, the interpretation of pushes heavily depends on whether the sentence ends with over the cliff or towards success. For this purpose, one can also construct RNNs backward connections. This means that the output for an instance $n$ within a sequence of $k$ instances, depends on the input instance $n$ and all following instances $n+1$...$k$. There are also bi-directional RNNs that have both forward and backward connections.

Due to the concatenation of long sequences, RNNs are particularly affected by the vanishing or exploding gradient problem (Hochreiter, 1991; Bengio et al., 1993). The vanishing or exploding gradient problem refers to the problem of gradients either becoming so small that the network barely converges to any minima or of becoming extremely large, which causes unstable predictions.

One of the many possible solutions to the vanishing gradient problem is to use gated RNNs (Goodfellow et al., 2016, 398). Gated RNNs use weights at the recurrent connections that regulate the influence of the recurrent connections depending on which instance of the sequence is currently processed. This means that they learn to conditionalize the influence of the sequential information. We will now take a closer look at one of the most popular gated RNN architectures:

LSTMs

The basic idea of long-short term memory neural nets (LSTMs) (Hochreiter and Schmidhuber, 1997) is that they are able to learn for each instance how much context (i.e. how many previous or following instances of the sequence) to incorporate. For instance, an LSTM may learn that the word push is highly dependent on the surrounding context and that the word president is rather independent.

To learn such a behavior they not only pass the activation $h_{t}$ of their cells, but also a internal cell state $c_t$. For instance, in time step $2$, our hidden layer would receive both the activations $h_{t_1}$ and the cell states $c_{t_1}$. Both $h_{t_2}$ and the cell states $c_{t_2}$, is then influenced by $h_{t_1}$, $c_{t_1}$ and the original input $x_t$ of $h_{t_2}$. This influence is regulated through so-called gates. Gates are realized as neural network layers that are not connected with all the nodes of previous layers or time steps, but that have only certain connections depending on their intended functionality. Hence, through back-propagation, the network can learn to appropriately mix past and current information. The connection between the different gates is realized with pointwise multiplication. An LSTM typically has a forget gate, an input gate and an output gate (Hochreiter and Schmidhuber, 1997). Figure 3.10 gives an overview on the sequence of the gate layers within an LSTM module. Now, we will have a closer look at each gate’s function:

At first, the forget gate decides how much of the activation of the previous time step $h_{t_{n-1}}$ and of the original input $x_t$ should be considered or forgotten. Formally, this operation is realized with:

Finally, given the updated cell state and the original input, the LSTM’s output gate learns what should be passed to the next layer and time step (i.e. $h_t$). Similar to the input gate, the output gate relies on two operations: a) We determine what parts of the original input can potentially serve as an output learns what should be passed to the next layer and time step (i.e. ). Similar to the input gate, the output gate relies on two operations: a) We determine what parts of the original input can potentially serve as an output and b) we filter this potential output based on the previously estimated cell state $C^{\prime \prime}_{t}$ : $O_{candidates,t}$ is determined using a sigmoid layer:

Convolutional Neural Networks (CNNs) (Le Cun et al., 1989) are responsible for many breakthroughs of deep learning – mainly in the area of computer vision. The basic idea of CNNs is that extracted features are expressive of several regions in the input. For data that is used in computer vision such a feature may be the edge of objects that occur in a picture. In the case of stance detection, such features could be the proximity of a negative word and a particular name.

To learn such reoccurring features, each neuron of an convolutional layer is only affected by spatially limited fraction of the input instance (Goodfellow et al., 2016, 325). The limited fraction of the data is determined through a so-called filter that slides over the data in a predefined way. To extract features that are meaningful at several regions of the input, all the neurons of a CNN-layer share the same weights. The matrix containing theses weights is called filter kernel.

Hence, the input $c$ of a single neuron of a convolutional layer $m$ is determined by the filtered input matrix $I_{j,k}$ and $m$’s filter kernel $F_{j,k}$ to weigh these inputs:

$c_m=\sum_{j,k}i_{j,j}w_{j,j}$

with $i_{j,j} \in I_{j,k}$ and $w_{j,j} \in F_{j,k}$ (Severyn and Moschitti, 2015). Typically, in addition to the actual convolutional layers CNNs have a pooling layer that aggregates the convolved representations. Figure 3.11 shows an example of a simple CNN with one convolutional and one generic pooling layer.

Filters operate on the those units into which we have segmented our data. This means that, if we segmented our data into tokens, the minimum size of a filter is a token. In the example in Figure 3.11 the convolution uses a filter size of three, which means that each filter extracts features from three tokens. In the case of NLP, one often uses two dimensional filters whose first dimension represents the number of words and whose second dimension represents the dimensionality of the used embeddings. In computer vision more flexible filter sizes are used.

Having defined a filter size, we can now slide the filter across the provided instances. The way in which the filter slides over the data is determined by its stride. If the stride is one, the filter shifts by one unit, if it is two it shifts by two units, and if it is $n$ the shift is $n$ units. Thereby, each step represents the input for exactly one neuron. As in other neural network architectures, the weights of the neurons of a convolutional layer are randomly initalized.

Pooling layers perform a fixed function and hence have no learnable parameters (Goodfellow et al., 2016, 330). Common pooling operations are: max pooling, which selects the maximal value from a convolved representation, min pooling, which selects the minimal value from a convolved representation, or average pooling that calculates the average of all values within the convolved representation. Pooling operations can be further distinguished by whether they are applied globally to the complete representation or only to a part of it.

In the sections above, we shed light on how to learn a mapping from given training instances to their labels. Theoretically, this goal could be satisfied by simply memorizing the training data. However, such a memorization would only be feasible in practice if all possible cases are covered by the training data. As we have seen in Section 3.2, such a data collection would be a hardly feasible and – due to the productivity of language – probably impossible.

Hence, we want a function to generalize to unseen data, and – in the ideal case – to be universally applicable. Therefore, we evaluate the quality of a learned function by measuring its performance on test data – data that has the same structure as the training data (i.e. that has been labeled in the same way), but that has not been involved in training the function.

By comparing the performance on the training data and the test data, we can assess if the learnt function over- or underfitts the training data (Bishop, 2006, p. 66). Overfitting refers to a function that learns the peculiarities of the training data to an extent that negatively affects the performance on training data. An example for an extreme form of overfitting is the above described memorizing of the mapping of training instances to their labels. Underfitting means that the model even fails to sufficiently describe the instances of training data.

In practice, before developing and training a model, one usually splits the available data into a train and a held-out test set.*It is also best practice to search for good hyperparamters by iteratively tuning a model on a development set. The development set represents a further fraction of the train set that is solely used for repeatedly testing the models. The division is usually done randomly. However, this split may again introduce a bias into the result as the randomly separated test data may have some peculiarities as well.

To tackle this problem, one often uses a technique known as $k$-fold cross-validation (Bishop, 2006, 32). In $k$-fold cross-validation, we split the available data, into $k$ subsets that ideally all have the same size. According to this split, we can define $k$ train and $k$ test sets. Next, we run $k$ iterations of training and testing our model. The overall performance of the model can then be calculated as the average of the performance of the $k$ iterations.

Scores

Now that we have defined a testbed we can actually measure the performance. Therefore, we now describe the measurements that are used in this work.

The simplest way to measure performance of a classification is to measure the accuracy. The accuracy is defined as the ratio of correctly predicted instances to the total number of instances:

Hence, we need measurements that take into account that there are two types of errors that can be made for each of the classes. For each class we can define a positive or negative prediction, where positive is the prediction of the respective class (e.g. $\ominus$ when being interested in $\ominus$) and negative is the non-prediction of the class (e.g. $\oplus$ when being interested in $\ominus$). The two types or error are then

• false positive: We predict the positive class for an instance, but the instance has a negative gold label.

• false negative: We predict the negative class for an instance, but the instance has a positive gold label.

Accordingly, we can also define cases of correct prediction in terms of true positives (correctly predicted as belonging to the positive class) and true negatives (correctly predicted as belonging to the negative class). Figure 3.12 visualizes these errors and correct prediction within a confusion matrix.

Based on this matrix, we can then determine the precision and the recall of the prediction for each class. Precision refers to the ratio of the number of instances that have been correctly classified as being positive and the number of instances that were classified as being positive (including the false positives):

In most cases, we want to have an individual score indicating the performance of the

classification of several classes. Therefore, one can average the scores over all classes. We can distinguish between two ways of averaging, such as the F₁ score:

• for the macro-averaged $F_1$ we simple compute the average precision and average recall over all classes.

• for the micro-averaged $F_1$ we calculate the individual true positives, false positives, and false negatives across all classes and then calculate the $F_1$ accordingly.

Since in regression tasks we have a continuous space of possible values, it seems inadequate to compute performance based on exact match of predicted and gold values. Imagine that, for example, if we have a gold value of $1$, both the predictions $1.0001$ and $3000$ would be considered equally wrong. Thus, we calculate the correlation between the values that have been predicted by the model and the gold values as a performance measure. Commonly, one uses Pearson’s correlation coefficient r, which is defined for tuples $x_i,y_i$ as:

Evaluation Strategies

Although with cross-validation and metrics we have the basic tools to evaluate stance detection systems, the complex and data-driven nature of these systems makes it difficult to directly infer general knowledge using these tools. For example, if researcher A claims that her system achieves a high macro $F_1$ in a ten-fold cross-validation on her dataset A, this may not mean that we can achieve a high micro $F_1$ in a five-fold cross-validation on our dataset B using the exact same system. While the goal of creating a universally applicable system is hard to reach,*As famously, pointed out by Wolpert and Macready (1997) in their no-free-lunch-theorem any optimization-based system that becomes better at a highly specialized problem, will perform worse on all other problems. In the case of stance detection that would mean that the better we are at one particular stance dataset, the worse we will be on other stance datasets. Of course, one can also try to make the algorithms valid for many (or all) stance datasets, but this is also adds even more complexity to the optimization problem. we here discuss strategies to infer more or less general knowledge.

Our first strategy is to participate in so-called shared tasks. A shared task is an evaluation initiative that is organized around a particular challenge (i.e. solving an NLP task on a particular dataset). A shared task usually involves a strict distinction between participants and organizers. The organizers provide the labeled training data, define the target metric and perform the evaluation of the participating systems. The test data is kept secret until the evaluation. Given this framework, the participants try to create a system that will perform well on the unknown test data (in other words they share the task). As the framework of the shared task is identical for all teams, one can now directly compare their performance. In addition, since the test data (both labels and raw data) is unknown during the development of the systems, the framework prevents (un-)accidental fitting to the data. From a practical perspective, a high variance of the participating systems provides insight into which general components are helpful for the task beyond the parameterizations of certain components. For instance, if several neural network-based approaches perform consistently better than traditional machine learning approaches, then this finding can potentially be generalized. Of course, the results obtained are only meaningful for the provided dataset.

In the sections above, we have shown that NLP systems and thus also stance detection systems can be seen as pipelines consisting of several components such as preprocessing steps or learning algorithms. If one focusses the pipeline character, one can distinguish between two types of evaluations of such systems (Palmer and Finin, 1990; Paroubek et al., 2007; Resnik and Lin, 2010): (i) end-to-end or black box approaches in which we evaluate the whole pipeline based on its input and output, and (i) component-wise or white box approaches in which if we evaluate each intermediate step. While black box evaluations are often sufficient for practical use, where we are just interested in the final result, white box approaches are necessary for understanding the functionality of the systems (Resnik and Lin, 2010).

However, obtaining a complete white box is difficult, as most of the components of a stance detection system have a range of parameters that may influence its effectiveness. Examples for such parameters include the hyperparameters of neural networks (e.g. how many layers to use? or How many nodes to use in each layer?), the kind of embedding resource we use, or if our POS-tagger should differentiate several classes of nouns (e.g. singular and plural nouns). To evaluate just one component one would require adequate gold labels (e.g. our data would need gold POS tags) and experiment with all or at least enough parameter configurations.

In addition to the large number of parameters, there may also be a complex interaction between the components, their parameters and the effectiveness of the system (Resnik and Lin, 2010). In some NLP pipelines an error that is made by one component will be propagated through the whole pipeline. Imagine, we use a rule to extract a feature based on patterns of POS-tags (e.g. adjectives and nouns), but the used POS-tagger performs poorly. In this case, our rule to extract the POS pattern feature may work perfectly, but the extracted feature will not be very helpful given the erroneous POS tags. Consequently, in such cases the individual components may perform reasonably, but the overall performance would be low. In other pipelines, individual components may be robust against the poor quality of certain components because components can compensate for errors or because several steps are independent. Thus, these pipelines may yield reasonable performance despite of erroneous components. These dependencies imply that for a complete white box approach we would have to not just evaluate all components individually but also, all possible combinations of them.

Since one of the main goals of this work is to understand how stance detection system work, we here adapt a gray box approach, in which we treat the whole system as blackbox, but still evaluate a few components that we suspect are particular meaningful for the effectiveness of the systems. To do this, we first generate hypotheses about the influence of certain components or parameterisations on the overall performance. For testing these hypotheses we use controlled experiments. Many of these experiments can be assigned to one of following three types:

Direct Evaluation: Directly evaluating the performance of a single component against gold labels or pseudo gold labels that have been heuristically derived.

Ablation Tests: Testing how the system’s overall effectiveness changes if we remove a component (Fawcett and Hoos, 2016). An ablation test tells us how important a component is for the system as a whole. If the loss of performance is high, we can conclude that the component has a high importance for the system. If the loss of performance is low, the component has a low importance for the system and may possibly be removed to simplify the system.

Oracle Conditions: Testing how the overall systems’ effectiveness changes if we assume a perfect performance for the component. The test allows us to judge how significant the error is that a particular component makes. If we observe a much better performance in an oracle condition, we can deduce that it is particular worthy to improve the performance of the particular component. For example, if we have a system that relies on POS tags and oracle POS tags lead to a significant improvement, one should try to optimize the performance of the POS tagger.

We now describe experiments that were conducted to better understand stance detection on single targets. Therefore we present our contributions to two of the first shared tasks on stance detection (SemEval 2016 task 6 Detecting Stance in Tweets (Mohammad et al., 2016) and the IberEval task Stance and Gender Detection in Tweets on Catalan Independence (Taulé et al., 2017)) and an appropriate part of a shared task (GermEval 2017) we have organized. The general set-up in all three initatives is comparable: For a given target such as the Catalan Independence, atheism or a railway operator, one has to classify whether the provided text express a three-way stance ($\oplus$, $\ominus$, and NONE).

Because our contributions build upon each other, we describe them in the order in which they were conducted and thus start by describing our submission to SemEval 2016 task six. For this system, we hypothesize that three-way stance detection actually involves two subtasks: First we have to decide whether a text contains any stance at all. Hence, we have to train a system that distinguished between the classes NONE and STANCE. Second, for the texts that have been classified as stance-bearing, we have to classify the polarity of the stance as $\oplus$ or $\ominus$. We first describe the dataset used in the competition, then we describe our approach, and subsequently we conduct an error analysis.

Data

The shared task SemEval 2016 task 6 includes two subtasks: Subtask A is a supervised scenario in which both test and train data is labeled. Subtask B is weakly supervised scenario in which only test data was labeled. For training, the participants were provided with a corpus of unlabelled texts relevant to the target. Since our focus was on the supervised task, we here only describe subtask A. For more information on subtask B, refer to the overview paper Mohammad et al. (2016) or our submission Wojatzki and Zesch (2016a).

The provided dataset consists of tweets that were collected based on target-specific hashtags such as #hillary4President. These tags have been removed from the record to prevent classifiers from relying on these obvious cues. The annotation was done via crowdsourcing.*crowdflower.com (now figure-eight); last accessed November 5th 2018

The data for subtask A contains the following targets: atheism, climate change is a real concern, feminist movement, Hillary Clinton, and legalization of abortion. In the following we shorten the targets climate change is a real concern and legalization of abortion to climate change resp. abortion.

Each tweet is only labeled with a stance towards one target. Thus the SemEval task is clearly a single-target stance detection task. In order to illustrate the dataset, below, we show a tweet that expresses a $\oplus$ stance, a $\ominus$ stance, and a NONE stance towards atheism:

• $\oplus$: IDC what the bible says, constitution says you can take it and shove it right back where it came from .. We aren’t a theocracy.

• $\ominus$: I would rather stand with #God and be judged by the world than stand with the world and be judged by God. #Godsnotdead #Truth

• NONE: ”They tried to bury us. They didn’t know we were #seeds.” #MexicanProverb #Tolerance #Coexistence #peace #Liberty #OneWorld

In Table 3.1 we visualize the number and distribution of classes for the different targets. The distribution shows that there are approximately as many $\ominus$ instances as the sum of $\oplus$ and NONE instances. Hence, the distribution of classes can be considered as being imbalanced. In addition, there are imbalances between the different targets. This variance is most obvious for the target Climate Change, which only has 26 $\ominus$ instances and is clearly biased towards $\oplus$ instances.

As an evaluation metric the organizers use the macro-average of $F_1(\oplus)$ and $F_1({\ominus})$. The intuition behind the metric is to focus on the stance-bearing classes ($\oplus$) and $\ominus$)). Note, that the detection of the NONE class is still indirectly evaluated as correctly or incorrectly classifying instances as NONE has an impact on $F_1(\oplus)$ and $F_1({\ominus})$.

Now, we describe how we use a sequence of stacked classifiers for first classifying whether the tweet contains any stance (classes: NONE and STANCE) and second to distinguish between $\oplus$ or $\ominus$ polarity for the instances which have been labeled with STANCE.

Before applying this classification, we preprocess the data with the DKPro Core framework*version 1.7.0 (Eckart de Castilho and Gurevych, 2014). For tokenization, we use the Twokenizer*version 0.3.2 by Gimpel et al. (2011). We determine sentence boundaries with the DKPro default sentence splitter. We also assign POS tags by using the OpenNLP PoS tagger.*maxent model in version 3.0.3 In addition, we identify hashtags with the help of the Arktweet PoS tagger (Gimpelet al., 2011).

On the basis of this preprocessing, we extract a large feature set of ngram, syntactic, sentiment and word-embedding features. However, after running an (cross-validated) ablation tests on the training data, sentiment and word-embedding features have not proved useful. We now briefly describe the remaining features.

Stance-lexicon

For capturing a stance-specific word polarity, we construct a stance lexicon. In this way, we compare the statistical association of each word with the two poles of each classification ($\ominus$ vs. $\oplus$ or STANCE/ NONE). In detail, we compute the gmean association of a word with both classes and subtract the scores. For instance, the gmean association (cf. Evert (2004)) of a word $x$ with $\oplus$ is computed as:

Bi- and Trigram Features

To capture longer expressions, we capture the 500 most frequent bi -and trigrams and derive binary features depending on their occurrences in the tweets. However, the resulting 500 features would outnumber the other features. Thus, we first train an SVM that is equipped with ngrams only and subsequnetly use the output of this classification as a feature.

Syntax Features

As syntax features we use the number of times conditional sentences and modal verbs, exclamation- and question marks, and negations occur in each instance.

Concept Features

While our lexicon feature captures wording that is associated with a single class, there are also words that are associated with both classes. We hypothesize that these words are proxies for important concepts in the debate. We try to retrieve these words by selecting the twelve most frequent nouns for each target. Next, we normalize (in the same way as above) the words and manually remove words that occur only with one class. Subsequently, we form tuples consisting of these words and their surrounding uni-, bi-, and trigrams. Then we train an SVM to classify for these tuples whether they occur in a $\ominus$ vs. $\oplus$ resp. STANCE/ NONE instance. We use the output of this classifier for each occurrence of the words as a feature.

Target Transfer Features

To exploit a potential overlap between the targets (e.g. both Abortion and Feminist Movement are related to the rights of women), we apply the trained target-specific models to the tweets of other targets. We use the resulting classification as an additional feature. This feature is only used for the AGAINST vs. FAVOR classification for the targets Climate Change is a Real Concern, Hillary Clinton and Legalization of Abortion.

Based on the described feature sets, we assemble a sequence of classifications in which we first determine if a tweet contains any stance and then further distinguish this stance into the classes $\ominus$ and $\oplus$. For both classification steps we use the Weka SVM classifier that is integrated in DKPro TC*version 0.8.0 (Daxenberger et al., 2014). Note that the feature sets in both classification steps are comparable, but that the normalization of the stance lexicon and the target transfer features are only used in the $\ominus$ vs. $\oplus$ classification. The sequence of stacked classifiers is visualized in Figure 3.13.

With the help of the implemented system, we now carry out experiments to better understand the task of single target stance detection.

Comparison with Other Approaches

In the official competition we are ranked twelfth of nineteen teams and achieved a score of $.62$ in the official metric. We compare the obtained score to the scores of other teams and baselines in Figure 3.14. The figure shows that even the best approach only reaches a performance score of $.69$, which leaves a lot of room for improvement. From this, we conclude that the task is a difficult challenge. This conclusion is also supported by the fact that less than a third of the teams were able to beat a simple majority class baseline.

Furthermore, we notice that the variance between the performances of the teams is small. With the exception of the submission Thomson Reuters, the difference between the best and worst ranking submission is just 10%. Hence, we cannot identify a team that stands out from the others in terms of superior performance. This is underlined by the fact that the best performing approach is the trigram (SVM) baseline of the organizers.

We analyze the systems of the other participants and identify two main strands of approaches: The first stand are knowledge-light, neural architectures – as for instance submitted by the two best teams. More precisely, the best scoring approach utilizes LSTM layers (Zarrella and Marsh, 2016) and the second best scoring approach uses a network with a CNN layer in its core (Wei et al., 2016a). The second strand includes more traditional classifiers such as SVMs that are equipped with features based on linguistic preprocessing.

In Figure 3.14, we show how these strands are distributed over all participants. We observe that no strand seems to be superior to the other. Interestingly, theperformance of both approaches is very similar, although they are based on fundamentallydifferent paradigms.

Ablation

While the set of implemented feature is useful on the training data, the selected configuration could still be overfitting to the training data. Thus, we conduct an additional ablation test in which we train our system on the training data and determine the loss of performance on the test data. This ablation test is visualized in Table 3.2. The results show that for almost all features the performance barely changes or even improves if one omits them. To some extent, this can be explained by the fact that the features are correleated (e.g. modal verbs are also unigrams). However, that also means that the features are not important for our system.

The only feature that is associated with a substantial loss across all targets is the stance lexicon. This means that the stance lexicon has a significant impact on the classification. In addition, we observe a loss if we remove the concept features for the targets atheism, climate change, and feminist movement. From this we conclude that the identified concepts indeed play a role in the classification.

As a result of the conducted ablation test, we additionally train a model with only the stance lexicon and the concept features. When training a model with this reduced feature set, the performance over all test data increases by about three percentage points (from .65 to .62). This indicates the other features (e.g. the transfer feature) caused our system to over-fit to the training data.

In summary, our participation in the shared task shows that a two-step procedure does not lead to substantial improvements over one-step procedures. If we compare our approach with the other systems and with the baseline systems, we can conclude that comparatively simple systems already lead to state-of-the-art performance.

As shown above, our system does not benefit from semantic knowledge in the form of word vectors. However, in a post-competition analysis, the organizers of the shared task show that the performance of an SVM equipped with ngrams increases if one adds features which are derived from word vectors that are created from a domain specific corpus (Mohammad et al., 2017). Similarly, several studies on sentiment analysis demonstrate that it is beneficial to rely on task-specific vectors instead of on general-purpose word vectors (Tang et al., 2016; Ren et al., 2016; Lan et al., 2016). To better understand why general purpose embeddings are not helpful in the present task, we now investigate how a given target influences the perceived semantic relationship of words. In detail, we examine how the perceived relatedness*Word relatedness refers to the degree to which the meaning of two words is related. Word relatedness is similar to the concept of word similarity, but is more general (Budanitsky and Hirst, 2006; Zesch and Gurevych, 2010). If two words are similar that means that two words describe a similar concept (e.g. electricity and power). Word relatedness also subsumes other relations such as antonymy (e.g. calm vs. rough), functional relationships (e.g. plant (produces) energy) or common associations (e.g. power and supply). of two words changes if we present them in the context of a target.

We hypothesize that people judge the relatedness of words differently, when presented a specific context. For example, without any context the words baby and mother should be closely related, while baby and murder should be more distantly related. However, in the context of a debate on abortion the words baby and murder seem to have a closer connection. In addition, we hypothesize that the performance of predicting the relatedness of words using general-purpose word vectors is lower compared to words without context.

Judging Word Relatedness in Context

In order to systematically examine these hypotheses, we construct a set of words that should be subject to such a semantic shift in the five SemEval targets. For constructing the sets, we first examine the degree of association between nouns and the classes $\oplus$ and $\ominus$ in the SemEval data using the statistical association measure Dice (Smadja et al., 1996). The idea behind this is that words that are particularly indicative for a stance could be particularly affected by the unique semantics of a debate (as opposed to words that do not contribute much to one’s stance, such as articles).

Next, for each of the five targets in the SemEval data, we manually form twenty pairs for which we hypothesize that the context causes a stronger relatedness between the words. This results in a set of 100 pairs. We also add 30 control pairs from the WordSim-353 dataset (Finkelstein et al., 2001). We select these 30 pairs according to an equal distribution across the spectrum of relatedness.

To measure the difference of perceived relatedness, we use a questionnaire which was submitted to a total of 109 participants. Specifically, we choose a between-group design with a treatment group, which was given a context (59 participants), and a control group, which was given no context (50 participants). To set the context for the experimental group, we provide the participants with a short definition and a symbolic image of the target which were taken from from corresponding Wikipedia articles. In each group, we let the participants estimate the relatedness of the pairs on a one-to-five scale where one means UNRELATED and five means HIGHLY RELATED. The data collection was conducted in a laboratory setting. As compensation the participants received 10€ or subject hour certificates (as needed by their study program).

After the survey, we average the ratings to obtain a relatedness scores for each word pair. To measure statistical significance, we aggregate the scores for each target and compare the scores using a T-test. The results of this comparison are shown in Table 3.3. As we are conducting multiple independent significant tests, there is a risk of erroneous significant results. Hence, we adjust the commonly used significant level of $.05$ using Bonferroni correction. As we are conducting five significant tests, we adjust the significance level to $\frac{.05}{15}=.003$. We show the complete list of averaged judgments per word pair in the appendix A.1.

For the targets climate change and abortion we observe a significant change when presenting a context to the participants. The change is an increase in relatedness. That means, that on average, the pairs are judged to be more strongly related.

If we differentiate between control variables and the domain-specific pairs, the effect becomes more pronounced. For the domain-specific pairs, we find a for all targets except for atheism a significant increase of relatedness. The mean difference for the target climate change is even $+.77$, which is an increase of almost 20%. For some of the individual pairs the increase is larger than 1.5 (e.g the score of the pair choice-body changes from $2.60$ to $4.22$ in the context of climate change), which is substantial given the one-to-five scale. A reason for the rather small change due to the context of atheism could be that many of the extracted pairs (e.g. bible-sins, lord-joy, or faith-superstitions) strongly evoke religious context by themselves.

For three of the five targets (abortion, feminism, and atheism) we observe that a presented context results in a significant decreased relatedness ratings for the control variables. In detail, we find this effect is especially strong for variables that have a relatively high relatedness score if presented without context. For instance, the score of the pair midday-noon decreases from $4.79$ to $3.27$ if presented in the context of atheism. Hence, presenting a special context seems to suppress other contexts in which there are particularly strong relationships. In contrast, we hardly find any effect by the context for control variables that have a relatively low relatedness without context. An example is that the perceived relatedness of king-cabbage changes only from $1.61$ to $1.63$ in the context of Abortion.

Automatically Measuring Word Relatedness in Context

Next, we analyze whether the found change has an impact on the efficiency of automatically estimating word relatedness using word vectors. Therefore, we download a set of publicly available pretrained word vectors and test how well they estimate the relatedness. For estimating relatedness, we simply calculate the cosine similarity of the vectors that correspond to the two words in a pair. Finally, we evaluate the performance of this approach by correlating the predicted with the collected relatedness.

We use the following pre-trained word vectors: (i) all three available versions of the FastText vectors (Bojanowski et al., 2017).*available from https://fasttext.cc/docs/en/english-vectors.html; last accesses November 5th 2018 The vectors were trained either from the CommonCrawl*commoncrawl.org; last accesses November 5th 2018 or on a mixture of Wikipedia and curated news news-sources. The latter is available in a version that was trained with and without sub-word information.

All FastText vectors have 300 dimensions. (ii) all available GloVE vectors (Pennington et al., 2014).*available from https://nlp.stanford.edu/projects/glove/; last accesses November 5th 2018 The GloVE vectors were trained on a Twitter corpus, a combination of Wikipedia and the GigaWord corpus (Parker et al., 2011), as well as on the Common-Crawl. The vectors are available with several different dimensionalities ranging from 25 to 300. (iii) the English Polyglot vectors (Al-Rfou et al., 2013), which are trained on Wikipedia.*available from https://sites.google.com/site/rmyeid/projects/polyglot; last accesses November5th 2018 The polyglott vectors have 64 dimensions.

We visualize the result of this evaluation in Figure 4.8. We find that both for the relatedness with and without context the prediction works substantially better on the control pairs than domain-specific pairs. For some vectors (e.g. FastText (d=300) common crawl) the difference between control and domain-specific pairs is over 60%. We conclude that the semantics of those words that are crucial for stance-detection is only insufficiently modelled.

There is a constant drop of about 10% between predicting relatedness with context and without context for the control variables. This is in line with our hypothesis that context influences the prediction of word-relatedness. However, for the domain-specific word pairs, we see this effect only for a few vectors (e.g. the FastText vectors). In some cases, the prediction for contextualized relatedness works even slightly better (e.g. GloVe (d=300) Wikipedia+Gigawords). Hence, the context does not seem to have a major impact on the prediction of word-relatedness the domain-specific word pairs. However, this could also be a result of the low performance of predicting word relatedness of our domain specific words. Overall, those vectors trained on curated text (e.g. Gigaword or Wikipedia) seem to perform slightly better than those trained on social media or web data.

In summary, our experiments on target-specific lexical semantics result in two findings: First, we show that whether one considers word-relatedness in the context of a target has a significant influence on human word-relatedness judgments. Second, we demonstrate that state-of-the-art methods on automatically measuring word-relatedness are not able to account for this influence, mainly since the relationship between words, which are particularly important for our debates, is poorly modelled. We conclude that methods that can account for influence the targets on word-relatedness would also be useful means to advance the state of the art in automatic stance detection.

In the previously described SemEval task, we identified two major strands of approaches: (i) Neural architectures that translate the input in a sequence of word vectors and feed it into neural networks, and (i) more traditional approaches that represent the input based on ngram, word-vector and sentiment features and than train a classification function with learning algorithms such as SVMs. As both strands are based on highly different learning paradigms, we hypothesize that a hybrid system that combines both strands could have the strengths of both strands and yield superior performance. We test this hypothesis by participating in the IberEval task Stance and Gender Detection in Tweets on Catalan Independence (Taulé et al., 2017) with such a hybrid system.

The goal of the IberEval competition is to compare systems that are able to detect the stance towards the target independence of Catalonia. The occasion is the referendum on the independence of Catalonia held on the first October 2017 by the regional government of Catalonia.

The dataset for the competition was collected in a similar way as the SemEval dataset. However, in the IberEval competition the tweets were collected in Spanish and Catalan. First, the organizers collected a set of tweets using the query hashtags #Independencia and #27S.*#27S refers to the date of the regional elections on 27th September 2015 in which a pro-independence coalition secured a majority in the regional parliament Next, the tweets were labeled with the polarity classes $\oplus$, $\ominus$, and NONE by three trained annotators. To give an impression of the data, below, we show a translated example (from Spanish) for a tweet that expresses a $\oplus$ stance, a $\ominus$ stance, and a NONE stance towards the independence of Catalonia:

• $\oplus$: Who can advocate that Catalan students receive only 5% of state scholarships and those in Madrid receive 58%? *translated version of the tweet: Quien puede defender que los estudiantes catalanes reciban solo el 5% de las becas del estado y los de Madrid reciban el 58%

• $\ominus$: High participation benefits the non-independence parties. Let’s go Catalans, let’s vote! Better united!*translated version of the tweet: Alta participacion beneficia a los partidos no independentista. Vamos catalanes, a votar! Mejor unidos!

• NONE: I’m going to vote today. All Catalans should go vote. Whoever does not vote, is not allowed to complain.*translated version of the tweet: Hoy voy a votar. Todos los catalanes deberian ir a votar. Quien no vote, que luego no se queje.

We show the class distribution for both languages in Table 3.4. The distribution shows a clear imbalance in both datasets. Less than 10% of the Catalan data is labeled with $\ominus$ stance. Contrary to this, less than 10% of the Spanish tweets are labeled with $\oplus$ stance.

A hybrid system combining neural and classical approaches can be successful if the individual approaches have strengths and weaknesses that balance each other out. For example, it would be conceivable that the SVM – as a linear classifier – generalizes better to unseen data, but reacts less sensitively to key sequences. Hence, for our hybrid system we train both an SVM and an LSTM classifier. Next, we design a system that decides whether to rely on the SVM or LSTM prediction. As both approaches require segmented text, we tokenize the tweets using the Twokenizer (Gimpel et al., 2011) from the DKPro Core framework (Eckart de Castilho and Gurevych, 2014).*30version 1.9.0

As the SVM classifier, we use a linear kernel SVM provided by the DKPro TC framework. We equip the SVM with uni-, bi-, and tri- word ngram features, and bi-, tri-, and four- character ngram features. We also use word vector features. Therefore, we first average the FastText (Bojanowski et al., 2017) vectors of all words of a tweet and then use each dimension of this averaged vector as a feature.

The LSTM classifier is implemented using the Keras framework and the Theano backend. In the initial layer of the network, we translate the input data into sequences of word vectors. We use the Spanish and Catalan FastText vectors provided by Bojanowski et al. (2017). The next layer is the bidirectional LSTM layer, which has 138 LSTM units. During development, we found that it is beneficial to add another dense layer after the LSTM layer. The final layer is a softmax classification layer.

In order to create the hybrid system, we first classify all tweets with both the SVM and the LSTM. Then, we label all tweets for each classifier and for each language with whether our prediction was wright (TRUE) or wrong (FALSE). Afterwards, we train a new classifier to automate this decision for each approach. For instance, we learn a classifier that predicts whether the SVM correctly or incorrectly classifies the tweets. To be able to explain the decisions of the hybrid system, our goal wasto base this classification ona small set of rules. Hence, we use a decision tree (weka’s J48) for this classification. For the decision tree, we represent the tweets with features that may affect whether a tweet canbetterbeclassifiedwithanSVMoranLSTM:

Number of Tokens per Tweet SVMs and LSTMs differ in the maximum length of the sequences that they can model. While the SVM is limited to the length of the ngrams, the LSTM can potentially capture whole tweets.

Text Similarity LSTMs and SVMs are both dependent on the test data being similar to the training data. We model the similarity between instances in terms of their ngram overlap.

Type-token-ratio Redundancy within the tweets may also affect their classifiability. Hence, we use the type-token-ratio as a feature to capture the lexical complexity of the tweets.

Word Vector Coverage As both approaches depend on pre-trained word-vectors, we measure the proportion of words that are covered by the resource of Bojanowski et al. (2017) and words that are not covered by the resource.

To conduct the final classification ($\ominus$, $\oplus$, or NONE), we follow the recommendation of the tree classifier and select a prediction accordingly. For example, if the tree predicts for a Spanish tweet that the LSTM is correct and the SVM is mispredicting, we choose the LSTM prediction. If the tree predicts that either both systems are correct or that both systems are wrong, we choose the SVM prediction as it is more reliable (cf. Table 3.5). Figure 3.16 visualizes the architecture of the hybrid system.

Before comparing the performance of our system to the performance of other submissions, we evaluate the performance of its components. For these evaluations, we measure their performance in a tenfold cross-validation on the training data. The official metric assigns equal weight to classes $\oplus$ and $\ominus$. However, since the class distribution is imbalanced, the rarer of the two positive classes ($\oplus$ and $\ominus$) is weighted disproportionately. For instance, on the Catalan dataset, a system that correctly predicts all 3331 $\oplus$ tweets ($F_1(\oplus)=1$) but that incorrectly predicts the 163 $\ominus$ tweets would receive a official score of only $.5$. Hence, we here report the micro-averaged $F_1$ over all classes. We show this evaluation in Table 3.5.

For the SVM we conduct an ablation test and find for the Catalan data that the performance decreases if we remove any feature type (word vector, character-, and word ngrams) from the model. For Spanish, however, we only observe a decrease for word ngrams. We conclude that, as in SemEval, ngrams are the most important feature. Since the word vector and character ngram features do not decrease the model’s performance on the Spanish data, we keep them in the model.

When comparing the performance of SVM, LSTM and hybrid prediction, we find a constantly better performance for Catalan than for Spanish. The difference is 10% for the SVM and hybrid approach, and 6% for the LSTM. We also observe that for both languages the SVM and the hybrid prediction outperform the LSTM.

Interestingly, we observe an almost identical performance for the SVM and the hybrid prediction. To examine this further, we inspect how similar the prediction of the models are. As a measure of the similarity of the predictions, we calculate the agreement metric Cohen’s $\kappa$ (cf. Section 4.1.2). We find rather high agreement scores for both Catalan (κ = .92, 169 differing predictions) and Spanish ($\kappa=.90$, 230 differing predictions). As the agreement between SVM and LSTM is rather low ($\kappa=.28$ for Spanish and $\kappa=.39$ for Catalan), the hybrid approach seems to lean mainly in the direction of SVM prediction.

Finally, we evaluate the decision tree’s performance for predicting whether tweets are correctly or incorrectly classified by SVMs or LSTMs. We find that the prediction works better for the SVM (Spanish: .75; Catalan: .66) than for the LSTM (Spanish: .75; Catalan: .66). The rather mediocre performance (given that random guessing would result in a score of .5) shows that this component is the weak point of the hybrid system and that the whole approach would benefit from further improving the type prediction. To nevertheless demonstrate the potential of the hybrid approach, we run an experiment with an oracle condition, in which the decision tree always correctly predicts whether a tweet is better classified with an SVM or an LSTM. The performance of a hybrid system that uses this oracle condition is shown in Table 3.5. In this oracle experiment, we observe that the performance increases by .09 for Catalan and by .13 for Spanish. This emphasizes that a hybrid approach could be superior over SVM- or LSTM-based approaches if the type prediction could be improved.

For comparing our system with the other submission we rely on the official metric $\frac{F_1(\oplus)+F_1(\ominus)}{2}$. As in the training data, our hybrid system performs very similarly to our SVM. Again, our LSTM performs worse than our other approaches.

In Figure 3.17 we compare the performance of our submissions to the performance of all submissions of the shared task. Each team was allowed to submit multiple predictions. However, we limit ourselves to the best submission per language.

We observe that no team obtained a score of over .5. The majority class baseline is outperformed by only a few systems and only by a small percentage. For Catalan the difference between the best performing system iTacos (Lai et al., 2017a) and the baseline is even less than one percent. This shows that stance detection is a difficult task. This finding is consistent with the findings of the SemEval task.

Among the systems there is again a variety of neural and more traditional (mostly SVM-based) approaches. We notice that the top scoring systems utilized rich feature sets, which include stylistic (e.g. POS ratio or number of capitalized words) and social media-specific features (e.g. occurrence url or hashtag). Interestingly, the neural approaches (attope, our LSTM, and deepCybErNet) are the lowest scoring submissions in both languages.

In summary, our participation in StanceCat@IberEval shows that our hybrid systemhas the potential to outperform state-of-the-art systems. However, the experiment also shows that we are not yet able to proof this claim under real-life conditions. We arguethat once weunderstand which instances can be predicted better with neural systemsand which instances can be predicted better with classical systems, we might be able to present superior hybridsystems.

For our final study on single target stance, we turn to a commercial topic. Specifically, we investigate texts from various social media sources that express a $\oplus$, or NEUTRAL evaluation towards the services of the largest German railway company Deutsche Bahn, which carries over twelve million passengers each day.*according to the companies own information: https://www.deutschebahn.com/en/group/ataglance/facts_figures-1776344; last accesses November 5th 2018. Since a large number of people use Deutsche Bahn services every day, it is not surprising that a large number of people share their corresponding experiences on social media platforms. These experiences can provide other customers with valuable insights on the quality of services and may be a source of feedback for the Deutsche Bahn. Hence, analyzing the feedback of Deutsche Bahn customers seems to be a valuable application domain for stance detection.

In contrast to the previous studies, we do not participate in a shared task, but organize it this time. Aim of the shared task is to model all subtasks that are important for an automatic analysis of customer reviews in the real world. Hence, we define four subtasks ranging from (A) determining relevance of reviews, (B) classifying which polarity ($\oplus$, $\ominus$, or NEUTRAL) is expressed towards the target as a whole, (C) classifying which polarity is expressed towards aspects of the service, to (D) extracting linguistic expressions which are used to express polarity on aspects. Note, that we choose to label the task as an aspect-based sentiment task as the term is more popular in the commercial domain.

Here, we only report our findings on subtasks A (relevance detection) and B (classification of expressed polarity towards the target) as they directly correspond to single target stance detection. Relevance detection is about identifying whether a review contains an evaluation of the target or whether it is off-topic. Thus, relevance detection corresponds to classifying whether a text contains any stance on the target or whether a text expresses a NONE stance. Classifying target-specific reviews into $\oplus$, $\ominus$, or NEUTRAL evaluations corresponds to the stance schema of the previously described shared tasks, where one has to decide if a text expresses a $\oplus$, $\ominus$, or NONE stance towards a target.

The data for our shared task was collected at Technische Universität Darmstadt as part of the project ABSA-DB: Aspect-based Sentiment Analysis for DB Products and Services (thus, the corresponding project members are the ones who deserve credit for creating the dataset). The raw data was collected from various social media sources such as social networking sites (e.g. posts on Facebook or Google plus), microblogs (e.g. Twitter), news forums (e.g. www.spiegel.de/forum), and Q&A sites (e.g. www.gutefrage.net). These sources were crawled for query terms such as deutsche bahn and – to reduce the massive amount of by-catch – additionally automatically filtered for relevance.

Next, for each month in the period from May 2015 to June 2016, about 1,500 reviews were randomly selected. After collecting the reviews, the data was randomly divided into a train (80% of the data), development (10% of the data) and test (10% of the data) set. In addition to the synchronic test set, which originates from the same period as the train data, a diachronic test set was created, which dates from the period from November 2016 to January 2017.

The collected customer feedback was annotated by a team consisting of six trained annotators and a curator. The team iteratively refined the annotation guidelines to improve the reliability of annotations.

Consequently, for subtask A, each review is annotated based on whether it contains feedback on the target Deutsche Bahn or if it is off-topic. We show the distribution of relevant and irrelevant documents across the different sets in Table 3.6. We notice that there are significantly more relevant than irrelevant documents in all sets. We attribute this to the pre-filtering of documents according to relevance.

Subsequently, the annotators labeled the reviews based on whether it refers to an aspect and whether this aspect is positively negatively evaluated. The aspects to be annotated were predefined within an aspect inventory, which also included an option for reviews that do not clearly address one aspect of the inventory (called general aspect). Based on these annotations, we derive a document sentiment (resp. stance) towards the target using a set of rules: If a review contains only positive (resp. negative) and neutral aspect evaluations, we define the overall stance of the review to be $\oplus$ (resp. $\ominus$). If a review contains both negative and positive evaluations, we set the stance to be NEUTRAL regardless of the number of negative and positive evaluations. Table 3.7 shows the resulting distribution of document stances across the different datasets. Again we observe a skewed class distribution, with more than twice as many NEUTRAL instances as the sum of $\oplus$ and $\ominus$ instances. It is noteworthy that there are substantially more $\ominus$ instances than $\oplus$. Hence, it seems to be more common to vent anger about delays or other inconveniences than to report positive experiences.

Subtask (A) attracted five teams and subtask (B) attracted eight teams. As each team was allowed to submit several predictions, we received a large variety of different approaches. Again, we will only report the best-performing submission for each team. For both subtasks, we report the micro-averaged F₁ score as an evaluation metric. We choose this metric because it weighs all instances equally and is thus not distorted by the imbalance of the classes. We also include the SVM that uses these advanced features – such as word vector features or features that are derived from an automatically expanded sentiment lexicon – in our comparison. We compare the performance of the approaches with a majority class baseline and a simple SVM-based text classification approach. This baseline SVM classifier is only equipped with frequency-weighted unigrams and sentiment features derived from the German sentiment lexicon by Waltinger (2010). The baseline system is described in more detail and with additional, more advanced features in Ruppert et al. (2017).

Several participating teams use hand-crafted rules to either normalize social media language (e.g. emoticons or URLs) or to derive features from it (Sayyed et al., 2017; Sidarenka, 2017; Mishra et al., 2017; Hövelmann and Friedrich, 2017). Most of the teams rely on semantic information by using some form of word vectors. The two teams Naderalvojoud et al. (2017) and Lee et al. (2017) even experiment with training task-specific word or sentence representations. In addition, most teams use some form of word polarity lexicons such as SentiWS (Remus et al., 2010) or the lexicon by Waltinger (2010). Among the used classifiers, we see linguistically motivated and threshold-based approaches (Schulz et al., 2017), neural approaches (e.g. Mishra et al. (2017)), and meta-learning approaches such as gradient boosting (e.g. Sayyed et al. (2017)) or other forms of ensemble learning (e.g. Sidarenka (2017) or Lee et al. (2017)).

We compare all these approaches in Table 3.8 for subtask (A) relevance detection and in Table 3.9 for subtask (B) sentiment classification. In the overall comparison, as expected, we find that the relevance subtask can be better solved than the sentiment subtask. Surprisingly, for both the relevance and sentiment classification we do not see large differences between the synchronic and diachronic test sets. In some cases, the diachronic set can be predicted even better than the synchronic set. From this, we conclude that the models learn little time-specific bias and generalize well to different time periods.

In the relevance classification most teams outperform both majority class baseline and the simple SVM classifier. Due to the skewed class distribution, the majority class baseline already yields a strong performance ($F_1=.816$), which is only exceeded by less than 10% – even by the best system. The SVM with the extended feature set is only outperformed by two approaches by a tight margin of less than 1%.

As in subtask (A), in subtask (B), we find that the majority class prediction and the simple SVM classifier are strong baselines, but that most teams are able to outperform them. We notice that – with a difference of less than 1% – the three top scoring teams perform very similarly. This is surprising, since the approaches chosen are very different: Naderalvojoud et al. (2017) combined a RNN with a sentiment lexicon, Hövelmann and Friedrich (2017) make use of linguistic preprocessing and the fastText classier, and Sidarenka (2017) learn an ensemble classifier from the predictions of an LSTM and an SVM. Another surprise is that the SVM with the advanced feature set beats even the best of the three by more than 1%.

All in all, we find that meta-learning strategies such as ensemble learning and the usage of sentiment lexicons are among the most successful strategies in both subtasks. However, the two baseline classifiers of Ruppert et al. (2017) show that traditional machine learning approaches are highly competitive and can even be superior. In addition, we find that social media-specific preprocessing and the use of word vectors also seem to be beneficial.

In this chapter, we reviewed the current state of the art of single-target stance detection systems. We first introduced a generic text classification framework, in which we can describe the majority of today’s stance detection systems. In particular, we discussed (i) more traditional machine learning approaches that are based on feature engineering and SVMs, and (ii) knowledge-light neural network architectures.

Subsequently, we reported on our participation in SemEval 2016 task 6 Detecting Stance in Tweets and IberEval 2017 task Stance and Gender Detection in Tweets on Catalan Independence, as well as the organization of GermEval 2017 Shared Task on Aspect-based Sentiment in Social Media Customer Feedback. Across all competitions, we find that single-target stance detection is a challenging task in which even simple majority baselines are hard to outperform. The submissions in the three tasks constantly show that the usage of word polarity lexicons leads to slight improvements of the systems. Further improvements can be achieved by incorporating the context (Augenstein et al., 2016; Du et al., 2017; Zhou et al., 2017).

The use of word vectors seems to be beneficial as well. However, our experiment on word-relatedness indicates that pre-trained word vectors may not sufficiently cover the lexical semantics of stance. The experiment also suggests that there is a potential for improving the state of the art by word vectors that model target-specific semantics.

Furthermore, we find that simple text classification systems that are equipped with ngram features only, yield a highly competitive performance and are not clearly inferior to neural network-based systems. In this point, stance detection differs from many other NLP tasks where neural approaches are the dominant paradigm. However, the results of IberEval 2017 and GermEval 2017 show that theres is a potential for improving performance by combining neural, non-neural, or rule based approaches.

In the next chapter, we will examine a more complex formalization of stance. Instead of stance that is expressed towards a single target, the subject of the next chapter models stance towards several, logically linked targets.

As people’s stances are often more complex than being $\oplus$ or $\ominus$ towards one target, in many scenarios it is necessary to adapt our formalization to this complexity. Consider the following three statements that may be uttered in a debate on wind energy:

• (A) The only pollution caused by windmills is noise. Fine with me.

• (B) Coal and gas will disappear. Wind is an inexhaustible source of energy.

• (C) While windmills are expensive, they make us less dependent on gas imports.

All three statements clearly communicate a $\oplus$ stance towards wind energy. However, it may be important to capture that all three actually favor different aspects of wind energy: One could argue that Example (A) addresses the aspect of noise pollution by wind energy, Example (B) the aspect of inexhaustibility of wind, and Example (B) the aspect independence of imports. Knowing which aspects are favored or disfavored may be important to many applications such as ranking positive and negative aspects, filtering debates for positions in which we are not interested, or identifying which aspects have not been discussed yet.

Additionally, utterances such as nuclear power is still the only way to go! may exist, which only implicitly express a $\ominus$ stance towards wind energy. However, that a person actually expresses a $\oplus$ stance towards nuclear power could be important for obtaining an adequate overview on the debate.

To capture this more fine grained notion of stance, in this chapter, we will explore multi-target stance approaches that formalize stance towards an overall target and a set of logically linked targets. In Figure 4.1, we visualize an applied annotation scheme that models both a overall stance towards wind energy and more specific stances towards the logically linked targets inexhaustibility of wind, fossil energy sources, wind energy is expensive and threat to ecosystems.

For predefining the sets of specific targets, we rely on prior knowledge on a debate topic. Prior knowledge refers to any external knowledge which cannot be obtained directly from the present dataset. Prior knowledge includes deriving targets from a knowledge source such as Wikipedia or a debate website, but also if the targets are manually selected on the basis of frequent words in the data. We will examine approaches that are independent of external knowledge in the subsequent Chapter 5. We donot distinguish between targets which are single words or entities and targets that are the whole statements. The target reality of climate change – which actually refers to the statement the climate change is real – shows that, in real world scenarios, the distinction is often hardly possible.

We study multi-target stance approaches by annotating the formalization to social media data and conducting a series of quantitative analyses. While there are already several datasets on multi-target stance or related formalisms, in this chapter, we describe how we created two datasets ourselves. We conduct our own annotation studies for three reasons: First, many multi-target approaches (e.g. most of the works in aspect-based sentiment) are annotated on data from the commercial domain (cf. Section 2.3.2). We want to contribute to the understanding of political and social issues with our work. Second, most multi-target stance approaches (e.g. Conrad et al. (2012), Hasan and Ng (2013), or Boltužić and Šnajder (2014)) utilize a binary stance polarity (i.e. $\oplus$ vs. $\ominus$). However, as we argued in Chapter 2, the NONE class is necessary for many social media use-cases. Third, by conducting our own data annotation, we have access to and control over the annotation process. This allows us to identify peculiarities in the annotation and to compare different variants of the annotation schemes.

Before reporting the findings of our annotation experiments, we take a look at how to annotate stance, as well as how one can estimate the reliability of annotations, and how to treat instances that implicitly express stance.

In the previous chapter, we described that machine learning approaches on stance detection are trained and evaluated with the help of gold labels. As stance is a highly semantic and pragmatic task (cf. Chapter 3), human judges are needed to interpret texts and assign these gold labels. If we want to annotate a stance towards the single target x of a text (cf. Chapter 3), we can ask human subjects to infer (Mohammad et al., 2016):

• whether the author of the text is in favor of $x$ ($\oplus$),

• whether the author of the text is against $x$ ($\ominus$), or

• whether the author of the text is neutral towards $x$ or if the text is off-topic (NONE).

If we want to collect stance towards the target set $Y$, we can simply repeat the procedure for all elements in $Y$ (Sobhani et al., 2017). We exemplify this process for two targets in Figure 4.3.

The annotation scheme does not restrict how much interpretation and what kind of inferences are allowed for the annotation. However, these degrees of freedom also make the annotation subjective. We will now take a closer look at the possible extent of subjectivity and which challenges arise from it. Subsequently, we will show how to quantify these difficulties by means of reliability measures.

The degrees of freedom, one has when interpreting utterances, are particularly problematic for multi-target stance scenarios, since all targets of the target set usually have a logical connection (e.g. part-of-relationships). Suppose we have the utterance Nuclear power poses unpredictable risks! and want to annotate stance towards the targets nuclear power, wind energy, and the more specific targets wind energy is safer than nuclear power, and nuclear power is cheaper than wind energy. An annotator could now make the following inferences:

• (A) The author stresses the risk of nuclear power and thus expresses a $\ominus$ stance towards nuclear power.

• (B) As the author rejects nuclear power, she should favor alternative energy sources ssuch as wind energy and hence has a $\oplus$ stance on wind energy.

• (C) Since the author thinks nuclear power is risky and favors wind energy, she has a $\oplus$ stance towards wind energy is safer than nuclear power.

• (D) As a consequence of the rejection of nuclear power and as risks are often associated with costs, the author opposes nuclear power is cheaper than wind energy.

Except for the first inference, without further context all the outlined inferences are highly speculative. For example, the author could be a supporter of coal-fired power plants and reject both nuclear and wind energy. The person could also have the opinion that nuclear power is less expensive than wind, but still be pro-wind for safety considerations. While this is unlikely, a situation might arise in which a person has $\oplus$ stance towards nuclear power despite having strong safety concerns (as indicated by the phrase unpredictable risks!).

While inferring a $\ominus$ stance towards nuclear power seems to be uncontroversial for most people, a $\oplus$ stance towards nuclear power – despite the expressed safety concerns – seems to be rather unlikely. Hence, the inferences differ in how likely we think they are intended in the given context. To understand how people estimate the intentions of a text, we will now shed light on the theories cooperative principle (Grice, 1970) and relevance theory (Sperber and Wilson, 1986, 1995; Wilson and Sperber, 1986, 2002). Both theories aim at explaining inference processes that occur in conscious conversation between two parties. Hence, they both assume a Shannon-Weaver-like*In the Mathematical Theory of Communication Shannon (1948) models information transfer as process in which a transmitter sends encoded information through a channel to a receiver which decodes the signal. The model was originally published 1948 and forms the basis of numerous more fine-grained communication models. process in which there is a sender that encodes content into a message and a receiver that decodes that message. In our set-up, the authors are unaware that annotators classify their posts. Hence, the authors do not encode their intentions in a way that is specific to the annotators. Therefore, we focus on those parts of the two theories that are concerned with decoding processes which correspond to annotating stance.

Cooperative Principle

The cooperative principle (Grice, 1970) states that successful conversation requires cooperation between a speaker and a listener. By the term successful, Grice means that the listener understands the speaker. The speaker’s part of the cooperation is to expresses the message in a way that the listener can understand in the given conversational context. Under the assumption that the speaker communicates cooperatively, the listener can now use the specific way in which the message is expressed to infer the intended meaning.

Grice (1970) postulates four conversational maxims which – if followed – shape a message in a way that allows the listener to effectively decode the intended meaning. These maxims are:

1. Quantity: The quantity maxim states that utterances should neither contain more or less information than required. For instance, if one wants to express that wind energy is more expensive than nuclear power, one should not provide additional information (e.g. wind energy is more expensive than nuclear power and loud), but also not less (e.g. wind energy is expensive).

2. Quality: The quality maxim means that anything one wants to communicate should be true. This includes that one should not state something one believes to be false and one should not state anything for which you cannot provide evidence. If someone deliberately provides misinformation, it is difficult to infer her intention, as the misinformation is in conflict with the assumption that the conversation is cooperative.

3. Relation: The relation maxim means that an utterance should be relevant (Grice himself summarizes the maxim with the simple phrase be relevant). One cannot derive an intention and thus a stance if an utterance is not relevant in the context. For instance, it is difficult to infer stance towards wind energy from the statement Dogs are better than cats.

4. Manner: The maxim states that the statement should be expressed in a clear and concise manner. This includes avoiding ambiguous, confusing and unnecessarily verbose expressions.

As shown by Deng et al. (2014) and Deng and Wiebe (2014) the cooperative principle can be used to develop rules that infer implicit $\oplus$ or $\ominus$ stance (they call them opinions) from explicitly expressed stance. Hence, by regulating inferences, the considerations of Grice’s assumptions can be used to limit the degrees of freedom inherent in the annotation of stance. Therefore, we first assume that the authors purposely wrote the texts to communicate their intentions to their audience. The audience can be the readers of their (micro-) blog or the group of people with whom the authors are currently in a debate. Next, we can assume that the intention of an utterance is to communicate stance on the topic of conversation in which she or he participates. Of course, this assumption is only valid if the utterance is not bye-catch. From the quantity, quality, relevance, and manner of contained information, we can now infer towards which of our targets the author may have intended to express a stance.

Relevance Theory

While the relevance theory (Sperber and Wilson, 1986, 1995; Wilson and Sperber, 1986, 2002) is grounded in Grice’s relevance maxim (be relevant), it does not assume that the basis of all communication is cooperation, but that communication can also have a deceptive nature. The core of the theory is that evolution has shaped the human cognition to seek the maximum relevance of stimuli such as perceived texts. Thereby, relevance is seen as the proportion of cognitive effects and cognitive effort. Positive cognitive effects positively contribute to an individual’s cognitive representation of the world. Cognitive effort refers to the mental effort that has to be done to obtain the cognitive effects.

Relevance theory claims that communicators exploit that their audience tries to maximize relevance (Wilson and Sperber, 2002). Messages generated by such a process thus contain clues that raise expectations regarding the relevance of the message. According to the relevance theory, these expectations enable a decoding process that allows understanding the intention of a message (Wilson and Sperber, 2002). In this decoding process, readers conduct a sequence of increasingly cognitive complex inference procedures until their expectations of relevance are satisfied. The sequence of inferences includes resolving references or ambiguities, or inferring implicit meanings.

In conclusion, both theories assume that authors provide cues on what intentions they want to communicate in their texts. Hence, if they want to communicate stance they also provide cues on their stance in their messages. By trying to recognize these clues, it may be possible to infer the implied stance.

However, in addition to the degrees of freedom in interpreting stance, there are various factors that make (multi-) stance annotations difficult. Because the texts were written in a different context than they are annotated, we could never infer the intention of a text with absolute certainty. For instance, irony, sarcasm, or communication for the sake of social interaction are factors that are often difficult to assess without knowing the original context.

Another substantial problem is that there is no guarantee that the selected target set matches the targets that are the subject of the discussions in our data the texts. We will examine this problem in more detail in the Sections 4.4 and 4.5. Furthermore, all above described inferences processes are highly subjective (Artstein and Poesio, 2008). This means that the recognition and evaluation of textual cues may be influenced by the annotator’s prior knowledge on the targets, her personal stance towards the targets, or her the motivation for conducting the annotation (e.g. whether the annotation is done out of personal interest or monetary reasons). In order to investigate how subjective the estimations of the annotators are, one can examine the reliability of the resulting annotations. Hence, in the following section, we will discuss how to measure the reliability of annotations.

As we train stance detection systems on labels that have been assigned by human annotators, these systems reflect the interpretation of the annotators. However, as we have discussed above, inferring the stance of texts may be, to some extent, subjective (Artstein and Poesio, 2008). As a result, even if a system achieves 100% accuracy, the resulting labels are unsatisfactory for a second person that interprets the texts in a different way. Hence, we want our data to be reliable. In our context, reliability means we get a consistent result for several annotators. This means that if another person re-annotates the data, we will get the same labels.

One way to measure the reliability of annotations is to have the data annotated by multiple ($>1$) annotators and then measure the agreement between them. The simplest way to measure the agreement between annotators is to calculate the ratio of the number of times the annotators agree on an annotation and the number of all annotations (Artstein and Poesio, 2008). This observed agreement $A_0$ can be calculated by:

Chance Corrected Agreement Measures

Using the percentage agreement may overestimate the reliability of annotations as it does not correct for both random agreement and the distribution of labels (Artstein and Poesio, 2008). For instance, if we randomly guess a three-way stance we will be correct in $\frac{1}{3}$ of the cases. In addition, consider a case in which one annotator annotated a dataset 90% with a $\oplus$, 5% with a $\ominus$, and 5% with a NONE stance. Now, if a second annotator always annotates $\oplus$ without even looking at the texts, we get a high percentage agreement of 90%.

One of the most popular measures, which corrects the observed inter-annotator agreement for chance and skewed class distributions, is Cohen’s $\kappa$ (Cohen, 1960). The basic idea of Cohen’s $\kappa$ is to standardize the observed agreement by the average probability with which a label is used by two annotators (Artstein and Poesio, 2008). The Cohen’s $\kappa$ agreement between two annotators is defined as:

There are a number of similar chance corrected inter-annotator agreement measures such as Bennett’s $S$ (Bennett et al., 1954) and Scott’s $\pi$ (Scott, 1955). They differ in how they calculate the agreement that can be expected by chance. Bennett’s $S$ does not estimate the expected agreement empirically, but assumes a uniform distribution of the labels (Artstein and Poesio, 2008). Hence, the $A_{E,S}(I)$ is simply calculated with:

Agreement Measures for Multiple Annotators

The above described agreement measures are defined for exactly two annotators. However, to understand the influence of subjectivity in greater detail, we want an item to be annotated by several people. Hence, we need measures that generalize to more than two annotators.

In order to calculate the agreement of more than two annotations we cannot iterate over each item and count if the same class was assigned by all annotators, as there may still be agreement between some annotators (Artstein and Poesio, 2008). For example, if a text has been labeled by the three annotators with a $\oplus$ (annotator A), a second $\oplus$ (annotator B), and a NONE (annotator C) stance, there is agreement between the annotators A and B, but disagreement between A and C, and B and C.

A large number of possibilities of measuring the reliability of multiple annotators have been suggested (Fleiss, 1971; Light, 1971; Hubert, 1977; Conger, 1980; Davies and Fleiss, 1982; Randolph, 2005). Here, we limit ourselves to the agreement measure which is most popular in computational linguistics, namely Fleiss’ $κ$ (Artstein and Poesio, 2008).

Fleiss (1971) proposes to calculate the agreement on an item $A_O(i)$ on the basis of all possible pairs of annotators that have labeled it (A—B, A—C, and B—C in the example). If we have a set of $N$ annotators, the number of all possible pairs of annotators can be calculated using: $|N|\cdot(|N| − 1)$ Hence, for calculating $A_O(i)$ we form the proportion of the agreement between pairs of annotators and the number of all possible pairs: