Applying quantitative appraisal analysis to the study of institutional discourse: the case of EU migration documents

Abstract

This article addresses the methodological issues related to applying the appraisal framework quantitatively with a corpus of institutional texts. The aim is to explore how to best annotate evaluative expressions in official institutional documents which are usually considered as factual and formal rather than attitudinal and affectual. Using a corpus of migration documents of the European Union as a point of reference, I outline a way of analysis in which each case of appraisal is systematically accounted for and annotated accurately with regard to the socio-cultural context of the EU, serving to deepen and widen the scope of the appraisal analysis. Ultimately, the study of evaluative patterns used in institutional discourse can reveal the ways in which evaluation is integrated into the structures of formal institutional language and how evaluation towards different phenomena (e.g. migration) is constructed.

Introduction

This article outlines the ways in which the appraisal framework can be applied quantitatively with a corpus of institutional texts. Several authors have approached the subject of conducting quantitative appraisal analysis (e.g. Aloy and Taboada 2017, Fuoli 2015, Macken-Horarik and Isaac 2014, Carretero and Taboada 2014, Kaltenbacher 2006), and this paper condenses those ideas that are the most suitable for analysing institutional discourse and connects them to my own insights in order to model an operational hands-on analysis based on the scrutiny of a corpus of migration documents of the European Union (EU). With the help of the appraisal framework, it is possible to unearth the attitudes integrated into the EU’s migration texts that demonstrate both continuity and change in the ways the EU views migration. As a conclusion, I will present a checklist of the main factors that are needed for a systematic analysis of evaluation that takes into account the specific socio-cultural context of EU migration documents. These documents involve a mixed set of, sometimes conflicting, ideologies, which complicates the appraisal coding process as deciphering all the nuances in the evaluations requires customised annotation practices.

The purpose of this research venture is to explore and encourage the usage of the appraisal framework in quantitative, corpus-based studies of institutional discourse which is an area where, despite its increasing popularity, it has not yet gained an established ground. Some corpus-based quantitative appraisal studies have been conducted; however, they have either used quite a small amount of data (e.g. corporate social reports (Fuoli 2012)), focused on certain keywords/expressions (e.g. concordance lines (Miller 2006)), have not systematically taken into account each instance of appraisal (e.g. quotations and reported voice (Jullian 2011)), or used a limited range of appraisal types/sub-systems (e.g. focus on appreciation (Kaltenbacher 2006)). Thus, more work is needed, and this article offers my contribution to bridging the gap between systemic functional linguistics and corpus linguistics (for previous research, see e.g. Matthiessen 2006); I focus on one specific area, namely quantifying evaluative expressions in institutional discourse. This study endeavours to deepen and widen the appraisal analysis. To increase depth, I use a wide range of attitude sub-systems and include both graduation and engagement into the analysis (see next section for a definition of appraisal concepts). Width, on the other hand, is increased by dealing with longer units of appraisal instead of individual words and by connecting the texts to their institutional contexts.

The data referred to in this article consists of official migration documents of the European Union and was collected for the purposes of my doctoral dissertation project that studies the attitudes and ideologies of EU institutions towards migration (Tupala in preparation n.d.). The corpus includes 56 EU documents with a total word count of about 209,000 words, and it encompasses statements, communications, opinions, directives and reports issued by the European Parliament, European Commission, European Council, Council of the European Union and European Economic and Social Committee between 1999 and 2015. The EU has been actively developing its common and comprehensive migration policy since 1999 in the spirit of freedom, security and justice (European Parliament 1999), and these principles still prevail. However, the refugee crisis of 2014/2015 acts as a watershed that led the EU to swiftly refocus its migration agenda, this time with a more robust approach on border management and stopping irregular migration (European Commission 2015). The 17-year time-span enables the study of how and if the EU’s attitudes towards migration and migrants have changed over the years. Although 209,000 words may seem a relatively modest number, it is the level of detail with which the data has been annotated that gives it its value and strength. Size and scope of the corpus are not enough instead what is required is the richness of the annotations and a program that is up to the task (Wu 2009: 132). The EU corpus was manually annotated with the help of the UAM CorpusTool which has been designed for the linguistic annotation of texts.¹

Migration can essentially be defined as movement of people (e.g. economic migrants or refugees) which varies in length, composition and cause (European Commission 2018). Through its migration policy, the EU aspires to control this movement and its effects on EU member states, and the documents the EU institutions issue to discuss the changing situations contain not only practical but also attitudinal responses towards migration. The EU’s migration policy focuses on enhancing economic development and solidarity with a strong commitment to fundamental rights; migration documents are drafted with these goals in mind, encouraging openness and mobility. However, nowadays they are also increasingly affected by security concerns and challenges of irregular migration, marking a change towards “managing” migration to suit the EU’s needs. Thus, as the increased population movement has switched the EU’s political priorities, it is even more important to study the EU’s attitudes towards migrants who are subjected to its policies by investigating the ways in which evaluation unfolds across texts as the EU juggles its diverse goals. I have identified three main issues in the appraisal framework that need to be considered in order to accurately annotate each instance of appraisal. These relate to meaningful units, contextual knowledge and layers in the annotations (see section Issues and Proposed Solutions).

Studying appraisal in the EU’s migration documents gives insight into the extent to which such texts carry evaluative and attitudinal meanings and what these reveal about the EU’s ideological base. Institutions use discourses to legitimise their existence and interests in order to cement certain social practices (Mayr 2008: 2). Therefore, institutional discourses do not just relay information but also promote agendas favourable to the institution’s cause which hints at the existence of attitudinal and evaluative positions; however, the language used in official documents is generally perceived as very rigid and factual. Any given text type has its own set of systemic probabilities, that is, the kinds of linguistic functions that are most likely to occur in certain types of texts (Halliday and Matthiessen 2014: 29), for instance, institutional texts are characterised by formulaic expressions and formal language structures. Evaluative and emotive lexis, on the other hand, is not (usually) associated with official institutional discourse. However, for this very reason, it is fruitful to study different types of evaluative expressions and their frequencies in the institutional documents of the EU as it can be assumed that the values and attitudes expressed in them have to be deep-rooted in order to penetrate the seemingly neutral legalese of the texts.

Appraisal framework

The appraisal framework is situated within systemic functional linguistics and relates to the interpersonal in language. It studies attitudinal and evaluative language, that is, how authors/speakers express their emotions or how they approve of or criticise the phenomena and people around them, and how they negotiate these positions with conversational partners (Martin and White 2005: 1). Appraisal can be divided into three types that have sub-systems (see Fig. 1). It is worth noting that there are no linguistic structures that have a purely evaluative purpose, instead evaluation functions at the level of wording and is parasitic of other language functions or linguistic structures, such as modality, polarity and the choice of pronouns.

Fig. 1

Appraisal types and sub-systemsFootnote

Produced with the UAM CorpusTool; examples from the EU corpus. For details see O’Donnell 2007; Martin and White 2005: 137; White 2015: 4.

Explicit and implicit appraisal

Within attitude, we can make a general distinction between explicit and implicit appraisal, and studying how they work together is an important part of decoding attitudes in official texts. Explicit appraisal can be roughly categorised as the “easy” case as it is realised by lexical items or wordings that directly indicate the positive or negative attitudes of the author (Hyatt 2007, 2009: 130) as illustrated in examples 1 and 2.

• (1) Smugglers and human traffickers exploit irregular migration.³

• (2) The positive effects of migration for host societies in Europe.⁴

In examples 1 and 2, exploit is explicitly negative and positive effects is explicitly positive describing the negative side effects of irregular migration and the benefits of migration for European societies respectively.

It is the implicit cases of appraisal that are the “problem” as they are not confined to individual words (Kaltenbacher 2006: 274). Implicit appraisal is conveyed by factual, superficially neutral expressions that can be interpreted as attitudinal in specific contexts by audiences that recognise the cultural or social reference (Hyatt 2007, 2009: 130) as demonstrated in examples 3 and 4.

• (3) This contributes to making immigrants more likely to take undeclared work.⁵

• (4) Europe’s population is ageing.⁶

In examples 3 and 4, evaluation is not expressed directly but in a way that prompts the reader to interpret the message negatively. Example 3 refers to how immigrants often encounter difficulties in finding jobs which results in them having to resort to undeclared work. This has negative consequences for both immigrants and the host society as it feeds black market economy. Example 4 is even more subtle and could be seen as a mere statement of fact but connected to discussions about labour shortages, it can be interpreted as a negative evaluation of the situation that needs to be addressed and to which migration is sometimes offered as a solution.

Kaltenbacher (2006: 274–275) contemplates ways of identifying implicit appraisal in corpora including keyword searches (evaluative items or targets of evaluation) and using explicit appraisal as a starting point as implicit cases often occur in the vicinity of explicit ones. Albeit both of these offer a viable place to start, neither is enough to enable finding all cases of implicit evaluation. With keyword searches, it is possible to overlook some (less common) items that might be important for the analysis. Relying on explicit appraisal does not take the analysis very far either as, with institutional texts, it is often the case that a large number of the evaluations are carried out implicitly due to the formal nature of the texts that does not encourage strong attitudinal viewpoints. These cases would be left outside the scope of the analysis if one only considered the immediate co-text of explicit appraisal. Nevertheless, explicit cases in earlier parts of the text guide the reader’s interpretation of the following implicit cases by creating a setting through which readers are conditioned to accept certain meanings or value positions (see e.g. ‘dynamic reader positioning’ in Coffin and O’Halloran 2005, Coffin and O’Halloran 2006; White 2008). Once a meaning has been established, for instance labour migrants having a positive impact on the EU’s economy, it can be backed up later on in the text by deploying implicit means, such as statements of fact about the decrease in the domestic working-age population.

Due to the diverse nature of the evaluative items and targets of evaluation in the data, I opted for a completely manual search. However, although keyword searches are not efficient enough for finding all instances of appraisal, they are useful when analysing evaluative patterns. I conducted keyword searches after the first round of manual annotations in order to trace how certain targets of evaluation, such as (im)migrants, (im)migration, Europe and the EU, are appraised in the data in terms of polarity and explicitness. It is important to tie the evaluative item to its target, and Mary Macken-Horarik and Anne Isaac (2014: 86) highlight the significance of co-text when coding appraisal as the textual environment shapes the readers’ perceptions. Studying the co-occurrence and alternation of explicit and implicit appraisal in texts can reveal how evaluative meanings are constructed.

Evaluative patterns

The detailed mapping of all cases of appraisal is significant when studying evaluative patterns in institutional texts. Once each instance of evaluation has been systematically charted and analysed, repeating patterns start to emerge making it possible to see how attitudinal stances are built throughout a text(s) and connect them to prevailing institutional value systems and ideologies. Evaluation is founded on earlier evaluations (see Coffin and O’Halloran 2005), and I argue that frequency strengthens the force of evaluation making it more solid. It is worth noting that isolated linguistic features or words alone do not reconstruct attitudes and ideologies, instead they come into being through the interplay between and the layeredness of linguistic resources that engender a higher meaning (Caffarel and Rechniewski 2009: 33). This meaning is moulded by what is valued or not valued in the surrounding society and is bigger than the sum of individual words. In the case of the EU, the attitudes and ideological networks it applies to migration need to be considered with respect to the EU’s fundamental values while bearing in mind the prevalent western economy-oriented way of thinking that quarrels with the notion of solidarity both within the EU and in its relations with third countries.

The EU’s migration documents often function as responses to previous ones showing how the discussion around migration has accumulated over the years in response to changing situations and how the EU uses discoursal means to build its stance towards migration. EU institutions repeatedly use certain explicit evaluative patterns, such as references to “enhancing prosperity” and “Europe that protects,” which are meant to demonstrate the economic benefits of migration and the EU’s commitment to human rights and solidarity. Coffin and O’Halloran (2005: 144) identify an ‘evaluative groove’ to study the ways in which meanings are piled up within a single text or through a string of subsequent texts in order to invite readers to interpret assumingly neutral expressions positively or negatively. Referring back to example 4, we can see how just mentioning the ageing population is enough to stir up an idea that this will cause problems in the long run, therefore justifying the need to attract workers from outside the EU.

Corpus tool

The EU corpus was annotated with the UAM CorpusTool. However, as it cannot code cases of appraisal automatically, its role is to assist the manual annotation process. This coincides with Geoffrey Leech’s (1991: 20) Data Retrieval Model in which the main function of a corpus program is to search, retrieve, store and count data while human intervention is needed to conduct the annotations. The corpus tool thus has a passive role, and tracing instances of appraisal manually emphasises the role of the analyst in ensuring that each annotational choice is justified and allows making connections with the specific context.

Auto-coding appraisal is generally not possible as it is not tied to any specific grammatical structures or words. The only exception to this in the UAM CorpusTool are those segments/words that have already been assigned a certain appraisal category as the program can be ordered to repeat it in similar cases throughout the data set. However, this only works with those cases of (explicit) appraisal that are carried by individual words as implicit cases of appraisal usually consist of longer units that most likely occur only once in the exact same form. In addition, since the same words may convey different meanings in different contexts, the auto-code function must be used with caution and the results checked manually. For instance, intolerance and racism are explicitly negative concepts and can be annotated as such. However, these words can also appear in contexts that condemn intolerance and racism and on such occasions, the overall effect is positive rather than negative. Nevertheless, searching for certain predictable items of interest in this way makes conducting the analysis easier since, instead of having to find all these instances manually, the researcher can check the auto-code results and refine them if needed. This step is necessary as the program is not able to decode the co-text, the understanding of which is essential in order to categorise the cases of appraisal accurately.

Let us now turn to the annotation scheme used for coding appraisal in the EU corpus. The UAM CorpusTool has a pre-made appraisal annotation scheme which allows editing, making it possible to increase or decrease the level of detail. I created an edited annotation scheme to suit the purposes of the present study (see Fig. 2). The edited scheme is deliberately elaborate when it comes to attitude enabling the charting of all the nuances in the evaluation of people/institutions and phenomena in the institutional context of the EU. I added the sub-categories of affect as the original scheme only makes the distinction between authorial and non-authorial affect (i.e. whose feelings are discussed) but does not include un/happiness, dis/satisfaction, in/security and dis/inclination (i.e. the distinction between different types of feelings). I also included the distinction between positive and negative cases within each sub-system of attitude instead of only dealing with polarity on a more general level. This enables a more detailed study of how and if polarity is used differently within different sub-systems. Conversely, I left out the sub-systems of engagement and graduation is only divided into force and focus as the sub-systems missing from the scheme are only dealt with qualitatively since their function is to support the attitude analysis.

Fig. 2

Appraisal annotation schemeFootnote

Produced with the UAM CorpusTool.

Issues and proposed solutions

In this section, I will discuss the three main issues that are worth considering when conducting quantitative appraisal analysis on a corpus of institutional texts. I will also propose some solutions that have proven to be useful as it was necessary to adapt some aspects of the appraisal framework to make it better suit the characteristics of institutional discourse.

There is a clear evaluative layer in the EU migration documents as their function is to negotiate how to best manage migration and to assert the EU’s power over migrants by laying down their rights and obligations. However, given the official nature of these texts, rather than being conveyed by transparently attitudinal words, the evaluative expressions used by the EU institutions are often interwoven within the structures of formal institutional discourse and require contextual knowledge to be interpreted correctly. Furthermore, it is common for the EU migration documents to impersonalise expressions of attitude as they entail layered evaluations where explicit appreciation of phenomena is used to veil implicit judgement towards people or their actions. Based on these characteristics, the discussion here revolves around identifying units of meaning, avoiding subjective interpretation and allowing layered codings.

One word vs. a phrase/sentence

The first issue is to determine whether to operate with individual words or with longer units (phrases/sentences) when annotating cases of appraisal. Appraisal analysis has to be connected to particular instances of language that are countable, otherwise there is a risk that the analysis relies on the general “feel” of the text instead of being grounded on concrete findings. This is also what Geoff Thompson (2014) is wary of and thus he puts much emphasis on the actual wording when categorising cases of appraisal. I agree with Thompson about tying the analysis to the level of wording, which does not mean that the annotations should be restricted to individual words or noun phrases. Prototypical examples of appraisal usually focus on depicting these isolated instances; however, more often than not an evaluation is carried by a longer unit within which it is not possible to identify only one evaluative lexical item but in which the whole sentence, for example, conveys an evaluative stance. My analysis thus deals with coherent units of meaning that connect cases of appraisal to their co-texts.

Example 5 demonstrates how the use of longer units in the annotations helps to capture the ideological context of evaluation.

• (5) Intolerance, racism and xenophobia are on the rise in Europe.⁸

In example 5, it would be possible to annotate the words intolerance, racism and xenophobia as separate instances but I argue for treating the whole sentence as one appraisal unit that expresses negative appreciation towards the phenomenon. The focus on individual words would make the analysis too scattered especially with a large data set since it is the units of meaning that matter when tracking down evaluative patterns in texts. As the interest here lies not only in what kinds of evaluations there are but also in studying what/who is being evaluated and why, it is essential to consider the immediate co-text around the evaluative item as it reveals the target of the evaluation and ties the evaluation to context. For instance, in the case of EU documents, the focus is on how many cases of appraisal and of what type are related to the EU, EU policies, migration and migrants, and whether they are positive or negative, explicit or implicit. This can be used as basis for determining the EU’s value positions towards migration, and the emerging patterns can shed light to the ways in which the institutional attitudes bleed into their discourses.

Relevance of contextual knowledge

The second issue that needs to be addressed is how to determine what can be considered attitudinal or evaluative in a given context. The annotations are often based on manual work conducted by one person, which allows subjective biases and preconceptions to penetrate the analytical process to some degree. For instance, if researchers start from the presumption that a certain institution engages in discrimination, there is a risk that they will evaluate the texts more negatively than if they started from an objective standpoint. Martin and White (2005: 162–163, 207) argue that subjective interpretation cannot be totally avoided as meanings are negotiated between the (authors of the) texts and the readers whose cultural backgrounds and personal experiences affect their reading position. The subjective bias can be mitigated by ensuring that the annotations are backed by valid and relevant background information on what is considered as positive/acceptable or negative/unacceptable in the society or institution under inspection.

As stated earlier, implicit evaluation might become clear only to those audiences that share the author’s understanding of a given society or culture. The evaluative expressions that utilise prior knowledge can be used to either persuade readers to accept the text’s position or alienate certain groups that do not belong into the target readership (see e.g. ‘ideal subject’ in Fairclough 1989: 49, 54; ‘in-group texts’ in Don 2016). Hence, in addition to subjective over-interpretation, there is the possibility that researchers might overlook the issues of which they do not have all the necessary information. In the case of official EU documents, it should also be borne in mind that although they are freely available to the public, they form a chain of communication between EU institutions and as such are written from one bureaucratic institution to another. Thus, these texts expect certain knowledge of the previous texts, and the evaluations they contain are affected by the previous ones. For this reason, the EU corpus includes text-chains that consist of documents from different EU institutions focusing on the same issue.

When studying EU migration discourse, what need to be taken into account are the institutional confinements of the EU that are reflected against its core values and agendas. The EU frames all its policies around the principles of prosperity, solidarity and security,⁹ and also its attitudinal responses towards migration are grounded on these three dimensions. The common denominator in the EU corpus is the legality of migration; legal migration is described in positive terms as it enhances economic development, partnership and security while irregular migration is viewed as having a negative effect on all three.

Ambiguity and overlap

The third issue is that the annotation of appraisal types, and especially the sub-systems of attitude, always involves certain ambiguity since wordings can be categorised into more than one system depending on the interpretation (Martin and White 2005: 60–61; Thompson 2004: 77). For instance, a sentence might be superficially interpreted as appreciation but the underlying evaluation entails judgement such as in examples 6 and 7.

• (6) The Council emphasises that the respect for the Charter of Fundamental rights and human rights lie at the basis of the EU migration policy and its relations with third countries.¹⁰

Example 6 includes an explicit positive appreciation towards the EU migration policy for its respect for fundamental and human rights and an implicit positive judgement towards the EU itself for enforcing a policy that embraces human rights. Furthermore, the example entails an engagement the Council emphasises that which conveys the Council’s recognition of the EU’s commitment to enhancing human rights. In addition, emphasises can be labelled as graduation as it is a stronger expression of engagement than, for instance, the verb ‘state’. It is common for explicit and implicit cases of appraisal to be intertwined and carry double meanings as often, especially in the case of judgement, the underlying target of evaluation is left opaque and expressed through appreciation of the situation rather than a direct judgement of (human) behaviour.

• (7) The Committee […] is alarmed because in some Member States the protection of people’s fundamental rights is being eroded.¹¹

Example 7 contains an explicit negative appreciation towards the described phenomenon, that is, the erosion of fundamental rights in some EU member states. However, when studied in context, there is also a deeper meaning in the form of an implicit negative judgement towards those member states’ governments and public officials which are letting it happen. Moreover, affect often coincides with engagement as the Committee is alarmed is both an emotional response and a way of taking a stand. As examples 6 and 7 demonstrate, layered codings are necessary to capture all relevant meanings. Monika Bednarek (2007) indicates that lexical items often carry multiple voices (polyphony) and this also applies to systems of appraisal which have a tendency to co-occur within the same wording. Appraisive language is not one-dimensional: there is often more than one evaluative layer at play and the underlying meanings might get lost if we only focus on the superficial layer. Therefore, I assert that layered codings enrich the analysis and help in decoding the overall meaning of the message.

The categorisation of underlying targets of evaluation, however, is not that straightforward. Thompson (2014: 57) contemplates underlying targets and the semantically-based view when discussing whether to label a case of appraisal as judgement or as appreciation. According to the semantically-based view, if the underlying target is human (or an institution whose actions are comparable to those of humans), the evaluation can be categorised as judgement. To clarify, I refer back to examples 6 and 7 which at the superficial level are appreciations towards the EU migration policy and (some) member states respectively. Nevertheless, they also contain implicit judgement which is targeted at the EU and the member states’ governments, hence revealing reverence towards the EU and condemnation towards the ill-behaved governments. However, Thompson (2014: 58–59) rejects the semantically-based view as he indicates that the actual (literal) wording should be used as a starting point for analysis instead of attempting to decipher the underlying meaning. Here, I differ from Thompson as I use both judgement and appreciation in cases like these for the sake of increased depth of the analysis. This helps in uncovering how the EU builds its attitudinal stance towards migration and different groups of migrants since preferring the use of impersonalised appreciation instead of judgement is a value choice in itself.

In order to increase the transparency of the annotations, I nonetheless share Thompson’s (2014: 64) notion of connecting the analysis to particular (strings of) words. Earlier, I defined a case of appraisal as a coherent unit of meaning that may consist of a phrase or a sentence; however, for the sake of clarity, it should not extend to multiple sentences or whole paragraphs. With too long units, there is an increased risk of ambiguity and over-interpretation; moreover some significant details might get ignored if the analysis only focused on the big picture. The overall tone of the text can be identified as generally positive or negative but determining how evaluation actually builds up within a text requires a much more focused approach. It is the level of wording where the evaluation is grounded, and the alternation and the dynamic between different types of appraisal is revealed by the systematic quantitative study of each instance.

Mixed methods

Even though the gist of this article is about operationalising quantitative appraisal analysis, it is beneficial to combine quantitative and qualitative methods. To gain a profound understanding of the ways in which the EU evaluates migration, the quantitative study of frequencies of appraisal types needs to be complemented with the qualitative study of the co−/contextual factors that tie the evaluative items to their targets and to the wider framework of the EU’s goals and value systems concerning the movement of people. In this section, I will discuss two approaches, one from each side, that can be used as reference points when modelling a way to conduct quantitative appraisal analysis on institutional discourse. The first one is Matteo Fuoli’s (2015) seven step method on systematic quantitative appraisal analysis that is intended to decrease subjectivity, and the second one is Macken-Horarik and Isaac’s (2014) cline of implicitness that focuses on detailed qualitative analysis.

Fuoli (2015) has proposed a step-by-step method for manual appraisal annotation to enhance replicability, reliability and transparency. The approach is based on finding the suitable annotation criteria for a specific data set and on the constant refining of the annotations to maximise accuracy. There are two main areas in Fuoli’s method that I make use of when analysing a specialised corpus of institutional texts: these are related to data processing and content. Firstly, when it comes to data processing, Fuoli (2015: 17–18) emphasises context-specificity: analysts should draft an annotation manual that specifies which appraisal categories they are going to use and how they are defined for the purposes of a particular data set so that the analysis can be repeated. This task can be eased by the use of a corpus tool where the annotation scheme can be edited to suit the researcher’s purposes so that it is custom-made for the project in question which gives the researcher control over how elaborate the analysis is. The UAM CorpusTool is suitable for this purpose, and the edited annotation scheme was presented in the third section. Fuoli (2015: 17) refers to the UAM’s auto-code function as potentially useful but, as I already pointed out, it does not work without strong analyst intervention. The EU corpus was subjected to three rounds of annotations: twice for appraisal and once for targets of evaluation. Secondly, data processing is connected to the level of detail, and the main factors I identify here are the depth (the number of sub-systems/categories accounted for in the analysis) and width (individual words vs. longer units) of the annotations.

The second area, content, relates to the specific features of the discourse type. The generic conventions of institutional discourse determine the means by which evaluation is conveyed (e.g. collective expressions of attitude¹² and impersonal attitude constructions¹³) and this also affects the annotations. Therefore, after the annotation manual has been created, category definitions should be tested against the data and possibly further adjusted according to its needs (Fuoli 2015: 17–18). My category definitions are mainly based on those of Martin and White (2005) as they offer clear general descriptions of the appraisal types and sub-systems. Nevertheless, they also leave room for manoeuvre, and I have expanded the definition of judgement as, in its strictest sense, it ought to be used when evaluating human behaviour. However, I use it to refer to the EU/EU institutions and their actions as well for it is ultimately people who are responsible for the EU’s decision-making and holding to its commitments in the name of the EU’s shared value-base. Therefore, I deem that the actions of the institutions can be judged along the same norms as those of individual people. Furthermore, an underlying actor can often be identified even if the message is relayed impersonally: for instance, attributing the respect for human rights to EU policies when it is actually the EU that commits the act of respecting (see example 6). The UAM CorpusTool is useful in managing the layered codings and providing the frequency statistics for the overlaps so that they can be examined. In addition to coding different types of appraisal, there is a need to consider the text surrounding them due to its significance for understanding how evaluation operates. Hence, Macken-Horarik and Isaac’s (2014) approach is helpful as they offer guidance on identifying (implicit) evaluation on different textual levels based rather on cultural understanding than on quantifiable patterns. They also accept the “fuzzyness” of categories, that is, the need for multiple codings to discover all additudinal layers.

Macken-Horarik and Isaac (2014: 89) present a cline of implicitness starting from the close scrutiny of individual words and wordings ranging all the way to the level of culture that requires an understanding of the norms and rules of a society. This approach is suitable for elaborate qualitative analyses, and although it is more oriented towards the analysis of literary texts, it has value in the study of institutional texts as well since much of the EU’s migration discourse relies on the understanding of the cultural background, values and procedures on which the EU is built. Evaluation naturally manifests itself differently in different types of texts; in literary texts, it might be used to guide the reader towards a certain reading position such as feeling sympathy for the protagonist while in institutional texts, evaluation is more about spreading the general attitudes and beliefs of the institution. These are subject to change in response to changes in society; for instance, the security dimension and the urge to curb irregular migration have become even more pronounced in EU documents after the onset of the refugee crisis (European Commission 2015). There is also diversity between different types of EU texts, statements and communications being more formal and opinions more informal containing more explicitly evaluative expressions.

If the merits of quantitative analysis lie in revealing evaluative patters in texts that can then be connected to the wider construction of value positions, qualitative approach ties the analysis to the socio-cultural context. The lexical patterns and linguistic structures we use do not rely only on our personal preferences but are affected by our linguistic and cultural backgrounds (Kaltenbacher 2006: 270). Hence, we can come to the conclusion that the study of lexical/evaluative patterns in corpora discloses things about the society in which the texts have been created (see e.g. Kaltenbacher 2006). Here, however, lies the complexity of conducting quantitative corpus-based appraisal analysis as decoding cases of appraisal automatically is practically impossible due to evaluation being influenced by interpersonal factors that are not visible in individual words (Miller 2006: 249). This underlines the significance of researcher involvement. Evaluation gains meaning from outside the realm of the text which shows that quantitative approach alone cannot grasp all evaluative dimensions if one wants to study the underlying motives of evaluation in institutional discourse.

To deal with the dilemma of how to quantify units of appraisal and to account for extra-linguistic factors, I have.

• 1. Conducted the appraisal analysis manually, sentence by sentence, taking into account the co-text of the cases of appraisal and the target and source of the evaluation.

and

• 2. Considered the wider institutional context of the texts and their impact and effectiveness.

In addition to appraisal annotation, point number one includes the annotation of the targets of evaluation: each case of appraisal is coded for whether it refers to the EU institutions, EU as a whole, EU policies, EU member states, non-EU countries, migration or migrants. There is a division between EU and non-EU targets which makes it possible to explore the differences between the ways in which the EU evaluates itself and the ways in which it evaluates migration. As suggested in point number two, this study is also interested in how the EU migration documents are situated in the context of the EU framework. Thompson and Hunston (2006: 4) argue that collecting texts into a corpus has the downside of separating texts from their contexts. However, in order to avoid this disconnection, I have studied the texts’ legal force and their impact on the EU’s decision-making and how the institutions communicate with each other through different types of texts when negotiating the migration policy (Tupala in prep. n.d.).

Conclusion: Checklist

As a conclusion, I will collect the most important points of applying quantitative appraisal analysis on a corpus of institutional texts into a checklist. Due to the formal nature of the EU’s institutional documents, rather than being overtly evaluative, the attitudinal positions they project are often discreet and require careful monitoring from the analyst to be analysed accurately. Therefore, to conduct the analysis as profoundly as possible, I declared it effective to mix quantitative and qualitative methods to be able to study evaluative patterns throughout the data and to closely inspect particular instances of evaluation in their specific contexts. There are six building blocks on which the appraisal analysis is based:

• 1. Manual annotation

• 2. Corpus tool assistance

• 3. Not limiting the analysis to individual words

• 4. Allowing layered codings

• 5. Quantifiability

• 6. Context-boundness

The first and second points on the list are closely related. The annotation process is based on manual work conducted with the help of the UAM CorpusTool, the main purpose of which is to act as a storing and counting device. It offers an interface for the user to conduct the annotations but is rather a passive participant for it cannot recognise appraisal on its own. Furthermore, a corpus tool helps in staying consistent, enabling constant revision and making changes with a few clicks. Since the corpus consists of institutional texts in which evaluations are usually very subtle and implicit appraisal is favoured, manual search was the most viable option as otherwise it would have been practically impossible to trace all the cases of appraisal. Keyword searches and reliance on explicit appraisal can be of help at the initial stages but they only get the analysis to a certain point due to diversity of evaluative items and targets of evaluation and due to many implicit cases of appraisal appearing “alone,” that is, without explicit instances in the vicinity.

Thirdly, since I have argued for the relevance of including co-text (and context), I did not limit the annotations to individual words but dealt with meaningful units instead. Especially in the case of implicit appraisal, it is not always possible to separate the evaluative item from its immediate co-text as it is entwined into it and as the correct understanding depends on it. The fourth point on the checklist is that I allow layered codings in the analysis in order to catch both the surface level and the underlying evaluations. Evaluation is multidimensional; for example, explicit appreciations towards EU policies or migration often convey implicit judgements towards the EU or migrants respectively. These kinds of veiled judgements tell a story of their own as there is a reason behind evading direct judgement towards the EU’s shortcomings for instance. It is also important to pay attention to those instances where evaluation towards a group of migrants is concealed by seemingly impersonal expressions which nonetheless entail an attitudinal stance towards them.

The fifth point revolves around countability. Every instance of appraisal has to be accounted for, and the corpus tool provides the statistics automatically (i.e. displays the exact numbers for each appraisal type) and enables making comparisons between targets of evaluation. Finally, point six is that all instances of appraisal have to be context-bound so that it is possible to determine who is evaluating what or who and how and eventually why. Even if official texts are often seen as plain and neutral, they are still tightly connected to certain culturally and socially bound ideas shared between the institution and its members (for details see e.g. ‘groups’ and ‘institutions’ in van Dijk 1998: 141, 186). The EU’s core values and the ideological beliefs it promotes largely determine what kinds of evaluative and attitudinal expressions appear in its discourses. However, it is the evaluations and attitudes that run counter to these that are of special interest, and therefore, their frequency and quality in different EU documents can reveal discrepancies between the EU’s projected values and its (discursive) actions.

Fundamental values and economic growth lie at the centre of the EU migration policy, and in its strategy, the EU envisions an open and secure Europe that recognises both the opportunities and challenges of international mobility (European Commission 2014). Increased flows of refugees in recent years combined with the EU member states suffering from the after-effects of the economic crisis have caused the EU to re-evaluate its views on migration, and this is also reflected in the migration documents which put an even stronger emphasis on economic values, security and protection than before. The ultimate goal of the appraisal analysis here is to map out the ways in which the evaluation of migration and migrants mirrors the EU’s ideological networks and the changes therein in order to infer the motives that drive the migration policy.

The checklist was originally created for the purposes of my doctoral dissertation project (Tupala in prep. n.d.), and the intention in this article was to clarify the annotation process and to propose ways to conduct quantitative appraisal analysis in a manner that connects it to the institutional and ideological context where the EU’s migration texts were produced. As I already indicated in the first section, previous quantitative appraisal studies have mainly focused on specific instances in rather restricted sets of data. My aim, however, is to promote the use of the appraisal framework in larger scale quantitative studies that not only account for each instance of appraisal and make use of a broad range of appraisal types (with their sub-systems/categories) and their layeredness but also ultimately chart the co−/context of evaluative expressions. Following the points on the checklist can hence lead to deeper and more grounded quantitative appraisal analyses. Detailed manual annotation, although very time-consuming with a large set of data, allows the analyst to find the rare, unpredictable and implicit instances of evaluation. The use of a corpus tool enables the coding of the co-text of cases of appraisal which, when combined with the appraisal annotation, binds the evaluative items to their targets of evaluation.

With the systematic account of all the cases of appraisal in a corpus, it is possible to see the ways in which evaluation towards different phenomena is constructed and how it is integrated into the structures of formal institutional discourse. It is common for EU institutions to use expressions of engagement, as meta-text before discussing the issue at hand, to demonstrate their level of commitment or stance towards it (see example 6). The scrutiny of implicit cases in particular can reveal underlying attitudes that have become the norm in institutional texts (see example 4: the EU referring to population ageing in order to rationalise the urgency to increase labour migration). The study of frequencies and tying the findings to the specific institutional context opens up new avenues for applying the appraisal framework. For instance, it allows the investigation of the extent to which a political institution, such as the EU, makes use of evaluative language in its official discourses and how (and if) evaluative choices reflect the ideological base of the institution. Furthermore, when combined with methods extracted, for example, from critical discourse analysis and political sociology, my method offers the possibility of examining the ways in which institutional constraints and social and economic forces shape attitudes in institutional discourses.¹⁴