1. Introduction

The study of emotions in language has received a wide interest in the last decades, especially in the specific alley of opinion mining, or sentiment analysis (Birjali et al., 2021; Pang & Lee, 2008). Emotions are a complex field where language, biology, psychology, society and even history (Boddice, 2018) interact (Barrett et al., 2018). In spite of the amount of recent research, there is still a lack of consensus on many levels. For example, there are not even consensual emotion categories that all or most scholars recognize. Many researchers, in fact, take a simplified view and only consider polarity (i.e., positive or negative). If we add the variable language, and the fact that each language has different lexical items and grammatical constructions to express emotion, it is clear that a lot remains to be done in this matter.

In this paper we consider reference to emotion in Portuguese. We are neither concerned with how texts may raise emotion in the readers nor how authors express their emotions. Our concern is to study the way words denoting emotion are used in text. We do not follow a particular theory of what an emotion is in psychological or physiological terms, and limit ourselves to the linguistic properties of emotion words. For surveys of other approaches and discussion of several theories, see the works by Maia and Santos (2018) and Santos and Maia (2018).

In this paper we want to evaluate a publicly available resource that provides emotion annotation in large Portuguese corpora (Santos, Simões & Mota, 2022). To build it, we used primarily our linguistic intuition as to whether a word denoted an emotion, and refined this process by consulting dictionaries, encyclopaedias, and corpora, producing 26 emotion groups in Portuguese, named by the most frequent members of the group.¹ See Figure 1 for a quantitative overview, and translation, of those emotion groups, from the original paper.

Figure 1

Table from Santos, Simões and Mota (2022), where the following properties are specified: the name of the group, its translation into English, the number of occurrences of words annotated with group, and the lexical diversity of each group in lemmas.

Note that we do not claim that there are only 26 types of emotions in Portuguese. In fact, we have a group called OUTRA (‘other’), which explicitly groups emotion words for which we have not yet created a dedicated group. In addition, we have a group called AUSENCIA (‘absence’) which marks words that concern absence of emotion, like impávido (‘unmoved’) – the justification to do this is the analogy with the semantics of colour and clothing, where we have postulated a similar absence category that deals with words like incolor (‘colourless’) or nu (‘naked’). We have also, just like in these other semantic categories, created a GEN (“generic”) group, which deals with emotion in general – again, examples from the colour and clothing domains would be colorido (‘coloured’) and vestir (‘to dress’). We use this group in the emotion domain for words like sentimento (‘feeling’) or emocionado (‘moved’).

So, the emotion groups are just a quantitatively-based first approach to the categories that are recognized by Portuguese. If we noticed that a considerable set of distinct words could be assigned to a group (which would thus have a high lexical diversity), we created the group. But one may also state that what we have (so far) achieved is the identification of 23 groups for which we assigned a label, and that every individual word in the OUTRA (‘other’) category is a potential emotion group in Portuguese. We will be mindful to treat the cases of AUSENCIA (‘absence’), OUTRA (‘other’) and GEN (“generic”) groups in a special way.

Then, we have devised a considerable number of rules, using the approach sketched by Santos and Mota (2010) for human-machine cooperation, in order to deal with the inescapable property of language that most words have more than one meaning. We exemplify in the aforementioned paper that, in the case of emotion, there are endless cases where a given word describes an emotion in some contexts, and something not emotional in others, and propose a lexical measure called the degree of emotionality of a word. We also note that it is common for words to convey more than one emotional meaning, which entails that some words are assigned to more than one group. All this has been discussed and exemplified in several papers by Mota and Santos (2015), Santos and Mota (2015), Ramos, Santos and Freitas (2020) and Santos, Simões and Mota (2022), but these facts result in additional problems for our studies on the basis of the annotated corpora. In particular, how to deal with words which are assigned to more than one emotion category. We will come back to this issue later.

The purpose of our studies on top of this material (the resources just described) is to evaluate it properly and independently, in order to assess its value for the community that deals with Portuguese at large. It would be extremely reassuring if one could find automatic methods that could confirm our intuitions² and/or help us improve the groups. Although Santos, Simões and Mota (2022) have presented some data that confirms our intuition, still the following questions remain: how can one evaluate it properly, and not in so general terms? How can one be sure that those corpora provide more than our subjective opinion of what an emotion is? And particularly, we want to investigate whether it is possible to independently motivate the postulated emotion groups – is there any statistical data that lend them support?

To be more concrete, let us take three examples:

• Can big data-based automatic methods support the existence of the two groups FELIZ (‘happiness’) and SATISFEITO (‘satisfaction’) that were postulated, or on the other hand do the methods suggest their merging?

• Can statistical processing support that concepts like mistrust and despair should belong to the same group as they are now in DESESPERO (‘despair’), or does language use show that they should be divided into two groups?

• Is there any statistical evidence that the group AMOR (‘love’) should also encompass friendship, as it does now?

So, the explorations described in the remainder of the paper are different attempts to use automatic techniques to evaluate the aforementioned annotation, and to investigate whether, by taking a quantitative bird’s eye view, one can generate further knowledge on emotions in Portuguese.

It should also be said upfront that, no matter how many revision rules were designed to improve the (automatic) annotation, it is humanly impossible to review and correct the million annotated cases. So, the studies reported here will help identify specific problems and uncover places to improve the annotation. This will be clear when we analyse our findings.

Emotion annotation is an on-going process, in the sense that, at the same time we are doing these studies, other annotators are painstakingly analysing particular cases and improving the rules, as reported e.g., by Ramos (2021). So, what we present here concerns a particular time slice of the annotated corpora (October 2021), and our most relevant contribution should be the methods we use and propose.

We will be using two different approaches: co-occurrence information, and word embeddings, a vectorial representation of words based on their co-occurrence in large corpora.

2. Further information about the data we use

The main corpus used is Todos, merging all AC/DC corpora (Santos, 2014) together and removing repeated material. The corpus is described in more detail in Santos, Simões and Mota (2022). It purports to include mainly written Portuguese in several genres (newspaper, academic writing, interviews), amounting to ca. 1.5 billion tokens (as of October 2021), most of them (3/4) in Brazilian Portuguese. A small part (6.5%) is transcribed oral data (see Santos, 2016, on the different kinds of oral corpora). A subset of Todos which we thought interesting to explore too was Literateca, containing all literary text after removing repeated texts, amounting to ca. 40 million words. Literateca features mostly old texts (not contemporary Portuguese) and, contrary to Todos, mostly Portuguese from Portugal (70%) (Todos includes Literateca).

We believe in the importance to document the options taken to deal with the material, because they may be essential for replication, and for interpretation. We have thus used the following information from the corpora: tokenization, sentence separation, lemmatization (done by the PALAVRAS parser, see Bick, 2000, 2007, 2014) and semantic annotation for the emotion domain, as displayed in Figure 2.

Figure 2

A fictive example of the material, where we show the three columns we made use of: word, lemma and semantic annotation. Note that the annotation does not mark the word sentimento in the context of sentimento de X (“feeling of X”), X being an emotion, as explained in Ramos and Freitas (2019). The example also illustrates one case of a verb with enclitics, whose lemma is the two lemmas concatenated by the + sign: trazer is the lemma of the verb form traz, and nós is the lemma of the pronoun form nos.

For the co-occurrence approach, no further preprocessing was necessary. In order to look at the particular groups Amor and Desespero, some manual pruning was done to the lemmas that were used in our graphs: we removed (a) cases where PALAVRAS had creatively added a derived lemma, such as atagonia or paragonimíase (incorrectly parsed as derivations from agonia); (b) cases of misspelled words that might be emotion words but appear too rarely to consider listing them in the emotion group, like afectuiar or deprezador or desesperadamante; (c) cases with non-standard capitalization or hyphenation, such as aMIGO or en-ternecer. We also removed hapax legomena, and added together all cases of clitics, so that lemmas amar+ele, amar+se, amar+ele+o and amar were lumped together under the lemma amar. This shrank the lemmas belonging to the Amor group from 1162 to 87, and those of the Desespero group from 246 to 64 cases.

For the creation of word embeddings, we additionally removed all capitalization, and used only words with a frequency higher than 5. Four kinds of word embeddings were created:

1. standard, using the bare corpus, without any kind of word annotation

2. changing the words marked as emotional to emo:word

3. changing the words marked as emotion to emo:word::group

4. replacing the words marked as emotion with their group: emo:group

To make this more concrete, the word empertigado (‘stiff’) from the Orgulho (‘pride’) group would have been coded like this in each approach:

1. empertigado

2. emo:empertigado

3. emo:empertigado::orgulho

4. emo:orgulho

This means that our word embeddings created on Todos had different sizes:

1. 1,171,525 word vectors

2. 1,177,040 word vectors

3. 1,177,420 word vectors

4. 1,163,962 word vectors

One should also note that clitics were not removed for the word embeddings, which means that for example the two words admirei and admirei-me count separately.

For the embeddings creation, we considered using Word2Vec (Mikolov et al., 2013), FastText (Bojanowski et al., 2017) or Glove (Pennington et al., 2014). Some authors (such as Romanov & Khusainova, 2021) claim that FastText gives better results for morphology queries, while the two others are more semantically aware. As to the choice between Word2Vec and Glove, the first uses a global word-to-word co-occurrence measure, while the second tries to work within local context. We chose Word2Vec.

To create the embeddings, we left most of Word2Vec’s default options unchanged, but we used a dimension of 300 instead of 200 and 20 training iterations instead of 5. Three hundred was chosen to be comparable to most other public word embeddings for Portuguese, and our expectation was that increasing the number of training iterations would increase quality.³

3. Creating a co-occurrence graph

One standard way to reduce large corpora to quantitative objects easy to manipulate, and reduce their dimensionality, is simply counting the number of times different concepts, or words, co-occur, taking this as a measure of relatedness or even similarity.

This was our first approach, which we applied to words annotated with emotion, in two different ways:

1. simply using the emotion group (any word annotated as belonging to the group Feliz (‘happiness’) would count as Feliz, so felicidade and ventura (both translateable by happiness) would both count as Feliz (‘happiness’)

2. or using the word itself, for particular emotion groups (so, in the Feliz group, felicidade and ventura would count as felicidade and ventura, respectively).

Since the corpora are parsed by PALAVRAS, we operationalized co-occurrence as “appearing in the same sentence”, marked by the structural attribute <s>. We believe this to be more linguistically motivated than deciding on a fixed window of N words.

Figures 3 and 4 show the relationship between the emotion groups in Todos and Literateca, drawn like a graph, whose vertices are the group labels, with size (diameter) proportional to their frequency, and whose edges correspond to the attested co-occurrences, drawn with thickness corresponding to the number of co-occurrences.

Figure 3

All emotion groups in Todos, using the R (R Development Core Team, 2008) package igraph (Csardi & Nepusz, 2006).

Figure 4

All emotion groups in literary text (Literateca).

It should be clarified that we use a random layout for drawing the graphs, which implies that the place where the particular categories appear is not meaningful – and therefore should not be compared across graphs. Also, it is important to explain that the sizes are relative to the universes, which differ widely in quantity: Literateca has about 40 million words, while Todos (which includes Literateca) amounts to a total of 1,315 million words.

With these caveats in mind, what can these graphical depictions tell us, as an initial overview of the annotated corpora? We can conclude that literary text in Portuguese (at least the one present in Literateca), has a stronger emphasis on AMOR (‘love’) and (In)Feliz (‘(un)happiness’) than other kinds of text. And that, in general, the most invoked emotion is Desejo (‘desire’) followed by Esperanca (‘hope’) and Amor (‘love’). Initially surprised by the unexpected prominence of Desejo, we soon understood its cause: words like querer (‘want’) and desejar (‘desire’) were considered as emotion words, even when they might arguably be considered only denoting volition or intent. If this were not the case, the Desejo group would considerably shrink. In fact, by investigating the matter closer, we also realized that our co-occurrence counting procedure counts words annotated, e.g., with desejo_amor for both the Desejo and Amor groups, therefore inflating even more the (possibly dubious) contribution of the verb desejar to the Portuguese emotion realm. In any case, this illustrates that the original decision to attribute as many emotion labels as deemed appropriate can have consequences on the further processing of the material. Since we are not sure what the best alternative to deal with these cases is – and suspect that they may, in fact, indicate a desirable merge of the groups in question –, we did not compute an alternative co-occurrence matrix.

In Figures 5, 6, 7 to 8, we present now the two groups Amor (‘love’) and Desespero (‘despair’) for both the Literateca and the Todos material, together with the pairs with more co-occurrences, in Tables 1 and 2.

We can thus observe several specific differences between the group Amor (‘love’) in general (in all genres present), and in literary text: it is easy to see that while romantic love is the most described in the literary texts of Literateca, amigos (‘friends’), gostar (‘to like’) and preferir (‘to prefer’) are the most common members of Amor in other genres. Interestingly, abraçar and beijar (‘hug’ and ‘kiss’) rank high in the literature list, preferir (‘to prefer’) or desejar (‘to wish for’) are more frequent, when one looks at all genres together.

In Table 2, we present again the most common co-occurrences, now for the Desespero group. Note that the quantities are much lower than in Table 1, because reference to this group is far less frequent in Portuguese, at least according to the corpora we are using (cf. Figure 1).

Although the material has far fewer instances to analyse, reference to mistrust – represented in Table 2 by the words desconfiança e desconfiar – does not occur often in literary texts, contrarily to general texts. And it seems that (from the most common co-occurrences only) the words related to mistrust and those related to despair keep separate. This is in sharp contrast with love and friendship, where there were plenty of co-occurrences, as can be appreciated in Table 1.

Figure 5

The words belonging to the Amor (‘love’) group in Todos.

Figure 6

The words belonging to the Amor (‘love’) group in literary text (Literateca).

Figure 7

The words belonging to the Desespero (‘despair’) group in Todos.

Figure 8

The words belonging to the Desespero (‘despair’) group in Literateca.

After this preliminary bird’s eye view, we have applied several clustering techniques to the co-occurrence material, which we proceed to describe below.

The idea of clustering, a non-supervised exploratory technique, is to identify meaningful groups (“clusters”) in large amounts of data. There are two major ways to proceed: divisive clustering, which starts by dividing the material, and agglomerative clustering, which proceeds bottom-up by joining the closest elements. In any case, these processes depend on a distance measure between the objects to be clustered, and there are many different distance measures to choose from.

For the co-occurrence data, we considered that co-occurrence between two words (or emotion groups) measured how close they were, and defined distance as the inverse of the co-occurrence number (so, if X co-occurred 43 times with Y, their distance would be 1/43). When there were no co-occurrences at all in the material, we assumed infinite distance. We then applied multidimensional scaling with two and three dimensions to the emotion group co-occurrence data. The result for two dimensions is in Figure 9.

Figure 9

Multidimensional scaling of the data obtained by co-occurrence of the emotion groups in Todos.

The result shows that dimension 2 clearly separates emotions from their absence. However, it is hard to understand what it is that dimension 1 captures, singling out two not very typical emotion groups: Ingrato (‘ingratitude’) and Inveja (‘envy’), in fact the two groups with lowest lexical diversity in Portuguese according to the corpora (cf. again Figure 1).

If we redo multidimensional scaling requiring three dimensions, see Figure 10, dimension 3 now singles out Saudade (‘longing’), which is the next least lexically diverse emotion group, apart from Insatisfeito (‘insatisfaction’).

Figure 10

Multidimensional scaling of the data obtained by co-occurrence of the emotion groups in Todos, with 3 dimensions.

Trying hierarchical clustering (with R’s hclust command), we get a similar result, see Figure 11.

Figure 11

Hierarchical clustering of the data obtained by co-occurrence of the emotion groups in Todos, by two different methods.

Table 3a shows the most frequent emotion group co-occurrences.

Most of these numbers can be interpreted straightforwardly: one joins quasi-synonyms (Feliz (‘happy’) and Satisfeito (‘satisfied’)), another shows the cohesiveness of the same group (Amor (‘love’)), while others join feelings that often come together, like Odio (‘hate’) and Vergonha (‘shame’), Vergonha (‘shame’) and Furia (‘anger’), or Desejo (‘desire’) and Esperanca (‘hope’), Amor (‘love’) and Feliz (‘happiness’), and Admirar (‘admiration’) and Humildade (‘humility’). Finally, one may also interpret Orgulho (‘pride’) as a cause for Furia (‘anger’), although obviously not always. It is nevertheless interesting that no antonyms come to the fore: all pairs are either both positive or both negative.

One should also recall that there is a significant number of word occurrences that are marked as belonging to Amor (‘love’) and Desejo (‘desire’) or Amor (‘love’) and Esperança (‘hope’), and these would inflate (artificially, in fact) the number of the co-occurrences of the two categories. Namely, all words marked as belonging to a double or triple category count as co-occurrences among these categories. This is something that we have to deal with, and an alternative Table 3b was therefore created without those cases.

We see that the quantities are considerably smaller than those of Table 3a, showing that many of these co-occurrences involved (or were a product of) vague categories. In this new table, there are three emotion groups that co-occur with themselves: Amor, Desejo and Medo. However, the categories that are included in vague classifications continue to be frequently co-occurring, which in a way vindicates the existence of words that convey both.

4. Investigating word embeddings

For almost a decade now, the technique of using large amounts of data to produce (static) word embeddings has been actively used in many different NLP tasks in order to provide a better representation of a word’s meaning, and has also been applied in other linguistic and literary contexts (Antoniak & Mimno, 2018). Although there are fortunately several word embeddings for Portuguese (see Batista, 2019, for an overview of Hartmann et al., 2017; Rodrigues & Branco, 2018, Grave et al., 2018 and Yamada et al., 2016; and Santos, 2021, for a recent comparison among them), we decided to create our own embeddings based on precisely the data we wanted to analyse, also because, as explained in Section 2 above, we tried four kinds of word embeddings.

However, one thing that stood out was that there is a scarcity of research that uses clustering over word embeddings. Tang et al. (2014) claimed that poor results of clustering over word embeddings are due to the fact that traditional word embeddings are based on substitutability, not similarity, and so “they cannot distinguish words with similar context but opposite sentiment polarity (e.g., good and bad)” (Tang et al., 2014, p. 1563). This means that, in a word embedding representation, antonyms are closer than unrelated words, since they are often substitutable. Another property of antonymy has been pointed out by Justeson and Katz (1992), who suggested that corpus co-occurrence is a textual marker for the antonymy lexical-semantic relation. In other words, antonyms tend to co-occur in text.

Before clustering, we tried to exploit the information gathered by the word embeddings in several ways, as we describe in what follows.

4.1 Emotions near emotions?

We first set out to investigate whether words annotated as emotions also have emotions as their nearest neighbours (most similar words) in word embeddings. In order to do this, we computed the most similar words for the 3 embedding models where emotions were explicitly marked (using Gensim’s (Rehurek & Sojka, 2010) method similar_by_word from its KeyedVectors module), and extracted the following statistics:

1. how many emotion words were included in the first 50 closest words

2. what was the position in the top 50 closest words of the first emotion (-1 if none was an emotion)

3. the sum of the inverse ranks of the 50 closest words which were considered an emotion.

Let us illustrate the third statistic with the help of Figure 12, which lists the 50 nearest neighbours of emo:amor. The rankings of emotion words (in bold) are then 2, 3, 4, 5, 12, 13, 14, 16, 18, 19, 20, 21, 25, 29, 30, 31, 33, 34, 36, 41, 43, 45, and 46, and the statistic amounts to 1/2 + 1/3 + 1/4 + 1/5 + 1/12 + 1/13 + 1/14 + 1/16 + 1/18 + 1/19 + 1/20 + 1/21 + 1/25 + 1/29 + 1/30 + 1/31 + 1/33 + 1/34 + 1/36 + 1/41 + 1/43 + 1/45 + 1/46, equalling 2.10.

Figure 12

The closest neighbours of emo:amor in the second kind of word embeddings, where we highlighted in bold those which were also marked as emotion by the prefix emo:. The corpora include spellings from all Portuguese varieties, and old orthographies.

Generically, $\sum _w\frac{1}{pos(w)}$ where w is each one of the similar (emotion) words, and pos(w) is the rank of that word.

This example at once shows several interesting features: the closest word to emo:amor is… amor itself! But this second amor was not considered an emotion in the context it appeared in – we can guess it probably was included in a proper noun like a movie or book title, and proper nouns were not annotated with emotions. Another observation is that encantamento (‘enchantment’), encanto (‘enchantment’), remorso (‘remorse’) and even arrebatamento (‘rapture’) do seem to us quite good emotion candidates, although they weren’t considered as such. Therefore, we can give this kind of feedback to enhance the lexicon and/or the rules, so that the corpus annotators can include these terms in the next round. Finally, the word coração (‘heart’) is also quite interesting because of the well-known relationship between body organs and emotions (see e.g., Enfield & Wierzbicka, 2002). Any Portuguese native speaker is used to the metaphor that love is located in the heart (a metaphor which is also incidentally shared by English and many other languages). Should one have also tagged (some) body parts as emotions? We have tagged body parts as possibly meaning emotions in another project (Freitas et al., 2015), but making use of a different syntax, and we have not so far converted that information into emotion annotation which could have been used in our experiments here.

These observations show that the results we obtain may not be final, and that several things could have be enhanced or done differently.

In order to make sense of all emotion words and not single out just one instance, we computed the three statistics given above for all the words of our 3 word embedding models. In Figure 13, we present a histogram of the results for the first statistic (how many words in the top 50 are also emotions), based on the second model, where words denoting emotion in context were prefixed with emo:

Figure 13

Distribution of the first statistic in the second (left) and third (right) kind of word embeddings, where words marked as emotion had the prefix emo:.

In Figure 13, we can see that there are wide differences between words. There are many emotion words (1,120 for the second word embeddings) that do not have an emotion among the closest neighbours, while others (216) have more than 40 out of 50. We assessed whether word frequency correlated with number of emotions as closest numbers, but the value of 0.11 for the Pearson’s product moment correlation coefficient means that there is no correlation at all.

We have also created a corresponding histogram for the third kind of word embeddings, because we have slightly different words (remember that the words here have a prefix added, see Section 2 above). The lexicon gets larger (the same word can be in many different groups) and embeddings may differ. In fact, the random component underlying Word2Vec’s algorithm has often been challenged with lack of stability (see Mihalcea, 2021), which means that different runs of the algorithm with the same parameters can actually produce wildly different results.

In Figure 14, we use the third kind of word embeddings (where we keep both word and group) and restricting our attention to those emotion words with at least one emotion in their closest 50 neighbours, we check whether they belong to the same emotion group. We create a fourth statistic that measures how many of the closest emotions belong to the same group.

Figure 14

Distribution of the fourth statistic for the third kind of word embeddings, where words marked as emotion were prefixed by their emotion group. Only words with at least one emotion neighbour are taken into consideration.

The results leave room for improvement. We believe that a decisive factor could be the property of antonyms that has been already discussed: words tend to co-occur with their antonyms, which are therefore closer to them than a word taken at random. So as a next step for future work, one should find a way to mark antonym emotion groups and check whether the percentages in Figure 14 would significantly increase.

4.2 Which emotion groups are the closest?

Using the fourth word embeddings, where we replaced all words of a particular group by their group name (keeping vague group names, like Amor_Desejo (‘love_desire’), as separate groups), we can use this representation to obtain similarities between groups.

The group Desejo (‘desire’) has as the closest and only neighbours the groups Amor_Desejo_Esperanca (‘love_desire_hope’), Amor (‘love’) and Desejo_Esperanca (‘desire_hope’), while Amor (‘love’) has many neighbours (16), in the following order: Feliz_Satisfeito, Feliz, Inveja, Desejo, Admirar, Odio, Satisfeito, Gen, Desespero, Esperanca, Surpresa, Desejo_Esperanca, Medo, Orgulho, Humildade_Admirar and Pena. The closest neighbours (21) of Desespero (‘despair’) are as follows: Medo, Infeliz, Surpresa, Medo_Surpresa, Furia, Insatisfeito_Outra, Alivio, Odio, Inveja, Infeliz_Desespero, Gen, Infeliz_Insatisfeito, Feliz, Ausencia, Furia_Odio, Esperança, Vergonha, Feliz_Satisfeito, Outra, Saudade and Infeliz_Vergonha.

These data are more difficult to interpret than those of the co-occurrence experiments, but we may hypothesize that the Desespero (‘despair’) group is the one with the most neighbours precisely because it has fewer occurrences and therefore it is more difficult to become autonomous. Conversely, the fact that Desejo (‘desire’) has only three group neighbours may show that either it is quite different/far away from the rest of emotions, and/or that it belongs to another (possible) cluster, namely that of volition.

One could use these similarities both to rank these groups in terms of their actual emotionality, and to measure their proximity with other emotions.

In Table 4 we show the raw results (for these groups).

Table 4

The closeness among some groups using the fourth kind of word embeddings: after the group comes the Gensim similarity.

emotion group	close groups	Gensim similarity
emo:desejo	emo:amor_desejo_esperanca	0,74783319
	emo:amor	0,60807204
	emo:desejo_esperanca	0,59662843
emo:amor	emo:feliz_satisfeito	0,65850675
	emo:feliz	0,62952852
	emo:inveja	0,62798548
	emo:desejo	0,60807198
	emo:admirar	0,60569620
	emo:odio	0,59930396
	emo:satisfeito	0,55990100
	emo:gen	0,54665804
	emo:desespero	0,53727543
	emo:esperanca	0,51752120
	emo:surpresa	0,50839704
	emo:desejo_esperanca|	0,48341280
	emo:medo	0.47919095
	emo:orgulho	0,47487900
	emo:humildade_admirar	0,47156897
	emo:pena	0,46538004
emo:desespero	emo:medo	0,79399341
	emo:infeliz	0,69591373
	emo:surpresa	0,68163306
	emo:medo_surpresa	0,67511058
	emo:furia	0,66599011
	emo:insatisfeito_outra	0,66186285
	emo:alivio	0,64180046
	emo:odio	0,64122468
	emo:inveja	0,64011359
	emo:infeliz_desespero	0,63227987
	emo:gen	0,63055670
	emo:infeliz_insatisfeito	0,62733608
	emo:feliz	0,62732589
	emo:ausencia	0,61817032
	emo:furia_odio	0,61506921
	emo:esperanca	0,61258727
	emo:vergonha	0,60081607
	emo:feliz_satisfeito	0,57563734
	emo:outra	0,57479715
	emo:saudade	0,55613655
	emo:infeliz_vergonha	0,54775155

In order to provide a better way to understand these data, we have also tried to create partial pictures for each emotion, in Figure 15.

Figure 15

Representing the data of Table 4 in graph form, by (manually) drawing the other emotions with a distance inversely proportional to their similarity.

4.3 What about the pure word embeddings?

So far, we have only used our three word embedding models that make use of the annotation. But could anything be learned, or discovered, from the traditional word embeddings that require no corpus annotation? We could even compare the results with other word embeddings for Portuguese.

So, we decided to try them out by choosing the words amor and desespero (the ones that, after all, gave the names to the groups we have been looking at) and manually identify their close neighbours that are emotions, in Figure 16.

Figure 16

The 50 closest words with the first word embeddings for the words amor and desespero, where the non-emotions are in bold, and words from the same group or antonyms are in italic.

We see that 21 out of 50 are not emotional words for amor, while only 11 out of 50 do not describe emotion for desespero. This means that regular word embeddings fare very well compared with those created with explicit annotation. However, these test cases may not have been the best ones, since they correspond to cases where the word (amor, desespero) is always an emotion, as opposed to pena (‘sorrow’, ‘feather’, ‘punishment’, ‘pen’, etc.) or reconhecer (‘be grateful’, ‘recognize’, etc.).

5. Related work

We report briefly on three different alleys in related work: looking at emotions in text and not only polarity; clustering emotions; and interpreting results with word embeddings.

Parallel to sentiment analysis, there is a (much smaller) research alley that pursues what has been called “emotion annotation” or “emotion detection”, represented by Maia (1994), Aman and Szpakowicz (2007), and Ptaszynski et al. (2014). For a review, see Seyeditabari et al. (2018). There has also been some work on precisely the literary domain to identify emotions, examples of which are Mohammad (2012) and Kim et al. (2017). Our work is definitely included in this tradition. Compared to the above works, we are using considerably larger amounts of text.

As to clustering emotions, that is, using empirical methods to identify how emotion is structured in a language, we are only aware of a few works: Feng et al. (2011) use a sentiment lexicon in order to cluster blogs, and extract the most common words in each of their (eight) clusters; Hu et al. (2009) classify Chinese lyrics by their emotions, using fuzzy clustering. Again, our work addresses far more textual data and more text genres.

The work closest to ours, Tang et al. (2014), creates what they call sentiment-specific word embeddings (SSWE). But, as the name indicates, they use “sentiment” (positive or negative) and not emotion. They also use the top-100 closest words in their word embeddings to evaluate the polarity consistency of different sentiment lexicons. Other researchers trying to adapt word embeddings to emotion or sentiment use techniques like retro-fitting or counter-fitting emotion lexicons onto the “ordinary” word embeddings, producing other kinds of models, see e.g. Speer and Chin (2019). Although these works are valuable to suggest new alleys for using word embeddings in the study of emotions, we believe our different attempts are also worth pursuing, and enrich this relatively young area.

6. Concluding remarks

These exploratory experiments only scratch the surface of what can be done having such a rich resource at our fingertips, the purpose of which is to provide a wide picture of the reference to emotions in Portuguese. We tried to explore this resource with several big data techniques, but we hope that this is just the beginning of a new research area in the years to come, especially because we have made the resources and the methods we employed in the explorations described here publicly available,⁴ so we hope to see others follow suit.

There are a number of experiments we still wish to perform, from merely testing different word embeddings approaches like FastText or GloVe, experimenting with different preprocessing strategies, adding other annotation information (part-of-speech, functional dependency, morphological or other semantic properties), investigating text genre, time period, and so on.

Also, there are other approaches in the word embeddings world that seem worth trying, from box embeddings (Abboud et al., 2020) to comparison with other public word embeddings for Portuguese, as well as the creation of word embeddings per genre, which has often been proposed in the literature (Tshitoyan et al., 2019).

Likewise, we only looked at the emotion groups Amor (‘love’) and Desespero (‘despair’). More than twenty further groups, as well as their possible mergings (for example, by joining together antonyms), could and should be investigated in order to learn more about emotion in Portuguese, and to evaluate thoroughly the particular annotation available.

Concerning the three initial motivating questions, our preliminary conclusions based on the available material are as follows:

• There is evidence to merge Feliz (‘happiness’) and Satisfeito (‘satisfaction’), from clustering word embeddings.

• The Desespero (‘despair’) group is apparently extremely wide, since it displays connections with many different emotions, which may mean that it joins disparate things and should be divided.

• There seems to be no reason to separate friendship and love. They seem to belong to the same group, called Amor (‘love’).

As one reviewer noted, there are no predefined predictions from previous studies of what emotions groups might look like in Portuguese, so we cannot contrast our groups with others in the literature.

Finally, and concerning the limitations of our approach to comparative work, clearly we are only discussing Portuguese here. Similar studies have to be done for other languages before one can contrast Portuguese with them. It is noteworthy to emphasize that we, with Wierzbicka (1999) and many others (Jackson et al., 2019), do not believe emotion concepts work similarly in different languages.⁵ Otherwise, there would be no point in doing emotion studies in Portuguese, given that there are more resources in general for English linguistics.

Competing Interests

The authors have no competing interests to declare.