# Chapter 7. Evaluations and Loss components¶

## Evaluations¶

This part evolves the main evaluation metrics for various tasks. These metrics derive from the save base class and have the same interface for easy use.

For classification problems, we implement: precision, recall, F1 and accuracy metrics

```
import torch
from graph4nlp.pytorch.modules.evaluation.accuracy import Accuracy
ground_truth = torch.Tensor([0, 1, 3, 1, 1]).long()
predict = torch.Tensor([0, 1, 3, 3, 1]).long()
metric = Accuracy(metrics=["precision", "recall", "F1", "accuracy"])
precision, recall, f1, accuracy = metric.calculate_scores(
ground_truth=ground_truth, predict=predict, average="micro")
```

For generation tasks, we implement 6 metrics including: 1) BLEU (BLEU@1-4), 2), BLEUTranslation(SacreBLEU), 3)CIDEr, 4)METEOR, 5) ROUGE (ROUGE-L), 6)SummarizationRouge (implemented by pyrouge).

```
from graph4nlp.pytorch.modules.evaluation import BLEU
bleu_metrics = BLEU(n_grams=[1, 2, 3, 4])
prediction = ["I am a PHD student.", "I am interested in Graph Neural Network."]
ground_truth = ["I am a student.", "She is interested in Math."]
scores = bleu_metrics.calculate_scores(ground_truth=ground_truth, predict=prediction)
```

## Loss components¶

We have implemented several specific loss functions for various tasks.

### Sequence generation loss.¶

We have wrapped the cross-entropy loss and the coverage loss(optional) to calculate the final loss for various sequence generation tasks (e.g., graph2seq, seq2seq).

```
from graph4nlp.pytorch.modules.loss.seq_generation_loss import SeqGenerationLoss
from graph4nlp.pytorch.models.graph2seq import Graph2Seq
loss_function = SeqGenerationLoss(ignore_index=0, use_coverage=True)
logits, enc_attn_weights, coverage_vectors = Graph2Seq(graph, tgt)
graph2seq_loss = SeqGenerationLoss(logits, tgt, enc_attn_weights=enc_attn_weights, coverage_vectors=coverage_vectors)
```

### Knowledge Graph Loss¶

In the state-of-the-art KGE models, loss functions were designed according to various pointwise, pairwise and multi-class approaches. Refers to Loss Functions in Knowledge Graph Embedding Models

**Pointwise Loss Function**

1. MSELoss
creates a criterion that measures the mean squared error (squared L2 norm)
between each element in the input \(x\) and target \(y\). It is the wrapper of `nn.MSELoss`

in pytorch.

2. SOFTMARGINLOSS Creates a criterion that optimizes a two-class classification
logistic loss between input tensor \(x\) and target tensor \(y\)
(containing 1 or -1). It is the wrapper of `nn.SoftMarginLoss`

in pytorch.

The number of positive and negative samples should be about the same, otherwise it’s easy to overfit.

**Pairwise Loss Function**

1. SoftplusLoss refers to the paper OpenKE: An Open Toolkit for Knowledge Embedding

```
class SoftplusLoss(nn.Module):
def __init__(self, adv_temperature=None):
super(SoftplusLoss, self).__init__()
self.criterion = nn.Softplus()
if adv_temperature != None:
self.adv_temperature = nn.Parameter(torch.Tensor([adv_temperature]))
self.adv_temperature.requires_grad = False
self.adv_flag = True
else:
self.adv_flag = False
def get_weights(self, n_score):
return torch.softmax(n_score * self.adv_temperature, dim=-1).detach()
def forward(self, p_score, n_score):
if self.adv_flag:
return (self.criterion(-p_score).mean() + (self.get_weights(n_score) * self.criterion(n_score)).sum(
dim=-1).mean()) / 2
else:
return (self.criterion(-p_score).mean() + self.criterion(n_score).mean()) / 2
```

2. SigmoidLoss refers to the paper OpenKE: An Open Toolkit for Knowledge Embedding

```
class SigmoidLoss(nn.Module):
def __init__(self, adv_temperature = None):
super(SigmoidLoss, self).__init__()
self.criterion = nn.LogSigmoid()
if adv_temperature != None:
self.adv_temperature = nn.Parameter(torch.Tensor([adv_temperature]))
self.adv_temperature.requires_grad = False
self.adv_flag = True
else:
self.adv_flag = False
def get_weights(self, n_score):
return torch.softmax(n_score * self.adv_temperature, dim = -1).detach()
def forward(self, p_score, n_score):
if self.adv_flag:
return -(self.criterion(p_score).mean() + (self.get_weights(n_score) * self.criterion(-n_score)).sum(dim = -1).mean()) / 2
else:
return -(self.criterion(p_score).mean() + self.criterion(-n_score).mean()) / 2
```

The implementations of `SoftplusLoss`

and `SigmoidLoss`

refer to OpenKE.

**Multi-Class Loss Function**

1. Binary Cross Entropy Loss
Creates a criterion that measures the Binary Cross Entropy between the target and the output. Note that the targets
\(y\) should be numbers between 0 and 1. It is the wrapper of `nn.BCELoss`

in pytorch.

Next it is a simple how to use code:

```
import torch
from graph4nlp.pytorch.modules.loss.kg_loss import KGLoss
loss_function = KGLoss(loss_type="BCELoss")
m = nn.Sigmoid()
input = torch.randn(3, requires_grad=True)
target = torch.empty(3).random_(2)
output = loss_function(m(input), target)
```

### General Loss¶

It includes the most used loss functions containing:

`NLL`

loss. It is the wrapper of`nn.NLLLoss`

in pytorch.`BCE`

loss. It is the wrapper of`nn.BCELoss`

in pytorch.`BCEWithLogits`

loss. It is the wrapper of`nn.BCEWithLogitsLoss`

in pytorch.`MultiLabelMargin`

loss. It is the wrapper of`nn.MultiLabelMarginLoss`

in pytorch.`SoftMargin`

loss. It is the wrapper of`nn.SoftMargin`

in pytorch.`CrossEntropy`

loss. It is the wrapper of`nn.CrossEntropy`

in pytorch.

Next it is a simple how to use code:

```
import torch
from graph4nlp.pytorch.modules.loss.general_loss import GeneralLoss
loss_function = GeneralLoss(loss_type="CrossEntropy")
input = torch.randn(3, 5)
target = torch.empty(3, dtype=torch.long).random_(5)
output = loss_function(input, target)
```