graph4nlp.loss

Losses

class graph4nlp.loss.CoverageLoss(cover_loss)

The loss function for coverage mechanism.

Parameters

cover_loss (float) – The weight for coverage loss.

Methods

add_module(name, module)	Adds a child module to the current module.
apply(fn)	Applies fn recursively to every submodule (as returned by .children()) as well as self.
bfloat16()	Casts all floating point parameters and buffers to bfloat16 datatype.
buffers([recurse])	Returns an iterator over module buffers.
children()	Returns an iterator over immediate children modules.
cpu()	Moves all model parameters and buffers to the CPU.
cuda([device])	Moves all model parameters and buffers to the GPU.
double()	Casts all floating point parameters and buffers to double datatype.
eval()	Sets the module in evaluation mode.
extra_repr()	Set the extra representation of the module
float()	Casts all floating point parameters and buffers to float datatype.
forward(enc_attn_weights, coverage_vectors)	The calculation function.
get_buffer(target)	Returns the buffer given by target if it exists, otherwise throws an error.
get_extra_state()	Returns any extra state to include in the module’s state_dict.
get_parameter(target)	Returns the parameter given by target if it exists, otherwise throws an error.
get_submodule(target)	Returns the submodule given by target if it exists, otherwise throws an error.
half()	Casts all floating point parameters and buffers to half datatype.
load_state_dict(state_dict[, strict])	Copies parameters and buffers from state_dict into this module and its descendants.
modules()	Returns an iterator over all modules in the network.
named_buffers([prefix, recurse])	Returns an iterator over module buffers, yielding both the name of the buffer as well as the buffer itself.
named_children()	Returns an iterator over immediate children modules, yielding both the name of the module as well as the module itself.
named_modules([memo, prefix, remove_duplicate])	Returns an iterator over all modules in the network, yielding both the name of the module as well as the module itself.
named_parameters([prefix, recurse])	Returns an iterator over module parameters, yielding both the name of the parameter as well as the parameter itself.
parameters([recurse])	Returns an iterator over module parameters.
register_backward_hook(hook)	Registers a backward hook on the module.
register_buffer(name, tensor[, persistent])	Adds a buffer to the module.
register_forward_hook(hook)	Registers a forward hook on the module.
register_forward_pre_hook(hook)	Registers a forward pre-hook on the module.
register_full_backward_hook(hook)	Registers a backward hook on the module.
register_parameter(name, param)	Adds a parameter to the module.
requires_grad_([requires_grad])	Change if autograd should record operations on parameters in this module.
set_extra_state(state)	This function is called from load_state_dict() to handle any extra state found within the state_dict.
share_memory()	See torch.Tensor.share_memory_()
state_dict([destination, prefix, keep_vars])	Returns a dictionary containing a whole state of the module.
to(*args, **kwargs)	Moves and/or casts the parameters and buffers.
to_empty(*, device)	Moves the parameters and buffers to the specified device without copying storage.
train([mode])	Sets the module in training mode.
type(dst_type)	Casts all parameters and buffers to dst_type.
xpu([device])	Moves all model parameters and buffers to the XPU.
zero_grad([set_to_none])	Sets gradients of all model parameters to zero.

__call__

forward(enc_attn_weights, coverage_vectors)

The calculation function.

Parameters

• enc_attn_weights (list[torch.Tensor]) – The list containing all decoding steps’ attention weights. The length should be the decoding step. Each element should be the tensor.

• coverage_vectors (list[torch.Tensor]) – The list containing all coverage vectors in decoding module.

Returns

coverage_loss – The loss.

Return type

torch.Tensor

class graph4nlp.loss.SeqGenerationLoss(ignore_index, use_coverage=False, coverage_weight=0.3)

The general loss for Graph2Seq model.

Parameters

• ignore_index (ignore_index) – The token index which will be ignored during calculation. Usually it is the padding index.

• use_coverage (bool, default=False) – Whether use coverage mechanism. If set True, the we will add the coverage loss.

• coverage_weight (float, default=0.3) – The weight of coverage loss.

Methods

add_module(name, module)	Adds a child module to the current module.
apply(fn)	Applies fn recursively to every submodule (as returned by .children()) as well as self.
bfloat16()	Casts all floating point parameters and buffers to bfloat16 datatype.
buffers([recurse])	Returns an iterator over module buffers.
children()	Returns an iterator over immediate children modules.
cpu()	Moves all model parameters and buffers to the CPU.
cuda([device])	Moves all model parameters and buffers to the GPU.
double()	Casts all floating point parameters and buffers to double datatype.
eval()	Sets the module in evaluation mode.
extra_repr()	Set the extra representation of the module
float()	Casts all floating point parameters and buffers to float datatype.
forward(logits, label[, enc_attn_weights, …])	The calculation method.
get_buffer(target)	Returns the buffer given by target if it exists, otherwise throws an error.
get_extra_state()	Returns any extra state to include in the module’s state_dict.
get_parameter(target)	Returns the parameter given by target if it exists, otherwise throws an error.
get_submodule(target)	Returns the submodule given by target if it exists, otherwise throws an error.
half()	Casts all floating point parameters and buffers to half datatype.
load_state_dict(state_dict[, strict])	Copies parameters and buffers from state_dict into this module and its descendants.
modules()	Returns an iterator over all modules in the network.
named_buffers([prefix, recurse])	Returns an iterator over module buffers, yielding both the name of the buffer as well as the buffer itself.
named_children()	Returns an iterator over immediate children modules, yielding both the name of the module as well as the module itself.
named_modules([memo, prefix, remove_duplicate])	Returns an iterator over all modules in the network, yielding both the name of the module as well as the module itself.
named_parameters([prefix, recurse])	Returns an iterator over module parameters, yielding both the name of the parameter as well as the parameter itself.
parameters([recurse])	Returns an iterator over module parameters.
register_backward_hook(hook)	Registers a backward hook on the module.
register_buffer(name, tensor[, persistent])	Adds a buffer to the module.
register_forward_hook(hook)	Registers a forward hook on the module.
register_forward_pre_hook(hook)	Registers a forward pre-hook on the module.
register_full_backward_hook(hook)	Registers a backward hook on the module.
register_parameter(name, param)	Adds a parameter to the module.
requires_grad_([requires_grad])	Change if autograd should record operations on parameters in this module.
set_extra_state(state)	This function is called from load_state_dict() to handle any extra state found within the state_dict.
share_memory()	See torch.Tensor.share_memory_()
state_dict([destination, prefix, keep_vars])	Returns a dictionary containing a whole state of the module.
to(*args, **kwargs)	Moves and/or casts the parameters and buffers.
to_empty(*, device)	Moves the parameters and buffers to the specified device without copying storage.
train([mode])	Sets the module in training mode.
type(dst_type)	Casts all parameters and buffers to dst_type.
xpu([device])	Moves all model parameters and buffers to the XPU.
zero_grad([set_to_none])	Sets gradients of all model parameters to zero.

__call__

forward(logits, label, enc_attn_weights=None, coverage_vectors=None)

The calculation method.

Parameters

• logits (torch.Tensor) – The probability with the shape of [batch_size, max_decoder_step, vocab_size]. Note that it is calculated by softmax.

• label (torch.Tensor) – The ground-truth with the shape of [batch_size, max_decoder_step].

• enc_attn_weights (list[torch.Tensor], default=None) – The list containing all decoding steps’ attention weights. The length should be the decoding step. Each element should be the tensor.

• coverage_vectors (list[torch.Tensor], default=None) – The list containing all coverage vectors in decoding module.

Returns

graph2seq_loss: torch.Tensor

class graph4nlp.loss.GeneralLoss(loss_type, weight=None, size_average=None, ignore_index=-100, reduce=None, reduction='mean', pos_weight=None)

This general loss are backended on the pytorch loss function. The detailed decription for each loss function can be found at:

pytorch loss function <https://pytorch.org/docs/stable/nn.html#loss-functions>

Parameters

loss_type: str

the loss function to select (NLL,``BCEWithLogits``, MultiLabelMargin,``SoftMargin`` ,``CrossEntropy`` )

NLL loss<https://pytorch.org/docs/stable/_modules/torch/nn/modules/loss.html#NLLLoss> measures the negative log likelihood loss. It is useful to train a classification problem with C classes.

BCEWithLogits loss <https://pytorch.org/docs/stable/_modules/torch/nn/modules/loss.html#BCEWithLogitsLoss> combines a Sigmoid layer and the BCELoss in one single class. This version is more numerically stable than using a plain Sigmoid`followed by a `BCELoss as, by combining the operations into one layer, we take advantage of the log-sum-exp trick for numerical stability.

BCE Loss<https://pytorch.org/docs/stable/_modules/torch/nn/modules/loss.html#BCELoss> creates a criterion that measures the Binary Cross Entropy between the target and the output.

MultiLabelMargin loss <https://pytorch.org/docs/stable/_modules/torch/nn/modules/loss.html#MultiLabelMarginLoss> creates a criterion that optimizes a multi-class multi-classification hinge loss (margin-based loss) between input \(x\) (a 2D mini-batch Tensor) and output \(y\) (which is a 2D Tensor of target class indices).

SoftMargin loss <https://pytorch.org/docs/stable/_modules/torch/nn/modules/loss.html#SoftMarginLoss> creates a criterion that optimizes a two-class classification logistic loss between input tensor \(x\) and target tensor \(y\) (containing 1 or -1).

CrossEntropy loss <https://pytorch.org/docs/stable/_modules/torch/nn/modules/loss.html#CrossEntropyLoss> ` combines pytorch function `nn.LogSoftmax and nn.NLLLoss in one single class. It is useful when training a classification problem with C classes.

weight: Tensor, optional

a manual rescaling weight given to the loss of each batch element. If given, has to be a Tensor of size nbatch. This parameter is not suitable for SoftMargin loss functions.

size_average: bool, optional

By default,the losses are averaged over each loss element in the batch. Note that for some losses, there are multiple elements per sample. If the field size_average is set to False, the losses are instead summed for each minibatch. Ignored when reduce is False. Default: True.

reduce: bool, optional

By default, the losses are averaged or summed over observations for each minibatch depending on size_average. When reduce is False, returns a loss per batch element instead and ignores size_average. Default: True

reduction: string, optional

Specifies the reduction to apply to the output: 'none' | 'mean' | 'sum'. 'none': no reduction will be applied,

'mean': the sum of the output will be divided by the number of elements in the output, 'sum': the output will be summed.

Note: size_average and reduce are in the process of being deprecated, and in the meantime, specifying either of those two args will override reduction. Default: 'mean'

pos_weight:Tensor, optional

A weight of positive examples. Must be a vector with length equal to the number of classes. This paramter is only suitable for BCEWithLogits loss function.

ignore_index: int, optional

Specifies a target value that is ignored and does not contribute to the input gradient. When size_average is True, the loss is averaged over non-ignored targets. This paramter is only suitable for CrossEntropy loss function.

Methods

add_module(name, module)	Adds a child module to the current module.
apply(fn)	Applies fn recursively to every submodule (as returned by .children()) as well as self.
bfloat16()	Casts all floating point parameters and buffers to bfloat16 datatype.
buffers([recurse])	Returns an iterator over module buffers.
children()	Returns an iterator over immediate children modules.
cpu()	Moves all model parameters and buffers to the CPU.
cuda([device])	Moves all model parameters and buffers to the GPU.
double()	Casts all floating point parameters and buffers to double datatype.
eval()	Sets the module in evaluation mode.
extra_repr()	Set the extra representation of the module
float()	Casts all floating point parameters and buffers to float datatype.
forward(input, target)	Compute the loss.
get_buffer(target)	Returns the buffer given by target if it exists, otherwise throws an error.
get_extra_state()	Returns any extra state to include in the module’s state_dict.
get_parameter(target)	Returns the parameter given by target if it exists, otherwise throws an error.
get_submodule(target)	Returns the submodule given by target if it exists, otherwise throws an error.
half()	Casts all floating point parameters and buffers to half datatype.
load_state_dict(state_dict[, strict])	Copies parameters and buffers from state_dict into this module and its descendants.
modules()	Returns an iterator over all modules in the network.
named_buffers([prefix, recurse])	Returns an iterator over module buffers, yielding both the name of the buffer as well as the buffer itself.
named_children()	Returns an iterator over immediate children modules, yielding both the name of the module as well as the module itself.
named_modules([memo, prefix, remove_duplicate])	Returns an iterator over all modules in the network, yielding both the name of the module as well as the module itself.
named_parameters([prefix, recurse])	Returns an iterator over module parameters, yielding both the name of the parameter as well as the parameter itself.
parameters([recurse])	Returns an iterator over module parameters.
register_backward_hook(hook)	Registers a backward hook on the module.
register_buffer(name, tensor[, persistent])	Adds a buffer to the module.
register_forward_hook(hook)	Registers a forward hook on the module.
register_forward_pre_hook(hook)	Registers a forward pre-hook on the module.
register_full_backward_hook(hook)	Registers a backward hook on the module.
register_parameter(name, param)	Adds a parameter to the module.
requires_grad_([requires_grad])	Change if autograd should record operations on parameters in this module.
set_extra_state(state)	This function is called from load_state_dict() to handle any extra state found within the state_dict.
share_memory()	See torch.Tensor.share_memory_()
state_dict([destination, prefix, keep_vars])	Returns a dictionary containing a whole state of the module.
to(*args, **kwargs)	Moves and/or casts the parameters and buffers.
to_empty(*, device)	Moves the parameters and buffers to the specified device without copying storage.
train([mode])	Sets the module in training mode.
type(dst_type)	Casts all parameters and buffers to dst_type.
xpu([device])	Moves all model parameters and buffers to the XPU.
zero_grad([set_to_none])	Sets gradients of all model parameters to zero.

__call__

forward(input, target)

Compute the loss.

Parameters

NLL loss:

Input: tensor.

\((N, C)\) where C = number of classes, or \((N, C, d_1, d_2, ..., d_K)\) with \(K \geq 1\) in the case of K-dimensional loss.

Target: tensor.

\((N)\) where each value is \(0 \leq \text{targets}[i] \leq C-1\), or \((N, d_1, d_2, ..., d_K)\) with \(K \geq 1\) in the case of K-dimensional loss.

Output: scalar.

If reduction is 'none', then the same size as the target: \((N)\), or \((N, d_1, d_2, ..., d_K)\) with \(K \geq 1\) in the case of K-dimensional loss.

BCE/BCEWithLogits loss:

Input: Tensor.

\((N, *)\) where \(*\) means, any number of additional dimensions

Target: Tensor.

\((N, *)\), same shape as the input

Output: scalar.

If reduction is 'none', then \((N, *)\), same shape as input.

MultiLabelMargin loss:

Input: Tensor.

\((C)\) or \((N, C)\) where N is the batch size and C is the number of classes.

Target: Tensor.

\((C)\) or \((N, C)\), label targets padded by -1 ensuring same shape as the input.

Output: Scalar.

If reduction is 'none', then \((N)\).

SoftMargin loss:

Input: Tensor.

\((*)\) where \(*\) means, any number of additional dimensions

Target: Tensor.

\((*)\), same shape as the input

Output: scalar.

If reduction is 'none', then same shape as the input

CrossEntropy:

Input: Tensor.

\((N, C)\) where C = number of classes, or \((N, C, d_1, d_2, ..., d_K)\) with \(K \geq 1\) in the case of K-dimensional loss.

Target: Tensor.

\((N)\) where each value is \(0 \leq \text{targets}[i] \leq C-1\), or \((N, d_1, d_2, ..., d_K)\) with \(K \geq 1\) in the case of K-dimensional loss.

Output: scalar.

If reduction is 'none', then the same size as the target: \((N)\), or \((N, d_1, d_2, ..., d_K)\) with \(K \geq 1\) in the case of K-dimensional loss.

class graph4nlp.loss.KGLoss(loss_type, size_average=None, reduce=None, reduction='mean', adv_temperature=None, weight=None)

In the state-of-the-art KGE models, loss functions were designed according to various pointwise, pairwise and multi-class approaches. Refers to Loss Functions in Knowledge Graph Embedding Models

Pointwise Loss Function

MSELoss Creates a criterion that measures the mean squared error (squared L2 norm) between each element in the input \(x\) and target \(y\).

SOFTMARGINLOSS Creates a criterion that optimizes a two-class classification logistic loss between input tensor \(x\) and target tensor \(y\) (containing 1 or -1). Tips: The number of positive and negative samples should be about the same,

otherwise it’s easy to overfit

\[\text{loss}(x, y) = \sum_i \frac{\log(1 + \exp(-y[i]*x[i]))}{\text{x.nelement}()}\]

Pairwise Loss Function

SoftplusLoss refers to the paper OpenKE: An Open Toolkit for Knowledge Embedding

SigmoidLoss refers to the paper OpenKE: An Open Toolkit for Knowledge Embedding

Multi-Class Loss Function

Binary Cross Entropy Loss Creates a criterion that measures the Binary Cross Entropy between the target and the output. Note that the targets \(y\) should be numbers between 0 and 1.

Methods

add_module(name, module)	Adds a child module to the current module.
apply(fn)	Applies fn recursively to every submodule (as returned by .children()) as well as self.
bfloat16()	Casts all floating point parameters and buffers to bfloat16 datatype.
buffers([recurse])	Returns an iterator over module buffers.
children()	Returns an iterator over immediate children modules.
cpu()	Moves all model parameters and buffers to the CPU.
cuda([device])	Moves all model parameters and buffers to the GPU.
double()	Casts all floating point parameters and buffers to double datatype.
eval()	Sets the module in evaluation mode.
extra_repr()	Set the extra representation of the module
float()	Casts all floating point parameters and buffers to float datatype.
forward([input, target, p_score, n_score])	Parameters
get_buffer(target)	Returns the buffer given by target if it exists, otherwise throws an error.
get_extra_state()	Returns any extra state to include in the module’s state_dict.
get_parameter(target)	Returns the parameter given by target if it exists, otherwise throws an error.
get_submodule(target)	Returns the submodule given by target if it exists, otherwise throws an error.
half()	Casts all floating point parameters and buffers to half datatype.
load_state_dict(state_dict[, strict])	Copies parameters and buffers from state_dict into this module and its descendants.
modules()	Returns an iterator over all modules in the network.
named_buffers([prefix, recurse])	Returns an iterator over module buffers, yielding both the name of the buffer as well as the buffer itself.
named_children()	Returns an iterator over immediate children modules, yielding both the name of the module as well as the module itself.
named_modules([memo, prefix, remove_duplicate])	Returns an iterator over all modules in the network, yielding both the name of the module as well as the module itself.
named_parameters([prefix, recurse])	Returns an iterator over module parameters, yielding both the name of the parameter as well as the parameter itself.
parameters([recurse])	Returns an iterator over module parameters.
register_backward_hook(hook)	Registers a backward hook on the module.
register_buffer(name, tensor[, persistent])	Adds a buffer to the module.
register_forward_hook(hook)	Registers a forward hook on the module.
register_forward_pre_hook(hook)	Registers a forward pre-hook on the module.
register_full_backward_hook(hook)	Registers a backward hook on the module.
register_parameter(name, param)	Adds a parameter to the module.
requires_grad_([requires_grad])	Change if autograd should record operations on parameters in this module.
set_extra_state(state)	This function is called from load_state_dict() to handle any extra state found within the state_dict.
share_memory()	See torch.Tensor.share_memory_()
state_dict([destination, prefix, keep_vars])	Returns a dictionary containing a whole state of the module.
to(*args, **kwargs)	Moves and/or casts the parameters and buffers.
to_empty(*, device)	Moves the parameters and buffers to the specified device without copying storage.
train([mode])	Sets the module in training mode.
type(dst_type)	Casts all parameters and buffers to dst_type.
xpu([device])	Moves all model parameters and buffers to the XPU.
zero_grad([set_to_none])	Sets gradients of all model parameters to zero.

__call__

forward(input=None, target=None, p_score=None, n_score=None)

Parameters

MSELoss

input: Tensor.

\((N,*)\) where \(*\) means any number of additional dimensions

target: Tensor.

\((N,*)\), same shape as the input

output:

If reduction is ‘none’, then same shape as the input

SoftMarginLoss

input: Tensor.

\((*)\) where * means, any number of additional dimensions

target: Tensor.

same shape as the input

output: scalar.

If reduction is ‘none’, then same shape as the input

SoftplusLoss

p_score: Tensor.

\((*)\) where * means, any number of additional dimensions

n_score: Tensor.

\((*)\) where * means, any number of additional dimensions. The dimension could be different from the p_score dimension.

output: scalar.

SigmoidLoss

p_score: Tensor.

\((*)\) where * means, any number of additional dimensions

n_score: Tensor.

\((*)\) where * means, any number of additional dimensions. The dimension could be different from the p_score dimension.

output: scalar.

BCELoss:

Input: Tensor.

\((N, *)\) where \(*\) means, any number of additional dimensions

Target: Tensor.

\((N, *)\), same shape as the input

Output: scalar.

If reduction is 'none', then \((N, *)\), same shape as input.