Skip to content

Commit 2d2a503

Browse files
committed
[LLM] fix bug when loss is None in llama modeling.py
1 parent daf2f3a commit 2d2a503

File tree

1 file changed

+5
-1
lines changed

1 file changed

+5
-1
lines changed

paddlenlp/transformers/llama/modeling.py

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1646,7 +1646,11 @@ def forward(self, prediction_scores, masked_lm_labels):
16461646
masked_lm_loss > 0, paddle.ones_like(masked_lm_loss), paddle.zeros_like(masked_lm_loss)
16471647
)
16481648
sum_ = paddle.sum(binary_sequence)
1649-
loss = 0 if sum_ == 0 else paddle.sum(masked_lm_loss * binary_sequence) / sum_
1649+
if sum_ == 0:
1650+
loss = paddle.zeros([], dtype=masked_lm_loss.dtype)
1651+
loss.stop_gradient = False
1652+
else:
1653+
loss = paddle.sum(masked_lm_loss * binary_sequence) / sum_
16501654

16511655
return loss
16521656

0 commit comments

Comments
 (0)