[Unified Checkpoint] update async save logic (#9274)

[DesmonDay]

• update async save signal

• fix async save hang

PR types

Others

PR changes

Others

Description

1. Add new training argument, output_signal_dir, which is used to save asynchronous saving signal.
2. Fix async saving hang, when the asynchronous process is exited, raise RuntimeError directly.

[paddle-bot]

Thanks for your contribution!

[codecov]

Codecov Report

Attention: Patch coverage is 26.38889% with 53 lines in your changes missing coverage. Please review.

Project coverage is 54.86%. Comparing base (0f428bb) to head (3fb6a6a).
Report is 57 commits behind head on release/2.8.

❗ Current head 3fb6a6a differs from pull request most recent head 18c385d

Please upload reports for the commit 18c385d to get more accurate results.

Files with missing lines	Patch %	Lines
paddlenlp/trainer/plugins/unified_checkpoint.py	12.50%	28 Missing ⚠️
paddlenlp/trainer/trainer.py	29.03%	22 Missing ⚠️
paddlenlp/trainer/trainer_utils.py	25.00%	3 Missing ⚠️

Additional details and impacted files

@@               Coverage Diff               @@
##           release/2.8    #9275      +/-   ##
===============================================
- Coverage        55.14%   54.86%   -0.28%     
===============================================
  Files              608      612       +4     
  Lines            94580    95769    +1189     
===============================================
+ Hits             52158    52546     +388     
- Misses           42422    43223     +801     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[wawltor]

LGTM