Skip to content

Nextflow eagerly terminates Azure Batch jobs during execution #5839

Open
@adamrtalbot

Description

@adamrtalbot

New feature

Based on comment: #3927 (comment)

Azure Batch jobs are terminated during pipeline closing down, see #3927.

During that PR, we discussed eagerly changing the setting of each job to onAllTasksComplete = 'terminate' but never looked further into it.

Use case

The main benefit of doing it this way is that if Nextflow fails to close properly (i.e. is killed), most active jobs will be marked as to be terminated and be removed from the active job quota.

Suggested implementation

We just need an "onProcessClose" event to notify when all tasks are created. I've already added it in a number of PRs, like this one: #3818

The suggestion from the first PR which could be implemented ☝

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions