Designing efficient workflows on LOTUS webinar

This webinar demonstrates two workflows available for using LOTUS.

This webinar demonstrates two workflows for using LOTUS*. The first example will show you how to build a workflow that can be re-submitted without re-running previously completed tasks and which reports failures in a traceable way. The second example will help you learn how to submit, monitor, and control high throughput workloads using the LSF “job array” utility. Both examples aim to increase users understanding of how to use LOTUS efficiently.

Warning: The batch scheduler LSF was replaced by SLURM in 2020. The first principles of batch computing presented in these past webinars are still valid but the batch commands used are not and should be replaced by SLURM. Please refer to the webinar ‘Transitioning  to SLURM webinar'. 

There are many benefits of using LOTUS (e.g. processing times can be significantly reduced) and all JASMIN users should aim to utilise this excellent resource for their data processing. This webinar will help show you how to set up and submit jobs to the LOTUS batch processing cluster.

The webinar covers:

• Why use batch compute?
• Introduction to LOTUS, batch processing and workflows
• Example 1: a framework for batch workflows
• Example 2: using job arrays to manage workflows

After attending the webinar you should feel confident to:

• Submit a script as a LOTUS job and view its status
• Understand the basic command-line options for the batch system
• Build a batch workflow that can be re-submitted without re-running previously completed tasks
• Run a set of batch jobs using a job array.

See links here for the video. 

Recorded on 13th December 2019 for the seventh webinar in a series provided by The Centre for Environmental Data Analysis (CEDA). If you have any queries, please email the CEDA helpdesk.

*LOTUS is a batch processing service on JASMIN which enables efficient submission and scheduling of large-scale and/or resource-intensive processing. See: https://help.ceda.ac.uk/category/107-batch-computing-on-lotus