Update on JASMIN issues 31 Aug 2023
Posted on August 31, 2023 (Last modified on October 19, 2023) • 2 min read • 278 wordsDear users,
We thank you for your patience while the system team has been working to resolve the network and storage issues of the past few days. The current situation is as follows:
Migration of data from /gws/smf/* (small-files group workspaces) and /apps/jasmin (used for software including the “modules” system) has now been completed and once other systems return to normal, no changes should be apparent to users.
At the time of writing, LOTUS nodes are currently in a state of either
Fixed but currently idle, pending (imminent) re-opening of queues
Stuck with failing or failed jobs, awaiting reboot
Shortly, queues will be re-enabled so will start processing jobs again, but initially at a slower rate than normal until nodes in state (a) have been rebooted. [STOP PRESS: now reopened]
Other systems needing to mount group workspace storage (including /gws/smf) will need rebooting in order to work correctly again.
General purpose sci VMs may need to be rebooted in due course
Managers of tenancy sci machines (*-sci-M tenancies in JASMIN cloud) are requested to reboot their own sci machines themselves via the cloud portal
Other services such as cron, cylc etc will be rebooted in due course.
The network issue is believed to be a recurrence of a hardware-related issue. It has been temporarily resolved but the possibility of further breaks remains. A further intervention is required (ideally to be scheduled in due course) but the risk of further interruptions does remain until this has been done.
In summary: users can start using services again, but systems are still “at risk” and LOTUS throughput may initially be slower than normal.
With apologies for any inconvenience caused,
JASMIN Team