This website uses cookies. By continuing to use this website you are agreeing to our use of cookies. 
CEDA logo

Update on JASMIN issues 31 Aug 2023

3 weeks, 5 days ago

Dear users,

 

We thank you for your patience while the system team has been working to resolve the network and storage issues of the past few days. The current situation is as follows:

 

  • Migration of data from /gws/smf/* (small-files group workspaces) and /apps/jasmin (used for software including the “modules” system) has now been completed and once other systems return to normal, no changes should be apparent to users.

  •  At the time of writing, LOTUS nodes are currently in a state of either

    • Fixed but currently idle, pending (imminent) re-opening of queues

    • Stuck with failing or failed jobs, awaiting reboot

    • Shortly, queues will be re-enabled so will start processing jobs again, but initially at a slower rate than normal until nodes in state (a) have been rebooted. [STOP PRESS: now reopened]

  • Other systems needing to mount group workspace storage (including /gws/smf) will need rebooting in order to work correctly again.

    • General purpose sci VMs may need to be rebooted in due course

    • Managers of tenancy sci machines (*-sci-M tenancies in JASMIN cloud) are requested to reboot their own sci machines themselves via the cloud portal

    • Other services such as cron, cylc etc will be rebooted in due course.

  • The network issue is believed to be a recurrence of a hardware-related issue. It has been temporarily resolved but the possibility of further breaks remains. A further intervention is required (ideally to be scheduled in due course) but the risk of further interruptions does remain until this has been done.

 

In summary: users can start using services again, but systems are still “at risk” and LOTUS throughput may initially be slower than normal.

 

With apologies for any inconvenience caused,

JASMIN Team

 

Recent Posts

Archive

2023
2022
2021
2020
2019
2018
2017
2016
2015
2014
2013
2012
2011
2010
2009
2008
2007
2006
2005
2004
2003
2002
2001
2000
1996

Categories

Tags

Authors

Feeds

RSS / Atom