Japan HP accidentally deleted 77TB data in Kyoto U. supercomputing system
I really appreciate the announcement from Hewlett Packard, which is very apologetic: https://www.iimc.kyoto-u.ac.jp/services/comp/pdf/file_loss_i...They do not try to blame it on complex systems or other factors.Users lost 1 day and 1/2 of recent work (which doesn't seem to be that bad).  About file loss in Luster file system in your supercomputer system, we are 100% responsible.
  We deeply apologize for causing a great deal of inconvenience due to the serious failure of the file loss.

  We would like to report the background of the file disappearance, its root cause and future countermeasures as follows:

  We believe that this file loss is 100% our responsibility.
  We will offer compensation for users who have lost files.

  [...]

  Impact:
  --

  Target file system: /LARGE0

  Deleted files: December 14, 2021 17:32 to December 16, 2021 12:43

  Files that were supposed to be deleted: Files that had not been updated since 17:32 on December 3, 2021

  [...]

  Cause:
  --

  The backup script uses the find command to delete log files that are older than 10 days.

  A variable name is passed to the delete process of the find command.

  A new improved version of the script was applied on the system.

  However, during deployment, there was a lack of consideration as the periodical script was not disabled.

  The modified shell script was reloaded from the middle.

  As a result, the find command containing undefined variables was executed and deleted the files.

  [...]

  Further measures:
  --

  In the future, the programs to be applied to the system will be fully verified and applied.

  We will examine the extent of the impact and make improvements so that similar problems do not occur.

  In addition, we will re-educate the engineers in charge of human error and risk prediction / prevention to prevent recurrence.

 We will thoroughly implement the measures.