Kyoto University, one of Japan’s most important research institutions, said that after Hewlett Packard Enterprise‘s software update, a script deleted 77 TB of research data backups. According to the announcement, 34 million files from 14 research groups were deleted, and a third of these data will not be recovered. The university blamed the HPE supercomputing system for the incident.
77 TB of research data
Hewlett Packard also made an announcement and took fool responsibility for the issue. The update error Cray/HPE system deleted most of the files older than ten days, rather than just the log files. The incident took place between December 14-16. HPE stated that the update was meant to update a script to improve visibility and readability.
Kyoto University is a renowned research institution, especially in chemistry, immunotherapy, materials sciences. The university didn’t announce which departments had lost their research data yet. HPE made an explanation about the incident,
« The backup script includes a find command to delete log files older than 10 days. In addition to functional improvement of the script, the variable name passed to the find command for deletion was changed to improve visibility and readability. However, there was a lack of consideration in the release procedure of this modified script.
We were not aware of the side effects of this behavior and released the [updated] script, overwriting a bash script while it was still running. This resulted in the reloading of the modified shell script in the middle of the execution, resulting in undefined variables. As a result, the original log files in /LARGE0 [backup disc storage] were deleted instead of the original process of deleting files saved in the log directory. »