UPDATE 3/22/23 4:30 PM: All images, signatures, and files have been restored for REDCap. Please email redcap@rt.cri.uchicago.edu with any questions or concerns. Thank you again for your patience as we diligently worked to restore all files.
UPDATE 3/21/23 10:15 PM: REDCap remains available for use. However, images and uploaded files may not be available yet. We are in the process of restoring the remaining files. Another outage will need to be scheduled to complete the process. This outage is scheduled for Wednesday, March 22 at 3pm for approximately 1-2 hours. Please email redcap@rt.cri.uchicago.edu with any questions or concerns. Thank you again for your patience as we diligently work to restore all REDCap files.
UPDATE 3/19/23 12:23 PM: Dear CRI Colleagues: We are pleased to notify you that PRFS and all related CRI services are now available. We will be continuing to monitor the system closely for any abnormalities. If you have experience any anomalies or have any questions, please submit a ticket by emailing support@rt.cri.uchicago.edu. If you have further questions or concerns, please feel free to email us at cri@bsd.uchicago.edu. Thank you again for your patience as we diligently worked to restore services. – Julie and Brian
UPDATE 3/17/23 5:06 PM: Dear CRI Colleagues: We have restored critical clinical functionality, and REDCap remains operational. However, PRFS remains unavailable for use as we continue performance stability testing. At this point in time, we plan to restore the system on Sunday, March 19. The root cause of the PRFS outage is still being investigated, and we will continue to monitor the system closely for the foreseeable future. Per our annual operating plan, PRFS is scheduled to be replaced in the coming months which will remediate the stability risk. We will provide an update on Sunday as to when the system will be available. Thank you for your patience as we diligently work to restore all services, and please feel free to contact us with any questions or concerns. – Julie and Brian
UPDATE 3/16/23 5:27 PM: Dear CRI Colleagues: PRFS remains offline, though we successfully completed the utility and found no inconsistencies with data integrity. The root cause is still being investigated, and as such, we have scheduled additional performance testing for Friday, 3/17 before we re-activate the system. We will have provide further updates on our performance testing tomorrow. In the meantime, REDCap remains operational, and we are actively restoring critical files. If you have a critical need, please contact cri@bsd.uchicago.edu. Thank you for your patience as we diligently work to restore all services, and please feel free to contact us with any questions or concerns. – Julie and Brian
UPDATE 3/15/23 5:16 PM: Dear CRI Colleagues: PRFS remains offline as we continue running the utility that will determine if there is any data corruption that must be repaired. Based on the rate at which this utility is running, we expect it to finish tomorrow. To date, we are pleased to say that we have not identified any data corruption but will know more tomorrow upon the utility’s completion. In the meantime, REDCap remains operational, and we are actively restoring critical files from our backup system. If you have a critical need, please contact cri@bsd.uchicago.edu. Thank you for your patience as we diligently work to restore all services, and please feel free to contact us with any questions or concerns. – Julie and Brian
UPDATE 3/14/23 5:29 PM: Dear CRI Colleagues: PRFS remains offline as we are currently running a utility that will determine if there is any data corruption that must be repaired. This utility must be run with the data offline, and it is scheduled to complete on Thursday. At that time, we will know the extent and duration of our recovery efforts. In the meantime, REDCap is up and running, and we are working through critical data restoration from our backup system. If you have a critical need, please contact cri@bsd.uchicago.edu. Thank you for your patience as we diligently work to restore all services, and please feel free to contact us with any questions or concerns. – Julie and Brian
UPDATE 3/14/23 9:22 AM: REDCap is now available for use. Please note, to ensure there is no data corruption, any project files created before today are still in the process of being restored and will be added when possible. This includes files stored in “File upload” field types, “Descriptive Text (with file attachments)” field types, and previous export files. Thank you for your patience, and please email redcap@rt.cri.uchicago.edu with any questions or concerns.
UPDATE 3/13/23 6:59 PM: Dear CRI Colleagues: On the afternoon of Thursday, March 8th, we encountered a critical error within our CRI PRFS parallel storage system. This critical error caused the storage system to become unavailable. Working throughout the weekend with our partners Lenovo and IBM, we have worked through the instability. To ensure there is no data corruption, the entire 2.5PB file system will need to be checked and verified and will be offline. We will provide an estimated restoration date by the end of the day on March 14th. In the interim, we are working on alternative means for critical functions. Please email cri@bsd.uchicago.edu if you have a critical need. Thank you for your patience, and do not hesitate to contact us with any questions or concerns. – Julie Johnson and Brian Schinker
UPDATE 3/12/23 8:27 PM: Services are still down. The vendor’s development team will continue to work over night on understanding the source of the error and finding a way to safely bypass it.
UPDATE 3/12/23 12:57 PM: Services are still unavailable. The vendor, having a global presence, is working on analyzing system logs and crash-dumps from the system around the clock. In the last 12 hours, there has been progress on narrowing down the location of the error, but it is still unclear what is required to correct the issue.
UPDATE 3/11/23 4:46 PM: Services are still unavailable. The team is troubleshooting with the vendors to resolve the issue. It is our top priority to restore services and we apologize for any inconvenience. Thank you for your patience.
UPDATE 3/10/23 12:34 PM: Unfortunately, services are still unavailable. The team is working to resolve the issue and it has been escalated with the vendor. We apologize for the inconvenience, and greatly appreciate your understanding and patience.
UPDATE 3/9/23 5:08 PM: The impacted services are still unavailable and the team is investigating the issue with the vendor. At this point, we do not have an estimate on when services will be restored.
UPDATE 3/9/23 2:25 PM: We are currently experiencing issues on PRFS storage that is affecting the use of CRI Lab shares, the Gardner HPC Cluster, Linux Stats, and other CRI & BSDIS systems, including REDCap. The team has identified the cause and are working on restoring services.
QUESTIONS: Please email support@rt.cri.uchicago.edu.
