RESOLVED
[RESOLVED – 10/19/2022]: The DOI BigFix team reported that the environment has been stable for over 24 hours, and all scheduled reports have been re-enabled. DOI now considers the issue resolved. The DOI BigFix team will not do maintenance on 10/20/2022. There will be no standard downtime this Thursday.
Local use of the BigFix Console or Web Reports can now be resumed.
The BigFix shut down action for computers that were non-complaint with DAR requirements was originally scheduled for 10/17/2022. This was postponed until the BigFix issue was resolved. The shutdown actions will start on Wednesday, 10/19/2022. Additional information can be found here.
[UPDATE 2]: The DOI BigFix team has made changes to the Top Level Relays and added a Windows relay in between the Core and Top Level Relays. They have also disabled all scheduled Web Reports, with plans to slowly re-enable them over the next few days.
The BigFix environment has remained stable during the last 8 hours, and the USGS ePatching Team has re-created, on schedule, the October monthly actions for both Fast Ring and Production. eAD and ePatching Production patching will proceed as originally scheduled.
The Department has not issued a final resolution notice for this outage. Local use of the BigFix Console or Web Reports should be avoided until the issue is announced as fully resolved. An update will be sent out when more information is available.
[UPDATE 1]: The restore was not successful. Over the weekend the DOI BigFix team then decided to restore back to current version from 10/12/2022 and made additional attempts to stabilize the environment, which were also not successful. There will be a meeting this morning to discuss the next steps, which will require the department to rebuild the BigFix environment and then restore the current database into it. An update will be sent out when more information is available. For now, continue to stay out of the BigFix console or Web Reports until further notice.
Updated Monday 10/17/2022 at 10AM Eastern.
The Department of Interior’s BigFix Team is restoring the BigFix Production Environment from a backup made on Sunday, 9/25/2022. The restore will begin on Friday 10/14/2022 at 3pm ET. The restore operation is expected to run through the entire weekend, with a goal to restore console functionality by start of business on Monday 10/17/2022.
The following actions have been completed in preparation for the rollback:
- The USGS BigFix Console Operators have made backups of all enterprise content that was created after 9/25/2022.
- They have also captured screenshots of all enterprise actions that were deployed and enterprise web report changes that were made since 9/25/2022.
When the Department informs the Bureaus that BigFix activity may resume, the USGS BigFix Operators will begin restoration activities, and the ePatching team will resume patching activities as soon as possible.
We are aware of the impact to ePatching activities and will continue to communicate as the situation evolves.
