Several weeks ago, One of our guys pressed the 'Upgrade' button of Jenkins, and after that it could not work any more.
Of course he should not do this as he totally did not know the result, while I also found I could not find a suitable snapshot of this VM, and the last one was created in last month.
It is my fault that I had not created the VM snapshots regularly. We could restore the last snapshot while we would lost all the changes after it. To make up my fault, I said to myself I must restore Jenkins with all the later changes.
I reviewed the directories and files under /var/lib/jenkins carefully, and decided to exclude jobs and workspace folders as they should be project files, and also exclude plugins folder as I would overwrite it. I chose a file as reference with timestamp just before the 'Upgrade' button was pressed, and run below command to get all the changed files.
find . \( -path ./workspace -o -path ./jobs -o -path ./plugins \) -prune -o -cnewer jenkins-jobs/jenkins-jobs.mv.db
The list was long and included below folders or files:
./org.jvnet.hudson.plugins.SSHBuildWrapper.xml ./hudson.model.UpdateCenter.xml ./secrets/whitelisted-callables.d/default.conf ./secrets/filepath-filters.d/30-default.conf ./global-build-stats.xml ./config.xml.bk ./nodeMonitors.xml ./queue.xml.bak ./updates ./updates/default.json ./updates/hudson.tasks.Maven.MavenInstaller ./updates/hudson.tools.JDKInstaller ./updates/hudson.tasks.Ant.AntInstaller ./updates/hudson.plugins.gradle.GradleInstaller ./jenkins.install.InstallUtil.lastExecVersion ./locale.xml ./config.xml.default ./org.jenkinsci.plugins.workflow.flow.GlobalDefaultFlowDurabilityLevel.xml ./config.xml ./hudson.plugins.timestamper.TimestamperConfig.xml ./jenkins.model.ArtifactManagerConfiguration.xml ./jenkins.model.JenkinsLocationConfiguration.xml ./hudson.scm.SubversionSCM.xml ./hudson.tasks.Shell.xml ./hudson.tasks.Mailer.xml ./hudson.triggers.SCMTrigger.xml ./.lastStarted ./.owner ./jenkins.plugins.publish_over_ssh.BapSshPublisherPlugin.xml ./com.cloudbees.hudson.plugins.folder.config.AbstractFolderConfiguration.xml
Then I compared them with the files restored using last snapshot, and finally restored below files:
./hudson.scm.SubversionSCM.xml ./jenkins.install.InstallUtil.lastExecVersion ./queue.xml.bak ./queue.xml ./config.yml （need to be modified to include all the later project information） ./plugins ./updates
And also restored the /usr/lib/jenkins/jenkins.war to previous version.
Then I restarted the jenkins service and it works well with all the later changes!
I hope such things will never happen again, so I will create a schedule to backup the VM according to our backup policy.
