I did not know why my NAS said one of the Basic volumes as 'Crashed' and I could not write files to it. At the beginning I though it should be easy task so I logged the system through SSH directly and umounted the filesystem and run a 'e2fsck' on it. It got some errors and fixed them then I could mount it without any issue, and of course I could write it without any problem.
It was fixed, right? Yes, until I rebooted the NAS.
After the reboot, I found this volume still was marked as 'Crashed' and the filesystem became readonly again. And this time I wanted to fix it permanently.
When I tried to assembly the raid array, I got below error message:
- dsm> mdadm -Av /dev/md3 /dev/sdd3
- mdadm: looking for devices for /dev/md3
- mdadm: /dev/sdd3 is identified as a member of /dev/md3, slot 0.
- mdadm: device 0 in /dev/md3 has wrong state in superblock, but /dev/sdd3 seems ok
- mdadm: added /dev/sdd3 to /dev/md3 as 0
- mdadm: /dev/md3 has been started with 1 drive.
- dsm> e2fsck /dev/md3
- e2fsck 1.42.6 (21-Sep-2012)
- 1.42.6-5644: is cleanly umounted, 809/91193344 files, 359249805/364756736 blocks
I checked the raid array and the disk partition, while could not find any issue:
- dsm> mdadm -D /dev/md3
- /dev/md3:
- Version : 1.2
- Creation Time : Thu Feb 4 15:03:34 2016
- Raid Level : raid1
- Array Size : 1459026944 (1391.44 GiB 1494.04 GB)
- Used Dev Size : 1459026944 (1391.44 GiB 1494.04 GB)
- Raid Devices : 1
- Total Devices : 1
- Persistence : Superblock is persistent
- Update Time : Sat Dec 30 16:13:47 2017
- State : clean
- Active Devices : 1
- Working Devices : 1
- Failed Devices : 0
- Spare Devices : 0
- Name : Gen8:3
- UUID : d0796c7b:c9b0ab70:211e65c0:843891e2
- Events : 40
- Number Major Minor RaidDevice State
- 0 8 51 0 active sync /dev/sdd3
- dsm> mdadm -E /dev/sdd3
- /dev/sdd3:
- Magic : a92b4efc
- Version : 1.2
- Feature Map : 0x0
- Array UUID : d0796c7b:c9b0ab70:211e65c0:843891e2
- Name : Gen8:3
- Creation Time : Thu Feb 4 15:03:34 2016
- Raid Level : raid1
- Raid Devices : 1
- Avail Dev Size : 2918053888 (1391.44 GiB 1494.04 GB)
- Array Size : 2918053888 (1391.44 GiB 1494.04 GB)
- Data Offset : 2048 sectors
- Super Offset : 8 sectors
- State : clean
- Device UUID : a9eafb18:84f3974f:42751688:7136418d
- Update Time : Sat Dec 30 16:59:19 2017
- Checksum : faa9b8c3 - correct
- Events : 40
- Device Role : Active device 0
- Array State : A ('A' == active, '.' == missing)
So for some unknown reason, I always got the wrong state in the /proc/mdstat:
- dsm> cat /proc/mdstat
- Personalities : [linear] [raid0] [raid1] [raid10] [raid6] [raid5] [raid4]
- md3 : active raid1 sdd3[0](E)
- 1459026944 blocks super 1.2 [1/1] [E]
- md2 : active raid1 sdc3[0]
- 17175149551 blocks super 1.2 [1/1] [U]
- md1 : active raid1 sdc2[0] sdd2[1]
- 2097088 blocks [12/2] [UU__________]
- md0 : active raid1 sdc1[0] sdd1[1]
- 2490176 blocks [12/2] [UU__________]
- unused devices: <none>
There was an '[E]' letter and I though maybe it meant 'Error', so how to clear it?
I searched and read lots of posts, and below one save me:
recovering-a-raid-array-in-e-state-on-a-synology-nas
I found similar solutions before reaching this post, while I did not dare to try it as I had not copied all the data, but the above information gave me more confidence and I run below commands:
- dsm> mdadm -Cf /dev/md3 -e1.2 -n1 -l1 /dev/sdd3 -ud0796c7b:c9b0ab70:211e65c0:843891e2
- mdadm: /dev/sdd3 appears to be part of a raid array:
- level=raid1 devices=1 ctime=Thu Feb 4 15:03:34 2016
- Continue creating array? y
- mdadm: array /dev/md3 started.
- dsm> cat /proc/mdstat
- Personalities : [linear] [raid0] [raid1] [raid10] [raid6] [raid5] [raid4]
- md3 : active raid1 sdd3[0]
- 1459026944 blocks super 1.2 [1/1] [U]
- md2 : active raid1 sdc3[0]
- 17175149551 blocks super 1.2 [1/1] [U]
- md1 : active raid1 sdc2[0] sdd2[1]
- 2097088 blocks [12/2] [UU__________]
- md0 : active raid1 sdc1[0] sdd1[1]
- 2490176 blocks [12/2] [UU__________]
- unused devices: <none>
- dsm> e2fsck -pvf -C0 /dev/md3
- 809 inodes used (0.00%, out of 91193344)
- 5 non-contiguous files (0.6%)
- 1 non-contiguous directory (0.1%)
- # of inodes with ind/dind/tind blocks: 0/0/0
- Extent depth histogram: 613/84/104
- 359249805 blocks used (98.49%, out of 364756736)
- 0 bad blocks
- 180 large files
- 673 regular files
- 127 directories
- 0 character device files
- 0 block device files
- 0 fifos
- 0 links
- 0 symbolic links (0 fast symbolic links)
- 0 sockets
- ------------
- 800 files
- dsm> cat /etc/fstab
- none /proc proc defaults 0 0
- /dev/root / ext4 defaults 1 1
- /dev/md3 /volume2 ext4 usrjquota=aquota.user,grpjquota=aquota.group,jqfmt=vfsv0,synoacl 0 0
- /dev/md2 /volume1 ext4 usrjquota=aquota.user,grpjquota=aquota.group,jqfmt=vfsv0,synoacl 0 0
- dsm> mount /dev/md3
- dsm> df -h
- Filesystem Size Used Avail Use% Mounted on
- /dev/root 2.3G 676M 1.6G 31% /
- /tmp 2.0G 120K 2.0G 1% /tmp
- /run 2.0G 2.5M 2.0G 1% /run
- /dev/shm 2.0G 0 2.0G 0% /dev/shm
- none 4.0K 0 4.0K 0% /sys/fs/cgroup
- /dev/bus/usb 2.0G 4.0K 2.0G 1% /proc/bus/usb
- /dev/md2 16T 14T 2.5T 85% /volume1
- /dev/md3 1.4T 1.4T 21G 99% /volume2
- dsm> ls /volume2
- @eaDir @tmp aquota.group aquota.user downloads lost+found synoquota.db
After that I checked the volume state again and the annoying 'Crashed' became 'Normal'. 
