TLDR{
A USB stick that I used for backup suddenly turned readonly and failed to mount
-
If it is still readable, make a backup first
-
If it is an ext[2,3,4] fs, use the
ro,noloadoptions to mount it
}TLDR
I've been using a SanDisk 128GB USB stick as a backup device for ~5 years now. Recently it failed to show up when plugged in, so I had to scramble to see what went wrong.
The annotated list of things that I tried is shown below:
Note: My notes/annotation is prefixed with a '--' character and the actual commands that I ran are prefixed with the '$' prompt and the output is copy pasted below it.
-- First make sure the device is found and the partitions are still there:
$ sudo fdisk -l /dev/sdb
Disk /dev/sdb: 114.6 GiB, 123060879360 bytes, 240353280 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x000afbc6
Device Boot Start End Sectors Size Id Type
/dev/sdb1 2048 33556479 33554432 16G 83 Linux
/dev/sdb2 33556480 240353279 206796800 98.6G 83 Linux
-- That means the partitions are still around and may be readable.
-- Let's see if there is anything in the syslog relating to it:
$ dmesg | grep sdb
[ 8.082633] sd 6:0:0:0: [sdb] 240353280 512-byte logical blocks: (123 GB/115 GiB)
[ 8.084141] sd 6:0:0:0: [sdb] Write Protect is on
[ 8.084144] sd 6:0:0:0: [sdb] Mode Sense: 43 00 80 00
[ 8.085116] sd 6:0:0:0: [sdb] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
[ 8.092715] sdb: sdb1 sdb2
[ 8.096027] sd 6:0:0:0: [sdb] Attached SCSI removable disk
-- Uh oh, "Write Protect is on" which means it has turned readonly
-- This USB stick does not have any manual means of turning off writes. so blame the firmware on the device, maybe?
-- In any case, let's see if it can be manually mounted:
$ sudo mount /dev/sdb1 /tmp/sdb1
mount: /dev/sdb1 is write-protected, mounting read-only
mount: wrong fs type, bad option, bad superblock on /dev/sdb1,
missing codepage or helper program, or other error
In some cases useful info is found in syslog - try
dmesg | tail or so.
-- regular "rw" mount fails because it is readonly
-- Check if the syslog had anything else to say:
$ dmesg | tail
[ 733.308075] EXT4-fs (sdb1): INFO: recovery required on readonly filesystem
[ 733.308080] EXT4-fs (sdb1): write access unavailable, cannot proceed
-- Since I didn't know what I should try next, I made the unwise choice of attempting an fsck. In hindsight DO NOT DO THIS - if this is a failing disk on its last legs, you may just make things worse.
$ sudo fsck -y /dev/sdb1
fsck from util-linux 2.27.1
e2fsck 1.42.13 (17-May-2015)
fsck.ext4: Read-only file system while trying to open /dev/sdb1
Disk write-protected; use the -n option to do a read-only
check of the device.
-- Thankfully, the fsck failed, but I was willing enough to attempt using the -n option
$ sudo fsck -n /dev/sdb1
fsck from util-linux 2.27.1
e2fsck 1.42.13 (17-May-2015)
Warning: skipping journal recovery because doing a read-only filesystem check.
/dev/sdb1: clean, 23460/1048576 files, 1565383/4194304 blocks
-- Now that I know the disk is still readable and in good shape, I decided to make a backup. I should have done this first before attempting anything else. BTW, this is an ext fs (ext4 to be specific), dump works on any ext fs (?)
$ sudo dump 0f /backup/128_gb_usb_stick_part1 /dev/sdb1
$ sudo dump 0f /backup/128_gb_usb_stick_part2 /dev/sdb2
-- If the disk was not in good shape (ie if the fsck -n had failed), you could attempt a raw dd of the disk (or give it up for dead and move on)
-- So now that we have a backup, we can attempt more radical stuff.
-- But let's start off by just trying to mount it readonly:
$ sudo mount -o ro /dev/sdb1 /tmp/sdb1
mount: wrong fs type, bad option, bad superblock on /dev/sdb1,
missing codepage or helper program, or other error
In some cases useful info is found in syslog - try
dmesg | tail or so.
$ dmesg | tail
[ 854.308075] EXT4-fs (sdb1): INFO: recovery required on readonly filesystem
[ 854.308080] EXT4-fs (sdb1): write access unavailable, cannot proceed
-- Sorry, no dice, even a readonly mount does now work. Huh, TIL.
-- What if I try to nuke the device, now that I have a backup.
$ sudo dd if=/dev/zero of=/dev/sdb bs=4096 count=1
dd: failed to open '/dev/sdb': Read-only file system
-- See if the individual partitions can be nuked
$ sudo dd if=/dev/zero of=/dev/sdb1 bs=4096 count=1
dd: failed to open '/dev/sdb1': Read-only file system
$ sudo dd if=/dev/zero of=/dev/sdb2 bs=4096 count=1
dd: failed to open '/dev/sdb2': Read-only file system
-- So none of those shenanigans works either.
-- Searching the web leads me to this command which I've not seen earlier:
$ sudo blockdev --setrw /dev/sdb
-- Appears to work since it passes without any warnings or messages, but it doesn't change the outcome of running any of the earlier commands. The device stays stubbornly readonly.
-- After even more searches on stackoverflow/reddit/... and filtering out the obvious nonsense, I finally stumbled upon this:
$ sudo mount -o ro,noload /dev/sdb1 /tmp/sdb1
$ df -k | grep sdb1
/dev/sdb1 16337876 5822192 9660440 38% /tmp/sdb1
-- Yay, success!
All in all, despite the fact that the USB stick died when it was only about ~20% full overall and lasted "only" 5 years, I'm glad that I was able to get access to all of the backup data stored on it. At least now I know that I don't need to worry about backing up my backups.
Reading up on Bunnie Huang/Xobs presentation about SD cards was a wake up call though - the bottomline is that you cannot trust any data stored on these devices.
zfs send/recv has been my backup method of choice with ext4 used as a short term convenience so I will now need to figure out if the zfs data is recoverable when a device acts up like this. Perhaps I can use the new 'blockdev' command that I learnt as part of this exercise to test that scenario.