TL;DR: wall of text, mostly a braindump, wait for the next(?) post if you’ve got a broken Drobo you’re trying to recover…
So I bought a broken Drobo FS (appeared to have had a failed firmware update) off eBay for $100, mostly just for shits and giggles, figured it’d be an interesting exercise to fix it, not sure I’d trust their weird “BeyondRaid” thing with important data but maybe I’ll use it as a third-tier storage for surveillance footage if I get it fixed.
This post isn’t detailing the fix because I’ve not actually fixed the damn thing yet (I fixed it briefly, then attempted to do a firmware update via Drobo Dashboard which screwed it up again, and it’s currently in some split-brain type state, once I actually fix it I’ll write a new post) so this is more or less a braindump of where I’m at currently.
So first thing you want to check out if you’re interested in doing things to a Drobo FS is this GitHub Repo, contains various info about the various components of the system including useful stuff like where to find the console connections and suchlike.
Now onto the first weirdness, Drobo’s (the FS at least, I gather others are the same) uses a dual-core processor but not in the “traditional” manner, rather than having one OS using the cores in parallel they actually run Linux on one core (handling all the interfacing type stuff) and VxWorks on the other (which actually talks to the disks and implements their “BeyondRaid” thing).
As a consequence of the weird dual-coreness there are two separate instances of uboot to bring up each OS separately, if you break boot on the Linux side you’ll have about 30 seconds before the system will freeze, initially I thought this was some kind of watchdog but it turns out it was the VxWorks core booting, thinking Linux had crashed and attempting to do something about it, if you break both uboot bootloaders it’ll let you spend as much time as you want to poke around.
Initially I powered the unit up and it eventually lit up drive slot 0 green indicating a disk should be inserted (which was already further than the seller told me it got), so I slapped a disk in, it did it’s orange/green flashy thing as expected then eventually lit up slot 1, stuck a disk in there too and it did it’s green/orange flashy dance and eventually both drives went green, so it looked like it was working at that point, however when I attempted to connect to it in Drobo Dashboard the unit wasn’t detected (which was as reported by the seller).
After confirming its status I cracked it open in order to jump onto a console, after a bit of poking around I discovered that there are two copies of the Linux image and two copies of the VxWorks image in flash, and uboot provides that “flashboot” command to select which one to boot, the bootup messaging suggested that the unit was booting image 1 so I tried booting image 2 on the Linux side instead by issuing “flashboot 2” at the prompt, lo and behold it was detected and working as expected in Drobo Dashboard.
Next port of call was to run a firmware update against the device with the theory being that that would unbreak image 1… In theory… In practise, I tried the update several times and each time it reported that it’d failed (with zero useful information as to WHY it failed…). The initial failure was a derp on my part, I’d downloaded the Drobo FS PRO image, when I realised what I’d done I assumed the failure to apply was just because it detected that it was the wrong device, but then I tried with the “normal” FS firmware and got pretty much the same result… I powered the unit down at this point figuring I’d come back to it later.
Later when I powered it back up it was no longer detected in Drobo Dashboard regardless of which image I booted, so presumably the “failed” firmware updates did SOMETHING but it’s anybody’s guess what :/
Now if I’d been smart, I’d have dumped the flash as soon as I got a successful boot, but dumping flash through “md” in uboot has never really been my favourite thing to do, and I’m now paying for that oversight… In my defence, at this point I’d not discovered that there was a parallel uboot process going on and I was operating under the assumption that the halt-after-30-seconds I was seeing in the Linux uboot was a watchdog (and I’d not found any obvious way to stop the watchdog) and trying to dump 128MB of Flash via “md” in uboot at 115200 30 seconds at a time wasn’t really a practical option…
At this point I decided to have a look at the VxWorks console and see what was going on there, at this point I discovered that it was running a separate uboot process and breaking that one behaved as expected (no halt), and realised that the halt I was seeing on the Linux side was probably actually due to VxWorks booting and being cranky that it couldn’t talk to Linux… Live and learn.
This is about when we start to run into some more weirdness, first off, I could’ve SWORN that the commands in the help on the VxWorks side uboot were missing from the Linux side, and so I spent a lot of time down a rabbit hole trying to get the VxWorks uboot to connect out to the network (it appears that the Linux core is the only one with access to the ethernet PHY, though there is some stuff in uboot that suggests it MIGHT be possible to switch the VxWorks core over to use the PHY.) And finally I resorted to an 84MB S19 record file upload over serial at 921600 (Which failed after about 9 hours because I had some bare wires connected to the power switch location and shorted them togehter moving stuff on my desk :/)… But then I ran “help” in the Linux uboot and there it was, “tdflashimg”, no idea if I missed it or if something changed during the process (Maybe a uboot update on the Linux side from one of the firmware upgrade attempts?).
OK, so we have the widget to flash the image, run “help tdflashimg” and it told me what I needed to call it, did some rummaging around to find a sufficiently large contiguous segment of memory to tftp the image to and attempted to update image 1 (and JUST image 1) directly with ‘tdflashimg 0x10000000 1 tftp “release.DroboNAS.tdf”’ the image got tftp’d over and the system flashed Image 1… And then proceeded to flash Image 2 as well… THANKS guys, nice that you bothered to actually implement that feature that you put in the help in your version of uboot… Of course once I saw it was updating Image 2 there wasn’t really anything I could do so I just had to let it run and hope for the best…
The best did not happen… The unit boots, is not detected by Drobo Dashboard AND now it doesn’t even get to the “insert disks” stage… It seems that the VxWorks side is waiting for the Linux side to send a command to enable the interface, but the Linux side is waiting for some command to complete on the VxWorks side before it does that :/
I’ve tried several different versions of the Drobo firmware on it, all of them so far have the same issue… I’m currently rummaging around inside “nasd” (via Ghidra and IDA) from the Drobo to try to figure out enough of the inter-core messaging system to figure out if I can either probe for more information as to WHY the two parts aren’t talking or alternatively to allow me to inject messages to try to get past the current split-brain issue.
More if/when I figure this bloody thing out…