&gt; And [0] indicates that you can live-replace a failed drive in a btrfs RAID array. (The man page makes it seem that you might want to also pass the -r flag to the replace operation.)<p>Good catch! I have to test on a rented server I&#x27;m running anyway in the next days - adding a an additional drive, running replace and removing the old drive seems to work - I&#x27;m about to find out if - deleting a device, rebooting and trying to replace an non-existent device also works - I guess it&#x27;s likely that I have to fumble with the provider-rescue system (so now control about the kernel version :&#x2F;)<p>I thought that the -o degraded dance was still necassary, looks like I was wrong. Let&#x27;s see how the disk replacement on that servers turns out...<p>&gt; One thing bothers me about that guy who was reporting the bug on the 4.1 kernel. He claimed that his mount options were:
    &#x2F;dev&#x2F;sdb &#x2F;media&#x2F;storage2 btrfs ro,noatime,compress=lzo,space_cache 0 0
but the issue he was reporting smelled like it could only happen on a rw volume. Wonder what was up with that.<p>That guy was me :)<p>Basically the disk got remounted ro after the failure and I just did a grep on &#x2F;proc&#x2F;mounts for the fs before posting (not very wise). This has been a bug that was fixed with 4.2 (didn&#x27;t saw it again at least) and btrfs check --repair fixed it.<p>&gt; It&#x27;s also interesting that noone replied to his messages. Both linux-btrfs and #btrfs have always been quite responsive when I&#x27;ve sent messages.<p>I had no time to hang around IRC at this time. Yes. Usually it&#x27;s very nice there. I also want to say I&#x27;m not ranting against any of the people involved. I&#x27;m just unhappy with the current state of the technology.<p>[ENOSPC]
&gt; Dunno. It looks like the Global Reserve got introduced in ~3.16, so it&#x27;s not likely related to that. Maybe you hit a hard-link limit or had really, really large xattrs attached to files or something? (I do know that btrfs has (had?) a surprisingly low limit on the number of hard links supported in a volume, but I have no idea how hitting that limit presents itself.)<p>Actually I don&#x27;t know - the filesystem in question ran a while with the 3.13 kernel and got later upgraded to something less old. Maybe that&#x27;s part of the problem. ENOSPC happened while a  misconfigured Hadoop NameNode wrote tons of data 4x without checkpointing on the disk and when restarting the checkpointing server it all crashed. It&#x27;s now running 4.2 and fine so far...<p>&gt; In the distant past, running a balance on empty chunks (in order to free them and free up otherwise unused metadata space) was one recommended way to work around mysterious ENOSPC issues. This isn&#x27;t supposed to be required anymore, but it&#x27;s a last-ditch cargo-cult thing that can&#x27;t do anything worse than waste one&#x27;s time.<p>Yeah did that balance dance and it somewhat worked.<p>&gt; Is it any OOM situation, or just particular ones? I know that I&#x27;ve had google-chrome + Hangouts run away and nom all my memory on multiple occasions, and had my btrfs volumes function just fine after the OOM killer had its way with Chrome.<p>Never got around to nail down this one (would love to being able to file a bug on this) - but at the time I was not really able to do much on the machines - it&#x27;s a NUMA system and basically ElasticSearch and Hadoop allocated far more than 64GB memory - OOM kicked in and _after a while_ the logs where full of hanging tasks related to btrfs - at that time I&#x27;ve found something related on the mailing list in regards to allocations and OOM however I&#x27;m lacking the intimate knowledge to be sure if it&#x27;s related.<p>I&#x27;ve screamed at everyone to avoid OOM situations so I&#x27;m not sure if it&#x27;s still a problem :)<p>&gt; Also, do you have a handle on roughly how much better it has become in 4.2, and is this problem something that the btrfs folks are aware of?<p>So far I did not see it again. I had not enough data to file a bug or investigate further. It was something lock related IIRC.<p>&gt; Anyway, I do REALLY hope I soon get the budget to set up a btrfs testbed. I&#x27;d really like to see (and help diagnose) all these failures that people keep talking about... in a carefully controlled environment, with data that I don&#x27;t care about -naturally-. :)<p>Good luck! You can try running Hadoop and some heavy jobs on it. Lot&#x27;s of threads that read and write lot&#x27;s of data. There are some benchmarks like <a href="https:&#x2F;&#x2F;github.com&#x2F;intel-hadoop&#x2F;HiBench" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;intel-hadoop&#x2F;HiBench</a> to stress the system. Should also work on a single system.<p>&gt; [1] And -for the record- I saw an eerily similar thing happen every other month at a Windows-only programming shop I used to work for. One or more files and&#x2F;or directories on the NTFS-backed Windows 2003 file server would inevitably become undeletable. [2] The sysadmin would have to take the whole thing down for a half hour to smear chicken guts in the right places to get rid of the offending files&#x2F;directories. Despite these regular issues, we never decided to stop using NTFS. ;)<p>Yes. There is worse stuff out there. I was using ZFS before with lots of really fucked up disks in a kind of &quot;don&#x27;t do this, you are stupid!&quot; setup and ZFS just did it&#x27;s job and never complained, same Hadoop workload + disks with thousands of bad sectors. Maybe it&#x27;s unfair to compare btrfs to this, however it was a ride full of 	disillusionment to think btrfs compares to ZFS.<p>Maybe it&#x27;s getting there and I was in at a bad time... I&#x27;m not so sure through.