Re: BUG #5166: readdir() failure on Mac OS-X is HFS "feature"

Some more information on the readdir() failure.

I investigated the way GNU handled the problem with rm for Darwin.
http://git.savannah.gnu.org/cgit/coreutils.git/tree/src/remove.c
[Note the current version of this file no longer uses readdir() - more below]

It appears that readdir() can fail:

1) On all Darwin/Mac OS-X for HFS+ and NFS
2) On SunOS on UFS

It appears to be triggered be a combination of the number of changes
and the total filename length of those changes forcing a new hash
table to be created. It appears that you are unlikely to observe this
bug if the number of files in the directory is no more than 180 or so.
But I think they found pathological cases as low as 16.

My pg_xlog currently has 259 files :-(

Looking at coreutils/src/remove.c as at 2006-09-29:

Changelog:
+2006-09-29 Jim Meyering <jim(at)meyering(dot)net>
+
+ Work around a readdir bug in Darwin 7.9.0 (MacOS X 10.3.9) on HFS+
+ and NFS, whereby rm would not remove all files in a directory.
+ * src/remove.c (CONSECUTIVE_READDIR_UNLINK_THRESHOLD): Reduce to 10.
+ (NEED_REWIND): New macro, so that we incur the cost of the work-around
+ rewinddir only on afflicted systems.
+ * NEWS: Clarify and correct.
+ * tests/rm/readdir-bug: New file. Test for the above fix.
+ * tests/rm/Makefile.am (TESTS): Add it.
+ Prompted by testing and analysis from Bruno Haible:
+ http://lists.gnu.org/archive/html/bug-coreutils/2006-09/msg00326.html

remove.c:
/* This is the maximum number of consecutive readdir/unlink calls that
can be made (with no intervening rewinddir or closedir/opendir) before
triggering a bug that makes readdir return NULL even though some
directory entries have not been processed. The bug afflicts SunOS's
readdir when applied to ufs file systems and Darwin 6.5's (and OSX
v.10.3.8's) HFS+. This maximum is conservative in that demonstrating
the problem requires a directory containing at least 16 deletable
entries (which doesn't count . and ..).
This problem also affects Darwin 7.9.0 (aka MacOS X 10.3.9) on HFS+
and NFS-mounted file systems, but not vfat ones. */
enum
{
CONSECUTIVE_READDIR_UNLINK_THRESHOLD = 10
};

#ifdef HAVE_WORKING_READDIR
# define NEED_REWIND(readdir_unlink_count) 0
#else
# define NEED_REWIND(readdir_unlink_count) \
(CONSECUTIVE_READDIR_UNLINK_THRESHOLD <= (readdir_unlink_count))
#endif

####

But just recently, on 2009-09-11, all those hacks were replaced. From
the changelog:
+** Improvements
+
+ rm: rewrite to use gnulib's fts
+ This makes rm -rf significantly faster (400-500%) in some pathological
+ cases, and slightly slower (20%) in at least one pathological case.

A comment that was removed:
-/* FIXME: in 2009, or whenever Darwin 7.9.0 (aka MacOS X 10.3.9) is no
- longer relevant, remove this work-around code. Then, there will be
- no need to perform the extra rewinddir call, ever. */
-#define NEED_REWIND(readdir_unlink_count) \
- (CONSECUTIVE_READDIR_UNLINK_THRESHOLD <= (readdir_unlink_count))

So apparently GNU thought that Apple would eventually fix readdir().
But I haven't been able to track down the documentation behind this
FIXME.

I can't see any specific Darwin code remaining in remove.c as of today.

An example of the 2006 discussion of the readdir problem is here:
http://lists.gnu.org/archive/html/bug-coreutils/2006-09/threads.html#00359

And the first annoucement of the complete rewrite of remove.c is here:
http://lists.gnu.org/archive/html/bug-coreutils/2009-08/msg00295.html

Stephen Tyler