Maximum allowed value of ulimit -n

[cyrilarnould]

I often perform grep -r searches, sometimes over very large directories. At some point, those searches fail with too many open files. I similarly run into this problem when I want to natively compile elisp packages for Emacs in parallel.

Searching online I've found that ulimit -n is responsible for this limitation. When I try to change the limit however, it fails saying cannot modify limit: Too many open files. I guess this probably also comes from cygwin. Curiously, I find very little discussion on this topic aside from people saying it's not possible, e.g. https://cygwin.com/pipermail/cygwin/2006-October/151485.html

It would be great to see this implemented, although there's probably a reason why this can't easily be changed. I'd be interested to find out why though. I'd also welcome any workarounds to my problems, it seems to me like I can't be the only one to regularly run into this.

[Compyx]

The issue with grep can be solved by using find with the -exec switch, for example:
find src \( -name '*.c' -o -name '*.h' \) -exec grep '#include' {} \;
will search for all .c/.h files under src/ and execute grep for each file found, one at a time.

As for the limit on open files, I wouldn't be surprised if that's hardcoded in Windows.

[cyrilarnould]

I'm currently running some tests to see what the limitations are. The limit is definitely not 3200.

If I just grep -r in my Windows %APPDATA% folder the error occurs. It's 57'296 files and a total of 18'255 folders.

But if I create a test folder with 200'000 files, I don't get any issues.

[1480c1]

Perhaps you need to simulate a ton of folders? I can only assume that a lot of the concurrently open files would be folders, but I haven't dug deep into grep's source code to see if it fails to close anything

[cyrilarnould]

Ok, I have a testcase. Should I describe it here or create a separate issue?

[1480c1]

If you truly found an issue, an issue would probably be better

[cyrilarnould]

I actually only realized afterwards that the 200'000 files also were problematic in that I only get results for the first 100'000 files (but no Too many open files message). I've created two separate issues for the two problems: #11675 and #11676.