Question: How can lsof report more open files than what ulimit says is the limit?

```
prod_web3(i-ca0b05aa):~$ sudo lsof | wc -l
4399
prod_web3(i-ca0b05aa):~$ ulimit -n
1024

```

Answer: 
That's a common mistake to compare results of raw `lsof` call with supposed limit.

For the global limit (`/proc/sys/fs/file-max`) you should have a look at `/proc/sys/fs/file-nr` -> the first value indicates what is used and the last value is the limit

The OpenFile limit is for each process, but it is defined on a user, see command `ulimit -Hn` for user limits and see `/etc/security/limits.conf` for definitions. Generally applied with "app user" eg:"tomcat": set limit to 65000 to user tomcat that will apply on java process it runs.

If you want to check limit applied on a process, get its PID and then do: `cat /proc/${PID}/limits`
If you want to check how many files are opened by a process, get its PID and then od: `ls -1 /proc/${PID}/fd | wc -l` (note: for ls it's 'minus one', not to confond with 'minus el')

If you want to know details with `lsof` but only for those file handlers that count for the limit, have a try with thoses commands:

* `lsof -p ${PID} | grep -P "^(\w+\s+){3}\d+\D+"`
* `lsof -p ${PID} -d '^cwd,^err,^ltx,^mem,^mmap,^pd,^rtd,^txt' -a`

Remark: the 'files' are files / pipe / TCP connections / etc.

Note that sometimes you'll probably need to be root or to use `sudo` to obtain correct result for the commands. Without privilege sometimes you don't have error, just less results.

Finally, if you want to know what 'files' on your filesystem are accessed by a process, have a look at: `lsof -p ${PID} | grep / | awk '{print $9}' | sort | uniq`

Have fun!
A bash(1) builtin, ulimit defines the maximum number of open files per process.

This is not a system-wide setting in any way.