Why would native Mac filesystem performance be MUCH worse than various virtualized disks on the same mac? I was attempting to profile a php application when I noticed it was running MUCH slower on my mac vs docker vs virtualbox (all on the same hardware). I started narrowing down possible issues, but now I'm not sure where to go. It seems to be related to the filesystem or core utilities itself, but I'm not sure.

So I started simplifying the problem and the running some tests. I dumped a large number of files from the app (~7.5K files, 74MB total size) as a sample set to perform operations on into a single dir, and then copied the dir to the container, and the vm.

Test 1: time -p find . -type f -exec md5sum {}  \; &gt; /dev/null

Native | Docker | VirtualBox 
  27s  |   6s   |    5s


Test 2: time -p find . -type f -exec cp {} /dev/null  \; &gt; /dev/null

Native | Docker | VirtualBox
  16s  |   9s   |    7s


Test 3: time -p { dd if=/dev/zero of=/tmp/delme bs=4k count=100000 &amp;&amp; sync; }

Native | Docker | VirtualBox
  0.7s |   0.8s |    0.4s


With the first 2 tests, I was trying to test reading and operating on many files using native utils. OSX was significantly worse. I was trying to test raw disk speed with the 3rd test.

I know this methodology may be flawed or there may be more accurate way to test, but this is just what I came up with and seemed representative enough for troubleshooting. (More details - I ran each test on each env sequentially, repeated the process 3 times, and took the average. Mac is Catlina 10.15.3 encrypted APFS (no errors reported on the drive). Docker container is debian:latest. VB VM is Ubuntu 16.04.

So what's possibly going on here? Am I overlooking something obvious? Is it likely the differences in filesystem performance? I also ran these test on a linux cloud container running on AWS, and had numbers aligned with the Docker/VB VM. Does it have something to do with how virtualized files are typically stored/read? Does the encryption on the native disk matter (since they are all ultimately on this disk)? Could it be some corporate endpoint security process intercepting native read/writes? (I'm unaware of any, but it's a company laptop, and I don't know everything installed, but when I run sudo htop, I'm not seeing anything unusual to me.)

Any ideas for further troubleshooting? Using a virtualized layer on top of the native hardware to get a 2-4x improvement just seems wrong.

Edit 1: Results with disk cache clearing

So after reading this link: https://stackoverflow.com/questions/28845524/echo-3-proc-sys-vm-drop-caches-on-mac-osx

I re-ran the tests with a flushed disk cache and noted the change (∆) between the first and 2nd run.

For Native:
sync &amp;&amp; sudo purge; run test; run test immediately again;

For Docker and VB:
echo 3 &gt; /proc/sys/vm/drop_caches; run test; run test immediately again;

Native | Docker | VirtualBox
  ∆5s  |   ∆3s  |    ∆3s


There was a clear difference between first and second run, but they all improved with disk caching, and the timings from my original tests indicate a primed a cache for all the original runs that I noted, and the performance difference still remains.

Edit 2: Sharing a script for others to test/verify

I wrote a short script to create some sample data files, simulate a workload similar to what I described, and print the results. Then I ran it on an older 2015 Macbook that I recently reset and installed Catalina on (so no unknown corporate apps, no encrypted APFS, etc.) I tested it natively and using Docker for Mac on that machine. Once again for the limited testing, the linux VM (running docker) on the Mac significantly outperformed OSX. I then tried with my original php app. Same result.

If you try the script and notice significantly different results, please let me know. If anyone has suggestions to improve the script/tests, please do, but otherwise there are certain scenarios/workloads that I just won't be using OSX for from now on and will be using a vm/docker on the Mac instead.
What you are seeing is the consequence of your hypervisor lying to you about disk write flushing, presumably because you have disk writeback caching enabled in the VM configuration. Disable the write caching, and you will find the performance is going to be much more similar (probably worse than bare metal).

As a secondary test, you can try on your host by LD_PRELOAD-ing libeatmydata and you should see that your bare metal performance reaches or surpasses the performance in VMs.