-
Type:
Task
-
Status: Open (View Workflow)
-
Priority:
Normal
-
Resolution: Unresolved
-
Component/s: ics_production
-
Labels:None
The VMs seem to be very slow, probably from IO.
I'll compare installing conda on shell-ics to an NFS-mounted /software, to installing on a physical oldish machine with not special spinning disks. I'd expect the NFS to maybe cause some slowdown, but I though there was a fast cache for that on the server? But it is much worse than that – bad enough to be a problem.
bash Miniconda3-latest-Linux-x86_64.sh -b -p /software/conda: 2m22 vs. 13s
conda update -y conda: 1m5s vs. 16s
conda install numpy cython twisted ply future astropy ruamel_yaml ipython: 6m48s vs. 2m42s
The disk controller has a battery-backed RAM cache, right? Are the xfs barriers turned off (along with disk caches)?
its due to a combination of xfs and NFS, so it does not depends on its storage device. ~1 digit performance degradation was measured between physical and NFS, as previously reported.
But XFS barriers do matter, especially on NFS servers – if you can legitimately turn them off because you have battery-backed cache in the right place, you gain enormously.
if it is an issue on bulk (or even sequential) read/write performance, such async or caching could work well, but the issue is handling of stat.
async option could help a bit for stat performance, but it is after NFS server-client communication (like server reply immediately after commit with no-op) and in filesystem + block device, so it would not have any significant change for our issue.
- actually, testing with noatime (client), async (server) does not change stat performance.
it is known/reported/discussed issue which I have not succeeded to resolve more than two/three years, that jfs+nfs has significant (~2 times) performance issue on appending to files more than 1-2TB, and xfs+jfs has significant (~1 digit / ~10 times) performance issue on any stat action to dir/file. so, for now PFS servers at IPMU uses jfs for its NFS server, but following Princeton order I configured the summit as xfs for NFS storage.
so, help wanted, or wontfix.