it is known/reported/discussed issue which I have not succeeded to resolve more than two/three years, that jfs+nfs has significant (~2 times) performance issue on appending to files more than 1-2TB, and xfs+jfs has significant (~1 digit / ~10 times) performance issue on any stat action to dir/file. so, for now PFS servers at IPMU uses jfs for its NFS server, but following Princeton order I configured the summit as xfs for NFS storage.
so, help wanted, or wontfix.

The disk controller has a battery-backed RAM cache, right? Are the xfs barriers turned off (along with disk caches)?

its due to a combination of xfs and NFS, so it does not depends on its storage device. ~1 digit performance degradation was measured between physical and NFS, as previously reported.

But XFS barriers do matter, especially on NFS servers – if you can legitimately turn them off because you have battery-backed cache in the right place, you gain enormously.

if it is an issue on bulk (or even sequential) read/write performance, such async or caching could work well, but the issue is handling of stat.
async option could help a bit for stat performance, but it is after NFS server-client communication (like server reply immediately after commit with no-op) and in filesystem + block device, so it would not have any significant change for our issue.

1. actually, testing with noatime (client), async (server) does not change stat performance.