Stat inefficiency

[Vladimir_Konjkov]

Hi Mike

I have been looking at an instability in VMC optimization when the sampled configurations are distributed very unevenly. In regions with few configurations, the Jastrow/backflow parameters can fit statistical noise rather than the underlying function, so the optimized function changes noticeably between independent samplings.

My current optimization scheme is varmin → emin → emin → emin. I am considering replacing the initial varmin step with madmin, since MAD is less sensitive to outliers and may produce a more stable starting wave function. The idea is that the subsequent emin steps can then optimize the energy without inheriting spurious features introduced by fitting the variance.

In the paper, this is described as a problem of local minima. The underlying issue is that a large statistical fluctuation in the tail leads to a poor approximation, and the new distribution generated from this incorrect approximation then reinforces the effect, making the instability self-perpetuating.

Vladimir