Stat inefficiency

[Vladimir_Konjkov]

Hello CASINO developers

In one of the calculations I noticed a warning " Warning: [POPSTATS_DUMP] Significant inefficiency due to population correlation. Be sure to understand the implications." and indeed:

Code: Select all

 
 Target weight                =     4267.000000000000
 Average population           =     4267.373640000001 +/-        0.068487247447
 Effective population         =     1955.022416256051
 Stat inefficiency (est)      =        1.005191070598 +/-        0.000529289365
 Stat inefficiency (measured) =        2.182802633226
 

how do I react or avoid such a message, what does it mean?

I added the results of the calculation to the attachment, in order to unpack them, you need to use the following command:
cat 0005_4267_*.tgz | tar xzf -

DMC energy is quite reasonable for such a wave function/Jastrow/Backflow.

Best, Vladimir.

[Neil Drummond]

Dear Vladimir,

That message means that the rate at which the walkers are branching is comparable with the rate at which they are decorrelating due to propagation in imaginary time. Hence each walker is similar to its sibling walkers (produced in the same branching event). Thus the walkers are not statistically independent and the effective walker population is small. This marks the onset of an exponential wall as a function of system size.

To avoid the situation you must somehow decrease the branching rate, which is controlled by the spread of local energies. The standard deviation of the local energy should ideally be << than the difference between the fixed-node first-excited-state and ground-state energies; if this is the case then branching only occurs occasionally and most walkers are independent of each other.

The standard deviation of the local energies can be reduced by using a smaller system size or by improving the trial wave function... Sorry not to have more helpful message.

Best wishes,

Neil.

[Vladimir_Konjkov]

Thanks Neil for the detailed explanation. if you mean the exponential wall described in the article. "Diffusion Monte Carlo: exponential scaling of computational cost for large systems" of Norbert Nemec, then my system is too small for it - a single beryllium atom. Today when recalculating, the warning did not appear:

Code: Select all

 Target weight                =     4267.000000000000
 Average population           =     4268.151816666666 +/-        0.065258516052
 Effective population         =     5903.111606659870
 Stat inefficiency (est)      =        1.005068307850 +/-        0.000471034390
 Stat inefficiency (measured) =        0.723044284765

It should be noted that my DMC energy -14.6648(2) for single beryllium atom (single determinant/jastrow/backflow) is very close to that calculated in article,
"Orbital–dependent backflow wave functions for real–space quantum Monte Carlo" Markus Holzmann, Saverio Moroni, I guess this is the best single determinant Be wave function. Also backlow term for same-spin electron don't contribute energy for that case (Be-Ne atoms) and maybe at all for non periodic systems.

Best wishes, Vladimir.

[Neil Drummond]

Sorry, the comments about system size are obviously irrelevant in this case. But somewhere in the local energy landscape in configuration space there must have been some feature that led to lots of configuration branching, even though your overall wave function is good. Note that this doesn't invalidate your DMC results, especially if you extrapolate to infinite population at the same time as zero time step.

Best wishes,

Neil.

[Vladimir_Konjkov]

Yes, I use a [dtdmc, population] sequence that satisfy dtdmc * population = const equation. Since linear term in the power expansion of timestep error has often opposite sign to linear term of finite population error, one can randomly choose such a const which zeros out the linear term of sequence [dtdmc, population], and the extrapolation curve will unexpectedly become parabolic.

If Hartree–Fock nodal surface and exact one are topologically non-equivalent and backflow is a homeomorphism then one can`t get exact nodal surface from Hartree–Fock by backflow transformations, instead, we actually get some features, but I'm not sure if they affect branching. I wouldn't be surprised if someone has already written an article about this. Maybe they even came up with another magical transformation that would do the job.

What do you think about uselessness of same-spin electron backflow in atomic system? Does such an effect exist in an electron gas calculations?
I mean that in Hartree–Fock approximation we may factor Slater determinant into electron spin components, i.e., ψD=ψαψβ. and after applying same-spin electron backflow we still can.

[Neil Drummond]

The Hartree-Fock nodes in the Be atom can easily be shown to be qualitatively wrong. However, since nodes for most other systems are also qualitatively wrong, I'm not sure why the Be atom should necessarily be so much worse.

I'm not sure why you say that backflow between same-spin electrons is useless. I agree that it doesn't mix up the factorisation of the Slater wave function, but it still describes correlation effects in a way that changes the nodal surface. It's generally true that same-spin backflow is less useful than opposite-spin backflow, because correlation has more work to do to keep opposite-spin electrons apart. However, for the fluid phase of an electron gas, one would certainly include backflow when comparing paramagnetic and ferromagnetic Fermi fluids. At long range, parallel- and antiparallel-spin two-body backflow functions behave in the same manner.

It would be easy enough to test this: one could freeze the opposite-spin backflow parameters at zero and investigate how much effect backflow then has. One could also freeze same-spin backflow parameters at zero and compare with a full backflow calculation.

[Pablo_Lopez_Rios]

Hi Vladimir,

Sorry that I am late to this discussion. Regarding your initial question, note that the "measured" inefficiency is given without an uncertainty - chances are the uncertainty in that estimator and/or the actual bias attained are very large for your calculation. The "estimated" inefficiency is ~ 1 (i.e., no inefficiency). You should just ignore the warning.

Regarding the nodal structure of the beryllium atom, it has indeed been the subject of studies in the literature, e.g., https://doi.org/10.1103/PhysRevB.86.115120 (I think the author had discussed much of this 10 years earlier in a book chapter). The fact that the HF nodes have the wrong topology does lead to a larger relative error for this few-electron system than for other first-row atoms, but backflow evens this out a bit, see, e.g., Fig. 1 of https://doi.org/10.1063/1.3554625 . The argument that a smooth continuous transformation is not capable of modifying the topology of the nodal surface is one that I like to present aided by the attached GIF showing how the "one-electron" node (a sphere in the HF wave function) is supposed to behave with the exact wave function as the other three electrons are moved:

nodes.gif (314.11 KiB) Viewed 32807 times

I can think of no smooth transformation that is capable of turning a sphere inside out like that.

Best,
Pablo

[Vladimir_Konjkov]

Dear Pablo and Neil.

Thanks for the answers, I just wanted to quote the article of Dario Bressanini "Implications of the two nodal domains conjecture for ground state fermionic wave functions" which stated that the quality of the HF wave function for four electron atoms can be greatly improved by adding 2s2->2p2 double excitation which corresponds to a near-degeneracy effect. The 2-configuration wave function can be obtained using CASSCF(2,4) method and consists of 4 determinants if graphical unitary group approach (GUGA) is used to construct configuration state functions (CSFs).
The nodal set of 2-configuration wave function is completely different from that of HF, independent nodal surfaces of the HF configuration merge, an “opening” appears where the two surfaces previously crossed, and only one hypersurface is left, with only two nodal pokets.

it is shown in the following figure:

Bressanini.png (396 KiB) Viewed 32794 times

If we want to get exact nodal surface given HF wave function for the Be atom and backflow transformation, we should transform the points on the right nodal surface using backflow to points on the left nodal surface and substitute these coordinates into HF wave function for the Be atom.
Because HF wave function for the Be atom is equal to zero when the points is on the left surface, but we need when they are on the right.
This should be understood taking into account the fact that the figures show 3-dimensional cuts of a 12-dimensional space and the backflow transforms points of the right cut to set of left cuts, because the coordinates of all electrons are transformed simultaneously. Since these surfaces are not topologically equivalent, it is not possible to transform one into the another using homeomorphism which backflow is. To understand what kind of nodal surface is obtained in the emin/madmin/varmin optimization process, it is necessary to imagine a nodal surface as close as possible to the ideal one (on the right), but topologically equivalent to HF one (on the left), that is, it should have 4 nodal pockets.

I belive that in this case an “opening” appeared in the case of an ideal nodal surface will be closed by a bilayers, but with a long enough dtdmc step the bilayer becomes permeable to walkers. I don`t know whether the angle between two independent nodes remains π/2, this is a consequence of finiteness of local energy -> zero value of the Laplacian of the wave function at the point of intersection of nodes -> if wfn approximate by quadratic form -> angle between two independent nodes = π/2.
But backflow transformation is not conformal mapping so angles are not preserved and it lead me to the conclusion that the backflow creates a far from ideal nodal surface no matter how functionally complex and ideal we make it, inhomogeneous or orbital–dependent both should be equally inaccurate and it turns out that this is so, but I did not finish the calculations.

Yep, this is not the whole my answer I need to think more. But your answers are quite interesting they give me a different point of view on backflow.

Best wishes, Vladimir.