Programming guide

For people with a developer license, here’s some advice for how to write new code intended for distribution with CASINO.

The main CASINO source code and many of the utilities are written in Fortran, which must conform to the Fortran 95 standard, since later standards are not yet necessarily supported by all compilers – see here. (Note that as of 2016 we are having to relax this a little bit since it appears that current openmpi packages use Fortran 2003 constructs). There are also a couple of simple C routines in the main code (for shared memory etc.). Utilities not in Fortran should generally be written as bash shell scripts (or possibly csh, though this is deprecated). Other languages add to problems with portability, but have occasionally been allowed in the past. There is a C++ pseudopotential conversion utility. The ADF converter in utils/wfn_converters/adf is written in Python, and the GAMESS converter in utils/wfn_converters/gamess is in Perl, though we provide no official support for these or any other programming languages. Please understand that CASINO is meant to compile, setup and run out of the box on any computer in the world, and the fewer languages we are dependent on the easier this is to achieve.

Style

CASINO has a Fortran95 ‘style’ which should be adhered to when writing code, both for the main source and for the Fortran utilities. This is because it is desirable that the package has a homogeneous look and feel (and because searching for text strings then works consistently). Everybody has their own style. Yours is different and may even be better, but we’ve decided on one for CASINO and there it is. If you don’t write your code like this, the likelihood is that MDT or someone else will reformat it for you, and they will probably accidentally delete a crucial minus sign while correcting your routine, an error which may take two weeks of your life to track down (and this could be much better spent doing other things). You can get the idea just by looking at the developer version source code in CASINO/src (the standard version is ‘obfuscated’ and practically unreadable) but let’s emphasize the main points:

  • Don’t use more than 1 module per physical file (Hitachi machines won’t allow it!).
  • Capitalization outside of character context: use upper-case letters for keywords whose use is primarily at compile time (statements that delimit program and subprogram boundaries, declaration statements of variables). Use lower-case letters for everything else, including the bulk of run-time code. Description of routines is to be enclosed in a little box with a description of what it does and an author. Later changes are to be documented at the bottom of this box. For example:SUBROUTINE rubbish
    !------------------------------------------------------------!
    ! Routine not to do anything at all. !
    ! !
    ! MDT 8.2000 !
    ! !
    ! Changes: !
    ! -------- !
    ! 3/2001 MDT - added capability to write 'Hello'. !
    !------------------------------------------------------------!
    USE dsp
    USE parallel
    IMPLICIT NONE
    INTEGER i,j,k,some_integer
    REAL(dp) a
    CHARACTER(llength)my_name
    i=0
    do j=1,10
    write(6,*)'Hello'
    do k=1,1000000
    i=i+1
    write(6,*)'Everyday Spanish'
    enddo
    enddo
    ! etc..
    END SUBROUTINE rubbish
  • Contents of if, do and case blocks to be indented by one space.
  • In general don’t put spaces between words, e.g., 

    if(coul_mpc)energy=energy1+energy2

     

    not


    if ( coul_mpc ) energy = energy1 + energy2

  • Use F90 versions of:  .eq. .ne. .lt.  .le.  etc.. i.e., ==, /-, <, <=
  • Maximum of 80 characters per line. If you go over this, use ‘&’ continuation characters at the end of the line and at the beginning of the next one (don’t include spaces between the ampersand and the text unless it’s part of a character string).
  • Two blank lines between subroutines in a given physical file.
  • Error conditions to be handled using the errstop, errstop_master, and errwarn routines (in the utilities.f90 module). Note that if calling errstop_master, all processes must be guaranteed to call it (and only the master will write the error); if not CASINO will hang.
  • Use ‘endif‘ and ‘enddo‘  not  ‘end if‘ and  ‘end do‘.
  • No double colons in simple variable declarations e.g.INTEGER ialloc except where required e.g.
    INTEGER,INTENT(in) :: n
  • In lists of declared variables, adhere to the following order:INTENTed dummy arguments first, order:

    in/inout/out

    INTEGER,INTENT(in)
    (INTEGER,INTENT(out) etc..)
    (INTEGER,ITENT(inout) etc..)
    REAL(sp),INTENT(in)
    REAL(dp),INTENT(in)
    COMPLEX(sp),INTENT(in)
    COMPLEX(dp),INTENT(in)
    LOGICAL,INTENT(in)
    CHARACTER(12),INTENT(in)
    TYPE(xx),INTENT(in)

    followed by things which are not arguments

    INTEGER
    REAL(sp)
    REAL(dp)
    COMPLEX(sp)
    COMPLEX(dp)
    LOGICAL
    CHARACTER(12)
    TYPE

    Within each class, put standard variables on the first line, followed by things with attributes like ALLOCATABLE, PARAMETER, etc., on subsequent lines, in whatever order seems aesthetically pleasing.Note the CHARACTER(12), not CHARACTER*12, which is not in the Fortran95 standard.

  • Don’t use tab characters anywhere in the code.
  • All units for reading and writing to be allocated unit numbers using the standard ‘call open_units’ procedure which you can figure out by looking at other examples in the source.
  • Initial letters of comments are to be in capitals. Comments are to be spelt correctly and end with a full stop/period (if they form a complete sentence). Comments must be useful.
    ! This is a legitimate comment.! this is an illegitimate comment X

    ! tihs one is evn more illetigimate as i cant spell XX

    ! allocate a XXX
    allocate(a(1))

  •  If you add a module USE statement anywhere, don’t forget to change the dependency list in the Makefile. This can be done automatically by running ‘update-makefile-tool‘ in the CASINO/src directory.
  • Module USE statements in alphabetical order, followed by USE ONLY’s, in alphabetical orderUSE a
    USE b
    USE c
    USE a1, ONLY : x
    USE b1, ONLY : y
    USE c1, ONLY : z
  • Don’t use ‘return‘ at the end of each routine – just use END SUBROUTINE or END FUNCTION etc.
  • Don’t use GOTO statements if you can avoid them (which you usually can).

Note you’ll find code written in this way is very readable if you turn on CASINO’s syntax highlighting – see the documentation in the manual.

C routines

The CASINO source currently contains two C routines etime.c and alloc_shm.c. Please make reference to these to get some idea of how to format new C routines. The only important point is not to use new-style C comments designated by // – there are still C compilers that we supposedly support which do not allow this. Use the old-style /* */ form instead.

Content

CASINO uses the git revision control system (see http://git-scm.com). There is an official CASINO document written by PLR which explains the most important features of this system, including how to submit content for inclusion into the main distribution. It can be found in the CASINO/doc directory as ‘git_guide.pdf‘ or online here).

Note that modifications to CASINO will not usually be accepted unless the patches are done on the most recent development version of the code via the standard git mechanism.

Even if modifications are made in the correct manner, you may find that they are still not incorporated into the public release. If you want this to happen, there are a variety of ways to lower the energy barrier for the inclusion of your routines. The general theme here is to try to reduce the amount of effort that MDT has to perform to validate and check your code. If your submission includes lots of things from the ‘Good things’ list, and none at all from the ‘Bad things’ list, then inclusion is semi-automatic and will normally take place within days. We are aware that it requires a lot of work to do the things on the Good list but of course we will have to do it if you don’t!

Good things

  • Well written absolutely standard Fortran95 in the standard CASINO format. Do not use features from later releases of the Fortran standard, or non-standard (albeit useful) extensions like ‘flush‘.
  • The results of extensive testing of your routine with a range
    of examples, with before and after numbers (including timings). This should be done with the autotest suite (see the README file in examples/TEST and the discussion in Appendix 2 of the manual).
  • A detailed description of whatever your code is supposed to do (I shouldn’t have to go through each line of your code at the same detailed level you did, or I might as well have written it myself.).
  • A bit of TeX for inclusion in the manual (if the code requires the user to know something over and above what the current manual describes). This should be well-written, comprehensive, and helpful.
  • Some example input/output which demonstrate the new capabilities which MDT can just run and look at, and perhaps incorporate into the CASINO/examples directory, without having to invent his own.
  • Evidence of testing on serial and parallel machines. Remember that in general only the master processor should write to the output file. Also, CASINO is supposed to compile and work on single-processor machines without an installed MPI (Message Passing Interface) library. Use the fake comms_serial.f90 module supplied both in the utils directory and with the main code (in general you only need worry about this if you introduce a call to an MPI routine which is not already used in CASINO and which has not been ‘faked’).
  • If you use MPI calls when doing the parallel bits, try to ensure they are included in the MPI-1 standard since, somewhat unbelievably, there are some computers in the world which claim not to be able to support the 14-year-old MPI-2 standard. Currently, the only MPI-2 routine used in CASINO is MPI_GATHER_IN_PLACE}, and PLR has implemented a fake version of this in the comms_parallel_mpi1.f90 routine which can automatically be used on machines which do not have an MPI-2 installation (if their CASINO_ARCH file tells them so).
  • Your routine uses the standard CASINO facilities for handling errors – errstop/errwarn etc.). These can be found in the run_control.f90 module.
  • The routine implements something that we desperately want or is in the TODO list.

Bad things

  • We do not sacrifice speed for additional functionality if at all possible. Think of another way to write it if it slows the code down.
  • We do not like to include things unless they are completely general. The development of general, complex electronic structure codes can be set back years by people choosing to implement functionality which works only for their current project. Non-general algorithms tend to get completely thrown away and re-implemented, which wastes the time of both parties. It is not usually much harder to write a general program than it is to write a specific one (we understand this is not always true!).
  • You should not use any calls to external libraries apart from MPI, BLAS and LAPACK.
  • When writing to the output file, the new routines should not write out lots of weird arrays (whose meaning is understood only by the author) in an incredibly scruffy format full of spelling mistakes. In general, be as beautiful and informative as you can when writing to output, or MDT will just have to make it so (which takes ages). If there are all sorts of special cases which require different output, then so be it  — ‘select case‘ and ‘if‘ blocks are very useful in this regard.
  • Prior agreement for modification of the code should have been obtained. Unsolicited submissions are not accepted (unless we really want to) and can lead to moody lack of cooperation on our part.

Testing and debugging CASINO

Here is some advice on how to go about testing and debugging CASINO after
you have made some modifications.

  • Type ‘make debug‘ in /CASINO or /CASINO/src/ in order to compile the code with debugging options set. The resulting binary can be run by the run script using ‘runqmc --debug‘ or, equivalently, ‘runqmc -d‘.
  • The best compilers for debugging on PCs are, in order of decreasing preference: NAG, G95 (now deprecated), Gfortran, PathScale, Intel. MDT has recently discovered that the Cray compiler is pretty good too.
  • It is a good idea to try running the code using several different compilers on several different architectures: this maximizes the chance of finding problems. There are bugs which require a combination of compilers to locate successfully. Note that you should always test that your modifications work on parallel computers, if possible.
  • If you encounter a problem, try changing options in the input file before looking in the code. For example, does the problem only occur when a Jastrow factor is used? Does it only occur when the cusp correction is used? Try to obtain the simplest set of circumstances capable of reproducing the problem.
  • Track down bugs either by (i) inserting write statements in the code and recompiling or (ii) using a debugger (such as gdb) in conjunction with binaries that have been compiled with the debugging flags set, which you can use by passing the --debugger=debugger-binary option to runqmc.
  • Any utilities you write should also be tested by compiling and running with full debugging flags. Use, e.g., ‘man f90‘ to find out about the available debugging flags for your compiler.
  • If you get stumped and have to send a bug report to us in Cambridge, then please make sure that everything we need to reproduce the problem is attached or otherwise available. Our record so far is an exchange of 24 emails—beginning with one stating ‘CASINO doesn’t work!’—before we discovered precisely what it was the user was complaining about (that is, what the problem was, rather than the solution). We shall leave it to you to guess which distinguished London-based professor we are referring to.

Performance

CASINO can print a detailed timing analysis at the end of the output file when it has finished a calculation. This information can be useful to the programmer in deciding what optimizations to do to make the program run faster. It is generally disabled by default since, annoyingly, the process of calculating the timings slows down the code significantly. To activate the detailed analysis, set ‘timing_info=T‘ in the input file.

The total time is printed as ‘CPU’ time, and the ratio between ‘System’ and ‘User’ time is displayed as well. These timings are broken down according to specific tasks, although by no means all activities are timed — only those thought likely to contribute significantly. For very small systems such as atoms a significant amount of CPU time will remain unaccounted for since the traditional heavyweight tasks are very fast. For medium and large systems, often the calculation of the orbitals and their derivatives will dominate, closely followed by Jastrow factors and Ewald interactions. In systems with pseudopotentials, the nonlocal integration is very expensive indeed (since it calls the orbital evaluation routine a lot) and this expense will increase the larger the nonlocal cutoff radius (lower ‘nlcutofftol‘ in the input file to see the effect of this).

The ‘User’ time is the time spent executing the instructions that make up the user code, including, e.g., calls to subroutine libraries linked to the code.

The ‘System’ time is the time spent in kernel mode, processing I/O requests, for example, or other situations which require the intervention of the operating system.

Together these comprise the total CPU time. Note that the User time is usually significantly greater than the System time. Indeed, a high ratio of system to user CPU time may indicate a problem. Exceptions such as page faults, misaligned memory references, and floating-point exceptions consume a large amount of system time. Note that time spent doing things like waiting for input from the terminal, seeking on a disk, or rewinding a tape does not appear in the CPU time; these situations do not require the CPU and thus it can (and generally does) work on something else while waiting.

The elapsed (‘wallclock’ or ‘real’) time is simply the total real time that passed during the execution of the program. This will typically be greater than the total CPU time due primarily to sharing the CPU with other programs. In addition, programs that perform a large number of I/O operations requiring more memory bandwidth than is available on the machine (i.e., the CPU spends significant time waiting for data to arrive from memory, or is paged or swapped) will show a relatively low ratio of CPU time used to elapsed time.

Bug reports

We cannot of course guarantee that CASINO is free of bugs, and if you find one, please tell us. If you don’t know what the problem is yourself, than please include as much detailed information you can about the nature of the error in order to ensure a quick fix. In particular, please send us all the relevant input files to enable us to reproduce your problem, and tell us precisely what version of the code you’re using.

Requests for new features

We are of course always happy to discuss such requests.

MDT 9/2001

Leave a Reply