http://lofs.io/api.php?action=feedcontributions&user=Admin-lofs-wiki&feedformat=atomLOFS - User contributions [en]2024-03-29T14:08:31ZUser contributionsMediaWiki 1.27.7http://lofs.io/index.php?title=Hdf2nc&diff=39Hdf2nc2023-08-09T21:29:52Z<p>Admin-lofs-wiki: </p>
<hr />
<div>Converts any subset of LOFS raw data to a single netCDF file.<br />
<br />
<code><br />
Typical usage: hdf2nc --time=[time] --histpath=[histpath] --base=[ncbase] --x0=[X0] --y0=[Y0] --x1=[X1] --y1=[Y1] --z0=[Z0] --z1=[Z1] [varname1 ... varnameN] <br />
</code><br />
<br />
<code>--time=[time](required)</code>: The model time requested in seconds<br />
<br />
<code>--histpath=[histpath] (required)</code>: Top level directory that contains the 3D model data<br />
<br />
<code>--ncbase=[ncbase] (optional)</code>: base name of netCDF file, otherwise generated from the LOFS basename<br />
<br />
<code>--x0=X0 (optional)</code> westmost index in X. Defaults to westmost index in X found in the saved data.<br />
<br />
<code>--x1=X1 (optional)</code> eastmost index in X. Defaults to eastmost index in X found in the saved data.<br />
<br />
<code>--y0=Y0 (optional)</code> southmost index in Y. Defaults to southmost index in Y found in the saved data.<br />
<br />
<code>--y1=Y1 (optional)</code> northmost index in Y. Defaults to northmost index in Y found in the saved data.<br />
<br />
<code>--z0=Z0 (optional)</code> bottommost index in Z. Defaults to 0.<br />
<br />
<code>--x0=X0 (optional)</code> topmost index in Z. Defaults to topmost index in Z found in the saved data.<br />
<br />
<code>(X0,Y0,Z0)(X1,Y1,Z1)</code>: This defines the volume (or space) that you wish to convert to netCDF fdata, with respect to integer array indices that span the full model domain, namely (0,0,0) to (nx-1,ny-1,nz-1). Each of these are optional. If none of these are passed to hdf2nc, the full 3D range of data saved will be converted to netCDF. If only some of them are provided, the remainder default to the min/max values from the saved data (the code will extract these values from what has been saved, making no assumptions).<br />
<br />
<code>varname1...varnameN (optional)</code> The list of variables, separated by whitespace, that you wish to convert.<br />
<br />
<code>--debug (optional)</code> Turn on debugging output.<br />
<br />
<code>--verbose (optional)</code> Turn on verbose output.<br />
<br />
<code>--recache (optional)</code> Force regeneration of cache files.<br />
<br />
<code>--swaths (optional)</code> Read and write all LOFS 2D swath data to netCDF files.<br />
<br />
<code>--zfp (optional)</code> Turn on lossy ZFP compression for 3D data saved in netCDF files.<br />
<br />
<code>--inprogress (optional)</code> Operate on a 3D directory where a simulation is in progress (since the last directory will be full of zeroed out files that haven't been written to disk yet).<br />
<br />
<code>--offset (optional)</code> Supplied X0,X1,Y0,Y1 values are with respect to what was saved, not (0,0). For instance, if only a subset of the domain was saved, say ranging from 500 to 1000 in X and 600 to 1100 in Y, the following two statements arguments would produce identical results:<br />
<pre><br />
--x0=50 --y0=100 --x1=250 --y1=300 --offset<br />
--x0=550 --y0=700 --x1=750 --y1=900<br />
</pre><br />
<br />
NOTE: u, v, and w (corresponding to ua, va, wa CM1 3D arrays that exist on the Arakawa C staggered mesh) must be saved in LOFS files for diagnostics to be calculated. This is enforced in order to achieve the highest order accuracy possible with diagnostics (specifically those involving derivatives of velocity components), rather than using velocity variables that have already been averaged to the scalar mesh. However, if u, v, and w have been saved and uinterp, vinterp, winterp have been requested, the code will interpolate those values from ua, va, and wa such so for post-processing and visualization, all variables lie on the same mesh. '''''It is strongly recommended for LOFS to save the native staggered velocity variables (ua, va, wa CM1 arrays, corresponding to output_[uvw] = 1 in the namelist.input file) if you wish to calculate diagnostics.''''' Alternatively, you may wish to calculate diagnostics from within CM1, and save interpolated velocity data. We choose to save the minimum amount of data possible to save on disk space, and push the diagnostic calculations to the post-processing stage.<br />
<br />
The following is a list of available additional fields that can be calculated within the hdf2nc code (u, v, w must be saved, as opposed to uinterp, vinterp, winterp):<br />
<br />
{| class="wikitable"<br />
|+Calculated fields<br />
|-<br />
|<code>uinterp</code><br />
|u component of wind interpolated to scalar mesh<br />
|-<br />
|<code>vinterp</code><br />
|v component of wind interpolated to scalar mesh<br />
|-<br />
|<code>winterp</code><br />
|w component of wind interpolated to scalar mesh<br />
|-<br />
|<code>hwin_sr</code><br />
|storm relative horizontal wind speed<br />
|-<br />
|<code>hwin_gr</code><br />
|ground relative horizontal wind speed<br />
|-<br />
|<code>xvort</code><br />
|x component of vorticity<br />
|-<br />
|<code>yvort</code><br />
|y component of vorticity<br />
|-<br />
|<code>zvort</code><br />
|z component of vorticity<br />
|-<br />
|<code>hvort</code><br />
|horizonal vorticity vector magnitude<br />
|-<br />
|<code>vortmag</code><br />
|3D vorticity vector magnitude<br />
|-<br />
|<code>hdiv</code><br />
|horizontal divergence (du/dx + dv/dy)<br />
|}<br />
<br />
<br />
<br />
== hdf2nc Example ==<br />
<br />
<pre><br />
h2ologin2:~/project-bagm/brainstorm2017/15m/history.fs8-15m-a% hdf2nc --histpath=3D --base=frob --time=7101 --x0=2300 --y0=2300 --x1=2400 --y1=2400 --z1=10 uinterp winterp dbz<br />
histpath = 3D<br />
ncbase = frob<br />
time = 7101.000000<br />
X0 = 2300<br />
Y0 = 2300<br />
X1 = 2400<br />
Y1 = 2400<br />
Z1 = 10<br />
Setting Z0 to default value of 0<br />
Read cached num_time_dirs of 146<br />
ntimedirs: 146<br />
Read cached sorted time dirs<br />
Read cached num node dirs<br />
Read cached nodedir<br />
Read cached firstfilename and all times<br />
<br />
We are requesting the following fields: uinterp winterp dbz <br />
<br />
Working on surface 2D thrhopert and dbz (<br />
acca<br />
czza<br />
czza<br />
aaaa<br />
<br />
acca<br />
czza<br />
czza<br />
aaaa<br />
)<br />
Working on uinterp (<br />
acca<br />
czza<br />
czza<br />
aaaa<br />
)<br />
Working on winterp (<br />
acca<br />
czza<br />
czza<br />
aaaa<br />
)<br />
Working on dbz (<br />
acca<br />
czza<br />
czza<br />
aaaa<br />
)<br />
h2ologin2:~/project-bagm/brainstorm2017/15m/history.fs8-15m-a% lt<br />
total 168656<br />
drwxr-xr-x 10 orf PRAC_bagm 4096 Jul 31 11:12 ./<br />
-rw-r--r-- 1 orf PRAC_bagm 1476356 Jul 31 11:12 frob.07101.000000.nc<br />
-rw-r--r-- 1 orf PRAC_bagm 116 Jul 31 11:12 frob.07101.000000.nc.cmd<br />
</pre><br />
<br />
=== Discussion ===<br />
<br />
If no array index options are provided to the command line, hdf2nc will attempt to convert the entire model domain to a netCDF file. In other words, X0,Y0,Z0,X1,Y1,Z1 are optional arguments to hdf2nc. However for typical use cases with large amounts of data, you will want to convert only a subset of the full model domain!<br />
<br />
The output of hdf2nc includes information on some basic metadata, plus some output that tracks the reading of the individual hdf5 files that comprise LOFS. Each letter that comprises the output that looks like<br />
<pre><br />
acca<br />
czza<br />
czza<br />
aaaa<br />
</pre><br />
represents the successful reading of data from a single hdf5 file. It's kind of a 'base 26' representation of the percentage of data (in the horizontal) requested from each file. If a <code>z</code> is printed, that means the full horizontal range of data was requested. If <code>a</code> is printed, a tiny piece of the horizontal range was selected. All intermediate letters represent the space between these two extremes. This output is for your entertainment only; you are essentially watching the assembly of the netCDF file from LOFS data in real time.<br />
<br />
hdf2nc always produces a 2D surface plot of density potential temperature perturbation from base state (proportional to buoyancy) and surface (calculated) radar reflectivity. These fields are used so much that they are always written to the netCDF files whether they are requested or not.<br />
<br />
Regarding the mention of cached data, the LOFS read routines will look for existing cache files before going out and getting all the metadata from hdf5 files, which, for large amounts of data, is very expensive. Since the data layout never changes (unlesss you change it) the cached files speed things up quite a bit. If you ever change your LOFS data (say, adding new time directories), you must remove the cache files and let LOFS regenerate them so they will contain the new information. Cache files all are prefixed by <code>.cm1hdf5_</code> and can always be removed, as they will always be regenerated.<br />
<br />
In this example, the output file name is <code>frob.07101.000000.nc</code>, indicating data that was retrieved at <code>t=07101.000000</code> seconds. Note that LOFS allows for the saving and retrieval of data saved in intervals of less than one second, as time is represented as a floating point variable.<br />
<br />
=== Rationale ===<br />
<br />
LOFS splits the model domain and times into files spread across hundreds directories in large simulations. Often times you may wish to analyze, plot, or visualize a subset of the full model domain at a given time, perhaps to make plots or to feed into visualization software that understands the netCDF format which is one of the most commonly used data formats used in atmospheric science.</div>Admin-lofs-wikihttp://lofs.io/index.php?title=Hdf2nc&diff=38Hdf2nc2023-08-09T20:57:46Z<p>Admin-lofs-wiki: </p>
<hr />
<div>Converts any subset of LOFS raw data to a single netCDF file.<br />
<br />
<code><br />
Typical usage: hdf2nc --time=[time] --histpath=[histpath] --base=[ncbase] --x0=[X0] --y0=[Y0] --x1=[X1] --y1=[Y1] --z0=[Z0] --z1=[Z1] [varname1 ... varnameN] <br />
</code><br />
<br />
<code>--time=[time](required)</code>: The model time requested in seconds<br />
<br />
<code>--histpath=[histpath] (required)</code>: Top level directory that contains the 3D model data<br />
<br />
<code>--ncbase=[ncbase] (optional)</code>: base name of netCDF file, otherwise generated from the LOFS basename<br />
<br />
<code>--x0=X0 (optional)</code> westmost index in X. Defaults to westmost index in X found in the saved data.<br />
<br />
<code>--x1=X1 (optional)</code> eastmost index in X. Defaults to eastmost index in X found in the saved data.<br />
<br />
<code>--y0=Y0 (optional)</code> southmost index in Y. Defaults to southmost index in Y found in the saved data.<br />
<br />
<code>--y1=Y1 (optional)</code> northmost index in Y. Defaults to northmost index in Y found in the saved data.<br />
<br />
<code>--z0=Z0 (optional)</code> bottommost index in Z. Defaults to 0.<br />
<br />
<code>--x0=X0 (optional)</code> topmost index in Z. Defaults to topmost index in Z found in the saved data.<br />
<br />
<code>(X0,Y0,Z0)(X1,Y1,Z1)</code>: This defines the volume (or space) that you wish to convert to netCDF fdata, with respect to integer array indices that span the full model domain, namely (0,0,0) to (nx-1,ny-1,nz-1). Each of these are optional. If none of these are passed to hdf2nc, the full 3D range of data saved will be converted to netCDF. If only some of them are provided, the remainder default to the min/max values from the saved data (the code will extract these values from what has been saved, making no assumptions).<br />
<br />
<code>varname1...varnameN (optional)</code> The list of variables, separated by whitespace, that you wish to convert.<br />
<br />
<code>--debug (optional)</code> Turn on debugging output.<br />
<br />
<code>--verbose (optional)</code> Turn on verbose output.<br />
<br />
<code>--recache (optional)</code> Force regeneration of cache files.<br />
<br />
<code>--swaths (optional)</code> Read and write all LOFS 2D swath data to netCDF files.<br />
<br />
<code>--zfp (optional)</code> Turn on lossy ZFP compression for 3D data saved in netCDF files.<br />
<br />
<code>--nc3 (optional)</code> Save NetCDF version 3 files with 64 bit offsets instead of the default option of NetCDF version 4 files (which are HDF5)<br />
<br />
<code>--interp (optional)</code> Read uinterp, vinterp, winterp directly from LOFS (for cases when u, v, w were not saved). NOTE! Diagnostics such as vorticity cannot be calculated with this option!<br />
<br />
<code>--offset (optional)</code> Supplied X0,X1,Y0,Y1 values are with respect to what was saved, not (0,0). For instance, if only a subset of the domain was saved, say ranging from 500 to 1000 in X and 600 to 1100 in Y, the following two statements arguments would produce identical results:<br />
<pre><br />
--x0=50 --y0=100 --x1=250 --y1=300 --offset<br />
--x0=550 --y0=700 --x1=750 --y1=900<br />
</pre><br />
<br />
NOTE: u, v, and w (corresponding to ua, va, wa CM1 3D arrays that exist on the Arakawa C staggered mesh) must be saved in LOFS files for diagnostics to be calculated. This is enforced in order to achieve the highest order accuracy possible with diagnostics (specifically those involving derivatives of velocity components), rather than using velocity variables that have already been averaged to the scalar mesh. However, if u, v, and w have been saved and uinterp, vinterp, winterp have been requested, the code will interpolate those values from ua, va, and wa such so for post-processing and visualization, all variables lie on the same mesh. '''''It is strongly recommended for LOFS to save the native staggered velocity variables (ua, va, wa CM1 arrays, corresponding to output_[uvw] = 1 in the namelist.input file) if you wish to calculate diagnostics.''''' Alternatively, you may wish to calculate diagnostics from within CM1, and save interpolated velocity data. We choose to save the minimum amount of data possible to save on disk space, and push the diagnostic calculations to the post-processing stage.<br />
<br />
The following is a list of available additional fields that can be calculated within the hdf2nc code (u, v, w must be saved, as opposed to uinterp, vinterp, winterp):<br />
<br />
{| class="wikitable"<br />
|+Calculated fields<br />
|-<br />
|<code>uinterp</code><br />
|u component of wind interpolated to scalar mesh<br />
|-<br />
|<code>vinterp</code><br />
|v component of wind interpolated to scalar mesh<br />
|-<br />
|<code>winterp</code><br />
|w component of wind interpolated to scalar mesh<br />
|-<br />
|<code>hwin_sr</code><br />
|storm relative horizontal wind speed<br />
|-<br />
|<code>hwin_gr</code><br />
|ground relative horizontal wind speed<br />
|-<br />
|<code>xvort</code><br />
|x component of vorticity<br />
|-<br />
|<code>yvort</code><br />
|y component of vorticity<br />
|-<br />
|<code>zvort</code><br />
|z component of vorticity<br />
|-<br />
|<code>hvort</code><br />
|horizonal vorticity vector magnitude<br />
|-<br />
|<code>vortmag</code><br />
|3D vorticity vector magnitude<br />
|-<br />
|<code>hdiv</code><br />
|horizontal divergence (du/dx + dv/dy)<br />
|}<br />
<br />
<br />
<br />
== hdf2nc Example ==<br />
<br />
<pre><br />
h2ologin2:~/project-bagm/brainstorm2017/15m/history.fs8-15m-a% hdf2nc --histpath=3D --base=frob --time=7101 --x0=2300 --y0=2300 --x1=2400 --y1=2400 --z1=10 uinterp winterp dbz<br />
histpath = 3D<br />
ncbase = frob<br />
time = 7101.000000<br />
X0 = 2300<br />
Y0 = 2300<br />
X1 = 2400<br />
Y1 = 2400<br />
Z1 = 10<br />
Setting Z0 to default value of 0<br />
Read cached num_time_dirs of 146<br />
ntimedirs: 146<br />
Read cached sorted time dirs<br />
Read cached num node dirs<br />
Read cached nodedir<br />
Read cached firstfilename and all times<br />
<br />
We are requesting the following fields: uinterp winterp dbz <br />
<br />
Working on surface 2D thrhopert and dbz (<br />
acca<br />
czza<br />
czza<br />
aaaa<br />
<br />
acca<br />
czza<br />
czza<br />
aaaa<br />
)<br />
Working on uinterp (<br />
acca<br />
czza<br />
czza<br />
aaaa<br />
)<br />
Working on winterp (<br />
acca<br />
czza<br />
czza<br />
aaaa<br />
)<br />
Working on dbz (<br />
acca<br />
czza<br />
czza<br />
aaaa<br />
)<br />
h2ologin2:~/project-bagm/brainstorm2017/15m/history.fs8-15m-a% lt<br />
total 168656<br />
drwxr-xr-x 10 orf PRAC_bagm 4096 Jul 31 11:12 ./<br />
-rw-r--r-- 1 orf PRAC_bagm 1476356 Jul 31 11:12 frob.07101.000000.nc<br />
-rw-r--r-- 1 orf PRAC_bagm 116 Jul 31 11:12 frob.07101.000000.nc.cmd<br />
</pre><br />
<br />
=== Discussion ===<br />
<br />
If no array index options are provided to the command line, hdf2nc will attempt to convert the entire model domain to a netCDF file. In other words, X0,Y0,Z0,X1,Y1,Z1 are optional arguments to hdf2nc. However for typical use cases with large amounts of data, you will want to convert only a subset of the full model domain!<br />
<br />
The output of hdf2nc includes information on some basic metadata, plus some output that tracks the reading of the individual hdf5 files that comprise LOFS. Each letter that comprises the output that looks like<br />
<pre><br />
acca<br />
czza<br />
czza<br />
aaaa<br />
</pre><br />
represents the successful reading of data from a single hdf5 file. It's kind of a 'base 26' representation of the percentage of data (in the horizontal) requested from each file. If a <code>z</code> is printed, that means the full horizontal range of data was requested. If <code>a</code> is printed, a tiny piece of the horizontal range was selected. All intermediate letters represent the space between these two extremes. This output is for your entertainment only; you are essentially watching the assembly of the netCDF file from LOFS data in real time.<br />
<br />
hdf2nc always produces a 2D surface plot of density potential temperature perturbation from base state (proportional to buoyancy) and surface (calculated) radar reflectivity. These fields are used so much that they are always written to the netCDF files whether they are requested or not.<br />
<br />
Regarding the mention of cached data, the LOFS read routines will look for existing cache files before going out and getting all the metadata from hdf5 files, which, for large amounts of data, is very expensive. Since the data layout never changes (unlesss you change it) the cached files speed things up quite a bit. If you ever change your LOFS data (say, adding new time directories), you must remove the cache files and let LOFS regenerate them so they will contain the new information. Cache files all are prefixed by <code>.cm1hdf5_</code> and can always be removed, as they will always be regenerated.<br />
<br />
In this example, the output file name is <code>frob.07101.000000.nc</code>, indicating data that was retrieved at <code>t=07101.000000</code> seconds. Note that LOFS allows for the saving and retrieval of data saved in intervals of less than one second, as time is represented as a floating point variable.<br />
<br />
=== Rationale ===<br />
<br />
LOFS splits the model domain and times into files spread across hundreds directories in large simulations. Often times you may wish to analyze, plot, or visualize a subset of the full model domain at a given time, perhaps to make plots or to feed into visualization software that understands the netCDF format which is one of the most commonly used data formats used in atmospheric science.</div>Admin-lofs-wikihttp://lofs.io/index.php?title=Hdf2nc&diff=37Hdf2nc2023-08-09T20:57:01Z<p>Admin-lofs-wiki: Making some much needed changes (1)</p>
<hr />
<div>Converts any subset of LOFS raw data to a single netCDF file.<br />
<br />
<code><br />
Typical usage: hdf2nc --time=[time] --histpath=[histpath] --base=[ncbase] --x0=[X0] --y0=[Y0] --x1=[X1] --y1=[Y1] --z0=[Z0] --z1=[Z1] [varname1 ... varnameN] <br />
</code><br />
<br />
<code>--time=[time](required)</code>: The model time requested in seconds<br />
<code>--histpath=[histpath] (required)</code>: Top level directory that contains the 3D model data<br />
<code>--ncbase=[ncbase] (optional)</code>: base name of netCDF file, otherwise generated from the LOFS basename<br />
<code>--x0=X0 (optional)</code> westmost index in X. Defaults to westmost index in X found in the saved data.<br />
<code>--x1=X1 (optional)</code> eastmost index in X. Defaults to eastmost index in X found in the saved data.<br />
<code>--y0=Y0 (optional)</code> southmost index in Y. Defaults to southmost index in Y found in the saved data.<br />
<code>--y1=Y1 (optional)</code> northmost index in Y. Defaults to northmost index in Y found in the saved data.<br />
<code>--z0=Z0 (optional)</code> bottommost index in Z. Defaults to 0.<br />
<code>--x0=X0 (optional)</code> topmost index in Z. Defaults to topmost index in Z found in the saved data.<br />
<code>(X0,Y0,Z0)(X1,Y1,Z1)</code>: This defines the volume (or space) that you wish to convert to netCDF fdata, with respect to integer array indices that span the full model domain, namely (0,0,0) to (nx-1,ny-1,nz-1). Each of these are optional. If none of these are passed to hdf2nc, the full 3D range of data saved will be converted to netCDF. If only some of them are provided, the remainder default to the min/max values from the saved data (the code will extract these values from what has been saved, making no assumptions).<br />
<code>varname1...varnameN (optional)</code> The list of variables, separated by whitespace, that you wish to convert.<br />
<code>--debug (optional)</code> Turn on debugging output.<br />
<code>--verbose (optional)</code> Turn on verbose output.<br />
<code>--recache (optional)</code> Force regeneration of cache files.<br />
<code>--swaths (optional)</code> Read and write all LOFS 2D swath data to netCDF files.<br />
<code>--zfp (optional)</code> Turn on lossy ZFP compression for 3D data saved in netCDF files.<br />
<code>--nc3 (optional)</code> Save NetCDF version 3 files with 64 bit offsets instead of the default option of NetCDF version 4 files (which are HDF5)<br />
<code>--interp (optional)</code> Read uinterp, vinterp, winterp directly from LOFS (for cases when u, v, w were not saved). NOTE! Diagnostics such as vorticity cannot be calculated with this option!<br />
<code>--offset (optional)</code> Supplied X0,X1,Y0,Y1 values are with respect to what was saved, not (0,0). For instance, if only a subset of the domain was saved, say ranging from 500 to 1000 in X and 600 to 1100 in Y, the following two statements arguments would produce identical results:<br />
<pre><br />
--x0=50 --y0=100 --x1=250 --y1=300 --offset<br />
--x0=550 --y0=700 --x1=750 --y1=900<br />
</pre><br />
NOTE: u, v, and w (corresponding to ua, va, wa CM1 3D arrays that exist on the Arakawa C staggered mesh) must be saved in LOFS files for diagnostics to be calculated. This is enforced in order to achieve the highest order accuracy possible with diagnostics (specifically those involving derivatives of velocity components), rather than using velocity variables that have already been averaged to the scalar mesh. However, if u, v, and w have been saved and uinterp, vinterp, winterp have been requested, the code will interpolate those values from ua, va, and wa such so for post-processing and visualization, all variables lie on the same mesh. '''''It is strongly recommended for LOFS to save the native staggered velocity variables (ua, va, wa CM1 arrays, corresponding to output_[uvw] = 1 in the namelist.input file) if you wish to calculate diagnostics.''''' Alternatively, you may wish to calculate diagnostics from within CM1, and save interpolated velocity data. We choose to save the minimum amount of data possible to save on disk space, and push the diagnostic calculations to the post-processing stage.<br />
<br />
The following is a list of available additional fields that can be calculated within the hdf2nc code (u, v, w must be saved, as opposed to uinterp, vinterp, winterp):<br />
<br />
{| class="wikitable"<br />
|+Calculated fields<br />
|-<br />
|<code>uinterp</code><br />
|u component of wind interpolated to scalar mesh<br />
|-<br />
|<code>vinterp</code><br />
|v component of wind interpolated to scalar mesh<br />
|-<br />
|<code>winterp</code><br />
|w component of wind interpolated to scalar mesh<br />
|-<br />
|<code>hwin_sr</code><br />
|storm relative horizontal wind speed<br />
|-<br />
|<code>hwin_gr</code><br />
|ground relative horizontal wind speed<br />
|-<br />
|<code>xvort</code><br />
|x component of vorticity<br />
|-<br />
|<code>yvort</code><br />
|y component of vorticity<br />
|-<br />
|<code>zvort</code><br />
|z component of vorticity<br />
|-<br />
|<code>hvort</code><br />
|horizonal vorticity vector magnitude<br />
|-<br />
|<code>vortmag</code><br />
|3D vorticity vector magnitude<br />
|-<br />
|<code>hdiv</code><br />
|horizontal divergence (du/dx + dv/dy)<br />
|}<br />
<br />
<br />
<br />
== hdf2nc Example ==<br />
<br />
<pre><br />
h2ologin2:~/project-bagm/brainstorm2017/15m/history.fs8-15m-a% hdf2nc --histpath=3D --base=frob --time=7101 --x0=2300 --y0=2300 --x1=2400 --y1=2400 --z1=10 uinterp winterp dbz<br />
histpath = 3D<br />
ncbase = frob<br />
time = 7101.000000<br />
X0 = 2300<br />
Y0 = 2300<br />
X1 = 2400<br />
Y1 = 2400<br />
Z1 = 10<br />
Setting Z0 to default value of 0<br />
Read cached num_time_dirs of 146<br />
ntimedirs: 146<br />
Read cached sorted time dirs<br />
Read cached num node dirs<br />
Read cached nodedir<br />
Read cached firstfilename and all times<br />
<br />
We are requesting the following fields: uinterp winterp dbz <br />
<br />
Working on surface 2D thrhopert and dbz (<br />
acca<br />
czza<br />
czza<br />
aaaa<br />
<br />
acca<br />
czza<br />
czza<br />
aaaa<br />
)<br />
Working on uinterp (<br />
acca<br />
czza<br />
czza<br />
aaaa<br />
)<br />
Working on winterp (<br />
acca<br />
czza<br />
czza<br />
aaaa<br />
)<br />
Working on dbz (<br />
acca<br />
czza<br />
czza<br />
aaaa<br />
)<br />
h2ologin2:~/project-bagm/brainstorm2017/15m/history.fs8-15m-a% lt<br />
total 168656<br />
drwxr-xr-x 10 orf PRAC_bagm 4096 Jul 31 11:12 ./<br />
-rw-r--r-- 1 orf PRAC_bagm 1476356 Jul 31 11:12 frob.07101.000000.nc<br />
-rw-r--r-- 1 orf PRAC_bagm 116 Jul 31 11:12 frob.07101.000000.nc.cmd<br />
</pre><br />
<br />
=== Discussion ===<br />
<br />
If no array index options are provided to the command line, hdf2nc will attempt to convert the entire model domain to a netCDF file. In other words, X0,Y0,Z0,X1,Y1,Z1 are optional arguments to hdf2nc. However for typical use cases with large amounts of data, you will want to convert only a subset of the full model domain!<br />
<br />
The output of hdf2nc includes information on some basic metadata, plus some output that tracks the reading of the individual hdf5 files that comprise LOFS. Each letter that comprises the output that looks like<br />
<pre><br />
acca<br />
czza<br />
czza<br />
aaaa<br />
</pre><br />
represents the successful reading of data from a single hdf5 file. It's kind of a 'base 26' representation of the percentage of data (in the horizontal) requested from each file. If a <code>z</code> is printed, that means the full horizontal range of data was requested. If <code>a</code> is printed, a tiny piece of the horizontal range was selected. All intermediate letters represent the space between these two extremes. This output is for your entertainment only; you are essentially watching the assembly of the netCDF file from LOFS data in real time.<br />
<br />
hdf2nc always produces a 2D surface plot of density potential temperature perturbation from base state (proportional to buoyancy) and surface (calculated) radar reflectivity. These fields are used so much that they are always written to the netCDF files whether they are requested or not.<br />
<br />
Regarding the mention of cached data, the LOFS read routines will look for existing cache files before going out and getting all the metadata from hdf5 files, which, for large amounts of data, is very expensive. Since the data layout never changes (unlesss you change it) the cached files speed things up quite a bit. If you ever change your LOFS data (say, adding new time directories), you must remove the cache files and let LOFS regenerate them so they will contain the new information. Cache files all are prefixed by <code>.cm1hdf5_</code> and can always be removed, as they will always be regenerated.<br />
<br />
In this example, the output file name is <code>frob.07101.000000.nc</code>, indicating data that was retrieved at <code>t=07101.000000</code> seconds. Note that LOFS allows for the saving and retrieval of data saved in intervals of less than one second, as time is represented as a floating point variable.<br />
<br />
=== Rationale ===<br />
<br />
LOFS splits the model domain and times into files spread across hundreds directories in large simulations. Often times you may wish to analyze, plot, or visualize a subset of the full model domain at a given time, perhaps to make plots or to feed into visualization software that understands the netCDF format which is one of the most commonly used data formats used in atmospheric science.</div>Admin-lofs-wikihttp://lofs.io/index.php?title=Main_Page&diff=36Main Page2022-02-03T18:13:49Z<p>Admin-lofs-wiki: /* What it is */</p>
<hr />
<div>== What it is ==<br />
<br />
LOFS (short for "Lack Of a File System" or "Leigh Orf File System") is (a) a methodology for organizing and writing numerical model output on distributed memory supercomputers (b) a set of tools for reading back the data for post-processing, visualization and analysis. LOFS is not a traditional file system like ext4, xfs, btrfs, NTFS, etc; LOFS sits atop a "real" file system and is entirely file based, hence the "lack of" a file system. LOFS was designed specifically for distributed memory supercomputer simulations where thousands to millions of MPI ranks must participate in I/O, and where I/O occurs frequently. The format of the files that comprise LOFS is [http://hdfgroup.org HDF5]. The HDF5 files that comprise LOFS can contain multiple time levels per file and be organized across many directories.<br />
<br />
LOFS source code can be found on Github at https://github.com/leighorf.<br />
<br />
On the write side, LOFS is comprised of Fortran 95 code that supplements the CM1 cloud model, but could be applied to other models utilizing MPI. LOFS utilizes the core HDF5 driver where HDF5 files are buffered to memory before being flushed to disk. For typical mappings of large runs to machines like the Blue Waters supercomputer, the CM1 model only comprises 4-5% of available memory on each node, which allows a lot of headroom for buffering data to memory. 50 to 100 time levels are typically buffered to memory before being flushed to disk.<br />
<br />
On the read side, a set of tools written in C and C++ provides an interface into the LOFS format that does not require any manual "stitching together" of files or any foreknowledge of the specific file layout. This is accomplished using low-level I/O calls that traverse the directory tree until any HDF5 file is found, after which metadata is extracted that reveals the full structure of the saved data, enabling data retrieval. The user specifies a variable name, model time, and array indices, relative to the full model domain, in each direction (i, j, and k), and a floating point buffer is returned. LOFS operations are serial - no "parallel I/O" options are used with HDF5. LOFS performs very well when one executes multiple serial LOFS reads on a multicore machine with a parallel file system such as Lustre. A single LOFS process may read steadily at, say, 20 MB/s, but on a modern parallel file system, a second LOFS read will also achieve 20 MB/s, such that reads scale linearly and one can easily saturate the I/O infrastructure of a modern computer. Hence, operations that require the same analysis applied concurrently to many model times perform quite efficiently.<br />
<br />
I developed LOFS for the specific purpose of being able to save large amounts of 3D floating data frequently for post-hoc analysis/visualization in thunderstorm simulations. LOFS has been used to save, analyze, and visualize extremely high resolution thunderstorm data as frequently as every model time step (see for example https://youtu.be/jcBzk3NJkGw). An istotropic supercell simulation spanning 1/4 trillion grid zones (ten meter domain-wide isotropic grid spacing) was being saved with LOFS, with I/O only taking up between a third and a half of total wallclock time with data being saved every 1/5 second. One of the ways this performance is achieved is through the use of lossy floating point compression, in this case, ZFP. A recent publication that goes into more detail about the usage of LOFS in a quarter trillion grid zone supercell thunderstorm simulation can be found here: https://doi.org/10.3390/atmos10100578. A link to a recent AGU poster presentation focusing on the use of ZFP compression in LOFS can be found here: http://orf.media/wp-content/uploads/2017/12/agu2017-orf.pdf.<br />
<br />
== Background ==<br />
<br />
Numerical models periodically output data to disk where it can be visualized and analyzed. Since the turn of the century, supercomputing hardware has transitioned from large shared memory machines to even larger distributed memory architectures with globally accessible file systems (such as Lustre or GPFS). Each compute node, itself made up of dozens to hundreds of compute cores, has direct write access to the underlying file system. It is no longer feasible (nor desirable) to save output data to a single file in very large simulations: Writing this way is inefficient, and extremely large output files are difficult to manage and archive. However, having each MPI rank save its own file during each save cycle will results in millions of files, each of which contains only a fragment of the full model domain. LOFS is a way of writing, organizing, and post-processing model data that lies between these two extremes. One file per compute node is written, and each file can contain dozens of time levels. This dramatically reduces the number of files written to disk. LOFS exploits the HDF5 core driver which allows HDF file to be grown in memory (buffered) before being written to disk. The use of the core driver reduces the number of times the underlying file system is accessed, reducing latency associated with many frequent writes to disk. Files are spread among many directories avoiding performance issues associated with holding tens of thousands or more files in a single directory.<br />
<br />
LOFS is essentially a form of "poor man's parallel IO", or using more modern terminology, it utilizes Multiple Independent Files (MIF), as opposed to Single Shared Files (SSF) and includes tools that allow access to the underlying model data. In summary, LOFS utilizes the MIF file structure but has a API that enables SSF operations. For more on the advantages and disadvantages of MIF as opposed to SSF, see this [https://www.hdfgroup.org/2017/03/mif-parallel-io-with-hdf5/ very informative blog posting] by Mark Miller at Lawrence Livermore.<br />
<br />
== Overview == <br />
<br />
At the time of this writing, the [http://www2.mmm.ucar.edu/people/bryan/cm1 CM1 model] offers several output formats including netCDF, a widely utilized data format in the atmospheric sciences. However, the netCDF I/O options for CM1 either do not scale well to many (tens of thousands or more) MPI ranks on supercomputing hardware, or scale well but produce too many files, with each file containing a small fragment of the full model domain. The latter situation makes post-processing and analysis very challenging.<br />
<br />
At this time, LOFS has been used only with CM1, but it could be applied to any model that utilizes a 2D (or with modifications, a 3D) domain decomposition strategy. One of the features of LOFS is the use of redundant metadata, with each HDF5 file containing enough information to reconstruct the layout of the entire file system.<br />
<br />
LOFS is designed specifically for large supercomputer simulations. Goals of the approach are to:<br />
<br />
* reduce the number of files written to a reasonable number<br />
* minimize the number of times you are doing actual I/O to the underlying file system<br />
* write big files (but not too big)<br />
* make it easy for users to read in data for analysis and visualization after it is written<br />
<br />
For simulations where under a few hundred MPI ranks are used, no significant benefits are found from using LOFS, and users are encouraged to use one of the existing CM1 output options.<br />
<br />
== Components of LOFS: Writing ==<br />
<br />
LOFS consists of modifications and additions to the CM1 model for writing data as well as tools written in C and C++ for reading data back and converting it to other formats such as netCDF. In CM1, LOFS completely replaces the I/O driver options of CM1. It consists of Fortran95 code that:<br />
<br />
* Creates a new MPI communicator containing reordered MPI ranks where each compute node contains a continuous chunk of the physical model domain<br />
* Creates directories and subdirectories on the fly to contain the HDF5 data<br />
* Applies a strict file and directory naming system wherein metadata can be extracted from the file and directory names themselves<br />
* Collects data from individual compute cores on a node and assembles them into 3D arrays that are stored in each HDF5 file<br />
* Optionally applies data compression utilizing existing HDF5 options, including lossless gzip and lossy ZFP<br />
<br />
Prior to model execution, the user must specify<br />
<br />
* the number of files to contain per directory<br />
* the number of times to buffer to an individual HDF5 file<br />
* the number of cores spanning each shared memory node in both dimensions (x and y) of the 2D decomposition (for example, on the Blue Waters supercomputer, which contains 16 floating point cores per node, corex=4 and corey=4)<br />
<br />
LOFS contains subroutines that handle the writing of both metadata and data, as well as the logic associated with assembling and flushing the data to disk. ''LOFS utilizes serial HDF, not parallel HDF, in its operations.''<br />
<br />
== Components of LOFS: Reading ==<br />
<br />
One of the problems of "poor man's parallel I/O" is that model data is spread across many files, fragmenting the physical model domain. While it is relatively straightforward to write code that assembles all the files into a single file, this is slow and results in 2x the data you started with.<br />
<br />
LOFS was designed from the start to not just spread data out into multiple files, but to provide an application programmer interface (API) into the data after it is written that requires no knowledge of the underlying filesystem. The API interface requires the user to specify a variable name, model time, and the beginning and end array index values in all three dimensions (x, y, and z). These indices are relative to the ''full model domain''. For instance, consider a simulation which spans 2200 points in x, 2200 points in y, and 380 points in z, spread over 10,000 MPI ranks with 16 MPI ranks per node. The user wishes to explore a 3D chunk of the domain in the the vicinity of the center of the model domain where the modeled storm is centered. This region spans (x0,y0,z0) to (x1,y1,z1) where x0=1000, x1=1200, y0=1000, y1=1200, z0=0, and z1=200. The user specifies these indexes along with the variable, and the LOFS code goes out and gets the 3D data from the proper files, assembles the data into an array, and sends back a pointer to the buffer that contains the requested chunk of data that can now be analyzed, visualized, etc. For retrieving 2D slices of data, the user may specify the start and end indices to be the same; for instance if one wished to view surface data, one would specify z0=z1=0.<br />
<br />
The primary tool for converting LOFS data is called '''hdf2nc'''. CF compliant netCDF files are output from the hdf2nc command. Such files can be read directly by popular visualization programs such as VisIt, Vapor, and Paraview.<br />
<br />
=== Conversion from LOFS to the netCDF format: [[hdf2nc]] ===</div>Admin-lofs-wikihttp://lofs.io/index.php?title=Main_Page&diff=35Main Page2021-11-26T15:50:53Z<p>Admin-lofs-wiki: </p>
<hr />
<div>== What it is ==<br />
<br />
LOFS (short for "Lack Of a File System" or "Leigh Orf File System") is (a) a methodology for organizing and writing numerical model output on distributed memory supercomputers (b) a set of tools for reading back the data for post-processing, visualization and analysis. LOFS is not a traditional file system like ext4, xfs, btrfs, NTFS, etc; LOFS sits atop a "real" file system and is entirely file based, hence the "lack of" a file system. LOFS was designed specifically for distributed memory supercomputer simulations where thousands to millions of MPI ranks must participate in I/O, and where I/O occurs frequently. The format of the files that comprise LOFS files is HDF5. Each HDF5 files that makes up LOFS can contain multiple time levels per file and be organized across many directories.<br />
<br />
LOFS source code can be found on Github at https://github.com/leighorf.<br />
<br />
On the write side, LOFS is comprised of Fortran 95 code that supplements the CM1 cloud model, but could be applied to other models utilizing MPI. LOFS utilizes the core HDF5 driver where HDF5 files are buffered to memory before being flushed to disk. For typical mappings of large runs to machines like the Blue Waters supercomputer, the CM1 model only comprises 4-5% of available memory on each node, which allows a lot of headroom for buffering data to memory. 50 to 100 time levels are typically buffered to memory before being flushed to disk.<br />
<br />
On the read side, a set of tools written in C and C++ provides an interface into the LOFS format that does not require any manual "stitching together" of files or any foreknowledge of the specific file layout. This is accomplished using low-level I/O calls that traverse the directory tree until any HDF5 file is found, after which metadata is extracted that reveals the full structure of the saved data, enabling data retrieval. The user specifies a variable name, model time, and array indices, relative to the full model domain, in each direction (i, j, and k), and a floating point buffer is returned. LOFS operations are serial - no "parallel I/O" options are used with HDF5. LOFS performs very well when one executes multiple serial LOFS reads on a multicore machine with a parallel file system such as Lustre. A single LOFS process may read steadily at, say, 20 MB/s, but on a modern parallel file system, a second LOFS read will also achieve 20 MB/s, such that reads scale linearly and one can easily saturate the I/O infrastructure of a modern computer. Hence, operations that require the same analysis applied concurrently to many model times perform quite efficiently.<br />
<br />
I developed LOFS for the specific purpose of being able to save large amounts of 3D floating data frequently for post-hoc analysis/visualization in thunderstorm simulations. LOFS has been used to save, analyze, and visualize extremely high resolution thunderstorm data as frequently as every model time step (see for example https://youtu.be/jcBzk3NJkGw). An istotropic supercell simulation spanning 1/4 trillion grid zones (ten meter domain-wide isotropic grid spacing) was being saved with LOFS, with I/O only taking up between a third and a half of total wallclock time with data being saved every 1/5 second. One of the ways this performance is achieved is through the use of lossy floating point compression, in this case, ZFP. A recent publication that goes into more detail about the usage of LOFS in a quarter trillion grid zone supercell thunderstorm simulation can be found here: https://doi.org/10.3390/atmos10100578. A link to a recent AGU poster presentation focusing on the use of ZFP compression in LOFS can be found here: http://orf.media/wp-content/uploads/2017/12/agu2017-orf.pdf.<br />
<br />
== Background ==<br />
<br />
Numerical models periodically output data to disk where it can be visualized and analyzed. Since the turn of the century, supercomputing hardware has transitioned from large shared memory machines to even larger distributed memory architectures with globally accessible file systems (such as Lustre or GPFS). Each compute node, itself made up of dozens to hundreds of compute cores, has direct write access to the underlying file system. It is no longer feasible (nor desirable) to save output data to a single file in very large simulations: Writing this way is inefficient, and extremely large output files are difficult to manage and archive. However, having each MPI rank save its own file during each save cycle will results in millions of files, each of which contains only a fragment of the full model domain. LOFS is a way of writing, organizing, and post-processing model data that lies between these two extremes. One file per compute node is written, and each file can contain dozens of time levels. This dramatically reduces the number of files written to disk. LOFS exploits the HDF5 core driver which allows HDF file to be grown in memory (buffered) before being written to disk. The use of the core driver reduces the number of times the underlying file system is accessed, reducing latency associated with many frequent writes to disk. Files are spread among many directories avoiding performance issues associated with holding tens of thousands or more files in a single directory.<br />
<br />
LOFS is essentially a form of "poor man's parallel IO", or using more modern terminology, it utilizes Multiple Independent Files (MIF), as opposed to Single Shared Files (SSF) and includes tools that allow access to the underlying model data. In summary, LOFS utilizes the MIF file structure but has a API that enables SSF operations. For more on the advantages and disadvantages of MIF as opposed to SSF, see this [https://www.hdfgroup.org/2017/03/mif-parallel-io-with-hdf5/ very informative blog posting] by Mark Miller at Lawrence Livermore.<br />
<br />
== Overview == <br />
<br />
At the time of this writing, the [http://www2.mmm.ucar.edu/people/bryan/cm1 CM1 model] offers several output formats including netCDF, a widely utilized data format in the atmospheric sciences. However, the netCDF I/O options for CM1 either do not scale well to many (tens of thousands or more) MPI ranks on supercomputing hardware, or scale well but produce too many files, with each file containing a small fragment of the full model domain. The latter situation makes post-processing and analysis very challenging.<br />
<br />
At this time, LOFS has been used only with CM1, but it could be applied to any model that utilizes a 2D (or with modifications, a 3D) domain decomposition strategy. One of the features of LOFS is the use of redundant metadata, with each HDF5 file containing enough information to reconstruct the layout of the entire file system.<br />
<br />
LOFS is designed specifically for large supercomputer simulations. Goals of the approach are to:<br />
<br />
* reduce the number of files written to a reasonable number<br />
* minimize the number of times you are doing actual I/O to the underlying file system<br />
* write big files (but not too big)<br />
* make it easy for users to read in data for analysis and visualization after it is written<br />
<br />
For simulations where under a few hundred MPI ranks are used, no significant benefits are found from using LOFS, and users are encouraged to use one of the existing CM1 output options.<br />
<br />
== Components of LOFS: Writing ==<br />
<br />
LOFS consists of modifications and additions to the CM1 model for writing data as well as tools written in C and C++ for reading data back and converting it to other formats such as netCDF. In CM1, LOFS completely replaces the I/O driver options of CM1. It consists of Fortran95 code that:<br />
<br />
* Creates a new MPI communicator containing reordered MPI ranks where each compute node contains a continuous chunk of the physical model domain<br />
* Creates directories and subdirectories on the fly to contain the HDF5 data<br />
* Applies a strict file and directory naming system wherein metadata can be extracted from the file and directory names themselves<br />
* Collects data from individual compute cores on a node and assembles them into 3D arrays that are stored in each HDF5 file<br />
* Optionally applies data compression utilizing existing HDF5 options, including lossless gzip and lossy ZFP<br />
<br />
Prior to model execution, the user must specify<br />
<br />
* the number of files to contain per directory<br />
* the number of times to buffer to an individual HDF5 file<br />
* the number of cores spanning each shared memory node in both dimensions (x and y) of the 2D decomposition (for example, on the Blue Waters supercomputer, which contains 16 floating point cores per node, corex=4 and corey=4)<br />
<br />
LOFS contains subroutines that handle the writing of both metadata and data, as well as the logic associated with assembling and flushing the data to disk. ''LOFS utilizes serial HDF, not parallel HDF, in its operations.''<br />
<br />
== Components of LOFS: Reading ==<br />
<br />
One of the problems of "poor man's parallel I/O" is that model data is spread across many files, fragmenting the physical model domain. While it is relatively straightforward to write code that assembles all the files into a single file, this is slow and results in 2x the data you started with.<br />
<br />
LOFS was designed from the start to not just spread data out into multiple files, but to provide an application programmer interface (API) into the data after it is written that requires no knowledge of the underlying filesystem. The API interface requires the user to specify a variable name, model time, and the beginning and end array index values in all three dimensions (x, y, and z). These indices are relative to the ''full model domain''. For instance, consider a simulation which spans 2200 points in x, 2200 points in y, and 380 points in z, spread over 10,000 MPI ranks with 16 MPI ranks per node. The user wishes to explore a 3D chunk of the domain in the the vicinity of the center of the model domain where the modeled storm is centered. This region spans (x0,y0,z0) to (x1,y1,z1) where x0=1000, x1=1200, y0=1000, y1=1200, z0=0, and z1=200. The user specifies these indexes along with the variable, and the LOFS code goes out and gets the 3D data from the proper files, assembles the data into an array, and sends back a pointer to the buffer that contains the requested chunk of data that can now be analyzed, visualized, etc. For retrieving 2D slices of data, the user may specify the start and end indices to be the same; for instance if one wished to view surface data, one would specify z0=z1=0.<br />
<br />
The primary tool for converting LOFS data is called '''hdf2nc'''. CF compliant netCDF files are output from the hdf2nc command. Such files can be read directly by popular visualization programs such as VisIt, Vapor, and Paraview.<br />
<br />
=== Conversion from LOFS to the netCDF format: [[hdf2nc]] ===</div>Admin-lofs-wikihttp://lofs.io/index.php?title=Main_Page&diff=34Main Page2021-11-26T15:50:19Z<p>Admin-lofs-wiki: </p>
<hr />
<div>== What it is ==<br />
<br />
LOFS (short for "Lack Of a File System" or "Leigh Orf File System") is (a) a methodology for organizing and writing numerical model output on distributed memory supercomputers (b) a set of tools for reading back the data for post-processing, visualization and analysis. LOFS is not a traditional file system like ext4, xfs, btrfs, NTFS, etc; LOFS sits atop a "real" file system and is entirely file based, hence the "lack of" a file system." LOFS was designed specifically for distributed memory supercomputer simulations where thousands to millions of MPI ranks must participate in I/O, and where I/O occurs frequently. The format of the files that comprise LOFS files is HDF5. Each HDF5 files that makes up LOFS can contain multiple time levels per file and be organized across many directories.<br />
<br />
LOFS source code can be found on Github at https://github.com/leighorf.<br />
<br />
On the write side, LOFS is comprised of Fortran 95 code that supplements the CM1 cloud model, but could be applied to other models utilizing MPI. LOFS utilizes the core HDF5 driver where HDF5 files are buffered to memory before being flushed to disk. For typical mappings of large runs to machines like the Blue Waters supercomputer, the CM1 model only comprises 4-5% of available memory on each node, which allows a lot of headroom for buffering data to memory. 50 to 100 time levels are typically buffered to memory before being flushed to disk.<br />
<br />
On the read side, a set of tools written in C and C++ provides an interface into the LOFS format that does not require any manual "stitching together" of files or any foreknowledge of the specific file layout. This is accomplished using low-level I/O calls that traverse the directory tree until any HDF5 file is found, after which metadata is extracted that reveals the full structure of the saved data, enabling data retrieval. The user specifies a variable name, model time, and array indices, relative to the full model domain, in each direction (i, j, and k), and a floating point buffer is returned. LOFS operations are serial - no "parallel I/O" options are used with HDF5. LOFS performs very well when one executes multiple serial LOFS reads on a multicore machine with a parallel file system such as Lustre. A single LOFS process may read steadily at, say, 20 MB/s, but on a modern parallel file system, a second LOFS read will also achieve 20 MB/s, such that reads scale linearly and one can easily saturate the I/O infrastructure of a modern computer. Hence, operations that require the same analysis applied concurrently to many model times perform quite efficiently.<br />
<br />
I developed LOFS for the specific purpose of being able to save large amounts of 3D floating data frequently for post-hoc analysis/visualization in thunderstorm simulations. LOFS has been used to save, analyze, and visualize extremely high resolution thunderstorm data as frequently as every model time step (see for example https://youtu.be/jcBzk3NJkGw). An istotropic supercell simulation spanning 1/4 trillion grid zones (ten meter domain-wide isotropic grid spacing) was being saved with LOFS, with I/O only taking up between a third and a half of total wallclock time with data being saved every 1/5 second. One of the ways this performance is achieved is through the use of lossy floating point compression, in this case, ZFP. A recent publication that goes into more detail about the usage of LOFS in a quarter trillion grid zone supercell thunderstorm simulation can be found here: https://doi.org/10.3390/atmos10100578. A link to a recent AGU poster presentation focusing on the use of ZFP compression in LOFS can be found here: http://orf.media/wp-content/uploads/2017/12/agu2017-orf.pdf.<br />
<br />
== Background ==<br />
<br />
Numerical models periodically output data to disk where it can be visualized and analyzed. Since the turn of the century, supercomputing hardware has transitioned from large shared memory machines to even larger distributed memory architectures with globally accessible file systems (such as Lustre or GPFS). Each compute node, itself made up of dozens to hundreds of compute cores, has direct write access to the underlying file system. It is no longer feasible (nor desirable) to save output data to a single file in very large simulations: Writing this way is inefficient, and extremely large output files are difficult to manage and archive. However, having each MPI rank save its own file during each save cycle will results in millions of files, each of which contains only a fragment of the full model domain. LOFS is a way of writing, organizing, and post-processing model data that lies between these two extremes. One file per compute node is written, and each file can contain dozens of time levels. This dramatically reduces the number of files written to disk. LOFS exploits the HDF5 core driver which allows HDF file to be grown in memory (buffered) before being written to disk. The use of the core driver reduces the number of times the underlying file system is accessed, reducing latency associated with many frequent writes to disk. Files are spread among many directories avoiding performance issues associated with holding tens of thousands or more files in a single directory.<br />
<br />
LOFS is essentially a form of "poor man's parallel IO", or using more modern terminology, it utilizes Multiple Independent Files (MIF), as opposed to Single Shared Files (SSF) and includes tools that allow access to the underlying model data. In summary, LOFS utilizes the MIF file structure but has a API that enables SSF operations. For more on the advantages and disadvantages of MIF as opposed to SSF, see this [https://www.hdfgroup.org/2017/03/mif-parallel-io-with-hdf5/ very informative blog posting] by Mark Miller at Lawrence Livermore.<br />
<br />
== Overview == <br />
<br />
At the time of this writing, the [http://www2.mmm.ucar.edu/people/bryan/cm1 CM1 model] offers several output formats including netCDF, a widely utilized data format in the atmospheric sciences. However, the netCDF I/O options for CM1 either do not scale well to many (tens of thousands or more) MPI ranks on supercomputing hardware, or scale well but produce too many files, with each file containing a small fragment of the full model domain. The latter situation makes post-processing and analysis very challenging.<br />
<br />
At this time, LOFS has been used only with CM1, but it could be applied to any model that utilizes a 2D (or with modifications, a 3D) domain decomposition strategy. One of the features of LOFS is the use of redundant metadata, with each HDF5 file containing enough information to reconstruct the layout of the entire file system.<br />
<br />
LOFS is designed specifically for large supercomputer simulations. Goals of the approach are to:<br />
<br />
* reduce the number of files written to a reasonable number<br />
* minimize the number of times you are doing actual I/O to the underlying file system<br />
* write big files (but not too big)<br />
* make it easy for users to read in data for analysis and visualization after it is written<br />
<br />
For simulations where under a few hundred MPI ranks are used, no significant benefits are found from using LOFS, and users are encouraged to use one of the existing CM1 output options.<br />
<br />
== Components of LOFS: Writing ==<br />
<br />
LOFS consists of modifications and additions to the CM1 model for writing data as well as tools written in C and C++ for reading data back and converting it to other formats such as netCDF. In CM1, LOFS completely replaces the I/O driver options of CM1. It consists of Fortran95 code that:<br />
<br />
* Creates a new MPI communicator containing reordered MPI ranks where each compute node contains a continuous chunk of the physical model domain<br />
* Creates directories and subdirectories on the fly to contain the HDF5 data<br />
* Applies a strict file and directory naming system wherein metadata can be extracted from the file and directory names themselves<br />
* Collects data from individual compute cores on a node and assembles them into 3D arrays that are stored in each HDF5 file<br />
* Optionally applies data compression utilizing existing HDF5 options, including lossless gzip and lossy ZFP<br />
<br />
Prior to model execution, the user must specify<br />
<br />
* the number of files to contain per directory<br />
* the number of times to buffer to an individual HDF5 file<br />
* the number of cores spanning each shared memory node in both dimensions (x and y) of the 2D decomposition (for example, on the Blue Waters supercomputer, which contains 16 floating point cores per node, corex=4 and corey=4)<br />
<br />
LOFS contains subroutines that handle the writing of both metadata and data, as well as the logic associated with assembling and flushing the data to disk. ''LOFS utilizes serial HDF, not parallel HDF, in its operations.''<br />
<br />
== Components of LOFS: Reading ==<br />
<br />
One of the problems of "poor man's parallel I/O" is that model data is spread across many files, fragmenting the physical model domain. While it is relatively straightforward to write code that assembles all the files into a single file, this is slow and results in 2x the data you started with.<br />
<br />
LOFS was designed from the start to not just spread data out into multiple files, but to provide an application programmer interface (API) into the data after it is written that requires no knowledge of the underlying filesystem. The API interface requires the user to specify a variable name, model time, and the beginning and end array index values in all three dimensions (x, y, and z). These indices are relative to the ''full model domain''. For instance, consider a simulation which spans 2200 points in x, 2200 points in y, and 380 points in z, spread over 10,000 MPI ranks with 16 MPI ranks per node. The user wishes to explore a 3D chunk of the domain in the the vicinity of the center of the model domain where the modeled storm is centered. This region spans (x0,y0,z0) to (x1,y1,z1) where x0=1000, x1=1200, y0=1000, y1=1200, z0=0, and z1=200. The user specifies these indexes along with the variable, and the LOFS code goes out and gets the 3D data from the proper files, assembles the data into an array, and sends back a pointer to the buffer that contains the requested chunk of data that can now be analyzed, visualized, etc. For retrieving 2D slices of data, the user may specify the start and end indices to be the same; for instance if one wished to view surface data, one would specify z0=z1=0.<br />
<br />
The primary tool for converting LOFS data is called '''hdf2nc'''. CF compliant netCDF files are output from the hdf2nc command. Such files can be read directly by popular visualization programs such as VisIt, Vapor, and Paraview.<br />
<br />
=== Conversion from LOFS to the netCDF format: [[hdf2nc]] ===</div>Admin-lofs-wikihttp://lofs.io/index.php?title=Main_Page&diff=33Main Page2020-05-11T18:38:55Z<p>Admin-lofs-wiki: /* What it is */</p>
<hr />
<div>== What it is ==<br />
<br />
LOFS (short for "Lack Of File System" or "Leigh Orf File System") is (a) a methodology for organizing and writing numerical model output on distributed memory supercomputers (b) a set of tools for reading back the data for post-processing, visualization and analysis. LOFS is not a traditional file system like ext4, xfs, btrfs, NTFS, etc; LOFS sits atop a "real" file system and is entirely file based, hence the "lack of" a file system." LOFS was designed specifically for distributed memory supercomputer simulations where thousands to millions of MPI ranks must participate in I/O, and where I/O occurs frequently. The format of the files that comprise LOFS files is HDF5. Each HDF5 files that makes up LOFS can contain multiple time levels per file and be organized across many directories.<br />
<br />
LOFS source code can be found on Github at https://github.com/leighorf.<br />
<br />
On the write side, LOFS is comprised of Fortran 95 code that supplements the CM1 cloud model, but could be applied to other models utilizing MPI. LOFS utilizes the core HDF5 driver where HDF5 files are buffered to memory before being flushed to disk. For typical mappings of large runs to machines like the Blue Waters supercomputer, the CM1 model only comprises 4-5% of available memory on each node, which allows a lot of headroom for buffering data to memory. 50 to 100 time levels are typically buffered to memory before being flushed to disk.<br />
<br />
On the read side, a set of tools written in C and C++ provides an interface into the LOFS format that does not require any manual "stitching together" of files or any foreknowledge of the specific file layout. This is accomplished using low-level I/O calls that traverse the directory tree until any HDF5 file is found, after which metadata is extracted that reveals the full structure of the saved data, enabling data retrieval. The user specifies a variable name, model time, and array indices, relative to the full model domain, in each direction (i, j, and k), and a floating point buffer is returned. LOFS operations are serial - no "parallel I/O" options are used with HDF5. LOFS performs very well when one executes multiple serial LOFS reads on a multicore machine with a parallel file system such as Lustre. A single LOFS process may read steadily at, say, 20 MB/s, but on a modern parallel file system, a second LOFS read will also achieve 20 MB/s, such that reads scale linearly and one can easily saturate the I/O infrastructure of a modern computer. Hence, operations that require the same analysis applied concurrently to many model times perform quite efficiently.<br />
<br />
I developed LOFS for the specific purpose of being able to save large amounts of 3D floating data frequently for post-hoc analysis/visualization in thunderstorm simulations. LOFS has been used to save, analyze, and visualize extremely high resolution thunderstorm data as frequently as every model time step (see for example https://youtu.be/jcBzk3NJkGw). An istotropic supercell simulation spanning 1/4 trillion grid zones (ten meter domain-wide isotropic grid spacing) was being saved with LOFS, with I/O only taking up between a third and a half of total wallclock time with data being saved every 1/5 second. One of the ways this performance is achieved is through the use of lossy floating point compression, in this case, ZFP. A recent publication that goes into more detail about the usage of LOFS in a quarter trillion grid zone supercell thunderstorm simulation can be found here: https://doi.org/10.3390/atmos10100578. A link to a recent AGU poster presentation focusing on the use of ZFP compression in LOFS can be found here: http://orf.media/wp-content/uploads/2017/12/agu2017-orf.pdf.<br />
<br />
== Background ==<br />
<br />
Numerical models periodically output data to disk where it can be visualized and analyzed. Since the turn of the century, supercomputing hardware has transitioned from large shared memory machines to even larger distributed memory architectures with globally accessible file systems (such as Lustre or GPFS). Each compute node, itself made up of dozens to hundreds of compute cores, has direct write access to the underlying file system. It is no longer feasible (nor desirable) to save output data to a single file in very large simulations: Writing this way is inefficient, and extremely large output files are difficult to manage and archive. However, having each MPI rank save its own file during each save cycle will results in millions of files, each of which contains only a fragment of the full model domain. LOFS is a way of writing, organizing, and post-processing model data that lies between these two extremes. One file per compute node is written, and each file can contain dozens of time levels. This dramatically reduces the number of files written to disk. LOFS exploits the HDF5 core driver which allows HDF file to be grown in memory (buffered) before being written to disk. The use of the core driver reduces the number of times the underlying file system is accessed, reducing latency associated with many frequent writes to disk. Files are spread among many directories avoiding performance issues associated with holding tens of thousands or more files in a single directory.<br />
<br />
LOFS is essentially a form of "poor man's parallel IO", or using more modern terminology, it utilizes Multiple Independent Files (MIF), as opposed to Single Shared Files (SSF) and includes tools that allow access to the underlying model data. In summary, LOFS utilizes the MIF file structure but has a API that enables SSF operations. For more on the advantages and disadvantages of MIF as opposed to SSF, see this [https://www.hdfgroup.org/2017/03/mif-parallel-io-with-hdf5/ very informative blog posting] by Mark Miller at Lawrence Livermore.<br />
<br />
== Overview == <br />
<br />
At the time of this writing, the [http://www2.mmm.ucar.edu/people/bryan/cm1 CM1 model] offers several output formats including netCDF, a widely utilized data format in the atmospheric sciences. However, the netCDF I/O options for CM1 either do not scale well to many (tens of thousands or more) MPI ranks on supercomputing hardware, or scale well but produce too many files, with each file containing a small fragment of the full model domain. The latter situation makes post-processing and analysis very challenging.<br />
<br />
At this time, LOFS has been used only with CM1, but it could be applied to any model that utilizes a 2D (or with modifications, a 3D) domain decomposition strategy. One of the features of LOFS is the use of redundant metadata, with each HDF5 file containing enough information to reconstruct the layout of the entire file system.<br />
<br />
LOFS is designed specifically for large supercomputer simulations. Goals of the approach are to:<br />
<br />
* reduce the number of files written to a reasonable number<br />
* minimize the number of times you are doing actual I/O to the underlying file system<br />
* write big files (but not too big)<br />
* make it easy for users to read in data for analysis and visualization after it is written<br />
<br />
For simulations where under a few hundred MPI ranks are used, no significant benefits are found from using LOFS, and users are encouraged to use one of the existing CM1 output options.<br />
<br />
== Components of LOFS: Writing ==<br />
<br />
LOFS consists of modifications and additions to the CM1 model for writing data as well as tools written in C and C++ for reading data back and converting it to other formats such as netCDF. In CM1, LOFS completely replaces the I/O driver options of CM1. It consists of Fortran95 code that:<br />
<br />
* Creates a new MPI communicator containing reordered MPI ranks where each compute node contains a continuous chunk of the physical model domain<br />
* Creates directories and subdirectories on the fly to contain the HDF5 data<br />
* Applies a strict file and directory naming system wherein metadata can be extracted from the file and directory names themselves<br />
* Collects data from individual compute cores on a node and assembles them into 3D arrays that are stored in each HDF5 file<br />
* Optionally applies data compression utilizing existing HDF5 options, including lossless gzip and lossy ZFP<br />
<br />
Prior to model execution, the user must specify<br />
<br />
* the number of files to contain per directory<br />
* the number of times to buffer to an individual HDF5 file<br />
* the number of cores spanning each shared memory node in both dimensions (x and y) of the 2D decomposition (for example, on the Blue Waters supercomputer, which contains 16 floating point cores per node, corex=4 and corey=4)<br />
<br />
LOFS contains subroutines that handle the writing of both metadata and data, as well as the logic associated with assembling and flushing the data to disk. ''LOFS utilizes serial HDF, not parallel HDF, in its operations.''<br />
<br />
== Components of LOFS: Reading ==<br />
<br />
One of the problems of "poor man's parallel I/O" is that model data is spread across many files, fragmenting the physical model domain. While it is relatively straightforward to write code that assembles all the files into a single file, this is slow and results in 2x the data you started with.<br />
<br />
LOFS was designed from the start to not just spread data out into multiple files, but to provide an application programmer interface (API) into the data after it is written that requires no knowledge of the underlying filesystem. The API interface requires the user to specify a variable name, model time, and the beginning and end array index values in all three dimensions (x, y, and z). These indices are relative to the ''full model domain''. For instance, consider a simulation which spans 2200 points in x, 2200 points in y, and 380 points in z, spread over 10,000 MPI ranks with 16 MPI ranks per node. The user wishes to explore a 3D chunk of the domain in the the vicinity of the center of the model domain where the modeled storm is centered. This region spans (x0,y0,z0) to (x1,y1,z1) where x0=1000, x1=1200, y0=1000, y1=1200, z0=0, and z1=200. The user specifies these indexes along with the variable, and the LOFS code goes out and gets the 3D data from the proper files, assembles the data into an array, and sends back a pointer to the buffer that contains the requested chunk of data that can now be analyzed, visualized, etc. For retrieving 2D slices of data, the user may specify the start and end indices to be the same; for instance if one wished to view surface data, one would specify z0=z1=0.<br />
<br />
The primary tool for converting LOFS data is called '''hdf2nc'''. CF compliant netCDF files are output from the hdf2nc command. Such files can be read directly by popular visualization programs such as VisIt, Vapor, and Paraview.<br />
<br />
=== Conversion from LOFS to the netCDF format: [[hdf2nc]] ===</div>Admin-lofs-wikihttp://lofs.io/index.php?title=Hdf2nc&diff=3Hdf2nc2019-10-06T18:00:22Z<p>Admin-lofs-wiki: First edit</p>
<hr />
<div>Converts any subset of LOFS raw data to a single netCDF file.<br />
<br />
<code><br />
Usage: hdf2nc --histpath=[histpath] --base=[ncbase] --x0=[X0] --y0=[Y0] --x1=[X1] --y1=[Y1] --z0=[Z0] --z1=[Z1] --time=[time] [varname1 ... varnameN] <br />
</code><br />
<br />
<code>histpath</code>: Top level directory that contains the 3D model data<br />
<br />
<code>ncbase</code>: base name of netCDF file<br />
<br />
<code>(X0,Y0,Z0)(X1,Y1,Z1)</code>: This defines the volume (or space) that you wish to convert to netCDF fdata, referenced by integer array indices that span the full model domain, namely (0,0,0) to (nx-1,ny-1,nz-1). Each of these are optional. If none of these are passed to hdf2nc, the full 3D range of data saved will be converted to netCDF. If only some of them are provided, the remainder default to the min/max values from the saved data (the code will extract these values from what has been saved, making no assumptions).<br />
<br />
<code>time</code>: The model time requested in seconds<br />
<br />
<code>varname1...varnameN</code> The list of variables, separated by whitespace, that you wish to convert<br />
<br />
== hdf2nc Example ==<br />
<br />
<pre><br />
h2ologin2:~/project-bagm/brainstorm2017/15m/history.fs8-15m-a% hdf2nc --histpath=3D --base=frob --time=7101 --x0=2300 --y0=2300 --x1=2400 --y1=2400 --z1=10 uinterp winterp dbz<br />
histpath = 3D<br />
ncbase = frob<br />
time = 7101.000000<br />
X0 = 2300<br />
Y0 = 2300<br />
X1 = 2400<br />
Y1 = 2400<br />
Z1 = 10<br />
Setting Z0 to default value of 0<br />
Read cached num_time_dirs of 146<br />
ntimedirs: 146<br />
Read cached sorted time dirs<br />
Read cached num node dirs<br />
Read cached nodedir<br />
Read cached firstfilename and all times<br />
<br />
We are requesting the following fields: uinterp winterp dbz <br />
<br />
Working on surface 2D thrhopert and dbz (<br />
acca<br />
czza<br />
czza<br />
aaaa<br />
<br />
acca<br />
czza<br />
czza<br />
aaaa<br />
)<br />
Working on uinterp (<br />
acca<br />
czza<br />
czza<br />
aaaa<br />
)<br />
Working on winterp (<br />
acca<br />
czza<br />
czza<br />
aaaa<br />
)<br />
Working on dbz (<br />
acca<br />
czza<br />
czza<br />
aaaa<br />
)<br />
h2ologin2:~/project-bagm/brainstorm2017/15m/history.fs8-15m-a% lt<br />
total 168656<br />
drwxr-xr-x 10 orf PRAC_bagm 4096 Jul 31 11:12 ./<br />
-rw-r--r-- 1 orf PRAC_bagm 1476356 Jul 31 11:12 frob.07101.000000.nc<br />
-rw-r--r-- 1 orf PRAC_bagm 116 Jul 31 11:12 frob.07101.000000.nc.cmd<br />
</pre><br />
<br />
=== Discussion ===<br />
<br />
If no array index options are provided to the command line, hdf2nc will attempt to convert the entire model domain to a netCDF file. In other words, X0,Y0,Z0,X1,Y1,Z1 are optional arguments to hdf2nc. However for typical use cases with large amounts of data, you will want to convert only a subset of the full model domain!<br />
<br />
The output of hdf2nc includes information on some basic metadata, plus some output that tracks the reading of the individual hdf5 files that comprise LOFS. Each letter that comprises the output that looks like<br />
<pre><br />
acca<br />
czza<br />
czza<br />
aaaa<br />
</pre><br />
represents the successful reading of data from a single hdf5 file. It's kind of a 'base 26' representation of the percentage of data (in the horizontal) requested from each file. If a <code>z</code> is printed, that means the full horizontal range of data was requested. If <code>a</code> is printed, a tiny piece of the horizontal range was selected. All intermediate letters represent the space between these two extremes. This output is for your entertainment only; you are essentially watching the assembly of the netCDF file from LOFS data in real time.<br />
<br />
hdf2nc always produces a 2D surface plot of density potential temperature perturbation from base state (proportional to buoyancy) and surface (calculated) radar reflectivity. These fields are used so much that they are always written to the netCDF files whether they are requested or not.<br />
<br />
Regarding the mention of cached data, the LOFS read routines will look for existing cache files before going out and getting all the metadata from hdf5 files, which, for large amounts of data, is very expensive. Since the data layout never changes (unlesss you change it) the cached files speed things up quite a bit. If you ever change your LOFS data (say, adding new time directories), you must remove the cache files and let LOFS regenerate them so they will contain the new information. Cache files all are prefixed by <code>.cm1hdf5_</code> and can always be removed, as they will always be regenerated.<br />
<br />
In this example, the output file name is <code>frob.07101.000000.nc</code>, indicating data that was retrieved at <code>t=07101.000000</code> seconds. Note that LOFS allows for the saving and retrieval of data saved in intervals of less than one second, as time is represented as a floating point variable.<br />
<br />
=== Rationale ===<br />
<br />
LOFS splits the model domain and times into files spread across hundreds directories in large simulations. Often times you may wish to analyze, plot, or visualize a subset of the full model domain at a given time, perhaps to make plots or to feed into visualization software that understands the netCDF format which is one of the most commonly used data formats used in atmospheric science.</div>Admin-lofs-wikihttp://lofs.io/index.php?title=Main_Page&diff=2Main Page2019-10-06T17:56:43Z<p>Admin-lofs-wiki: First edit</p>
<hr />
<div>== What it is ==<br />
<br />
LOFS (short for "Leigh Orf File System" or "Lack Of File System" or "Logical Output For Simulations<ref>LOFS is not a traditional file system like ext4, xfs, btrfs, NTFS, etc; LOFS sits atop a "real" file system and is entirely file based, hence the "lack of" a file system.</ref>") is (a) a methodology for organizing and writing numerical model output on distributed memory supercomputers (b) a set of tools for reading back the data for post-processing, visualization and analysis. LOFS was designed specifically for distributed memory supercomputer simulations where thousands to millions of MPI ranks must participate in I/O, and where I/O occurs frequently. The core format of LOFS files is HDF5. The HDF5 files that make up LOFS contain multiple time levels per file and are organized across many directories.<br />
<br />
On the write side, LOFS is comprised of Fortran 95 code that supplements the CM1 cloud model, but could be applied to other models utilizing MPI. LOFS utilizes the core HDF5 driver where HDF5 files are buffered to memory before being flushed to disk. For typical mappings of large runs to machines like the Blue Waters supercomputer, the CM1 model only comprises 4-5% of available memory on each node, which allows a lot of headroom for buffering data to memory. 50 to 100 time levels are typically buffered to memory before being flushed to disk.<br />
<br />
On the read side, a set of tools written in C and C++ provides an interface into the LOFS format that does not require any manual "stitching together" of files or any foreknowledge of the specific file layout. This is accomplished using low-level I/O calls that traverse the directory tree until any HDF5 file is found, after which metadata is extracted that reveals the full structure of the saved data, enabling data retrieval. The user specifies a variable name, model time, and array indices, relative to the full model domain, in each direction (i, j, and k), and a floating point buffer is returned. LOFS operations are serial - no "parallel I/O" options are used with HDF5. LOFS performs very well when one executes multiple serial LOFS reads on a multicore machine with a parallel file system such as Lustre. A single LOFS process may read steadily at, say, 20 MB/s, but on a modern parallel file system, a second LOFS read will also achieve 20 MB/s, such that reads scale linearly and one can easily saturate the I/O infrastructure of a modern computer. Hence, operations that require analysis spanning many model times can be executed efficiently in parallel. Scripts are available for exploiting the "embarrassingly parallel" nature of typical operations.<br />
<br />
I developed LOFS for the specific purpose of being able to save large amounts of volumetric data frequently for post-hoc analysis/visualization in thunderstorm simulations. LOFS has been used to save, analyze, and visualize extremely high resolution thunderstorm data as frequently as every model time step (see for example https://youtu.be/jcBzk3NJkGw). An istotropic supercell simulation spanning 1/4 trillion grid zones (ten meter domain-wide isotropic grid spacing) is being saved with LOFS, with I/O only taking up a small fraction of total wallclock time. One of the ways this performance is achieved is through the use of lossy floating point compression, in this case, ZFP. A link to a recent AGU poster presentation on the use of ZFP compression in LOFS can be found here: http://orf.media/wp-content/uploads/2017/12/agu2017-orf.pdf<br />
<br />
== Background ==<br />
<br />
Numerical models periodically output data to disk where it can be visualized and analyzed. Since the turn of the century, supercomputing hardware has transitioned from large shared memory machines to even larger distributed memory architectures with globally accessible file systems (such as Lustre or GPFS). Each compute node, itself made up of dozens to hundreds of compute cores, has direct write access to the underlying file system. It is no longer feasible (nor desirable) to save output data to a single file in very large simulations: Writing this way is inefficient, and extremely large output files are difficult to manage and archive. However, having each MPI rank save its own file during each save cycle will results in millions of files, each of which contains only a fragment of the full model domain. LOFS is a way of writing, organizing, and post-processing model data that lies between these two extremes. One file per compute node is written, and each file can contain dozens of time levels. This dramatically reduces the number of files written to disk. LOFS exploits the HDF5 core driver which allows HDF file to be grown in memory (buffered) before being written to disk. The use of the core driver reduces the number of times the underlying file system is accessed, reducing latency associated with many frequent writes to disk. Files are spread among many directories avoiding performance issues associated with holding tens of thousands or more files in a single directory.<br />
<br />
LOFS is essentially a form of [https://visitbugs.ornl.gov/projects/hpc-hdf5/wiki/Poor_Man's_vs_Rich_Mans'_Parallel_IO "poor man's parallel IO"], or using more modern terminology, it utilizes Multiple Independent Files (MIF), as opposed to Single Shared Files (SSF) and includes tools that allow access to the underlying model data. In summary, LOFS utilizes the MIF file structure but has a API that enables SSF operations. For more on the advantages and disadvantages of MIF as opposed to SSF, see this [https://www.hdfgroup.org/2017/03/mif-parallel-io-with-hdf5/ very informative blog posting] by Mark Miller at Lawrence Livermore.<br />
<br />
== Overview == <br />
<br />
At the time of this writing, the [http://www2.mmm.ucar.edu/people/bryan/cm1 CM1 model] offers several output formats including netCDF, a widely utilized data format in the atmospheric sciences. However, the netCDF I/O options for CM1 either do not scale well to many (tens of thousands or more) MPI ranks on supercomputing hardware, or scale well but produce too many files, with each file containing a small fragment of the full model domain. The latter situation makes post-processing and analysis very challenging.<br />
<br />
At this time, LOFS has been used only with CM1, but it could be applied to any model that utilizes a 2D (or with modifications, a 3D) domain decomposition strategy. One of the features of LOFS is the use of redundant metadata, with each HDF5 file containing enough information to reconstruct the layout of the entire file system.<br />
<br />
LOFS is designed specifically for large supercomputer simulations. Goals of the approach are to:<br />
<br />
* reduce the number of files written to a reasonable number<br />
* minimize the number of times you are doing actual I/O to the underlying file system<br />
* write big files (but not too big)<br />
* make it easy for users to read in data for analysis and visualization after it is written<br />
<br />
For simulations where under a few hundred MPI ranks are used, no significant benefits are found from using LOFS, and users are encouraged to use one of the existing CM1 output options.<br />
<br />
== Components of LOFS: Writing ==<br />
<br />
LOFS consists of modifications and additions to the CM1 model for writing data as well as tools written in C and C++ for reading data back and converting it to other formats such as netCDF. In CM1, LOFS completely replaces the I/O driver options of CM1. It consists of Fortran95 code that:<br />
<br />
* Creates a new MPI communicator containing reordered MPI ranks where each compute node contains a continuous chunk of the physical model domain<br />
* Creates directories and subdirectories on the fly to contain the HDF5 data<br />
* Applies a strict file and directory naming system wherein metadata can be extracted from the file and directory names themselves<br />
* Collects data from individual compute cores on a node and assembles them into 3D arrays that are stored in each HDF5 file<br />
* Optionally applies data compression utilizing existing HDF5 options, including lossless gzip and lossy ZFP<br />
<br />
Prior to model execution, the user must specify<br />
<br />
* the number of files to contain per directory<br />
* the number of times to buffer to an individual HDF5 file<br />
* the number of cores spanning each shared memory node in both dimensions (x and y) of the 2D decomposition (for example, on the Blue Waters supercomputer, which contains 16 floating point cores per node, corex=4 and corey=4)<br />
<br />
LOFS contains subroutines that handle the writing of both metadata and data, as well as the logic associated with assembling and flushing the data to disk. ''LOFS utilizes serial HDF, not parallel HDF, in its operations.''<br />
<br />
== Components of LOFS: Reading ==<br />
<br />
One of the problems of "poor man's parallel I/O" is that model data is spread across many files, fragmenting the physical model domain. While it is relatively straightforward to write code that assembles all the files into a single file, this is slow and results in 2x the data you started with.<br />
<br />
LOFS was designed from the start to not just spread data out into multiple files, but to provide an application programmer interface (API) into the data after it is written that requires no knowledge of the underlying filesystem. The API interface requires the user to specify a variable name, model time, and the beginning and end array index values in all three dimensions (x, y, and z). These indices are relative to the ''full model domain''. For instance, consider a simulation which spans 2200 points in x, 2200 points in y, and 380 points in z, spread over 10,000 MPI ranks with 16 MPI ranks per node. The user wishes to explore a 3D chunk of the domain in the the vicinity of the center of the model domain where the modeled storm is centered. This region spans (x0,y0,z0) to (x1,y1,z1) where x0=1000, x1=1200, y0=1000, y1=1200, z0=0, and z1=200. The user specifies these indexes along with the variable, and the LOFS code goes out and gets the 3D data from the proper files, assembles the data into an array, and sends back a pointer to the buffer that contains the requested chunk of data that can now be analyzed, visualized, etc. For retrieving 2D slices of data, the user may specify the start and end indices to be the same; for instance if one wished to view surface data, one would specify z0=z1=0.<br />
<br />
The primary tool for converting LOFS data is called '''hdf2nc'''. CF compliant netCDF files are output from the hdf2nc command. Such files can be read directly by popular visualization programs such as VisIt, Vapor, and Paraview.<br />
<br />
=== Conversion from LOFS to the netCDF format: [[hdf2nc]] ===</div>Admin-lofs-wiki