***************************************************************************** * You can find a PDF version of this file from * * http://hdfeos.org/software/aug_eos5/Readme.pdf * * * * Where to find two other referred documents: * * * * 1. Usage document of this tool * * A. PDF Version * * http://hdfeos.org/software/aug_eos5/Usage_Of_Aug_Tool.pdf * * B. Text Version * * usage.txt under the top directory * * * * 2. Validation program document of this tool * * A. PDF Version * * http://hdfeos.org/software/aug_eos5/Validation_Of_Aug_Tool.pdf * * B. Text Version * * validation.txt under the validprog directory * * * ***************************************************************************** 1. Introduction This package contains a program that augments an HDF-EOS5 file so that the netCDF-4 library may access the augmented file by following the netCDF enhanced data model. This program mainly adds HDF5 dimension scales by following the HDF5 dimension scale specification. Therefore, the netCDF-4 library can correctly obtain the dimension information of a variable. Be default, this tool only calculates the latitude and longitude for HDF-EOS5 grids. For other dimensions in an HDF-EOS5 grid object, and all dimensions in an HDF-EOS5 swath object or an HDF-EOS5zonal average object, a user needs to provide missing dimension or dimension scale information by means of a mapping file. Please check the Usage_of_Aug_Tool.pdf or usage.txt for more details about the usage and options of the augmentation tool. We also provide three simple programs for users to validate whether an augmented file can be accessed by the netCDF-4 APIs. Please check the Validation_Of_Aug_Tool.pdf or validation.txt for more details. 2. Limitations We are assuming that all HDF5 datasets in an HDF-EOS5 file can be accessed via the HDF-EOS5 APIs. In other words, the HDF-EOS5 file to be augmented should not contain any HDF5 datasets that are created by directly using the HDF5APIs since such datasets are no longer visible by the HDF-EOS5 APIs. This limitation is caused by the fact that we use the HDF-EOS5 APIs to collect all information. If an HDF-EOS5 file contains an HDF5 dataset that is invisible through the HDF-EOS5 APIs, this program does not add any dimension scales for that dataset, which may cause the netCDF-4 library unable to read the augmented file by following the netCDF-4 enhanced data model. We also assume that HDF-EOS5 object, field and attribute names in an HDF-EOS5 file do not contain any special characters not allowed by the netCDF-4 library. HDF-EOS5 swath may contain dimension maps. The augmentation tool does not support this. An exception will be thrown if dimension maps exist. Please see section 4 in Validation_of_Aug_Tool.pdf or validation.txt for other limitations. 3. Requirements HDF-EOS5 C library (1.12 or higher) HDF5 C library (1.8, prefer 1.8.7 or higher) szlib and zlib libraries (if HDF5 is built with these two libraries) netCDF-4 library (Only required if building the simple validation programs) This augmentation program does not need the netCDF-4 library. The netCDF-4 library is only required if a user wants to test whether an augmented file can be read by the netCDF-4 library. 4. Software Package After uncompressing the package, three directories, src, test, validprog and test will be created under the installation directory. The src directory includes the source code files, Makefile and a makefile template. This template can be used to build the tool in case the configuration script does not work on users' machines. The test directory includes a few shell scripts that can be used to test the tool with real HDF-EOS5 files. The validprog directory includes the source code files and sample HDF-EOS5 files for the simple validation programs. The examples directory holds some sample HDF-EOS5 files and mapping files. The sample HDF-EOS5 files are single_swath_2.he5 and single_zonal_average.he5. The sample mapping files are single_swath_2_mapping_file.txt for single_swath_2.he5 and single_zonal_average_mapping_file.txt for single_zonal_average.he5. Please see Usage_Of_Aug_Tool.pdf or usage.txt for more detailed information about the concept of a mapping file. 5. Installation 5.1 How to Build Augmentation Tool 5.1.1 Use autoconf and automake 5.1.1.1 Configuration 5.1.1.1.1 Use h5cc Script (Recommended) Assume the HDF5 library path is /path/to/hdf5. set CC to be /path/to/hdf5/bin/h5cc. Use setenv for csh and export for bash. Optionally, a user can also build the HDF-EOS5 C library following http://hdfeos.org/software/hdfeos.php#ref_sec:hdf-eos5 After uncompressing the package, executing the configuration script under the top directory with the appropriate path: . /configure --with-hdfeos5=/path/to/hdfeos5 5.1.1.1.2 Use gcc Compiler After uncompressing the package, executing the configuration script with the appropriate path names: . /configure \ --with-zlib=/path/to/zlib \ --with-szlib=/path/to/szlib \ --with-hdf5=/path/to/hdf5 \ --with-hdfeos5=/path/to/hdfeos5 If the HDF5 library is not built with szlib, a user doesn't need to include --with-szlib. Also on most Linux platforms the zlib library is installed in their system path. Under that case, a user doesn't need to include --with-zlib either. The above configuration script will build a dynamically linked executable. Adding LDFLAGS=-static to your configuration script will build a statically linked executable instead. 5.1.1.2 Make and Install After successful configuration, typing the following command: make install This generates the executable file aug_eos5 of the augmentation tool. It is under the /path/to/augmentation/aug_eos5/bin and /path/to/augmentation/src or under the /path/to/installation/bin if you use the prefix option during the configuration. If the configuration script does not work on your machine, a user can also generate the same executable file by executing a makefile template under the /src directory, makefile.augment, as follows: make -f makefile.augment The executable aug_eos5 will be generated under the /src directory. For makefile.augment to work properly, a user should modify the library path based on your own system configuration, such as HDF5_DIR=/path/to /hdf5 HDFEOS5_DIR=/path/to/hdfeos5 ...... Note: For Mac users, ignoring the warnings like: run.c: In function 'gridaugalloc_info': run.c:56: warning: format '%4d' expects type 'int', but argument 3 has type 'size_t' run.c:56: warning: format '%4d' expects type 'int', but argument 4 has type 'size_t' The executable file will work properly in spite of these warnings. 5.2 How to Build Validation Executables Validation executables are used to check the accessibility of an augmented file by the netCDF-4 APIs. They can be built by typing the following command under the validprog directory: make -f makefile.check This generates three executable files, check_c, check_f and check_za_f, which are capable of validating whether the netCDF-4 C or Fortran APIs can access augmented files as expected. For makefile.check to work properly, you should modify library path based on your own system configuration, such as: NCCONFIG=/path/to/netCDF4/bin/nc-config 6. Usage The following is an example on how to use the tool. Please see Usage_Of_Aug_Tool.pdf or usage.txt for more information. Go to the examples directory, at the command prompt, typing the following: aug_eos5 -f single_swath_2_mapping_file.txt single_swath_2.he5 single_swath_2.he5 is an HDF-EOS5 swath file to be augmented. The relationship between the dimension and the corresponding dimension scale is missing from this file. single_swath_2_mapping_file.txt is a mapping file that a user provides to specify this relationship. The layout of single_swath_2_mapping_file.txt is as follows: 0 nTimes Time 0 nLevels Pressure where nTimes and nLevels are dimension names, and Time and Pressure are the data fields in single_swath_2.he5. There is another dimension called nCloudTypes in this file. Since this dimension is not used by any data fields at all, users don't need to provide any dimension information in the mapping file. However, to follow the netCDF enhanced data model, the augmentation tool still adds the dimension scale of nCloudTypes. The values of this dimension scale are filled with zeros. The netCDF-4 will not output the dimension scales at all. It should be mentioned that users should give the write permission to the original file since the augmentation tool writes the augmented information to the original file. To find more details about the concept and usage of a mapping file, please see the Usage_Of_Aug_Tool.pdf or usage.txt. 7. Test 7.1. Single-File Test The purpose of this test is to validate whether an augmented file can be accessed by the netCDF-4 library. The following examples demonstrate how to use the three testing programs, check_c, check_f and check_za_f, to perform the validation. The source code files can be found under the validprog directory. check_c uses the APIS in the netCDF-4 C library. check_f and check_za_f use the APIs in the netCDF-4 Fortran library. By providing command-line parameters, a user can use check_c to validate any data field in an augmented file. In this release, the data field to be validated is hardcoded in check_f and check_za_f .These programs have some limitations. Please see Validation_Of_Aug_Tool.pdf or validation.txt for more details. After building validation programs under the directory validprog, at the command prompt, typing the following command: ./check_c -t 0 -g HIRDLS -v CFC11 single_swath_2.he5 check_c checks the data field CFC11 of the object HIRDLS in single_swath_2.he5. Please check Validation_Of_Aug_Tool.pdf or validation.txt on how to specify the command-line parameters used by check_c. Typing the following command: ./check_f check_f checks the data field CFC11 of the object HIRDLS in single_swath_2.he5. Typing the following command: ./check_za_f check_za_f checks the data field Day of the object HIRDLS in single_zonal_average.he5. We also strongly recommend using netCDF-4's ncdump utility to check the augmented file like: ncdump -h single_swath_2.he5 ncdump -v CFC11 single_swath_2.he5 ncdump -h will dump the header of a netCDF file. The following is part of the header information of single_swath_2.he5 obtained via ncdump. Users should clearly see the correct dimension information assigned to the data fields. group: HIRDLS { dimensions: nCloudTypes = 1 ; nLevels = 145 ; nTimes = 5554 ; variables: float nLevels(nLevels) ; nLevels:_FillValue = -999.f ; nLevels:MissingValue = -999.f ; nLevels:Units = "hPa" ; nLevels:UniqueFieldDefinition = "Aura-Shared" ; float nTimes(nTimes) ; nTimes:MissingValue = -999.f ; nTimes:Units = "deg" ; nTimes:UniqueFieldDefinition = "Aura-Shared" ; nTimes:_FillValue = -999.f ; group: Data\ Fields { variables: float CFC11(nTimes, nLevels) ; CFC11:_FillValue = -999.f ; CFC11:MissingValue = -999.f ; CFC11:Title = "CFC11" ; CFC11:Units = "vmr" ; CFC11:UniqueFieldDefinition = "Aura-Shared" ; } // group Data\ Fields The current NetCDF-4.2 fails to read the fixed-length HDF5 string dataset, so ncdump single_swath_2.he5 may get the segmentation fault. However, variables with other HDF5 data types in an HDF-EOS5 file can still be accessed by the netCDF-4 APIs. Please see section 9 for more details. 7.2 Batch Test (optional) This is an optional sanitized test for users who want to check if this tool works properly with typical NASA HDF-EOS5 files. Generally users do not need to check this step. The test described in section 7.1 should be sufficient. To run this test, the HDF group provides some sample NASA HDF-EOS5 files. This test essentially compares the augmented sample HDF-EOS5 files generated by users with the augmented sample HDF-EOS5 files generated by the HDF group. For proper testing, the directory containing h5diff binary should be included in the system path. Before testing, sample NASA HDF-EOS5 files need to be downloaded. Except for Mac users, this can be done by typing the following command: cd test ./download.sh For Mac users, typing the following command instead: cd test ./download_mac.sh After this step, the following files should be available under /path/to/installation/test/sample_files_for_batch_test/ HIRDLS-Aura_L2_v02-04-09-c03_2008d001.he5 MLS-Aura_L2GP-BrO_v02-23-c01_2010d255.he5 MLS-Aura_L2GP-Temperature_v02-23-c01_2010d058.he5 OMI-Aura_L2-OMBRO_2010m1005t1835-o33108_v003-2010m1006t023445.he5 OMI-Aura_L3-OMAEROe_2010m0830_v003-2010m0902t032205.he5 TES-Aura_L3-O3_r0000011812_C01_F01_07.he5 TES-Aura_L2-SUPPLEMENTAL-SO_r0000011860_C01_F01_07.he5 HIR3ZA-test2.he5 GSSTF.2b.2008.01.01.he5 then typing the following command: ./test.sh The shell script stops on error and details is printed out subsequently if any error occurs, otherwise it means that aug_eos5 works as expected. 8 Tested Products We have tested augmentation tool against Aura HIRDLS, MLS, TES, OMI, and NASA MEaSUREs GSSTF products on 32-bit and 64-bit Linux, as well as 64-bit Mac. 9 Known Problems The current netCDF-4.2 fails to read long fixed-length HDF5 string. So dumping all the contents of an augmented file with ncdump may get "Segmentation Fault". The long fixed-length HDF5 string (such as StructMetadata.0) in an HDF-EOS5 file mainly stores some metadata information essential for the HDF-EOS5 applications. Thus currently we think that the long fixed-length HDF5 string the netCDF-4 APIs fails to read is not important to netCDF-4 users. We already reported this problem to the netCDF-4 team. Hopefully this problem will be fixed in their future release. Starting from this release, the augmentation tool can no longer be compiled with HDF5 1.6 APIs. Users should go back to the previous version if HDF5 1.6 is used to compile this tool.