Read HDF files

[Arnaud Mosnier]

Hi Geo UseRs,

I am searching a way to read and manage HDF files in R (like files provided
by the ocean color web site, http://oceancolor.gsfc.nasa.gov/).

Thanks

Arnaud

        [[alternative HTML version deleted]]

_______________________________________________
R-sig-Geo mailing list
[hidden email]
https://stat.ethz.ch/mailman/listinfo/r-sig-geo

[Alexandre Castagna]

I'm couldn't either.

My strategy was to export geotiffs with IDRISI because the current version
of the program that permits batch processing with the
SeaDas<http://seadas.gsfc.nasa.gov/>(
SeaBatch <http://seabatch.com/>, developed by Mike Brown) doesn't accept L3
input data. But if you are not working with level 3 data, you could use
SeaBacht and SeaDas.

But I really wanted to open directly into R...

--
Alexandre


2011/9/1 Arnaud Mosnier <[hidden email]>

> Hi Geo UseRs,
>
> I am searching a way to read and manage HDF files in R (like files provided
> by the ocean color web site, http://oceancolor.gsfc.nasa.gov/).
>
> Thanks
>
> Arnaud
>
>        [[alternative HTML version deleted]]
>
> _______________________________________________
> R-sig-Geo mailing list
> [hidden email]
> https://stat.ethz.ch/mailman/listinfo/r-sig-geo
>

--
Alexandre Castagna Mourão e Lima
Laboratório de Radioecologia e Mudanças Globais - UERJ
Rua São Francisco Xavier, 524, Maracanã
Pav. Haroldo Lisboa da Cunha, Sub-solo

        [[alternative HTML version deleted]]


_______________________________________________
R-sig-Geo mailing list
[hidden email]
https://stat.ethz.ch/mailman/listinfo/r-sig-geo

[Paul Hiemstra-2]

 On 09/01/2011 05:20 PM, Arnaud Mosnier wrote:

> Hi Geo UseRs,
>
> I am searching a way to read and manage HDF files in R (like files provided
> by the ocean color web site, http://oceancolor.gsfc.nasa.gov/).
>
> Thanks
>
> Arnaud
>
> [[alternative HTML version deleted]]
>
> _______________________________________________
> R-sig-Geo mailing list
> [hidden email]
> https://stat.ethz.ch/mailman/listinfo/r-sig-geo

The answer can be found by googling for "R read hdf". That leads to the
HDF5 package or building GDAL with HDF support. Alternatively, find a
(command)-line tool which can translate HDF5 to geotiff or netCDF (e.g.
gdal_translate or ) and then use readGDAL or the ncdf package.

Paul

--
Paul Hiemstra, Ph.D.
Global Climate Division
Royal Netherlands Meteorological Institute (KNMI)
Wilhelminalaan 10 | 3732 GK | De Bilt | Kamer B 3.39
P.O. Box 201 | 3730 AE | De Bilt
tel: [phone]

http://intamap.geo.uu.nl/~paul
http://nl.linkedin.com/pub/paul-hiemstra/20/30b/770

_______________________________________________
R-sig-Geo mailing list
[hidden email]
https://stat.ethz.ch/mailman/listinfo/r-sig-geo

[janverbesselt]

You can try something like this;

# check out the raster and rgdal packages
library(raster)
library(rgdal)
# have the files in your current wd()
# for MODIS data MOD13C1 I do the following
fl <- list.files(pattern="\\.hdf$", recursive = TRUE) # without directories
GDALinfo(modis) # check out the name using this function
fndvi <- paste("HDF4_EOS:EOS_GRID:",fl[1],":MODIS_Grid_16Day_VI_CMG:CMG 0.05 Deg 16 days NDVI",sep='')

# then extract raster info and data and much more
system.time(data <- readGDAL(fndvi, offset=c(0,0), region.dim=c(1, 1))) # centre is at (0,0)
parameters <- gridparameters(data)
res <- parameters$cellsize[1]
res

good luck.
Jan
http://www.grs.wur.nl/UK/Staff/Jan+Verbesselt/
http://bfast.r-forge.r-project.org/

[Arnaud Mosnier]

Hi Paul,

Thanks for your answer ... in fact I googled for that before asking this R
list.
Oceancolor HDF files are in HDF4 format and provided as two HDF4 format, in
SIM (Standard Mapped Image) or BIN (binned).
I first tried to use the BIN format without success. I use the gdal version
provided with FWtools and gdal_info correctly identify the file as a HDF4
format, but it refuses to convert it to a geotiff.
I finally succeeded using the SIM format.

But, now I am interested in doing that directly into R.
The version of gdal called by the library rgdal do not include support for
HDF4 format.
So my question is now ... can someone give me a step by step instruction
that I can use to build a GDAL version including the support for HDF4 format
(as it is the case in the version compiled with FWtools) ?

Thanks,

Arnaud

2011/9/2 Paul Hiemstra <[hidden email]>

>  On 09/01/2011 05:20 PM, Arnaud Mosnier wrote:
> > Hi Geo UseRs,
> >
> > I am searching a way to read and manage HDF files in R (like files
> provided
> > by the ocean color web site, http://oceancolor.gsfc.nasa.gov/).
> >
> > Thanks
> >
> > Arnaud
> >
> >       [[alternative HTML version deleted]]
> >
> > _______________________________________________
> > R-sig-Geo mailing list
> > [hidden email]
> > https://stat.ethz.ch/mailman/listinfo/r-sig-geo
> The answer can be found by googling for "R read hdf". That leads to the
> HDF5 package or building GDAL with HDF support. Alternatively, find a
> (command)-line tool which can translate HDF5 to geotiff or netCDF (e.g.
> gdal_translate or ) and then use readGDAL or the ncdf package.
>
> Paul
>
> --
> Paul Hiemstra, Ph.D.
> Global Climate Division
> Royal Netherlands Meteorological Institute (KNMI)
> Wilhelminalaan 10 | 3732 GK | De Bilt | Kamer B 3.39
> P.O. Box 201 | 3730 AE | De Bilt
> tel: [phone]
>
> http://intamap.geo.uu.nl/~paul
> http://nl.linkedin.com/pub/paul-hiemstra/20/30b/770
>
>

        [[alternative HTML version deleted]]

_______________________________________________
R-sig-Geo mailing list
[hidden email]
https://stat.ethz.ch/mailman/listinfo/r-sig-geo

[janverbesselt]

Please have a look at the example I provided, it should work for your HDF4 files. The MODIS data I am dealing with comes in HDF4 format. I did not need to build a specific GDAL version that supports HDF4 format.
I am using GDAL available via http://www.kyngchaos.com/software/frameworks for Mac OSx, and the latest version of rgdal for R2.13.1. In which environment are your working?
Best,
Jan


On 02/09/2011, at 3:36 PM, Arnaud Mosnier [via R-sig-geo] wrote:

> Hi Paul,
>
> Thanks for your answer ... in fact I googled for that before asking this R
> list.
> Oceancolor HDF files are in HDF4 format and provided as two HDF4 format, in
> SIM (Standard Mapped Image) or BIN (binned).
> I first tried to use the BIN format without success. I use the gdal version
> provided with FWtools and gdal_info correctly identify the file as a HDF4
> format, but it refuses to convert it to a geotiff.
> I finally succeeded using the SIM format.
>
> But, now I am interested in doing that directly into R.
> The version of gdal called by the library rgdal do not include support for
> HDF4 format.
> So my question is now ... can someone give me a step by step instruction
> that I can use to build a GDAL version including the support for HDF4 format
> (as it is the case in the version compiled with FWtools) ?
>
> Thanks,
>
> Arnaud
>
> 2011/9/2 Paul Hiemstra <[hidden email]>
>
> >  On 09/01/2011 05:20 PM, Arnaud Mosnier wrote:
> > > Hi Geo UseRs,
> > >
> > > I am searching a way to read and manage HDF files in R (like files
> > provided
> > > by the ocean color web site, http://oceancolor.gsfc.nasa.gov/).
> > >
> > > Thanks
> > >
> > > Arnaud
> > >
> > >       [[alternative HTML version deleted]]
> > >
> > > _______________________________________________
> > > R-sig-Geo mailing list
> > > [hidden email]
> > > https://stat.ethz.ch/mailman/listinfo/r-sig-geo
> > The answer can be found by googling for "R read hdf". That leads to the
> > HDF5 package or building GDAL with HDF support. Alternatively, find a
> > (command)-line tool which can translate HDF5 to geotiff or netCDF (e.g.
> > gdal_translate or ) and then use readGDAL or the ncdf package.
> >
> > Paul
> >
> > --
> > Paul Hiemstra, Ph.D.
> > Global Climate Division
> > Royal Netherlands Meteorological Institute (KNMI)
> > Wilhelminalaan 10 | 3732 GK | De Bilt | Kamer B 3.39
> > P.O. Box 201 | 3730 AE | De Bilt
> > tel: [phone]
> >
> > http://intamap.geo.uu.nl/~paul
> > http://nl.linkedin.com/pub/paul-hiemstra/20/30b/770
> >
> >
>
>         [[alternative HTML version deleted]]
>
> _______________________________________________
> R-sig-Geo mailing list
> [hidden email]
> https://stat.ethz.ch/mailman/listinfo/r-sig-geo
>
>
> If you reply to this email, your message will be added to the discussion below:
> http://r-sig-geo.2731867.n2.nabble.com/Read-HDF-files-tp6750856p6753885.html
> To unsubscribe from Read HDF files, click here.

[Arnaud Mosnier]

Hi Jan,

Thanks for you answer.
I tried your script but unfortunatly it does not work for my case.
If someone want to make some test with the kind of image I want to use, just
try this script to download and uncompress it.

TestFolder <- choose.dir()

download.file("
http://oceandata.sci.gsfc.nasa.gov/cgi/getfile/A20072132007243.L3m_MO_CHL_chlor_a_4km.bz2",
f<-paste(TestlFolder, "/hdf4test.bz2", sep = "")
library(R.utils)
gunzip(f, f2<-paste(TestFolder, "/testhdf4.hdf", sep = "")
GDALinfo(f2) # in my case it results on this message
`C:/test/testhdf4.hdf' not recognised as a supported file format.



Here is my sessionInfo() result.

R version 2.12.0 (2010-10-15)
Platform: x86_64-pc-mingw32/x64 (64-bit)

locale:
[1] LC_COLLATE=French_Canada.1252  LC_CTYPE=French_Canada.1252
LC_MONETARY=French_Canada.1252 LC_NUMERIC=C
LC_TIME=French_Canada.1252

attached base packages:
[1] grDevices datasets  splines   graphics  stats     tcltk     utils
methods   base

other attached packages:
 [1] R.utils_1.7.8     R.oo_1.8.1        R.methodsS3_1.2.1 rgdal_0.7-1
raster_1.9-5      sp_0.9-84         svSocket_0.9-51   TinnR_1.0.3
R2HTML_2.2        Hmisc_3.8-3       survival_2.35-8

loaded via a namespace (and not attached):
[1] cluster_1.14.0  grid_2.12.0     lattice_0.19-33 svMisc_0.9-61
tools_2.12.0


Thanks for any help !

Arnaud

        [[alternative HTML version deleted]]

_______________________________________________
R-sig-Geo mailing list
[hidden email]
https://stat.ethz.ch/mailman/listinfo/r-sig-geo

[Roger Bivand]

On Sat, 3 Sep 2011, Arnaud Mosnier wrote:

> Hi Jan,
>
> Thanks for you answer.
> I tried your script but unfortunatly it does not work for my case.

No, you did not. He is using rgdal built against an OSX framework GDAL
that includes an HDF driver. If you type gdalDrivers() on your CRAN rgdal
built statically against a minimal GDAL, you will see that you do not have
an HDF driver. You need either to install rgdal from source linked to a
GDAL that has the HDF driver (which I assume you do not want to do - see
file.show(system.file("README.windows", package="rgdal")) for details if
you do), or to use external software, such as a Windows version of
gdal_translate to convert your HDF file to for example GTiff outside R,
and then use the minimal set of drivers included in the CRAN rgdal to read
the data.

GDAL potentially has well over 100 drivers, many of which must be linked
against external libraries. It is not practically possible to build the
CRAN Windows rgdal binaries (or the CRAN Extras OSX rgdal binaries) for
more than a minimal set with very few external dependencies. A source of
Windows GDAL binaries is:

http://www.gisinternals.com/sdk/

linked from:

http://trac.osgeo.org/gdal/wiki/DownloadingGdalBinaries

from:

http://www.gdal.org/

Hope this helps,

Roger

> If someone want to make some test with the kind of image I want to use, just
> try this script to download and uncompress it.
>
> TestFolder <- choose.dir()
>
> download.file("
> http://oceandata.sci.gsfc.nasa.gov/cgi/getfile/A20072132007243.L3m_MO_CHL_chlor_a_4km.bz2",
> f<-paste(TestlFolder, "/hdf4test.bz2", sep = "")
> library(R.utils)
> gunzip(f, f2<-paste(TestFolder, "/testhdf4.hdf", sep = "")
> GDALinfo(f2) # in my case it results on this message
> `C:/test/testhdf4.hdf' not recognised as a supported file format.
>
>
>
> Here is my sessionInfo() result.
>
> R version 2.12.0 (2010-10-15)
> Platform: x86_64-pc-mingw32/x64 (64-bit)
>
> locale:
> [1] LC_COLLATE=French_Canada.1252  LC_CTYPE=French_Canada.1252
> LC_MONETARY=French_Canada.1252 LC_NUMERIC=C
> LC_TIME=French_Canada.1252
>
> attached base packages:
> [1] grDevices datasets  splines   graphics  stats     tcltk     utils
> methods   base
>
> other attached packages:
> [1] R.utils_1.7.8     R.oo_1.8.1        R.methodsS3_1.2.1 rgdal_0.7-1
> raster_1.9-5      sp_0.9-84         svSocket_0.9-51   TinnR_1.0.3
> R2HTML_2.2        Hmisc_3.8-3       survival_2.35-8
>
> loaded via a namespace (and not attached):
> [1] cluster_1.14.0  grid_2.12.0     lattice_0.19-33 svMisc_0.9-61
> tools_2.12.0
>
>
> Thanks for any help !
>
> Arnaud
>
> [[alternative HTML version deleted]]
>
> _______________________________________________
> R-sig-Geo mailing list
> [hidden email]
> https://stat.ethz.ch/mailman/listinfo/r-sig-geo
>

--
Roger Bivand
Department of Economics, NHH Norwegian School of Economics,
Helleveien 30, N-5045 Bergen, Norway.
voice: [phone]; fax [phone]
e-mail: [hidden email]

_______________________________________________
R-sig-Geo mailing list
[hidden email]
https://stat.ethz.ch/mailman/listinfo/r-sig-geo

[Arnaud Mosnier]

Thanks for making that crystal clear.
For the moment, I will probably stick to FWtools to make my conversions, but
I hope I will have time to learn how to build gdal from source including the
drivers I need.

Thanks again,

Arnaud

2011/9/4 Roger Bivand <[hidden email]>

> On Sat, 3 Sep 2011, Arnaud Mosnier wrote:
>
>  Hi Jan,
>>
>> Thanks for you answer.
>> I tried your script but unfortunatly it does not work for my case.
>>
>
> No, you did not. He is using rgdal built against an OSX framework GDAL that
> includes an HDF driver. If you type gdalDrivers() on your CRAN rgdal built
> statically against a minimal GDAL, you will see that you do not have an HDF
> driver. You need either to install rgdal from source linked to a GDAL that
> has the HDF driver (which I assume you do not want to do - see
> file.show(system.file("README.**windows", package="rgdal")) for details if
> you do), or to use external software, such as a Windows version of
> gdal_translate to convert your HDF file to for example GTiff outside R, and
> then use the minimal set of drivers included in the CRAN rgdal to read the
> data.
>
> GDAL potentially has well over 100 drivers, many of which must be linked
> against external libraries. It is not practically possible to build the CRAN
> Windows rgdal binaries (or the CRAN Extras OSX rgdal binaries) for more than
> a minimal set with very few external dependencies. A source of Windows GDAL
> binaries is:
>
> http://www.gisinternals.com/**sdk/ <http://www.gisinternals.com/sdk/>
>
> linked from:
>
> http://trac.osgeo.org/gdal/**wiki/DownloadingGdalBinaries<http://trac.osgeo.org/gdal/wiki/DownloadingGdalBinaries>
>
> from:
>
> http://www.gdal.org/
>
> Hope this helps,
>
> Roger
>
>  If someone want to make some test with the kind of image I want to use,
>> just
>> try this script to download and uncompress it.
>>
>> TestFolder <- choose.dir()
>>
>> download.file("
>> http://oceandata.sci.gsfc.**nasa.gov/cgi/getfile/**
>> A20072132007243.L3m_MO_CHL_**chlor_a_4km.bz2<http://oceandata.sci.gsfc.nasa.gov/cgi/getfile/A20072132007243.L3m_MO_CHL_chlor_a_4km.bz2>
>> ",
>> f<-paste(TestlFolder, "/hdf4test.bz2", sep = "")
>> library(R.utils)
>> gunzip(f, f2<-paste(TestFolder, "/testhdf4.hdf", sep = "")
>> GDALinfo(f2) # in my case it results on this message
>> `C:/test/testhdf4.hdf' not recognised as a supported file format.
>>
>>
>>
>> Here is my sessionInfo() result.
>>
>> R version 2.12.0 (2010-10-15)
>> Platform: x86_64-pc-mingw32/x64 (64-bit)
>>
>> locale:
>> [1] LC_COLLATE=French_Canada.1252  LC_CTYPE=French_Canada.1252
>> LC_MONETARY=French_Canada.1252 LC_NUMERIC=C
>> LC_TIME=French_Canada.1252
>>
>> attached base packages:
>> [1] grDevices datasets  splines   graphics  stats     tcltk     utils
>> methods   base
>>
>> other attached packages:
>> [1] R.utils_1.7.8     R.oo_1.8.1        R.methodsS3_1.2.1 rgdal_0.7-1
>> raster_1.9-5      sp_0.9-84         svSocket_0.9-51   TinnR_1.0.3
>> R2HTML_2.2        Hmisc_3.8-3       survival_2.35-8
>>
>> loaded via a namespace (and not attached):
>> [1] cluster_1.14.0  grid_2.12.0     lattice_0.19-33 svMisc_0.9-61
>> tools_2.12.0
>>
>>
>> Thanks for any help !
>>
>> Arnaud
>>
>>        [[alternative HTML version deleted]]
>>
>> ______________________________**_________________
>> R-sig-Geo mailing list
>> [hidden email]
>> https://stat.ethz.ch/mailman/**listinfo/r-sig-geo<https://stat.ethz.ch/mailman/listinfo/r-sig-geo>
>>
>>
> --
> Roger Bivand
> Department of Economics, NHH Norwegian School of Economics,
> Helleveien 30, N-5045 Bergen, Norway.
> voice: [phone]; fax [phone]
> e-mail: [hidden email]
>
>

        [[alternative HTML version deleted]]

_______________________________________________
R-sig-Geo mailing list
[hidden email]
https://stat.ethz.ch/mailman/listinfo/r-sig-geo