Automate the extraction of climate variable at different time depths from netcdf in r

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

Automate the extraction of climate variable at different time depths from netcdf in r

Barnabas Daru
Dear all,
I write to seek help with extracting climate data at various times and
depth from NETCDF in R.
The climate data is sea surface temperature monthly means from 1854 to
2014.
My goal is to automate the extraction of SST climate data for a set of
points with different time depths.

I have successfully used the 'brick' function in the raster package to get
climate data at various time depths as follows:

b <- brick("~sst.mnmean.nc", varname="sst")

mydate <- b[["X1869.09.01"]] # SST for September in 1869

mydata <- read.csv("~Species one.csv")

mydataSPDF <- SpatialPointsDataFrame(mydata[,5:6],data.frame(mydata))

extract.mydata <- extract(mydate, mydataSPDF, sp=TRUE)

extract.mydata <- data.frame(extract.mydata)

write.csv(extract.mydata, file = "Species_one_extracted.csv")

My major challenge is that I have to keep extracting manually for each time
depth, then copy and paste in the main dataframe and the data has about
3000 observations, meaning I have to keep copying and pasting ~3000 times
for one species!
Is there a way I can iterate the process in R to loop through each month's
climate and extract the data for each point and time depth rather than to
do it manually?

Here's how my data frame looks like:
  Date year month day lon lat SST  1855-01-01 1855 1 1 11.6861 57.7254
3.440000057  1859-07-26 1859 7 26 25.46122 60.26766 13.25  1860-01-01 1860 1
1 10.9154 53.9484 15.10999966  1861-07-10 1861 7 10 7.79588 58.08673
15.71000004  1861-07-26 1861 7 26 -2.84072 54.0778  1861-08-17 1861 8 17
11.9792 57.51298  1862-01-01 1862 1 1 22.20467 60.27955  1862-08-05 1862 8 5
11.78316 57.61649  1862-08-23 1862 8 23 11.9792 57.51298  1863-08-26 1863 8
26 10.72237 59.97258  1863-08-28 1863 8 28 15.53721 56.20091  1863-09-22
1863 9 22 16.37849 56.84255  1864-07-05 1864 7 5 -4.07097 51.09542
1864-07-18 1864 7 18 -4.21368 51.0928  1864-08-26 1864 8 26 -4.07097
51.09542  1865-09-07 1865 9 7 10.76144 59.91271  1865-09-27 1865 9 27
10.53107 60.43395  1867-08-28 1867 8 28 12.82669 55.86822  1868-01-01 1868 1
1 12.8944 56.6897  1868-01-01 1868 1 1 4.82685 53.13793
Thanks and kind regards
Barnabas Daru

--

  \-/
   /\
  /--|
 /---/ *Daru, Barnabas Haruna*
 |--/  PhD Candidate,
 \-/   African Centre for DNA Barcoding,
 /\    University of Johannesburg,
/--\   PO BOX 524 Auckland Park, 2006,
|---\  South Africa
 \---\ Lab: +27 11 559 3477
  \--| Mobile: +27 7381 89 583
   \-/
   /\  My homepage: http://barnabasdaru.com
<http://acdb.co.za/index.php?page=mr-barnabas-h-daru>
  /--\

        [[alternative HTML version deleted]]

_______________________________________________
R-sig-Geo mailing list
[hidden email]
https://stat.ethz.ch/mailman/listinfo/r-sig-geo
Reply | Threaded
Open this post in threaded view
|

Re: Automate the extraction of climate variable at different time depths from netcdf in r

Robert Hijmans
Barnabas, ,
You can try something like this:

b <- brick("~sst.mnmean.nc", varname="sst")
# loop over species, or combine all species into one data.frame?
mydata <- read.csv("~Species one.csv")
extract.mydata <- extract(b, mydata[,5:6])
write.csv(extract.mydata, file = "Species_one_extracted.csv")

Robert

On Sat, Sep 6, 2014 at 10:10 AM, Barnabas Daru <[hidden email]> wrote:

> Dear all,
> I write to seek help with extracting climate data at various times and
> depth from NETCDF in R.
> The climate data is sea surface temperature monthly means from 1854 to
> 2014.
> My goal is to automate the extraction of SST climate data for a set of
> points with different time depths.
>
> I have successfully used the 'brick' function in the raster package to get
> climate data at various time depths as follows:
>
> b <- brick("~sst.mnmean.nc", varname="sst")
>
> mydate <- b[["X1869.09.01"]] # SST for September in 1869
>
> mydata <- read.csv("~Species one.csv")
>
> mydataSPDF <- SpatialPointsDataFrame(mydata[,5:6],data.frame(mydata))
>
> extract.mydata <- extract(mydate, mydataSPDF, sp=TRUE)
>
> extract.mydata <- data.frame(extract.mydata)
>
> write.csv(extract.mydata, file = "Species_one_extracted.csv")
>
> My major challenge is that I have to keep extracting manually for each time
> depth, then copy and paste in the main dataframe and the data has about
> 3000 observations, meaning I have to keep copying and pasting ~3000 times
> for one species!
> Is there a way I can iterate the process in R to loop through each month's
> climate and extract the data for each point and time depth rather than to
> do it manually?
>
> Here's how my data frame looks like:
>   Date year month day lon lat SST  1855-01-01 1855 1 1 11.6861 57.7254
> 3.440000057  1859-07-26 1859 7 26 25.46122 60.26766 13.25  1860-01-01 1860 1
> 1 10.9154 53.9484 15.10999966  1861-07-10 1861 7 10 7.79588 58.08673
> 15.71000004  1861-07-26 1861 7 26 -2.84072 54.0778  1861-08-17 1861 8 17
> 11.9792 57.51298  1862-01-01 1862 1 1 22.20467 60.27955  1862-08-05 1862 8 5
> 11.78316 57.61649  1862-08-23 1862 8 23 11.9792 57.51298  1863-08-26 1863 8
> 26 10.72237 59.97258  1863-08-28 1863 8 28 15.53721 56.20091  1863-09-22
> 1863 9 22 16.37849 56.84255  1864-07-05 1864 7 5 -4.07097 51.09542
> 1864-07-18 1864 7 18 -4.21368 51.0928  1864-08-26 1864 8 26 -4.07097
> 51.09542  1865-09-07 1865 9 7 10.76144 59.91271  1865-09-27 1865 9 27
> 10.53107 60.43395  1867-08-28 1867 8 28 12.82669 55.86822  1868-01-01 1868 1
> 1 12.8944 56.6897  1868-01-01 1868 1 1 4.82685 53.13793
> Thanks and kind regards
> Barnabas Daru
>
> --
>
>   \-/
>    /\
>   /--|
>  /---/ *Daru, Barnabas Haruna*
>  |--/  PhD Candidate,
>  \-/   African Centre for DNA Barcoding,
>  /\    University of Johannesburg,
> /--\   PO BOX 524 Auckland Park, 2006,
> |---\  South Africa
>  \---\ Lab: +27 11 559 3477
>   \--| Mobile: +27 7381 89 583
>    \-/
>    /\  My homepage: http://barnabasdaru.com
> <http://acdb.co.za/index.php?page=mr-barnabas-h-daru>
>   /--\
>
>         [[alternative HTML version deleted]]
>
> _______________________________________________
> R-sig-Geo mailing list
> [hidden email]
> https://stat.ethz.ch/mailman/listinfo/r-sig-geo

_______________________________________________
R-sig-Geo mailing list
[hidden email]
https://stat.ethz.ch/mailman/listinfo/r-sig-geo
Reply | Threaded
Open this post in threaded view
|

Re: Automate the extraction of climate variable at different time depths from netcdf in r

Barnabas Daru
Hi Robert,
Thanks very much for your suggestion.
I tried the code you provided and it work! Many thanks.
However, it tends to extract the SST values for all the points and for time periods including even the years that I don't desire. This means I will have to delete one-by-one, the values for the years that I do not desire and keep only the ones I want.
I thought there is a way I could simply ask R to print in one column, the extracted values associated with only the years for which point data is available.
Thanks and kind regards
Barnabas



  \-/
   /\
  /--|    
 /---/ Barnabas Daru
 |--/  PhD Candidate,
 \-/   African Centre for DNA Barcoding,
 /\    University of Johannesburg,
/--\   PO Box 524, Auckland Park, 2006,
|---\  Johannesburg, South Africa.
 \---\ Lab: +27 11 559 3477        
  \--| Mobile: +277 3818 9583
   \-/ Website: www.barnabasdaru.com
   /\        
  /--\
 
#�if you can think it, you can do it.
 




On Sep 7, 2014, at 5:23 AM, Robert J. Hijmans wrote:

> Barnabas, ,
> You can try something like this:
>
> b <- brick("~sst.mnmean.nc", varname="sst")
> # loop over species, or combine all species into one data.frame?
> mydata <- read.csv("~Species one.csv")
> extract.mydata <- extract(b, mydata[,5:6])
> write.csv(extract.mydata, file = "Species_one_extracted.csv")
>
> Robert

        [[alternative HTML version deleted]]


_______________________________________________
R-sig-Geo mailing list
[hidden email]
https://stat.ethz.ch/mailman/listinfo/r-sig-geo
Reply | Threaded
Open this post in threaded view
|

Re: Automate the extraction of climate variable at different time depths from netcdf in r

Kristin
In reply to this post by Robert Hijmans
Hello
I´m having similar problem, do not seem to find obvious solution on the net.  

I have a raster brick of 464 layers, each layer representing time defined as number from 1 to 464.
The number is stored here:
rasterbrick@z$days

and has values from 1 to 464

Then I have 2800 space time locations of field data sampling. These are both irregular sampling locations and some irregular sampling dates at same sampling locations. Dates are also defined as numbers from 1 to 464. Each individual point is then identified by lat and lon at different times.
This is what my dataframe looks like:

Lat, Lon, days
64   -21    23
63   -19    52
63   -19    53
66   -25    23
....

So what I want to do is to extract values from the raster brick at a certain location, but only where the date numbers from both the raster brick and the locations match, but not for all times for all locations as the extract from raster brick does.

atb,
Kristin
 
Reply | Threaded
Open this post in threaded view
|

Re: Automate the extraction of climate variable at different time depths from netcdf in r

Pascal Oettli
Hi Kristin,

Please start a new thread for your own question, thank you.

Regards,
Pascal

On Mon, Sep 8, 2014 at 1:57 AM, Kristin <[hidden email]> wrote:

> Hello
> I´m having similar problem, do not seem to find obvious solution on the net.
>
> I have a raster brick of 464 layers, each layer representing time defined as
> number from 1 to 464.
> The number is stored here:
> rasterbrick@z$days
>
> and has values from 1 to 464
>
> Then I have 2800 space time locations of field data sampling. These are both
> irregular sampling locations and some irregular sampling dates at same
> sampling locations. Dates are also defined as numbers from 1 to 464. Each
> individual point is then identified by lat and lon at different times.
> This is what my dataframe looks like:
>
> Lat, Lon, days
> 64   -21    23
> 63   -19    52
> 63   -19    53
> 66   -25    23
> ....
>
> So what I want to do is to extract values from the raster brick at a certain
> location, but only where the date numbers from both the raster brick and the
> locations match, but not for all times for all locations as the extract from
> raster brick does.
>
> atb,
> Kristin
>
>
>
>
> --
> View this message in context: http://r-sig-geo.2731867.n2.nabble.com/Automate-the-extraction-of-climate-variable-at-different-time-depths-from-netcdf-in-r-tp7587070p7587080.html
> Sent from the R-sig-geo mailing list archive at Nabble.com.
>
> _______________________________________________
> R-sig-Geo mailing list
> [hidden email]
> https://stat.ethz.ch/mailman/listinfo/r-sig-geo



--
Pascal Oettli
Project Scientist
JAMSTEC
Yokohama, Japan

_______________________________________________
R-sig-Geo mailing list
[hidden email]
https://stat.ethz.ch/mailman/listinfo/r-sig-geo
Reply | Threaded
Open this post in threaded view
|

Re: Automate the extraction of climate variable at different time depths from netcdf in r

rubenfcasal
In reply to this post by Barnabas Daru
Hello,

     I use the attached script to extract climate data from the OISST
sea surface temperature netCDF "sst.wkmean.1990-present.nc". This script
is a slight modification of one written by Luke Miller (downloaded from
his web page http://lukemiller.org, some years ago...  ) to convert the
data to sp/spacetime classes (easily to handle, subset,...).

     I just realised that Luke Miller has updated the original script.
See:
http://lukemiller.org/index.php/2014/01/noaa-oisst-v2-high-resolution-daily-sea-surface-temperatures-with-r/

     I hope it helps...

     Best Regards, Ruben FC


El 06/09/2014 19:10, Barnabas Daru escribió:

> Dear all,
> I write to seek help with extracting climate data at various times and
> depth from NETCDF in R.
> The climate data is sea surface temperature monthly means from 1854 to
> 2014.
> My goal is to automate the extraction of SST climate data for a set of
> points with different time depths.
>
> I have successfully used the 'brick' function in the raster package to get
> climate data at various time depths as follows:
>
> b <- brick("~sst.mnmean.nc", varname="sst")
>
> mydate <- b[["X1869.09.01"]] # SST for September in 1869
>
> mydata <- read.csv("~Species one.csv")
>
> mydataSPDF <- SpatialPointsDataFrame(mydata[,5:6],data.frame(mydata))
>
> extract.mydata <- extract(mydate, mydataSPDF, sp=TRUE)
>
> extract.mydata <- data.frame(extract.mydata)
>
> write.csv(extract.mydata, file = "Species_one_extracted.csv")
>
> My major challenge is that I have to keep extracting manually for each time
> depth, then copy and paste in the main dataframe and the data has about
> 3000 observations, meaning I have to keep copying and pasting ~3000 times
> for one species!
> Is there a way I can iterate the process in R to loop through each month's
> climate and extract the data for each point and time depth rather than to
> do it manually?
>
> Here's how my data frame looks like:
>    Date year month day lon lat SST  1855-01-01 1855 1 1 11.6861 57.7254
> 3.440000057  1859-07-26 1859 7 26 25.46122 60.26766 13.25  1860-01-01 1860 1
> 1 10.9154 53.9484 15.10999966  1861-07-10 1861 7 10 7.79588 58.08673
> 15.71000004  1861-07-26 1861 7 26 -2.84072 54.0778  1861-08-17 1861 8 17
> 11.9792 57.51298  1862-01-01 1862 1 1 22.20467 60.27955  1862-08-05 1862 8 5
> 11.78316 57.61649  1862-08-23 1862 8 23 11.9792 57.51298  1863-08-26 1863 8
> 26 10.72237 59.97258  1863-08-28 1863 8 28 15.53721 56.20091  1863-09-22
> 1863 9 22 16.37849 56.84255  1864-07-05 1864 7 5 -4.07097 51.09542
> 1864-07-18 1864 7 18 -4.21368 51.0928  1864-08-26 1864 8 26 -4.07097
> 51.09542  1865-09-07 1865 9 7 10.76144 59.91271  1865-09-27 1865 9 27
> 10.53107 60.43395  1867-08-28 1867 8 28 12.82669 55.86822  1868-01-01 1868 1
> 1 12.8944 56.6897  1868-01-01 1868 1 1 4.82685 53.13793
> Thanks and kind regards
> Barnabas Daru
>

_______________________________________________
R-sig-Geo mailing list
[hidden email]
https://stat.ethz.ch/mailman/listinfo/r-sig-geo

NOAA_OISST_extraction_v2.r (7K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Automate the extraction of climate variable at different time depths from netcdf in r

Robert Hijmans
In reply to this post by Barnabas Daru
Barnabas,
I doubt you will need to "delete one-by-one, the values for the years
that I do not desire". There surely are basic R tricks for that.
Perhaps the below example can get you there:

## example data
#raster
b <- brick(system.file("external/rlogo.grd", package="raster"))
# points
set.seed(0)
p <- xyFromCell(b, sample(ncell(b), 10))
# "years"
y <- cbind(y1=sample(0:1, 10, replace=TRUE), y2=sample(0:1, 10,
replace=TRUE), y3=sample(0:1, 10, replace=TRUE))


# extract all
e <- extract(b, p)
# combine data
x <- data.frame(x=p[,1], y=p[,2], year=rep(1:3, each=nrow(p)),
include=as.vector(y), value=as.vector(e))
# select values you want
x[x$include==1, ]

Whether the above works for you really depends on how you need to get
the values for further processing.

Robert



On Sun, Sep 7, 2014 at 8:58 AM, Barnabas Daru <[hidden email]> wrote:

> Hi Robert,
> Thanks very much for your suggestion.
> I tried the code you provided and it work! Many thanks.
> However, it tends to extract the SST values for all the points and for time
> periods including even the years that I don't desire. This means I will have
> to delete one-by-one, the values for the years that I do not desire and keep
> only the ones I want.
> I thought there is a way I could simply ask R to print in one column, the
> extracted values associated with only the years for which point data is
> available.
> Thanks and kind regards
> Barnabas
>
>
>
>   \-/
>    /\
>   /--|
>  /---/ Barnabas Daru
>  |--/  PhD Candidate,
>  \-/   African Centre for DNA Barcoding,
>  /\    University of Johannesburg,
> /--\   PO Box 524, Auckland Park, 2006,
> |---\  Johannesburg, South Africa.
>  \---\ Lab: +27 11 559 3477
>   \--| Mobile: +277 3818 9583
>    \-/ Website: www.barnabasdaru.com
>    /\
>   /--\
>
> #…if you can think it, you can do it.
>
>
>
>
>
> On Sep 7, 2014, at 5:23 AM, Robert J. Hijmans wrote:
>
> Barnabas, ,
> You can try something like this:
>
> b <- brick("~sst.mnmean.nc", varname="sst")
> # loop over species, or combine all species into one data.frame?
> mydata <- read.csv("~Species one.csv")
> extract.mydata <- extract(b, mydata[,5:6])
> write.csv(extract.mydata, file = "Species_one_extracted.csv")
>
> Robert
>
>

_______________________________________________
R-sig-Geo mailing list
[hidden email]
https://stat.ethz.ch/mailman/listinfo/r-sig-geo
Reply | Threaded
Open this post in threaded view
|

Re: Automate the extraction of climate variable at different time depths from netcdf in r

Barnabas Daru
Hi Robert,
Thank you very much for the code. It works.
Thanks and kind regards
Barnabas Daru


  \-/
   /\
  /--|    
 /---/ Barnabas Daru
 |--/  PhD Candidate,
 \-/   African Centre for DNA Barcoding,
 /\    University of Johannesburg,
/--\   PO Box 524, Auckland Park, 2006,
|---\  Johannesburg, South Africa.
 \---\ Lab: +27 11 559 3477        
  \--| Mobile: +277 3818 9583
   \-/ Website: www.barnabasdaru.com
   /\        
  /--\
 
#�if you can think it, you can do it.
 




On Sep 11, 2014, at 6:47 PM, Robert J. Hijmans wrote:

> Barnabas,
> I doubt you will need to "delete one-by-one, the values for the years
> that I do not desire". There surely are basic R tricks for that.
> Perhaps the below example can get you there:
>
> ## example data
> #raster
> b <- brick(system.file("external/rlogo.grd", package="raster"))
> # points
> set.seed(0)
> p <- xyFromCell(b, sample(ncell(b), 10))
> # "years"
> y <- cbind(y1=sample(0:1, 10, replace=TRUE), y2=sample(0:1, 10,
> replace=TRUE), y3=sample(0:1, 10, replace=TRUE))
>
>
> # extract all
> e <- extract(b, p)
> # combine data
> x <- data.frame(x=p[,1], y=p[,2], year=rep(1:3, each=nrow(p)),
> include=as.vector(y), value=as.vector(e))
> # select values you want
> x[x$include==1, ]
>
> Whether the above works for you really depends on how you need to get
> the values for further processing.
>
> Robert
>
>
>
> On Sun, Sep 7, 2014 at 8:58 AM, Barnabas Daru <[hidden email]> wrote:
>> Hi Robert,
>> Thanks very much for your suggestion.
>> I tried the code you provided and it work! Many thanks.
>> However, it tends to extract the SST values for all the points and for time
>> periods including even the years that I don't desire. This means I will have
>> to delete one-by-one, the values for the years that I do not desire and keep
>> only the ones I want.
>> I thought there is a way I could simply ask R to print in one column, the
>> extracted values associated with only the years for which point data is
>> available.
>> Thanks and kind regards
>> Barnabas
>>
>>
>>
>>  \-/
>>   /\
>>  /--|
>> /---/ Barnabas Daru
>> |--/  PhD Candidate,
>> \-/   African Centre for DNA Barcoding,
>> /\    University of Johannesburg,
>> /--\   PO Box 524, Auckland Park, 2006,
>> |---\  Johannesburg, South Africa.
>> \---\ Lab: +27 11 559 3477
>>  \--| Mobile: +277 3818 9583
>>   \-/ Website: www.barnabasdaru.com
>>   /\
>>  /--\
>>
>> #�if you can think it, you can do it.
>>
>>
>>
>>
>>
>> On Sep 7, 2014, at 5:23 AM, Robert J. Hijmans wrote:
>>
>> Barnabas, ,
>> You can try something like this:
>>
>> b <- brick("~sst.mnmean.nc", varname="sst")
>> # loop over species, or combine all species into one data.frame?
>> mydata <- read.csv("~Species one.csv")
>> extract.mydata <- extract(b, mydata[,5:6])
>> write.csv(extract.mydata, file = "Species_one_extracted.csv")
>>
>> Robert
>>
>>

        [[alternative HTML version deleted]]


_______________________________________________
R-sig-Geo mailing list
[hidden email]
https://stat.ethz.ch/mailman/listinfo/r-sig-geo