Replacing values in a RasterBrick

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Replacing values in a RasterBrick

Terry B
Hi.

I'm working with the Raster package. I have a RasterBrick of 374 monthly rainfall rasters called Rain2.


> Rain2
class       : RasterBrick
dimensions  : 374, 315, 117810, 374  (nrow, ncol, ncell, nlayers)
resolution  : 0.05, 0.05  (x, y)
extent      : 137.925, 153.675, -29.275, -10.575  (xmin, xmax, ymin, ymax)
coord. ref. : +proj=longlat +ellps=GRS80 +no_defs
data source : K:\RainStuff\RainRaster.grd
names       :    X198601,    X198602,    X198603,    X198604 ...
min values  :       0.00,       0.00,       0.00,       0.00 ...
max values  :  962.01929,  534.42188,  929.24780,  790.83087 ...



I want to replace the 374 monthly rainfall pixel values at a particular location (defined by the intersection of a single row and a single column) with a different series of 374 values. But when I try the reassignment, I run into memory issues....



> Rain2[100,100]<-c(1:374)

Error: cannot allocate vector of size 336.2 Mb



I've also tried using setValues to reassign the values, as per the "Writing functions with the 'raster' package" vignette, but without success e.g. ...

> f2<-

+ function(x,row,col,a){

+ x[row,col]<-setValues(x[row,col],a)

+ return(x)}

>

> a<-c(1:374)

> f2(Rain2,100,101,a)



Error in (function (classes, fdef, mtable)  :

  unable to find an inherited method for function 'setValues' for signature '"matrix"'





I don't work much with the Raster package and I'm sure I'm making an elementary error here, but I'd appreciate any advice on the best way replace these values efficiently.

Many thanks

Terry

------------------------------
The information in this email together with any attachments is intended only for the person or entity to which it is addressed and may contain confidential and/or privileged material. There is no waiver of any confidentiality/privilege by your inadvertent receipt of this material.
Any form of review, disclosure, modification, distribution and/or publication of this email message is prohibited, unless as a necessary part of Departmental business.
If you have received this message in error, you are asked to inform the sender as quickly as possible and delete this message and any copies of this message from your computer and/or your computer system network.
------------------------------

        [[alternative HTML version deleted]]

_______________________________________________
R-sig-Geo mailing list
[hidden email]
https://stat.ethz.ch/mailman/listinfo/r-sig-geo
Reply | Threaded
Open this post in threaded view
|

Re: Replacing values in a RasterBrick

Michael Sumner-2
On Thu, 7 Sep 2017 at 08:22 BEUTEL Terry S <[hidden email]>
wrote:

> Hi.
>
> I'm working with the Raster package. I have a RasterBrick of 374 monthly
> rainfall rasters called Rain2.
>
>
> > Rain2
> class       : RasterBrick
> dimensions  : 374, 315, 117810, 374  (nrow, ncol, ncell, nlayers)
> resolution  : 0.05, 0.05  (x, y)
> extent      : 137.925, 153.675, -29.275, -10.575  (xmin, xmax, ymin, ymax)
> coord. ref. : +proj=longlat +ellps=GRS80 +no_defs
> data source : K:\RainStuff\RainRaster.grd
> names       :    X198601,    X198602,    X198603,    X198604 ...
> min values  :       0.00,       0.00,       0.00,       0.00 ...
> max values  :  962.01929,  534.42188,  929.24780,  790.83087 ...
>
>
>
> I want to replace the 374 monthly rainfall pixel values at a particular
> location (defined by the intersection of a single row and a single column)
> with a different series of 374 values. But when I try the reassignment, I
> run into memory issues....
>
>
As far as I understand it there is no actual partial-write for the this
format, it's actually a complicated topic given the range of ways raster
can store data either on file or in memory.

A Raster in memory is a long-form matrix with the matrix flattened out
(left right top to bottom, different to R's matrix), and the columns are
"layers". On disk, the rasterfile (.grd) is a flat binary with an
orientation ordering (either band sequential, band interleaved by line or
band interleaved by pixel) and so that can't be easily overridden without
knowing the details.

Given the size of the file I would first try pulling it all into memory and
rewriting in whole:

Rain2 <- readAll(Rain2)

## then run your updates
## ...
##  now write back to file

writeRaster(Rain2, "RainRaster_test.grd")

and check.

If that fails, because your system can't perform this (wasteful, brute
force) read and write then you could use ff - it's possible to
transparently map an ff array to an existing RasterBrick (in rasterfile.grd
format) and update it using ff's efficient memory mapping.

Let  me know if you that sounds promising and you want to explore more,
it's a simple thing I should finish up and put on CRAN.

Cheers, Mike.


>
> > Rain2[100,100]<-c(1:374)
>
> Error: cannot allocate vector of size 336.2 Mb
>
>
>
> I've also tried using setValues to reassign the values, as per the
> "Writing functions with the 'raster' package" vignette, but without success
> e.g. ...
>
> > f2<-
>
> + function(x,row,col,a){
>
> + x[row,col]<-setValues(x[row,col],a)
>
> + return(x)}
>
> >
>
> > a<-c(1:374)
>
> > f2(Rain2,100,101,a)
>
>
>
> Error in (function (classes, fdef, mtable)  :
>
>   unable to find an inherited method for function 'setValues' for
> signature '"matrix"'
>
>
>
>
>
> I don't work much with the Raster package and I'm sure I'm making an
> elementary error here, but I'd appreciate any advice on the best way
> replace these values efficiently.
>
> Many thanks
>
> Terry
>
> ------------------------------
> The information in this email together with any attachments is intended
> only for the person or entity to which it is addressed and may contain
> confidential and/or privileged material. There is no waiver of any
> confidentiality/privilege by your inadvertent receipt of this material.
> Any form of review, disclosure, modification, distribution and/or
> publication of this email message is prohibited, unless as a necessary part
> of Departmental business.
> If you have received this message in error, you are asked to inform the
> sender as quickly as possible and delete this message and any copies of
> this message from your computer and/or your computer system network.
> ------------------------------
>
>         [[alternative HTML version deleted]]
>
> _______________________________________________
> R-sig-Geo mailing list
> [hidden email]
> https://stat.ethz.ch/mailman/listinfo/r-sig-geo
>
--
Dr. Michael Sumner
Software and Database Engineer
Australian Antarctic Division
203 Channel Highway
Kingston Tasmania 7050 Australia

        [[alternative HTML version deleted]]

_______________________________________________
R-sig-Geo mailing list
[hidden email]
https://stat.ethz.ch/mailman/listinfo/r-sig-geo
Reply | Threaded
Open this post in threaded view
|

Re: Replacing values in a RasterBrick

Terry B
Thanks Michael.
I tried pulling it all into memory and running on a heftier computer and it now works. I also changed the datatype to INT2U which sped things up further without losing significant information in my data.

Consequently the ff option now probably isn’t necessary for me, but if its simple to do as you say, and speeds up processing and/or pushes up the size of raster/bricks/stacks that users can process it would always help, and I’d be happy to trial it on data here if that would help.

Cheers
Terry


From: Michael Sumner [mailto:[hidden email]]
Sent: Thursday, 7 September 2017 6:38 PM
To: BEUTEL Terry S; [hidden email]
Subject: Re: [R-sig-Geo] Replacing values in a RasterBrick


On Thu, 7 Sep 2017 at 08:22 BEUTEL Terry S <[hidden email]<mailto:[hidden email]>> wrote:
Hi.

I'm working with the Raster package. I have a RasterBrick of 374 monthly rainfall rasters called Rain2.


> Rain2
class       : RasterBrick
dimensions  : 374, 315, 117810, 374  (nrow, ncol, ncell, nlayers)
resolution  : 0.05, 0.05  (x, y)
extent      : 137.925, 153.675, -29.275, -10.575  (xmin, xmax, ymin, ymax)
coord. ref. : +proj=longlat +ellps=GRS80 +no_defs
data source : K:\RainStuff\RainRaster.grd
names       :    X198601,    X198602,    X198603,    X198604 ...
min values  :       0.00,       0.00,       0.00,       0.00 ...
max values  :  962.01929,  534.42188,  929.24780,  790.83087 ...



I want to replace the 374 monthly rainfall pixel values at a particular location (defined by the intersection of a single row and a single column) with a different series of 374 values. But when I try the reassignment, I run into memory issues....

As far as I understand it there is no actual partial-write for the this format, it's actually a complicated topic given the range of ways raster can store data either on file or in memory.

A Raster in memory is a long-form matrix with the matrix flattened out (left right top to bottom, different to R's matrix), and the columns are "layers". On disk, the rasterfile (.grd) is a flat binary with an orientation ordering (either band sequential, band interleaved by line or band interleaved by pixel) and so that can't be easily overridden without knowing the details.

Given the size of the file I would first try pulling it all into memory and rewriting in whole:

Rain2 <- readAll(Rain2)

## then run your updates
## ...
##  now write back to file

writeRaster(Rain2, "RainRaster_test.grd")

and check.

If that fails, because your system can't perform this (wasteful, brute force) read and write then you could use ff - it's possible to transparently map an ff array to an existing RasterBrick (in rasterfile.grd format) and update it using ff's efficient memory mapping.

Let  me know if you that sounds promising and you want to explore more, it's a simple thing I should finish up and put on CRAN.

Cheers, Mike.



> Rain2[100,100]<-c(1:374)

Error: cannot allocate vector of size 336.2 Mb



I've also tried using setValues to reassign the values, as per the "Writing functions with the 'raster' package" vignette, but without success e.g. ...

> f2<-

+ function(x,row,col,a){

+ x[row,col]<-setValues(x[row,col],a)

+ return(x)}

>

> a<-c(1:374)

> f2(Rain2,100,101,a)



Error in (function (classes, fdef, mtable)  :

  unable to find an inherited method for function 'setValues' for signature '"matrix"'





I don't work much with the Raster package and I'm sure I'm making an elementary error here, but I'd appreciate any advice on the best way replace these values efficiently.

Many thanks

Terry

------------------------------
The information in this email together with any attachments is intended only for the person or entity to which it is addressed and may contain confidential and/or privileged material. There is no waiver of any confidentiality/privilege by your inadvertent receipt of this material.
Any form of review, disclosure, modification, distribution and/or publication of this email message is prohibited, unless as a necessary part of Departmental business.
If you have received this message in error, you are asked to inform the sender as quickly as possible and delete this message and any copies of this message from your computer and/or your computer system network.
------------------------------

        [[alternative HTML version deleted]]

_______________________________________________
R-sig-Geo mailing list
[hidden email]<mailto:[hidden email]>
https://stat.ethz.ch/mailman/listinfo/r-sig-geo<https://urldefense.proofpoint.com/v2/url?u=https-3A__stat.ethz.ch_mailman_listinfo_r-2Dsig-2Dgeo&d=DwMFaQ&c=tpTxelpKGw9ZbZ5Dlo0lybSxHDHIiYjksG4icXfalgk&r=WSiYvK4_AlHC5jVoClfL7UaU2gIlHzsDOL_wKj5virk&m=ytXH4ODNC7WcBTPkAGOxRKX_UKtPAi_340UheMnZFnM&s=bC3e32XH9U0dtqHsyPoiSfF3PsFtd3JgsCByvId2nOI&e=>
--
Dr. Michael Sumner
Software and Database Engineer
Australian Antarctic Division
203 Channel Highway
Kingston Tasmania 7050 Australia

        [[alternative HTML version deleted]]

_______________________________________________
R-sig-Geo mailing list
[hidden email]
https://stat.ethz.ch/mailman/listinfo/r-sig-geo
Reply | Threaded
Open this post in threaded view
|

Re: Replacing values in a RasterBrick

Michael Sumner-2
On Sun, 10 Sep 2017 at 10:45 BEUTEL Terry S <[hidden email]>
wrote:

> Thanks Michael.
>
> I tried pulling it all into memory and running on a heftier computer and
> it now works. I also changed the datatype to INT2U which sped things up
> further without losing significant information in my data.
>
>
>
> Consequently the ff option now probably isn’t necessary for me, but if its
> simple to do as you say, and speeds up processing and/or pushes up the size
> of raster/bricks/stacks that users can process it would always help, and
> I’d be happy to trial it on data here if that would help.
>
>
>

Thanks!  I'll take you up on that when I look at it again.

Cheers, Mike.



> Cheers
>
> Terry
>
>
>
>
>
> *From:* Michael Sumner [mailto:[hidden email]]
> *Sent:* Thursday, 7 September 2017 6:38 PM
> *To:* BEUTEL Terry S; [hidden email]
> *Subject:* Re: [R-sig-Geo] Replacing values in a RasterBrick
>
>
>
>
>
> On Thu, 7 Sep 2017 at 08:22 BEUTEL Terry S <[hidden email]>
> wrote:
>
> Hi.
>
> I'm working with the Raster package. I have a RasterBrick of 374 monthly
> rainfall rasters called Rain2.
>
>
> > Rain2
> class       : RasterBrick
> dimensions  : 374, 315, 117810, 374  (nrow, ncol, ncell, nlayers)
> resolution  : 0.05, 0.05  (x, y)
> extent      : 137.925, 153.675, -29.275, -10.575  (xmin, xmax, ymin, ymax)
> coord. ref. : +proj=longlat +ellps=GRS80 +no_defs
> data source : K:\RainStuff\RainRaster.grd
> names       :    X198601,    X198602,    X198603,    X198604 ...
> min values  :       0.00,       0.00,       0.00,       0.00 ...
> max values  :  962.01929,  534.42188,  929.24780,  790.83087 ...
>
>
>
> I want to replace the 374 monthly rainfall pixel values at a particular
> location (defined by the intersection of a single row and a single column)
> with a different series of 374 values. But when I try the reassignment, I
> run into memory issues....
>
>
>
> As far as I understand it there is no actual partial-write for the this
> format, it's actually a complicated topic given the range of ways raster
> can store data either on file or in memory.
>
>
>
> A Raster in memory is a long-form matrix with the matrix flattened out
> (left right top to bottom, different to R's matrix), and the columns are
> "layers". On disk, the rasterfile (.grd) is a flat binary with an
> orientation ordering (either band sequential, band interleaved by line or
> band interleaved by pixel) and so that can't be easily overridden without
> knowing the details.
>
>
>
> Given the size of the file I would first try pulling it all into memory
> and rewriting in whole:
>
>
>
> Rain2 <- readAll(Rain2)
>
>
>
> ## then run your updates
>
> ## ...
>
> ##  now write back to file
>
>
>
> writeRaster(Rain2, "RainRaster_test.grd")
>
>
>
> and check.
>
>
>
> If that fails, because your system can't perform this (wasteful, brute
> force) read and write then you could use ff - it's possible to
> transparently map an ff array to an existing RasterBrick (in rasterfile.grd
> format) and update it using ff's efficient memory mapping.
>
>
>
> Let  me know if you that sounds promising and you want to explore more,
> it's a simple thing I should finish up and put on CRAN.
>
>
>
> Cheers, Mike.
>
>
>
>
>
> > Rain2[100,100]<-c(1:374)
>
> Error: cannot allocate vector of size 336.2 Mb
>
>
>
> I've also tried using setValues to reassign the values, as per the
> "Writing functions with the 'raster' package" vignette, but without success
> e.g. ...
>
> > f2<-
>
> + function(x,row,col,a){
>
> + x[row,col]<-setValues(x[row,col],a)
>
> + return(x)}
>
> >
>
> > a<-c(1:374)
>
> > f2(Rain2,100,101,a)
>
>
>
> Error in (function (classes, fdef, mtable)  :
>
>   unable to find an inherited method for function 'setValues' for
> signature '"matrix"'
>
>
>
>
>
> I don't work much with the Raster package and I'm sure I'm making an
> elementary error here, but I'd appreciate any advice on the best way
> replace these values efficiently.
>
> Many thanks
>
> Terry
>
> ------------------------------
> The information in this email together with any attachments is intended
> only for the person or entity to which it is addressed and may contain
> confidential and/or privileged material. There is no waiver of any
> confidentiality/privilege by your inadvertent receipt of this material.
> Any form of review, disclosure, modification, distribution and/or
> publication of this email message is prohibited, unless as a necessary part
> of Departmental business.
> If you have received this message in error, you are asked to inform the
> sender as quickly as possible and delete this message and any copies of
> this message from your computer and/or your computer system network.
> ------------------------------
>
>         [[alternative HTML version deleted]]
>
> _______________________________________________
> R-sig-Geo mailing list
> [hidden email]
> https://stat.ethz.ch/mailman/listinfo/r-sig-geo
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__stat.ethz.ch_mailman_listinfo_r-2Dsig-2Dgeo&d=DwMFaQ&c=tpTxelpKGw9ZbZ5Dlo0lybSxHDHIiYjksG4icXfalgk&r=WSiYvK4_AlHC5jVoClfL7UaU2gIlHzsDOL_wKj5virk&m=ytXH4ODNC7WcBTPkAGOxRKX_UKtPAi_340UheMnZFnM&s=bC3e32XH9U0dtqHsyPoiSfF3PsFtd3JgsCByvId2nOI&e=>
>
> --
>
> Dr. Michael Sumner
> Software and Database Engineer
> Australian Antarctic Division
> 203 Channel Highway
> Kingston Tasmania 7050 Australia
>
--
Dr. Michael Sumner
Software and Database Engineer
Australian Antarctic Division
203 Channel Highway
Kingston Tasmania 7050 Australia

        [[alternative HTML version deleted]]

_______________________________________________
R-sig-Geo mailing list
[hidden email]
https://stat.ethz.ch/mailman/listinfo/r-sig-geo