How to reduce the buffering time with the function "buffer" (package raster)

classic Classic list List threaded Threaded
12 messages Options
Reply | Threaded
Open this post in threaded view
|

How to reduce the buffering time with the function "buffer" (package raster)

Nelly Reduan

Hello,


I would like to build 100-m buffers (representing ecological corridors) around land cover attributes in a raster layer and to assign a given value (i.e., value of 13) to buffer cells.


To do this, I'm using the function "buffer" (package raster) to draw the buffers around particular raster cells (i.e., cell values of 2 and 5). For example, in the image below, the buffers are represented in blue (cell values of 13) and the particular raster cells are represented in yellow (cell values of 2).


The problem is that the time to run the function "buffer" is very low because I have a raster of 5000000 cells. Does anyone know how I can reduce the buffering time ?


Thanks a lot for your time.

Have a nice day.

Nell



_______________________________________________
R-sig-Geo mailing list
[hidden email]
https://stat.ethz.ch/mailman/listinfo/r-sig-geo
Reply | Threaded
Open this post in threaded view
|

Re: How to reduce the buffering time with the function "buffer" (package raster)

Forrest Stevens-2
Five million cells isn't all that many, how slow is too slow? Buffering a raster of about 10 million cells on my laptop takes on the order of 20 seconds or so for a binary raster. Is it possible that you're fighting memory problems?

In the past when doing multiple ring buffers I've found it faster to calculate a distance-to raster first, and then apply multiple logical comparisons on the distance raster.  I don't know if this applies to your situation or not but it's one trick that might help.

Sincerely,
Forrest

On Mon, Apr 11, 2016 at 10:58 AM Nelly Reduan <[hidden email]> wrote:

Hello,


I would like to build 100-m buffers (representing ecological corridors) around land cover attributes in a raster layer and to assign a given value (i.e., value of 13) to buffer cells.


To do this, I'm using the function "buffer" (package raster) to draw the buffers around particular raster cells (i.e., cell values of 2 and 5). For example, in the image below, the buffers are represented in blue (cell values of 13) and the particular raster cells are represented in yellow (cell values of 2).


The problem is that the time to run the function "buffer" is very low because I have a raster of 5000000 cells. Does anyone know how I can reduce the buffering time ?


Thanks a lot for your time.

Have a nice day.

Nell


Capture.PNG

_______________________________________________
R-sig-Geo mailing list
[hidden email]
https://stat.ethz.ch/mailman/listinfo/r-sig-geo

_______________________________________________
R-sig-Geo mailing list
[hidden email]
https://stat.ethz.ch/mailman/listinfo/r-sig-geo
Reply | Threaded
Open this post in threaded view
|

Re: How to reduce the buffering time with the function "buffer" (package raster)

Nelly Reduan

Thank you very much Forrest for your answer. I don't think I have memory problems (my computer has a 32 GB memory).


Here is my code to draw the 100-m buffers with the function "buffer" (package raster):

r1 <- r ## r is the original raster and has a resolution of 10m
r1[(r1[]!=2 & r1[]!=5)]=NA
plot(r1)
r2 <- buffer(r1, 10)


I also tested the function distance (package raster):

r1 <- r ## r is the original raster and has a resolution of 10m
r1[(r1[]!=2 & r1[]!=5)]=NA
plot(r1)
r3 <- distance(r1)


In the two cases, I stopped the functions afer 5 hours as this was too long ! For the moment, I haven't find solutions.


Thanks a lot for your time.

Have a nice day.

Nell





De : Forrest Stevens <[hidden email]>
Envoyé : lundi 11 avril 2016 09:09
À : Nelly Reduan; [hidden email]
Objet : Re: [R-sig-Geo] How to reduce the buffering time with the function "buffer" (package raster)
 
Five million cells isn't all that many, how slow is too slow? Buffering a raster of about 10 million cells on my laptop takes on the order of 20 seconds or so for a binary raster. Is it possible that you're fighting memory problems?

In the past when doing multiple ring buffers I've found it faster to calculate a distance-to raster first, and then apply multiple logical comparisons on the distance raster.  I don't know if this applies to your situation or not but it's one trick that might help.

Sincerely,
Forrest

On Mon, Apr 11, 2016 at 10:58 AM Nelly Reduan <[hidden email]> wrote:

Hello,


I would like to build 100-m buffers (representing ecological corridors) around land cover attributes in a raster layer and to assign a given value (i.e., value of 13) to buffer cells.


To do this, I'm using the function "buffer" (package raster) to draw the buffers around particular raster cells (i.e., cell values of 2 and 5). For example, in the image below, the buffers are represented in blue (cell values of 13) and the particular raster cells are represented in yellow (cell values of 2).


The problem is that the time to run the function "buffer" is very low because I have a raster of 5000000 cells. Does anyone know how I can reduce the buffering time ?


Thanks a lot for your time.

Have a nice day.

Nell


Capture.PNG

_______________________________________________
R-sig-Geo mailing list
[hidden email]
https://stat.ethz.ch/mailman/listinfo/r-sig-geo

_______________________________________________
R-sig-Geo mailing list
[hidden email]
https://stat.ethz.ch/mailman/listinfo/r-sig-geo
Reply | Threaded
Open this post in threaded view
|

Re: How to reduce the buffering time with the function "buffer" (package raster)

Forrest Stevens-2
And this is for a raster with around 5 milliion cells? Your results surprise me a bit, especially if the bottleneck is at the call to buffer()/distance().  Your best bet is to wait for a response from Robert Hijmans or someone else working on the package, they might have some other tricks up their sleeves. In any case, I agree, 5+ hours is extremely excessive.

Sincerely,
Forrest

On Mon, Apr 11, 2016 at 6:57 PM Nelly Reduan <[hidden email]> wrote:

Thank you very much Forrest for your answer. I don't think I have memory problems (my computer has a 32 GB memory).


Here is my code to draw the 100-m buffers with the function "buffer" (package raster):

r1 <- r ## r is the original raster and has a resolution of 10m
r1[(r1[]!=2 & r1[]!=5)]=NA
plot(r1)
r2 <- buffer(r1, 10)


I also tested the function distance (package raster):

r1 <- r ## r is the original raster and has a resolution of 10m
r1[(r1[]!=2 & r1[]!=5)]=NA
plot(r1)
r3 <- distance(r1)


In the two cases, I stopped the functions afer 5 hours as this was too long ! For the moment, I haven't find solutions.


Thanks a lot for your time.

Have a nice day.

Nell





De : Forrest Stevens <[hidden email]>
Envoyé : lundi 11 avril 2016 09:09
À : Nelly Reduan; [hidden email]
Objet : Re: [R-sig-Geo] How to reduce the buffering time with the function "buffer" (package raster)
 
Five million cells isn't all that many, how slow is too slow? Buffering a raster of about 10 million cells on my laptop takes on the order of 20 seconds or so for a binary raster. Is it possible that you're fighting memory problems?

In the past when doing multiple ring buffers I've found it faster to calculate a distance-to raster first, and then apply multiple logical comparisons on the distance raster.  I don't know if this applies to your situation or not but it's one trick that might help.

Sincerely,
Forrest

On Mon, Apr 11, 2016 at 10:58 AM Nelly Reduan <[hidden email]> wrote:

Hello,


I would like to build 100-m buffers (representing ecological corridors) around land cover attributes in a raster layer and to assign a given value (i.e., value of 13) to buffer cells.


To do this, I'm using the function "buffer" (package raster) to draw the buffers around particular raster cells (i.e., cell values of 2 and 5). For example, in the image below, the buffers are represented in blue (cell values of 13) and the particular raster cells are represented in yellow (cell values of 2).


The problem is that the time to run the function "buffer" is very low because I have a raster of 5000000 cells. Does anyone know how I can reduce the buffering time ?


Thanks a lot for your time.

Have a nice day.

Nell


_______________________________________________
R-sig-Geo mailing list
[hidden email]
https://stat.ethz.ch/mailman/listinfo/r-sig-geo

_______________________________________________
R-sig-Geo mailing list
[hidden email]
https://stat.ethz.ch/mailman/listinfo/r-sig-geo

Capture.PNG (16K) Download Attachment
Capture.PNG (16K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: How to reduce the buffering time with the function "buffer" (package raster)

Nelly Reduan
Yes, I tested the functions "distance" and "buffer" by using a cropped raster with around 5 million cells. But my original raster has 48096864 cells. Ideally, I would like to build the buffers from my original raster. Here are some characteristics of the original raster:


class       : RasterLayer

dimensions  : 4876, 9864, 48096864  (nrow, ncol, ncell)

extent      : 188384.5, 484304.5, 4914481, 5060761  (xmin, xmax, ymin, ymax)

coord. ref. : +proj=tmerc +lat_0=0 +lon_0=-73.5 +k=0.9999 +x_0=304800 +y_0=0 +ellps=GRS80 +towgs84=0,0,0,0,0,0,0 +units=m +no_defs


Thanks a lot for your help.

Have a nice day.

Nell



________________________________
De : Forrest Stevens <[hidden email]>
Envoy� : lundi 11 avril 2016 18:55
� : Nelly Reduan; Forrest Stevens; [hidden email]
Objet : Re: [R-sig-Geo] How to reduce the buffering time with the function "buffer" (package raster)

And this is for a raster with around 5 milliion cells? Your results surprise me a bit, especially if the bottleneck is at the call to buffer()/distance().  Your best bet is to wait for a response from Robert Hijmans or someone else working on the package, they might have some other tricks up their sleeves. In any case, I agree, 5+ hours is extremely excessive.

Sincerely,
Forrest

On Mon, Apr 11, 2016 at 6:57 PM Nelly Reduan <[hidden email]<mailto:[hidden email]>> wrote:

Thank you very much Forrest for your answer. I don't think I have memory problems (my computer has a 32 GB memory).


Here is my code to draw the 100-m buffers with the function "buffer" (package raster):

r1 <- r ## r is the original raster and has a resolution of 10m
r1[(r1[]!=2 & r1[]!=5)]=NA
plot(r1)
r2 <- buffer(r1, 10)


I also tested the function distance (package raster):

r1 <- r ## r is the original raster and has a resolution of 10m
r1[(r1[]!=2 & r1[]!=5)]=NA
plot(r1)
r3 <- distance(r1)


In the two cases, I stopped the functions afer 5 hours as this was too long ! For the moment, I haven't find solutions.


Thanks a lot for your time.

Have a nice day.

Nell



________________________________
De : Forrest Stevens <[hidden email]<mailto:[hidden email]>>
Envoy� : lundi 11 avril 2016 09:09
� : Nelly Reduan; [hidden email]<mailto:[hidden email]>
Objet : Re: [R-sig-Geo] How to reduce the buffering time with the function "buffer" (package raster)

Five million cells isn't all that many, how slow is too slow? Buffering a raster of about 10 million cells on my laptop takes on the order of 20 seconds or so for a binary raster. Is it possible that you're fighting memory problems?

In the past when doing multiple ring buffers I've found it faster to calculate a distance-to raster first, and then apply multiple logical comparisons on the distance raster.  I don't know if this applies to your situation or not but it's one trick that might help.

Sincerely,
Forrest

On Mon, Apr 11, 2016 at 10:58 AM Nelly Reduan <[hidden email]<mailto:[hidden email]>> wrote:

Hello,


I would like to build 100-m buffers (representing ecological corridors) around land cover attributes in a raster layer and to assign a given value (i.e., value of 13) to buffer cells.


To do this, I'm using the function "buffer" (package raster) to draw the buffers around particular raster cells (i.e., cell values of 2 and 5). For example, in the image below, the buffers are represented in blue (cell values of 13) and the particular raster cells are represented in yellow (cell values of 2).


The problem is that the time to run the function "buffer" is very low because I have a raster of 5000000 cells. Does anyone know how I can reduce the buffering time ?


Thanks a lot for your time.

Have a nice day.

Nell


[Capture.PNG]

_______________________________________________
R-sig-Geo mailing list
[hidden email]<mailto:[hidden email]>
https://stat.ethz.ch/mailman/listinfo/r-sig-geo

        [[alternative HTML version deleted]]


_______________________________________________
R-sig-Geo mailing list
[hidden email]
https://stat.ethz.ch/mailman/listinfo/r-sig-geo
Reply | Threaded
Open this post in threaded view
|

Re: How to reduce the buffering time with the function "buffer" (package raster)

Michael Sumner-2
I think we still haven't seen a print of the raster object?

It may be a tiled file on disk? Try readAll to pull into memory, if it is
tiled and read in on demand via gdal tools like extract will be very slow
since access is done line by line.

But just guessing, please give full details and ideally a reproducible
example.

Cheers, Mije

On Wed, 13 Apr 2016, 02:33 Nelly Reduan <[hidden email]> wrote:

> Yes, I tested the functions "distance" and "buffer" by using a cropped
> raster with around 5 million cells. But my original raster has 48096864
> cells. Ideally, I would like to build the buffers from my original raster.
> Here are some characteristics of the original raster:
>
>
> class       : RasterLayer
>
> dimensions  : 4876, 9864, 48096864  (nrow, ncol, ncell)
>
> extent      : 188384.5, 484304.5, 4914481, 5060761  (xmin, xmax, ymin,
> ymax)
>
> coord. ref. : +proj=tmerc +lat_0=0 +lon_0=-73.5 +k=0.9999 +x_0=304800
> +y_0=0 +ellps=GRS80 +towgs84=0,0,0,0,0,0,0 +units=m +no_defs
>
>
> Thanks a lot for your help.
>
> Have a nice day.
>
> Nell
>
>
>
> ________________________________
> De : Forrest Stevens <[hidden email]>
> Envoyé : lundi 11 avril 2016 18:55
> À : Nelly Reduan; Forrest Stevens; [hidden email]
> Objet : Re: [R-sig-Geo] How to reduce the buffering time with the function
> "buffer" (package raster)
>
> And this is for a raster with around 5 milliion cells? Your results
> surprise me a bit, especially if the bottleneck is at the call to
> buffer()/distance().  Your best bet is to wait for a response from Robert
> Hijmans or someone else working on the package, they might have some other
> tricks up their sleeves. In any case, I agree, 5+ hours is extremely
> excessive.
>
> Sincerely,
> Forrest
>
> On Mon, Apr 11, 2016 at 6:57 PM Nelly Reduan <[hidden email]<mailto:
> [hidden email]>> wrote:
>
> Thank you very much Forrest for your answer. I don't think I have memory
> problems (my computer has a 32 GB memory).
>
>
> Here is my code to draw the 100-m buffers with the function "buffer"
> (package raster):
>
> r1 <- r ## r is the original raster and has a resolution of 10m
> r1[(r1[]!=2 & r1[]!=5)]=NA
> plot(r1)
> r2 <- buffer(r1, 10)
>
>
> I also tested the function distance (package raster):
>
> r1 <- r ## r is the original raster and has a resolution of 10m
> r1[(r1[]!=2 & r1[]!=5)]=NA
> plot(r1)
> r3 <- distance(r1)
>
>
> In the two cases, I stopped the functions afer 5 hours as this was too
> long ! For the moment, I haven't find solutions.
>
>
> Thanks a lot for your time.
>
> Have a nice day.
>
> Nell
>
>
>
> ________________________________
> De : Forrest Stevens <[hidden email]<mailto:
> [hidden email]>>
> Envoyé : lundi 11 avril 2016 09:09
> À : Nelly Reduan; [hidden email]<mailto:[hidden email]>
> Objet : Re: [R-sig-Geo] How to reduce the buffering time with the function
> "buffer" (package raster)
>
> Five million cells isn't all that many, how slow is too slow? Buffering a
> raster of about 10 million cells on my laptop takes on the order of 20
> seconds or so for a binary raster. Is it possible that you're fighting
> memory problems?
>
> In the past when doing multiple ring buffers I've found it faster to
> calculate a distance-to raster first, and then apply multiple logical
> comparisons on the distance raster.  I don't know if this applies to your
> situation or not but it's one trick that might help.
>
> Sincerely,
> Forrest
>
> On Mon, Apr 11, 2016 at 10:58 AM Nelly Reduan <[hidden email]
> <mailto:[hidden email]>> wrote:
>
> Hello,
>
>
> I would like to build 100-m buffers (representing ecological corridors)
> around land cover attributes in a raster layer and to assign a given value
> (i.e., value of 13) to buffer cells.
>
>
> To do this, I'm using the function "buffer" (package raster) to draw the
> buffers around particular raster cells (i.e., cell values of 2 and 5). For
> example, in the image below, the buffers are represented in blue (cell
> values of 13) and the particular raster cells are represented in yellow
> (cell values of 2).
>
>
> The problem is that the time to run the function "buffer" is very low
> because I have a raster of 5000000 cells. Does anyone know how I can reduce
> the buffering time ?
>
>
> Thanks a lot for your time.
>
> Have a nice day.
>
> Nell
>
>
> [Capture.PNG]
>
> _______________________________________________
> R-sig-Geo mailing list
> [hidden email]<mailto:[hidden email]>
> https://stat.ethz.ch/mailman/listinfo/r-sig-geo
>
>         [[alternative HTML version deleted]]
>
> _______________________________________________
> R-sig-Geo mailing list
> [hidden email]
> https://stat.ethz.ch/mailman/listinfo/r-sig-geo

--
Dr. Michael Sumner
Software and Database Engineer
Australian Antarctic Division
203 Channel Highway
Kingston Tasmania 7050 Australia

        [[alternative HTML version deleted]]

_______________________________________________
R-sig-Geo mailing list
[hidden email]
https://stat.ethz.ch/mailman/listinfo/r-sig-geo
Reply | Threaded
Open this post in threaded view
|

Re: How to reduce the buffering time with the function "buffer" (package raster)

Nelly Reduan
I put a cropped raster with 3581676 cells in the link below:


https://www.dropbox.com/sh/ewaz4yncdqt7kxj/AAAEHyuWyHzBE7gSINA1oayTa?dl=0


Here is my code to built the buffers:

rf <- clip_buff

rf [(rf != 2 & rf != 3)] <- NA

plot(rf)

system.time(corr <- buffer(rf, 10))

system.time(dist_rf <- distance(rf))


I stopped the code with the functions "buffer" and "distance" after two hours.


Thanks a lot for your time.

Have a nice day.

Nell



________________________________
De : Michael Sumner <[hidden email]>
Envoy� : mardi 12 avril 2016 20:36
� : Nelly Reduan; Forrest Stevens; [hidden email]
Objet : Re: [R-sig-Geo] How to reduce the buffering time with the function "buffer" (package raster)


I think we still haven't seen a print of the raster object?

It may be a tiled file on disk? Try readAll to pull into memory, if it is tiled and read in on demand via gdal tools like extract will be very slow since access is done line by line.

But just guessing, please give full details and ideally a reproducible example.

Cheers, Mije

On Wed, 13 Apr 2016, 02:33 Nelly Reduan <[hidden email]<mailto:[hidden email]>> wrote:
Yes, I tested the functions "distance" and "buffer" by using a cropped raster with around 5 million cells. But my original raster has 48096864 cells. Ideally, I would like to build the buffers from my original raster. Here are some characteristics of the original raster:


class       : RasterLayer

dimensions  : 4876, 9864, 48096864  (nrow, ncol, ncell)

extent      : 188384.5, 484304.5, 4914481, 5060761  (xmin, xmax, ymin, ymax)

coord. ref. : +proj=tmerc +lat_0=0 +lon_0=-73.5 +k=0.9999 +x_0=304800 +y_0=0 +ellps=GRS80 +towgs84=0,0,0,0,0,0,0 +units=m +no_defs


Thanks a lot for your help.

Have a nice day.

Nell



________________________________
De : Forrest Stevens <[hidden email]<mailto:[hidden email]>>
Envoy� : lundi 11 avril 2016 18:55
� : Nelly Reduan; Forrest Stevens; [hidden email]<mailto:[hidden email]>
Objet : Re: [R-sig-Geo] How to reduce the buffering time with the function "buffer" (package raster)

And this is for a raster with around 5 milliion cells? Your results surprise me a bit, especially if the bottleneck is at the call to buffer()/distance().  Your best bet is to wait for a response from Robert Hijmans or someone else working on the package, they might have some other tricks up their sleeves. In any case, I agree, 5+ hours is extremely excessive.

Sincerely,
Forrest

On Mon, Apr 11, 2016 at 6:57 PM Nelly Reduan <[hidden email]<mailto:[hidden email]><mailto:[hidden email]<mailto:[hidden email]>>> wrote:

Thank you very much Forrest for your answer. I don't think I have memory problems (my computer has a 32 GB memory).


Here is my code to draw the 100-m buffers with the function "buffer" (package raster):

r1 <- r ## r is the original raster and has a resolution of 10m
r1[(r1[]!=2 & r1[]!=5)]=NA
plot(r1)
r2 <- buffer(r1, 10)


I also tested the function distance (package raster):

r1 <- r ## r is the original raster and has a resolution of 10m
r1[(r1[]!=2 & r1[]!=5)]=NA
plot(r1)
r3 <- distance(r1)


In the two cases, I stopped the functions afer 5 hours as this was too long ! For the moment, I haven't find solutions.


Thanks a lot for your time.

Have a nice day.

Nell



________________________________
De : Forrest Stevens <[hidden email]<mailto:[hidden email]><mailto:[hidden email]<mailto:[hidden email]>>>
Envoy� : lundi 11 avril 2016 09:09
� : Nelly Reduan; [hidden email]<mailto:[hidden email]><mailto:[hidden email]<mailto:[hidden email]>>
Objet : Re: [R-sig-Geo] How to reduce the buffering time with the function "buffer" (package raster)

Five million cells isn't all that many, how slow is too slow? Buffering a raster of about 10 million cells on my laptop takes on the order of 20 seconds or so for a binary raster. Is it possible that you're fighting memory problems?

In the past when doing multiple ring buffers I've found it faster to calculate a distance-to raster first, and then apply multiple logical comparisons on the distance raster.  I don't know if this applies to your situation or not but it's one trick that might help.

Sincerely,
Forrest

On Mon, Apr 11, 2016 at 10:58 AM Nelly Reduan <[hidden email]<mailto:[hidden email]><mailto:[hidden email]<mailto:[hidden email]>>> wrote:

Hello,


I would like to build 100-m buffers (representing ecological corridors) around land cover attributes in a raster layer and to assign a given value (i.e., value of 13) to buffer cells.


To do this, I'm using the function "buffer" (package raster) to draw the buffers around particular raster cells (i.e., cell values of 2 and 5). For example, in the image below, the buffers are represented in blue (cell values of 13) and the particular raster cells are represented in yellow (cell values of 2).


The problem is that the time to run the function "buffer" is very low because I have a raster of 5000000 cells. Does anyone know how I can reduce the buffering time ?


Thanks a lot for your time.

Have a nice day.

Nell


[Capture.PNG]

_______________________________________________
R-sig-Geo mailing list
[hidden email]<mailto:[hidden email]><mailto:[hidden email]<mailto:[hidden email]>>
https://stat.ethz.ch/mailman/listinfo/r-sig-geo

        [[alternative HTML version deleted]]

_______________________________________________
R-sig-Geo mailing list
[hidden email]<mailto:[hidden email]>
https://stat.ethz.ch/mailman/listinfo/r-sig-geo
--
Dr. Michael Sumner
Software and Database Engineer
Australian Antarctic Division
203 Channel Highway
Kingston Tasmania 7050 Australia


        [[alternative HTML version deleted]]


_______________________________________________
R-sig-Geo mailing list
[hidden email]
https://stat.ethz.ch/mailman/listinfo/r-sig-geo
Reply | Threaded
Open this post in threaded view
|

Re: How to reduce the buffering time with the function "buffer" (package raster)

Forrest Stevens-2
So you're correct, the buffer() and distance() functions you're using will
take a long period of time.  The code below illustrates the shortcut that
I've used in the past when buffering only by the limit of one pixel, since
it uses queens-case connectance and distance between cell-centers based on
that connectance rule. This uses the gridDistance() function which is much
faster than calculating straight line distances. You can see that the
distance-based calculation takes under 40 seconds. But do notice that the
buffer it creates by setting the threshold is around both classes and is
indiscriminate of the nearest class:

library(raster)

clip_buff <- raster("clip_study_area_2010.tif")

system.time(d <- gridDistance(clip_buff, c(2,3)))
#   user  system elapsed
#  37.36   11.00   48.41

rf[(rf != 2 & rf != 3)] <- NA

## Set buffer to 10m just for fun and assign
## value of 4 to that buffered region:
rf[ d > 0 & d <= 10 ] <- 4
plot(rf)

If you want to buffer by more than one pixel dimension then you'll probably
want straight line distances and I'd suggest using GDAL and using the
gdal_proximity utility via a system() call. Alternatively, you could look
into other utilities that have better-optimized raster-based distance
tools. Hopefully this gives you some ideas on how to proceed?

Sincerely,
Forrest


On Wed, Apr 13, 2016 at 9:18 PM Nelly Reduan <[hidden email]> wrote:

> I put a cropped raster with 3581676 cells in the link below:
>
>
> https://www.dropbox.com/sh/ewaz4yncdqt7kxj/AAAEHyuWyHzBE7gSINA1oayTa?dl=0
>
> Here is my code to built the buffers:
>
> rf <- clip_buff
>
> rf [(rf != 2 & rf != 3)] <- NA
>
> plot(rf)
>
> system.time(corr <- buffer(rf, 10))
>
> system.time(dist_rf <- distance(rf))
>
>
> I stopped the code with the functions "buffer" and "distance" after two
> hours.
>
>
> Thanks a lot for your time.
>
> Have a nice day.
>
> Nell
>
>
>
>
> ------------------------------
> *De :* Michael Sumner <[hidden email]>
> *Envoyé :* mardi 12 avril 2016 20:36
>
> *À :* Nelly Reduan; Forrest Stevens; [hidden email]
> *Objet :* Re: [R-sig-Geo] How to reduce the buffering time with the
> function "buffer" (package raster)
>
> I think we still haven't seen a print of the raster object?
>
> It may be a tiled file on disk? Try readAll to pull into memory, if it is
> tiled and read in on demand via gdal tools like extract will be very slow
> since access is done line by line.
>
> But just guessing, please give full details and ideally a reproducible
> example.
>
> Cheers, Mije
>
> On Wed, 13 Apr 2016, 02:33 Nelly Reduan <[hidden email]> wrote:
>
>> Yes, I tested the functions "distance" and "buffer" by using a cropped
>> raster with around 5 million cells. But my original raster has 48096864
>> cells. Ideally, I would like to build the buffers from my original raster.
>> Here are some characteristics of the original raster:
>>
>>
>> class       : RasterLayer
>>
>> dimensions  : 4876, 9864, 48096864  (nrow, ncol, ncell)
>>
>> extent      : 188384.5, 484304.5, 4914481, 5060761  (xmin, xmax, ymin,
>> ymax)
>>
>> coord. ref. : +proj=tmerc +lat_0=0 +lon_0=-73.5 +k=0.9999 +x_0=304800
>> +y_0=0 +ellps=GRS80 +towgs84=0,0,0,0,0,0,0 +units=m +no_defs
>>
>>
>> Thanks a lot for your help.
>>
>> Have a nice day.
>>
>> Nell
>>
>>
>>
>> ________________________________
>> De : Forrest Stevens <[hidden email]>
>> Envoyé : lundi 11 avril 2016 18:55
>> À : Nelly Reduan; Forrest Stevens; [hidden email]
>> Objet : Re: [R-sig-Geo] How to reduce the buffering time with the
>> function "buffer" (package raster)
>>
>> And this is for a raster with around 5 milliion cells? Your results
>> surprise me a bit, especially if the bottleneck is at the call to
>> buffer()/distance().  Your best bet is to wait for a response from Robert
>> Hijmans or someone else working on the package, they might have some other
>> tricks up their sleeves. In any case, I agree, 5+ hours is extremely
>> excessive.
>>
>> Sincerely,
>> Forrest
>>
>> On Mon, Apr 11, 2016 at 6:57 PM Nelly Reduan <[hidden email]
>> <mailto:[hidden email]>> wrote:
>>
>> Thank you very much Forrest for your answer. I don't think I have memory
>> problems (my computer has a 32 GB memory).
>>
>>
>> Here is my code to draw the 100-m buffers with the function "buffer"
>> (package raster):
>>
>> r1 <- r ## r is the original raster and has a resolution of 10m
>> r1[(r1[]!=2 & r1[]!=5)]=NA
>> plot(r1)
>> r2 <- buffer(r1, 10)
>>
>>
>> I also tested the function distance (package raster):
>>
>> r1 <- r ## r is the original raster and has a resolution of 10m
>> r1[(r1[]!=2 & r1[]!=5)]=NA
>> plot(r1)
>> r3 <- distance(r1)
>>
>>
>> In the two cases, I stopped the functions afer 5 hours as this was too
>> long ! For the moment, I haven't find solutions.
>>
>>
>> Thanks a lot for your time.
>>
>> Have a nice day.
>>
>> Nell
>>
>>
>>
>> ________________________________
>> De : Forrest Stevens <[hidden email]<mailto:
>> [hidden email]>>
>> Envoyé : lundi 11 avril 2016 09:09
>> À : Nelly Reduan; [hidden email]<mailto:[hidden email]>
>> Objet : Re: [R-sig-Geo] How to reduce the buffering time with the
>> function "buffer" (package raster)
>>
>> Five million cells isn't all that many, how slow is too slow? Buffering a
>> raster of about 10 million cells on my laptop takes on the order of 20
>> seconds or so for a binary raster. Is it possible that you're fighting
>> memory problems?
>>
>> In the past when doing multiple ring buffers I've found it faster to
>> calculate a distance-to raster first, and then apply multiple logical
>> comparisons on the distance raster.  I don't know if this applies to your
>> situation or not but it's one trick that might help.
>>
>> Sincerely,
>> Forrest
>>
>> On Mon, Apr 11, 2016 at 10:58 AM Nelly Reduan <[hidden email]
>> <mailto:[hidden email]>> wrote:
>>
>> Hello,
>>
>>
>> I would like to build 100-m buffers (representing ecological corridors)
>> around land cover attributes in a raster layer and to assign a given value
>> (i.e., value of 13) to buffer cells.
>>
>>
>> To do this, I'm using the function "buffer" (package raster) to draw the
>> buffers around particular raster cells (i.e., cell values of 2 and 5). For
>> example, in the image below, the buffers are represented in blue (cell
>> values of 13) and the particular raster cells are represented in yellow
>> (cell values of 2).
>>
>>
>> The problem is that the time to run the function "buffer" is very low
>> because I have a raster of 5000000 cells. Does anyone know how I can reduce
>> the buffering time ?
>>
>>
>> Thanks a lot for your time.
>>
>> Have a nice day.
>>
>> Nell
>>
>>
>> [Capture.PNG]
>>
>> _______________________________________________
>> R-sig-Geo mailing list
>> [hidden email]<mailto:[hidden email]>
>> https://stat.ethz.ch/mailman/listinfo/r-sig-geo
>>
>>         [[alternative HTML version deleted]]
>>
>> _______________________________________________
>> R-sig-Geo mailing list
>> [hidden email]
>> https://stat.ethz.ch/mailman/listinfo/r-sig-geo
>
> --
> Dr. Michael Sumner
> Software and Database Engineer
> Australian Antarctic Division
> 203 Channel Highway
> Kingston Tasmania 7050 Australia
>
>

        [[alternative HTML version deleted]]

_______________________________________________
R-sig-Geo mailing list
[hidden email]
https://stat.ethz.ch/mailman/listinfo/r-sig-geo
Reply | Threaded
Open this post in threaded view
|

Re: How to reduce the buffering time with the function "buffer" (package raster)

Nelly Reduan
Thank you very much Forrest for your answer. How can I call the function "gdal_proximity" in R via system() ?


Thanks a lot for your time.

Nell

________________________________
De : Forrest Stevens <[hidden email]>
Envoy� : mercredi 13 avril 2016 22:37:01
� : Nelly Reduan; Michael Sumner; Forrest Stevens; [hidden email]
Objet : Re: [R-sig-Geo] How to reduce the buffering time with the function "buffer" (package raster)

So you're correct, the buffer() and distance() functions you're using will take a long period of time.  The code below illustrates the shortcut that I've used in the past when buffering only by the limit of one pixel, since it uses queens-case connectance and distance between cell-centers based on that connectance rule. This uses the gridDistance() function which is much faster than calculating straight line distances. You can see that the distance-based calculation takes under 40 seconds. But do notice that the buffer it creates by setting the threshold is around both classes and is indiscriminate of the nearest class:

library(raster)

clip_buff <- raster("clip_study_area_2010.tif")

system.time(d <- gridDistance(clip_buff, c(2,3)))
#   user  system elapsed
#  37.36   11.00   48.41

rf[(rf != 2 & rf != 3)] <- NA

## Set buffer to 10m just for fun and assign
## value of 4 to that buffered region:
rf[ d > 0 & d <= 10 ] <- 4
plot(rf)

If you want to buffer by more than one pixel dimension then you'll probably want straight line distances and I'd suggest using GDAL and using the gdal_proximity utility via a system() call. Alternatively, you could look into other utilities that have better-optimized raster-based distance tools. Hopefully this gives you some ideas on how to proceed?

Sincerely,
Forrest


On Wed, Apr 13, 2016 at 9:18 PM Nelly Reduan <[hidden email]<mailto:[hidden email]>> wrote:

I put a cropped raster with 3581676 cells in the link below:


https://www.dropbox.com/sh/ewaz4yncdqt7kxj/AAAEHyuWyHzBE7gSINA1oayTa?dl=0


Here is my code to built the buffers:

rf <- clip_buff

rf [(rf != 2 & rf != 3)] <- NA

plot(rf)

system.time(corr <- buffer(rf, 10))

system.time(dist_rf <- distance(rf))


I stopped the code with the functions "buffer" and "distance" after two hours.


Thanks a lot for your time.

Have a nice day.

Nell



________________________________
De : Michael Sumner <[hidden email]<mailto:[hidden email]>>
Envoy� : mardi 12 avril 2016 20:36

� : Nelly Reduan; Forrest Stevens; [hidden email]<mailto:[hidden email]>
Objet : Re: [R-sig-Geo] How to reduce the buffering time with the function "buffer" (package raster)

I think we still haven't seen a print of the raster object?

It may be a tiled file on disk? Try readAll to pull into memory, if it is tiled and read in on demand via gdal tools like extract will be very slow since access is done line by line.

But just guessing, please give full details and ideally a reproducible example.

Cheers, Mije

On Wed, 13 Apr 2016, 02:33 Nelly Reduan <[hidden email]<mailto:[hidden email]>> wrote:
Yes, I tested the functions "distance" and "buffer" by using a cropped raster with around 5 million cells. But my original raster has 48096864 cells. Ideally, I would like to build the buffers from my original raster. Here are some characteristics of the original raster:


class       : RasterLayer

dimensions  : 4876, 9864, 48096864  (nrow, ncol, ncell)

extent      : 188384.5, 484304.5, 4914481, 5060761  (xmin, xmax, ymin, ymax)

coord. ref. : +proj=tmerc +lat_0=0 +lon_0=-73.5 +k=0.9999 +x_0=304800 +y_0=0 +ellps=GRS80 +towgs84=0,0,0,0,0,0,0 +units=m +no_defs


Thanks a lot for your help.

Have a nice day.

Nell



________________________________
De : Forrest Stevens <[hidden email]<mailto:[hidden email]>>
Envoy� : lundi 11 avril 2016 18:55
� : Nelly Reduan; Forrest Stevens; [hidden email]<mailto:[hidden email]>
Objet : Re: [R-sig-Geo] How to reduce the buffering time with the function "buffer" (package raster)

And this is for a raster with around 5 milliion cells? Your results surprise me a bit, especially if the bottleneck is at the call to buffer()/distance().  Your best bet is to wait for a response from Robert Hijmans or someone else working on the package, they might have some other tricks up their sleeves. In any case, I agree, 5+ hours is extremely excessive.

Sincerely,
Forrest

On Mon, Apr 11, 2016 at 6:57 PM Nelly Reduan <[hidden email]<mailto:[hidden email]><mailto:[hidden email]<mailto:[hidden email]>>> wrote:

Thank you very much Forrest for your answer. I don't think I have memory problems (my computer has a 32 GB memory).


Here is my code to draw the 100-m buffers with the function "buffer" (package raster):

r1 <- r ## r is the original raster and has a resolution of 10m
r1[(r1[]!=2 & r1[]!=5)]=NA
plot(r1)
r2 <- buffer(r1, 10)


I also tested the function distance (package raster):

r1 <- r ## r is the original raster and has a resolution of 10m
r1[(r1[]!=2 & r1[]!=5)]=NA
plot(r1)
r3 <- distance(r1)


In the two cases, I stopped the functions afer 5 hours as this was too long ! For the moment, I haven't find solutions.


Thanks a lot for your time.

Have a nice day.

Nell



________________________________
De : Forrest Stevens <[hidden email]<mailto:[hidden email]><mailto:[hidden email]<mailto:[hidden email]>>>
Envoy� : lundi 11 avril 2016 09:09
� : Nelly Reduan; [hidden email]<mailto:[hidden email]><mailto:[hidden email]<mailto:[hidden email]>>
Objet : Re: [R-sig-Geo] How to reduce the buffering time with the function "buffer" (package raster)

Five million cells isn't all that many, how slow is too slow? Buffering a raster of about 10 million cells on my laptop takes on the order of 20 seconds or so for a binary raster. Is it possible that you're fighting memory problems?

In the past when doing multiple ring buffers I've found it faster to calculate a distance-to raster first, and then apply multiple logical comparisons on the distance raster.  I don't know if this applies to your situation or not but it's one trick that might help.

Sincerely,
Forrest

On Mon, Apr 11, 2016 at 10:58 AM Nelly Reduan <[hidden email]<mailto:[hidden email]><mailto:[hidden email]<mailto:[hidden email]>>> wrote:

Hello,


I would like to build 100-m buffers (representing ecological corridors) around land cover attributes in a raster layer and to assign a given value (i.e., value of 13) to buffer cells.


To do this, I'm using the function "buffer" (package raster) to draw the buffers around particular raster cells (i.e., cell values of 2 and 5). For example, in the image below, the buffers are represented in blue (cell values of 13) and the particular raster cells are represented in yellow (cell values of 2).


The problem is that the time to run the function "buffer" is very low because I have a raster of 5000000 cells. Does anyone know how I can reduce the buffering time ?


Thanks a lot for your time.

Have a nice day.

Nell


[Capture.PNG]

_______________________________________________
R-sig-Geo mailing list
[hidden email]<mailto:[hidden email]><mailto:[hidden email]<mailto:[hidden email]>>
https://stat.ethz.ch/mailman/listinfo/r-sig-geo

        [[alternative HTML version deleted]]

_______________________________________________
R-sig-Geo mailing list
[hidden email]<mailto:[hidden email]>
https://stat.ethz.ch/mailman/listinfo/r-sig-geo
--
Dr. Michael Sumner
Software and Database Engineer
Australian Antarctic Division
203 Channel Highway
Kingston Tasmania 7050 Australia


        [[alternative HTML version deleted]]


_______________________________________________
R-sig-Geo mailing list
[hidden email]
https://stat.ethz.ch/mailman/listinfo/r-sig-geo
Reply | Threaded
Open this post in threaded view
|

Re: How to reduce the buffering time with the function "buffer" (package raster)

Nelly Reduan
I have tried the following code to compute distances from raster pixels of 5 via the function "gdal_proximity":


    pypath <- Sys.which("C:\\OSGeo4W64\\bin\\gdal_proximity")
    owd <- getwd()
    on.exit(setwd(owd))
    setwd(dirname(pypath))
    rastpath <- "H:\\Project\\grassland.tif"
    outpath <- "H:\\Project\\distance.shp"
    maxDist <- 50
    system2("C:\\OSGeo4W64\\OSGeo4W.bat", args=c(pypath,rastpath,outpath,maxDist))


But I obtain the error message:

    gdal_proximity.py srcfile dstfile [-srcband n] [-dstband n]
                      [-of format] [-co name=value]*
                      [-ot Byte/Int16/Int32/Float32/etc]
                      [-values n,n,n] [-distunits PIXEL/GEO]
                      [-maxdist n] [-nodata n] [-fixed-buf-val n] [-q]
    Warning message:
    running command '"C:\OSGeo4W64\OSGeo4W.bat" C:\OSGEO4~1\bin\GDAL_P~2.BAT H:\Project\grassland.tif H:\Project\distance.shp 50' had status 1


Here are some details about the raster "grassland.tif"

    > grassland
    class       : RasterLayer
    dimensions  : 9887, 30532, 301869884  (nrow, ncol, ncell)
    resolution  : 10, 10  (x, y)
    extent      : -515893.7, -210573.7, 110739.6, 209609.6  (xmin, xmax, ymin, ymax)
    coord. ref. : +proj=lcc +lat_1=46 +lat_2=60 +lat_0=44 +lon_0=-68.5 +x_0=0 +y_0=0 +ellps=GRS80 +towgs84=0,0,0,0,0,0,0 +units=m +no_defs
    values      : 5, 5  (min, max)


Thanks a lot for your time.
Have a nice day
Nell


________________________________
De : Nelly Reduan
Envoy� : vendredi 6 mai 2016 12:00:36
� : Forrest Stevens; Michael Sumner; [hidden email]
Objet : RE: [R-sig-Geo] How to reduce the buffering time with the function "buffer" (package raster)


Thank you very much Forrest for your answer. How can I call the function "gdal_proximity" in R via system() ?


Thanks a lot for your time.

Nell

________________________________
De : Forrest Stevens <[hidden email]>
Envoy� : mercredi 13 avril 2016 22:37:01
� : Nelly Reduan; Michael Sumner; Forrest Stevens; [hidden email]
Objet : Re: [R-sig-Geo] How to reduce the buffering time with the function "buffer" (package raster)

So you're correct, the buffer() and distance() functions you're using will take a long period of time.  The code below illustrates the shortcut that I've used in the past when buffering only by the limit of one pixel, since it uses queens-case connectance and distance between cell-centers based on that connectance rule. This uses the gridDistance() function which is much faster than calculating straight line distances. You can see that the distance-based calculation takes under 40 seconds. But do notice that the buffer it creates by setting the threshold is around both classes and is indiscriminate of the nearest class:

library(raster)

clip_buff <- raster("clip_study_area_2010.tif")

system.time(d <- gridDistance(clip_buff, c(2,3)))
#   user  system elapsed
#  37.36   11.00   48.41

rf[(rf != 2 & rf != 3)] <- NA

## Set buffer to 10m just for fun and assign
## value of 4 to that buffered region:
rf[ d > 0 & d <= 10 ] <- 4
plot(rf)

If you want to buffer by more than one pixel dimension then you'll probably want straight line distances and I'd suggest using GDAL and using the gdal_proximity utility via a system() call. Alternatively, you could look into other utilities that have better-optimized raster-based distance tools. Hopefully this gives you some ideas on how to proceed?

Sincerely,
Forrest


On Wed, Apr 13, 2016 at 9:18 PM Nelly Reduan <[hidden email]<mailto:[hidden email]>> wrote:

I put a cropped raster with 3581676 cells in the link below:


https://www.dropbox.com/sh/ewaz4yncdqt7kxj/AAAEHyuWyHzBE7gSINA1oayTa?dl=0


Here is my code to built the buffers:

rf <- clip_buff

rf [(rf != 2 & rf != 3)] <- NA

plot(rf)

system.time(corr <- buffer(rf, 10))

system.time(dist_rf <- distance(rf))


I stopped the code with the functions "buffer" and "distance" after two hours.


Thanks a lot for your time.

Have a nice day.

Nell



________________________________
De : Michael Sumner <[hidden email]<mailto:[hidden email]>>
Envoy� : mardi 12 avril 2016 20:36

� : Nelly Reduan; Forrest Stevens; [hidden email]<mailto:[hidden email]>
Objet : Re: [R-sig-Geo] How to reduce the buffering time with the function "buffer" (package raster)

I think we still haven't seen a print of the raster object?

It may be a tiled file on disk? Try readAll to pull into memory, if it is tiled and read in on demand via gdal tools like extract will be very slow since access is done line by line.

But just guessing, please give full details and ideally a reproducible example.

Cheers, Mije

On Wed, 13 Apr 2016, 02:33 Nelly Reduan <[hidden email]<mailto:[hidden email]>> wrote:
Yes, I tested the functions "distance" and "buffer" by using a cropped raster with around 5 million cells. But my original raster has 48096864 cells. Ideally, I would like to build the buffers from my original raster. Here are some characteristics of the original raster:


class       : RasterLayer

dimensions  : 4876, 9864, 48096864  (nrow, ncol, ncell)

extent      : 188384.5, 484304.5, 4914481, 5060761  (xmin, xmax, ymin, ymax)

coord. ref. : +proj=tmerc +lat_0=0 +lon_0=-73.5 +k=0.9999 +x_0=304800 +y_0=0 +ellps=GRS80 +towgs84=0,0,0,0,0,0,0 +units=m +no_defs


Thanks a lot for your help.

Have a nice day.

Nell



________________________________
De : Forrest Stevens <[hidden email]<mailto:[hidden email]>>
Envoy� : lundi 11 avril 2016 18:55
� : Nelly Reduan; Forrest Stevens; [hidden email]<mailto:[hidden email]>
Objet : Re: [R-sig-Geo] How to reduce the buffering time with the function "buffer" (package raster)

And this is for a raster with around 5 milliion cells? Your results surprise me a bit, especially if the bottleneck is at the call to buffer()/distance().  Your best bet is to wait for a response from Robert Hijmans or someone else working on the package, they might have some other tricks up their sleeves. In any case, I agree, 5+ hours is extremely excessive.

Sincerely,
Forrest

On Mon, Apr 11, 2016 at 6:57 PM Nelly Reduan <[hidden email]<mailto:[hidden email]><mailto:[hidden email]<mailto:[hidden email]>>> wrote:

Thank you very much Forrest for your answer. I don't think I have memory problems (my computer has a 32 GB memory).


Here is my code to draw the 100-m buffers with the function "buffer" (package raster):

r1 <- r ## r is the original raster and has a resolution of 10m
r1[(r1[]!=2 & r1[]!=5)]=NA
plot(r1)
r2 <- buffer(r1, 10)


I also tested the function distance (package raster):

r1 <- r ## r is the original raster and has a resolution of 10m
r1[(r1[]!=2 & r1[]!=5)]=NA
plot(r1)
r3 <- distance(r1)


In the two cases, I stopped the functions afer 5 hours as this was too long ! For the moment, I haven't find solutions.


Thanks a lot for your time.

Have a nice day.

Nell



________________________________
De : Forrest Stevens <[hidden email]<mailto:[hidden email]><mailto:[hidden email]<mailto:[hidden email]>>>
Envoy� : lundi 11 avril 2016 09:09
� : Nelly Reduan; [hidden email]<mailto:[hidden email]><mailto:[hidden email]<mailto:[hidden email]>>
Objet : Re: [R-sig-Geo] How to reduce the buffering time with the function "buffer" (package raster)

Five million cells isn't all that many, how slow is too slow? Buffering a raster of about 10 million cells on my laptop takes on the order of 20 seconds or so for a binary raster. Is it possible that you're fighting memory problems?

In the past when doing multiple ring buffers I've found it faster to calculate a distance-to raster first, and then apply multiple logical comparisons on the distance raster.  I don't know if this applies to your situation or not but it's one trick that might help.

Sincerely,
Forrest

On Mon, Apr 11, 2016 at 10:58 AM Nelly Reduan <[hidden email]<mailto:[hidden email]><mailto:[hidden email]<mailto:[hidden email]>>> wrote:

Hello,


I would like to build 100-m buffers (representing ecological corridors) around land cover attributes in a raster layer and to assign a given value (i.e., value of 13) to buffer cells.


To do this, I'm using the function "buffer" (package raster) to draw the buffers around particular raster cells (i.e., cell values of 2 and 5). For example, in the image below, the buffers are represented in blue (cell values of 13) and the particular raster cells are represented in yellow (cell values of 2).


The problem is that the time to run the function "buffer" is very low because I have a raster of 5000000 cells. Does anyone know how I can reduce the buffering time ?


Thanks a lot for your time.

Have a nice day.

Nell


[Capture.PNG]

_______________________________________________
R-sig-Geo mailing list
[hidden email]<mailto:[hidden email]><mailto:[hidden email]<mailto:[hidden email]>>
https://stat.ethz.ch/mailman/listinfo/r-sig-geo

        [[alternative HTML version deleted]]

_______________________________________________
R-sig-Geo mailing list
[hidden email]<mailto:[hidden email]>
https://stat.ethz.ch/mailman/listinfo/r-sig-geo
--
Dr. Michael Sumner
Software and Database Engineer
Australian Antarctic Division
203 Channel Highway
Kingston Tasmania 7050 Australia


        [[alternative HTML version deleted]]


_______________________________________________
R-sig-Geo mailing list
[hidden email]
https://stat.ethz.ch/mailman/listinfo/r-sig-geo
Reply | Threaded
Open this post in threaded view
|

Re: How to reduce the buffering time with the function "buffer" (package raster)

Alexander Brown
I've attempted to use "OSGeo4W.bat" and the like, and have also seen such
error messages.

Shell functions like "OSGeo4W.bat" are transcriptions of Linux/UNIX shell
scripts and can't be expected to work in a Windows environment, and are a
very low priority in open source spatial software, which have many more
important problems.  (Google "OSGeo4W.bat" to see a variety of problem
reports.)  It's well worth the effort of assembling a Linux environment and
learning to use it, for any important problems to be solved with open
source tools, including GDAL.

>From https://translate.google.com,

Les fonctions "Shell" comme "OSGeo4W.bat" sont des transcriptions de Linux
/ UNIX ''shell scripts" -- ne peuvent pas être appelés à travailler dans un
environnement Windows, et sont une priorité très faible en logiciel open
source spatiale, qui ont des problèmes beaucoup plus importants. (Google
"OSGeo4W.bat" pour voir une variété de rapports de problèmes.) Il vaut bien
l'effort d'assemblage d'un environnement Linux et d'apprendre à l'utiliser,
pour tous les problèmes importants à résoudre avec des outils open source,
y compris GDAL.

Sorry, best I can do.  Good luck!

On Mon, May 9, 2016 at 1:36 PM, Nelly Reduan <[hidden email]> wrote:

> I have tried the following code to compute distances from raster pixels of
> 5 via the function "gdal_proximity":
>
>
>     pypath <- Sys.which("C:\\OSGeo4W64\\bin\\gdal_proximity")
>     owd <- getwd()
>     on.exit(setwd(owd))
>     setwd(dirname(pypath))
>     rastpath <- "H:\\Project\\grassland.tif"
>     outpath <- "H:\\Project\\distance.shp"
>     maxDist <- 50
>     system2("C:\\OSGeo4W64\\OSGeo4W.bat",
> args=c(pypath,rastpath,outpath,maxDist))
>
>
> But I obtain the error message:
>
>     gdal_proximity.py srcfile dstfile [-srcband n] [-dstband n]
>                       [-of format] [-co name=value]*
>                       [-ot Byte/Int16/Int32/Float32/etc]
>                       [-values n,n,n] [-distunits PIXEL/GEO]
>                       [-maxdist n] [-nodata n] [-fixed-buf-val n] [-q]
>     Warning message:
>     running command '"C:\OSGeo4W64\OSGeo4W.bat"
> C:\OSGEO4~1\bin\GDAL_P~2.BAT H:\Project\grassland.tif
> H:\Project\distance.shp 50' had status 1
>
>
> Here are some details about the raster "grassland.tif"
>
>     > grassland
>     class       : RasterLayer
>     dimensions  : 9887, 30532, 301869884  (nrow, ncol, ncell)
>     resolution  : 10, 10  (x, y)
>     extent      : -515893.7, -210573.7, 110739.6, 209609.6  (xmin, xmax,
> ymin, ymax)
>     coord. ref. : +proj=lcc +lat_1=46 +lat_2=60 +lat_0=44 +lon_0=-68.5
> +x_0=0 +y_0=0 +ellps=GRS80 +towgs84=0,0,0,0,0,0,0 +units=m +no_defs
>     values      : 5, 5  (min, max)
>
>
> Thanks a lot for your time.
> Have a nice day
> Nell
>
>
> ________________________________
> De : Nelly Reduan
> Envoyé : vendredi 6 mai 2016 12:00:36
> À : Forrest Stevens; Michael Sumner; [hidden email]
> Objet : RE: [R-sig-Geo] How to reduce the buffering time with the function
> "buffer" (package raster)
>
>
> Thank you very much Forrest for your answer. How can I call the function
> "gdal_proximity" in R via system() ?
>
>
> Thanks a lot for your time.
>
> Nell
>
> ________________________________
> De : Forrest Stevens <[hidden email]>
> Envoyé : mercredi 13 avril 2016 22:37:01
> À : Nelly Reduan; Michael Sumner; Forrest Stevens; [hidden email]
> Objet : Re: [R-sig-Geo] How to reduce the buffering time with the function
> "buffer" (package raster)
>
> So you're correct, the buffer() and distance() functions you're using will
> take a long period of time.  The code below illustrates the shortcut that
> I've used in the past when buffering only by the limit of one pixel, since
> it uses queens-case connectance and distance between cell-centers based on
> that connectance rule. This uses the gridDistance() function which is much
> faster than calculating straight line distances. You can see that the
> distance-based calculation takes under 40 seconds. But do notice that the
> buffer it creates by setting the threshold is around both classes and is
> indiscriminate of the nearest class:
>
> library(raster)
>
> clip_buff <- raster("clip_study_area_2010.tif")
>
> system.time(d <- gridDistance(clip_buff, c(2,3)))
> #   user  system elapsed
> #  37.36   11.00   48.41
>
> rf[(rf != 2 & rf != 3)] <- NA
>
> ## Set buffer to 10m just for fun and assign
> ## value of 4 to that buffered region:
> rf[ d > 0 & d <= 10 ] <- 4
> plot(rf)
>
> If you want to buffer by more than one pixel dimension then you'll
> probably want straight line distances and I'd suggest using GDAL and using
> the gdal_proximity utility via a system() call. Alternatively, you could
> look into other utilities that have better-optimized raster-based distance
> tools. Hopefully this gives you some ideas on how to proceed?
>
> Sincerely,
> Forrest
>
>
> On Wed, Apr 13, 2016 at 9:18 PM Nelly Reduan <[hidden email]<mailto:
> [hidden email]>> wrote:
>
> I put a cropped raster with 3581676 cells in the link below:
>
>
> https://www.dropbox.com/sh/ewaz4yncdqt7kxj/AAAEHyuWyHzBE7gSINA1oayTa?dl=0
>
>
> Here is my code to built the buffers:
>
> rf <- clip_buff
>
> rf [(rf != 2 & rf != 3)] <- NA
>
> plot(rf)
>
> system.time(corr <- buffer(rf, 10))
>
> system.time(dist_rf <- distance(rf))
>
>
> I stopped the code with the functions "buffer" and "distance" after two
> hours.
>
>
> Thanks a lot for your time.
>
> Have a nice day.
>
> Nell
>
>
>
> ________________________________
> De : Michael Sumner <[hidden email]<mailto:[hidden email]>>
> Envoyé : mardi 12 avril 2016 20:36
>
> À : Nelly Reduan; Forrest Stevens; [hidden email]<mailto:
> [hidden email]>
> Objet : Re: [R-sig-Geo] How to reduce the buffering time with the function
> "buffer" (package raster)
>
> I think we still haven't seen a print of the raster object?
>
> It may be a tiled file on disk? Try readAll to pull into memory, if it is
> tiled and read in on demand via gdal tools like extract will be very slow
> since access is done line by line.
>
> But just guessing, please give full details and ideally a reproducible
> example.
>
> Cheers, Mije
>
> On Wed, 13 Apr 2016, 02:33 Nelly Reduan <[hidden email]<mailto:
> [hidden email]>> wrote:
> Yes, I tested the functions "distance" and "buffer" by using a cropped
> raster with around 5 million cells. But my original raster has 48096864
> cells. Ideally, I would like to build the buffers from my original raster.
> Here are some characteristics of the original raster:
>
>
> class       : RasterLayer
>
> dimensions  : 4876, 9864, 48096864  (nrow, ncol, ncell)
>
> extent      : 188384.5, 484304.5, 4914481, 5060761  (xmin, xmax, ymin,
> ymax)
>
> coord. ref. : +proj=tmerc +lat_0=0 +lon_0=-73.5 +k=0.9999 +x_0=304800
> +y_0=0 +ellps=GRS80 +towgs84=0,0,0,0,0,0,0 +units=m +no_defs
>
>
> Thanks a lot for your help.
>
> Have a nice day.
>
> Nell
>
>
>
> ________________________________
> De : Forrest Stevens <[hidden email]<mailto:
> [hidden email]>>
> Envoyé : lundi 11 avril 2016 18:55
> À : Nelly Reduan; Forrest Stevens; [hidden email]<mailto:
> [hidden email]>
> Objet : Re: [R-sig-Geo] How to reduce the buffering time with the function
> "buffer" (package raster)
>
> And this is for a raster with around 5 milliion cells? Your results
> surprise me a bit, especially if the bottleneck is at the call to
> buffer()/distance().  Your best bet is to wait for a response from Robert
> Hijmans or someone else working on the package, they might have some other
> tricks up their sleeves. In any case, I agree, 5+ hours is extremely
> excessive.
>
> Sincerely,
> Forrest
>
> On Mon, Apr 11, 2016 at 6:57 PM Nelly Reduan <[hidden email]<mailto:
> [hidden email]><mailto:[hidden email]<mailto:
> [hidden email]>>> wrote:
>
> Thank you very much Forrest for your answer. I don't think I have memory
> problems (my computer has a 32 GB memory).
>
>
> Here is my code to draw the 100-m buffers with the function "buffer"
> (package raster):
>
> r1 <- r ## r is the original raster and has a resolution of 10m
> r1[(r1[]!=2 & r1[]!=5)]=NA
> plot(r1)
> r2 <- buffer(r1, 10)
>
>
> I also tested the function distance (package raster):
>
> r1 <- r ## r is the original raster and has a resolution of 10m
> r1[(r1[]!=2 & r1[]!=5)]=NA
> plot(r1)
> r3 <- distance(r1)
>
>
> In the two cases, I stopped the functions afer 5 hours as this was too
> long ! For the moment, I haven't find solutions.
>
>
> Thanks a lot for your time.
>
> Have a nice day.
>
> Nell
>
>
>
> ________________________________
> De : Forrest Stevens <[hidden email]<mailto:
> [hidden email]><mailto:[hidden email]<mailto:
> [hidden email]>>>
> Envoyé : lundi 11 avril 2016 09:09
> À : Nelly Reduan; [hidden email]<mailto:[hidden email]
> ><mailto:[hidden email]<mailto:[hidden email]>>
> Objet : Re: [R-sig-Geo] How to reduce the buffering time with the function
> "buffer" (package raster)
>
> Five million cells isn't all that many, how slow is too slow? Buffering a
> raster of about 10 million cells on my laptop takes on the order of 20
> seconds or so for a binary raster. Is it possible that you're fighting
> memory problems?
>
> In the past when doing multiple ring buffers I've found it faster to
> calculate a distance-to raster first, and then apply multiple logical
> comparisons on the distance raster.  I don't know if this applies to your
> situation or not but it's one trick that might help.
>
> Sincerely,
> Forrest
>
> On Mon, Apr 11, 2016 at 10:58 AM Nelly Reduan <[hidden email]
> <mailto:[hidden email]><mailto:[hidden email]<mailto:
> [hidden email]>>> wrote:
>
> Hello,
>
>
> I would like to build 100-m buffers (representing ecological corridors)
> around land cover attributes in a raster layer and to assign a given value
> (i.e., value of 13) to buffer cells.
>
>
> To do this, I'm using the function "buffer" (package raster) to draw the
> buffers around particular raster cells (i.e., cell values of 2 and 5). For
> example, in the image below, the buffers are represented in blue (cell
> values of 13) and the particular raster cells are represented in yellow
> (cell values of 2).
>
>
> The problem is that the time to run the function "buffer" is very low
> because I have a raster of 5000000 cells. Does anyone know how I can reduce
> the buffering time ?
>
>
> Thanks a lot for your time.
>
> Have a nice day.
>
> Nell
>
>
> [Capture.PNG]
>
> _______________________________________________
> R-sig-Geo mailing list
> [hidden email]<mailto:[hidden email]><mailto:
> [hidden email]<mailto:[hidden email]>>
> https://stat.ethz.ch/mailman/listinfo/r-sig-geo
>
>         [[alternative HTML version deleted]]
>
> _______________________________________________
> R-sig-Geo mailing list
> [hidden email]<mailto:[hidden email]>
> https://stat.ethz.ch/mailman/listinfo/r-sig-geo
> --
> Dr. Michael Sumner
> Software and Database Engineer
> Australian Antarctic Division
> 203 Channel Highway
> Kingston Tasmania 7050 Australia
>
>
>         [[alternative HTML version deleted]]
>
>
> _______________________________________________
> R-sig-Geo mailing list
> [hidden email]
> https://stat.ethz.ch/mailman/listinfo/r-sig-geo
>

        [[alternative HTML version deleted]]

_______________________________________________
R-sig-Geo mailing list
[hidden email]
https://stat.ethz.ch/mailman/listinfo/r-sig-geo
Reply | Threaded
Open this post in threaded view
|

Re: How to reduce the buffering time with the function "buffer" (package raster)

Alex Mandel-2
In reply to this post by Nelly Reduan
On 05/09/2016 10:36 AM, Nelly Reduan wrote:
> But I obtain the error message:
>
>     gdal_proximity.py srcfile dstfile [-srcband n] [-dstband n]
>                       [-of format] [-co name=value]*
>                       [-ot Byte/Int16/Int32/Float32/etc]
>                       [-values n,n,n] [-distunits PIXEL/GEO]
>                       [-maxdist n] [-nodata n] [-fixed-buf-val n] [-q]
>     Warning message:
>     running command '"C:\OSGeo4W64\OSGeo4W.bat" C:\OSGEO4~1\bin\GDAL_P~2.BAT H:\Project\grassland.tif H:\Project\distance.shp 50' had status 1

I would test the command directly on the osgeo4w shell first to figure
out what's wrong. Note, I think that command outputs a raster not a shp.

Thanks,
Alex

_______________________________________________
R-sig-Geo mailing list
[hidden email]
https://stat.ethz.ch/mailman/listinfo/r-sig-geo