Long time to load shapefiles

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

Long time to load shapefiles

Filoche
Hi everyone.
I'm trying to open some shapefiles, but the process is still not complete after 25 minutes.  Looking at the shapefiles in ArcGIS, I find that these files have roughly 500K entries. I was wondering if this processing time to open the shapefiles is normal. Here is my configuration:
Intel i5-2500 @ 3.3 GHz16 GB of RAMWindows 7 (64 bits)
For information, you can download the shapefiles here:
http://www.usna.edu/Users/oceano/pguth/srtm/rivers.zip
Here is my code:
library(rgdal)shape=readOGR("af_riv_15s.shp", layer="af_riv_15s")

Thank you for your help,Phil



     
        [[alternative HTML version deleted]]

_______________________________________________
R-sig-Geo mailing list
[hidden email]
https://stat.ethz.ch/mailman/listinfo/r-sig-geo
GD
Reply | Threaded
Open this post in threaded view
|

Re: Long time to load shapefiles

GD
Hi Phil,

I have also found readOGR to be very slow when reading a large shapefile (in my case, a shapefile with 85000 polygons and ~ 200 columns in the attribute table).

In my case, I would repeatedly re-read the shapefile each time I was working on the script.

To speed things up, I found it best to use readOGR to get the file into R the first time, but then save the resulting object as an RDS file (using saveRDS). From then on, you can load the object with readRDS. In my case, that would happen in about 10s, whereas it might take 20min for readOGR to directly read the shapefile.

Cheers,
Gareth.  
Reply | Threaded
Open this post in threaded view
|

Re: Long time to load shapefiles

Michael Sumner-2
In reply to this post by Filoche
I have a similar spec to yours, with i7 instead.

I have successfully read it, but first time it crashed (I think
because memory was exhausted).

It took about 20min.

I would work on subsetting the part you want with an ogr2ogr
incantation if you want to stay away from GIS:
http://www.gdal.org/ogr2ogr.html

QGIS or similar would be able to read / crop this no-sweat, but it
depends on exactly how you want to subset it, and that may be a topic
for another forum

Cheers, Mike

On Thu, Oct 17, 2013 at 2:18 AM, philippe massicotte
<[hidden email]> wrote:

> Hi everyone.
> I'm trying to open some shapefiles, but the process is still not complete after 25 minutes.  Looking at the shapefiles in ArcGIS, I find that these files have roughly 500K entries. I was wondering if this processing time to open the shapefiles is normal. Here is my configuration:
> Intel i5-2500 @ 3.3 GHz16 GB of RAMWindows 7 (64 bits)
> For information, you can download the shapefiles here:
> http://www.usna.edu/Users/oceano/pguth/srtm/rivers.zip
> Here is my code:
> library(rgdal)shape=readOGR("af_riv_15s.shp", layer="af_riv_15s")
>
> Thank you for your help,Phil
>
>
>
>
>         [[alternative HTML version deleted]]
>
> _______________________________________________
> R-sig-Geo mailing list
> [hidden email]
> https://stat.ethz.ch/mailman/listinfo/r-sig-geo



--
Michael Sumner
Hobart, Australia
e-mail: [hidden email]

_______________________________________________
R-sig-Geo mailing list
[hidden email]
https://stat.ethz.ch/mailman/listinfo/r-sig-geo
Reply | Threaded
Open this post in threaded view
|

Re: Long time to load shapefiles

Filoche
Thank you everyone for your answers. I think I'll just load it and save the workspace for further uses. I still wonder why this is taking such long time to load whereas in QGIS or ArcGIS it is a matter of seconds.
Have a great day,Phil

> Date: Thu, 17 Oct 2013 17:19:39 +1100
> Subject: Re: [R-sig-Geo] Long time to load shapefiles
> From: [hidden email]
> To: [hidden email]
> CC: [hidden email]
>
> I have a similar spec to yours, with i7 instead.
>
> I have successfully read it, but first time it crashed (I think
> because memory was exhausted).
>
> It took about 20min.
>
> I would work on subsetting the part you want with an ogr2ogr
> incantation if you want to stay away from GIS:
> http://www.gdal.org/ogr2ogr.html
>
> QGIS or similar would be able to read / crop this no-sweat, but it
> depends on exactly how you want to subset it, and that may be a topic
> for another forum
>
> Cheers, Mike
>
> On Thu, Oct 17, 2013 at 2:18 AM, philippe massicotte
> <[hidden email]> wrote:
> > Hi everyone.
> > I'm trying to open some shapefiles, but the process is still not complete after 25 minutes.  Looking at the shapefiles in ArcGIS, I find that these files have roughly 500K entries. I was wondering if this processing time to open the shapefiles is normal. Here is my configuration:
> > Intel i5-2500 @ 3.3 GHz16 GB of RAMWindows 7 (64 bits)
> > For information, you can download the shapefiles here:
> > http://www.usna.edu/Users/oceano/pguth/srtm/rivers.zip
> > Here is my code:
> > library(rgdal)shape=readOGR("af_riv_15s.shp", layer="af_riv_15s")
> >
> > Thank you for your help,Phil
> >
> >
> >
> >
> >         [[alternative HTML version deleted]]
> >
> > _______________________________________________
> > R-sig-Geo mailing list
> > [hidden email]
> > https://stat.ethz.ch/mailman/listinfo/r-sig-geo
>
>
>
> --
> Michael Sumner
> Hobart, Australia
> e-mail: [hidden email]
     
        [[alternative HTML version deleted]]

_______________________________________________
R-sig-Geo mailing list
[hidden email]
https://stat.ethz.ch/mailman/listinfo/r-sig-geo
Reply | Threaded
Open this post in threaded view
|

Re: Long time to load shapefiles

Roger Bivand
Administrator
On Thu, 17 Oct 2013, philippe massicotte wrote:

> Thank you everyone for your answers. I think I'll just load it and save
> the workspace for further uses. I still wonder why this is taking such
> long time to load whereas in QGIS or ArcGIS it is a matter of seconds.

Why do you wonder? You could check instead, taking subsets of the fields,
and subsets of the features in your object. You could also use alternative
formats, but I don't think that the choice of driver is important here.
Note that few statisticians would consider global analysis of data sets
this large sensible unless very specific conditions were met; the
additional variability after the first several thousand observations
probably doesn't tell you much. Once the data are converted to sp classes,
things go rather faster, as has been pointed out.

Roger

> Have a great day,Phil
>
>> Date: Thu, 17 Oct 2013 17:19:39 +1100
>> Subject: Re: [R-sig-Geo] Long time to load shapefiles
>> From: [hidden email]
>> To: [hidden email]
>> CC: [hidden email]
>>
>> I have a similar spec to yours, with i7 instead.
>>
>> I have successfully read it, but first time it crashed (I think
>> because memory was exhausted).
>>
>> It took about 20min.
>>
>> I would work on subsetting the part you want with an ogr2ogr
>> incantation if you want to stay away from GIS:
>> http://www.gdal.org/ogr2ogr.html
>>
>> QGIS or similar would be able to read / crop this no-sweat, but it
>> depends on exactly how you want to subset it, and that may be a topic
>> for another forum
>>
>> Cheers, Mike
>>
>> On Thu, Oct 17, 2013 at 2:18 AM, philippe massicotte
>> <[hidden email]> wrote:
>>> Hi everyone.
>>> I'm trying to open some shapefiles, but the process is still not complete after 25 minutes.  Looking at the shapefiles in ArcGIS, I find that these files have roughly 500K entries. I was wondering if this processing time to open the shapefiles is normal. Here is my configuration:
>>> Intel i5-2500 @ 3.3 GHz16 GB of RAMWindows 7 (64 bits)
>>> For information, you can download the shapefiles here:
>>> http://www.usna.edu/Users/oceano/pguth/srtm/rivers.zip
>>> Here is my code:
>>> library(rgdal)shape=readOGR("af_riv_15s.shp", layer="af_riv_15s")
>>>
>>> Thank you for your help,Phil
>>>
>>>
>>>
>>>
>>>         [[alternative HTML version deleted]]
>>>
>>> _______________________________________________
>>> R-sig-Geo mailing list
>>> [hidden email]
>>> https://stat.ethz.ch/mailman/listinfo/r-sig-geo
>>
>>
>>
>> --
>> Michael Sumner
>> Hobart, Australia
>> e-mail: [hidden email]
>
> [[alternative HTML version deleted]]
>
> _______________________________________________
> R-sig-Geo mailing list
> [hidden email]
> https://stat.ethz.ch/mailman/listinfo/r-sig-geo
>

--
Roger Bivand
Department of Economics, NHH Norwegian School of Economics,
Helleveien 30, N-5045 Bergen, Norway.
voice: +47 55 95 93 55; fax +47 55 95 95 43
e-mail: [hidden email]

_______________________________________________
R-sig-Geo mailing list
[hidden email]
https://stat.ethz.ch/mailman/listinfo/r-sig-geo
Roger Bivand
Department of Economics
Norwegian School of Economics
Helleveien 30
N-5045 Bergen, Norway
Reply | Threaded
Open this post in threaded view
|

Re: Long time to load shapefiles

Filoche
For information, I'm not a the step of statistical analyses. I'm currently working to extract the needed data. I have several hundred points around the world for which I want to extract some information from these large shapefiles.

> Date: Thu, 17 Oct 2013 14:51:24 +0200
> From: [hidden email]
> To: [hidden email]
> CC: [hidden email]; [hidden email]
> Subject: Re: [R-sig-Geo] Long time to load shapefiles
>
> On Thu, 17 Oct 2013, philippe massicotte wrote:
>
> > Thank you everyone for your answers. I think I'll just load it and save
> > the workspace for further uses. I still wonder why this is taking such
> > long time to load whereas in QGIS or ArcGIS it is a matter of seconds.
>
> Why do you wonder? You could check instead, taking subsets of the fields,
> and subsets of the features in your object. You could also use alternative
> formats, but I don't think that the choice of driver is important here.
> Note that few statisticians would consider global analysis of data sets
> this large sensible unless very specific conditions were met; the
> additional variability after the first several thousand observations
> probably doesn't tell you much. Once the data are converted to sp classes,
> things go rather faster, as has been pointed out.
>
> Roger
>
> > Have a great day,Phil
> >
> >> Date: Thu, 17 Oct 2013 17:19:39 +1100
> >> Subject: Re: [R-sig-Geo] Long time to load shapefiles
> >> From: [hidden email]
> >> To: [hidden email]
> >> CC: [hidden email]
> >>
> >> I have a similar spec to yours, with i7 instead.
> >>
> >> I have successfully read it, but first time it crashed (I think
> >> because memory was exhausted).
> >>
> >> It took about 20min.
> >>
> >> I would work on subsetting the part you want with an ogr2ogr
> >> incantation if you want to stay away from GIS:
> >> http://www.gdal.org/ogr2ogr.html
> >>
> >> QGIS or similar would be able to read / crop this no-sweat, but it
> >> depends on exactly how you want to subset it, and that may be a topic
> >> for another forum
> >>
> >> Cheers, Mike
> >>
> >> On Thu, Oct 17, 2013 at 2:18 AM, philippe massicotte
> >> <[hidden email]> wrote:
> >>> Hi everyone.
> >>> I'm trying to open some shapefiles, but the process is still not complete after 25 minutes.  Looking at the shapefiles in ArcGIS, I find that these files have roughly 500K entries. I was wondering if this processing time to open the shapefiles is normal. Here is my configuration:
> >>> Intel i5-2500 @ 3.3 GHz16 GB of RAMWindows 7 (64 bits)
> >>> For information, you can download the shapefiles here:
> >>> http://www.usna.edu/Users/oceano/pguth/srtm/rivers.zip
> >>> Here is my code:
> >>> library(rgdal)shape=readOGR("af_riv_15s.shp", layer="af_riv_15s")
> >>>
> >>> Thank you for your help,Phil
> >>>
> >>>
> >>>
> >>>
> >>>         [[alternative HTML version deleted]]
> >>>
> >>> _______________________________________________
> >>> R-sig-Geo mailing list
> >>> [hidden email]
> >>> https://stat.ethz.ch/mailman/listinfo/r-sig-geo
> >>
> >>
> >>
> >> --
> >> Michael Sumner
> >> Hobart, Australia
> >> e-mail: [hidden email]
> >
> > [[alternative HTML version deleted]]
> >
> > _______________________________________________
> > R-sig-Geo mailing list
> > [hidden email]
> > https://stat.ethz.ch/mailman/listinfo/r-sig-geo
> >
>
> --
> Roger Bivand
> Department of Economics, NHH Norwegian School of Economics,
> Helleveien 30, N-5045 Bergen, Norway.
> voice: +47 55 95 93 55; fax +47 55 95 95 43
> e-mail: [hidden email]
>
     
        [[alternative HTML version deleted]]

_______________________________________________
R-sig-Geo mailing list
[hidden email]
https://stat.ethz.ch/mailman/listinfo/r-sig-geo
Reply | Threaded
Open this post in threaded view
|

Re: Long time to load shapefiles

Tim Keitt-3
My advise is to use PostGIS first, then read the summaries into R for
analysis.

THK


On Thu, Oct 17, 2013 at 8:42 AM, philippe massicotte <
[hidden email]> wrote:

> For information, I'm not a the step of statistical analyses. I'm currently
> working to extract the needed data. I have several hundred points around
> the world for which I want to extract some information from these large
> shapefiles.
>
> > Date: Thu, 17 Oct 2013 14:51:24 +0200
> > From: [hidden email]
> > To: [hidden email]
> > CC: [hidden email]; [hidden email]
> > Subject: Re: [R-sig-Geo] Long time to load shapefiles
> >
> > On Thu, 17 Oct 2013, philippe massicotte wrote:
> >
> > > Thank you everyone for your answers. I think I'll just load it and save
> > > the workspace for further uses. I still wonder why this is taking such
> > > long time to load whereas in QGIS or ArcGIS it is a matter of seconds.
> >
> > Why do you wonder? You could check instead, taking subsets of the fields,
> > and subsets of the features in your object. You could also use
> alternative
> > formats, but I don't think that the choice of driver is important here.
> > Note that few statisticians would consider global analysis of data sets
> > this large sensible unless very specific conditions were met; the
> > additional variability after the first several thousand observations
> > probably doesn't tell you much. Once the data are converted to sp
> classes,
> > things go rather faster, as has been pointed out.
> >
> > Roger
> >
> > > Have a great day,Phil
> > >
> > >> Date: Thu, 17 Oct 2013 17:19:39 +1100
> > >> Subject: Re: [R-sig-Geo] Long time to load shapefiles
> > >> From: [hidden email]
> > >> To: [hidden email]
> > >> CC: [hidden email]
> > >>
> > >> I have a similar spec to yours, with i7 instead.
> > >>
> > >> I have successfully read it, but first time it crashed (I think
> > >> because memory was exhausted).
> > >>
> > >> It took about 20min.
> > >>
> > >> I would work on subsetting the part you want with an ogr2ogr
> > >> incantation if you want to stay away from GIS:
> > >> http://www.gdal.org/ogr2ogr.html
> > >>
> > >> QGIS or similar would be able to read / crop this no-sweat, but it
> > >> depends on exactly how you want to subset it, and that may be a topic
> > >> for another forum
> > >>
> > >> Cheers, Mike
> > >>
> > >> On Thu, Oct 17, 2013 at 2:18 AM, philippe massicotte
> > >> <[hidden email]> wrote:
> > >>> Hi everyone.
> > >>> I'm trying to open some shapefiles, but the process is still not
> complete after 25 minutes.  Looking at the shapefiles in ArcGIS, I find
> that these files have roughly 500K entries. I was wondering if this
> processing time to open the shapefiles is normal. Here is my configuration:
> > >>> Intel i5-2500 @ 3.3 GHz16 GB of RAMWindows 7 (64 bits)
> > >>> For information, you can download the shapefiles here:
> > >>> http://www.usna.edu/Users/oceano/pguth/srtm/rivers.zip
> > >>> Here is my code:
> > >>> library(rgdal)shape=readOGR("af_riv_15s.shp", layer="af_riv_15s")
> > >>>
> > >>> Thank you for your help,Phil
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>         [[alternative HTML version deleted]]
> > >>>
> > >>> _______________________________________________
> > >>> R-sig-Geo mailing list
> > >>> [hidden email]
> > >>> https://stat.ethz.ch/mailman/listinfo/r-sig-geo
> > >>
> > >>
> > >>
> > >> --
> > >> Michael Sumner
> > >> Hobart, Australia
> > >> e-mail: [hidden email]
> > >
> > >     [[alternative HTML version deleted]]
> > >
> > > _______________________________________________
> > > R-sig-Geo mailing list
> > > [hidden email]
> > > https://stat.ethz.ch/mailman/listinfo/r-sig-geo
> > >
> >
> > --
> > Roger Bivand
> > Department of Economics, NHH Norwegian School of Economics,
> > Helleveien 30, N-5045 Bergen, Norway.
> > voice: +47 55 95 93 55; fax +47 55 95 95 43
> > e-mail: [hidden email]
> >
>
>         [[alternative HTML version deleted]]
>
> _______________________________________________
> R-sig-Geo mailing list
> [hidden email]
> https://stat.ethz.ch/mailman/listinfo/r-sig-geo
>



--
http://www.keittlab.org/

        [[alternative HTML version deleted]]

_______________________________________________
R-sig-Geo mailing list
[hidden email]
https://stat.ethz.ch/mailman/listinfo/r-sig-geo