CRS in rgdal does not function with PROJ 6.2.0

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

CRS in rgdal does not function with PROJ 6.2.0

Cal Abel
I have a Ubuntu 18.0.4 with R 3.6.1 system with the following releases from OSGeo for GEOS 3.8.0, GDAL 3.0.1, PROJ 6.2.0  I have an application that requires using gstat and thus I have to use rgdal as sf objects/classes aren’t supported in gstat, only sp. (This is a bit frustrating because rpostgis uses sf, so when I import my data it is an sf datatype, convert it to sp for gstat, and then back to sf to upload…) I thought the issue originally resided in sp, but after reviewing the sp source, it comes down to the calls that the sp CRS function makes using rgdal::. Below is a summary that shows my problem, sf links with the necessary system files and gets the correct CRS, rgdal does not. I think the issue is in how proj_api.h is included, specifically with not recognizing proj.db and instead relying on proj_def.dat which was removed in PROJ 6.1 as I recall. I can confirm that there is only one set of OSGeo libraries installed and those are those are the ones that I’ve compiled and are in /usr/local.

I can trick rgdal to use the proj_def.dat file by adding gdal/ and proj/ directories from the latest windows binary *.zip to ~/R/x86_64-pc-linux-gnu-library/3.6/rgdal. After doing this, rgdal will execute CRS, but complains that it can’t find proj.db   Once those files are removed it finds proj.db, but then can’t do CRS stuff. Win some, loose some I guess… 

After reading through a bit of the source, it seems that rgdal defaults to using the deperecated proj4 strings and does not access the newer formats in proj.db even though during compilation rgdal finds the db with no problem. I am not too familiar with either package, so this is conjecture based on my limited testing/research.

As a note, if $PROJ_LIB is specified as an environment variable the result is Path to PROJ.4 shared files: /usr/local/share/proj.  But, when it is not specified the result is Path to PROJ.4 shared files: (autodetected). The behavior of rgdal remains unchanged in either case.


> library(sf)
Linking to GEOS 3.8.0, GDAL 3.0.1, PROJ 6.2.0
> library(rgdal)
Loading required package: sp
rgdal: version: 1.5-1, (SVN revision 879)
 Geospatial Data Abstraction Library extensions to R successfully loaded
 Loaded GDAL runtime: GDAL 3.0.1, released 2019/06/28
 Path to GDAL shared files: 
 GDAL binary built with GEOS: TRUE 
 Loaded PROJ.4 runtime: Rel. 6.2.0, September 1st, 2019, [PJ_VERSION: 620]
 Path to PROJ.4 shared files: /usr/local/share/proj
 Linking to sp version: 1.3-1 
> CRS("+init=epsg:4269")
Error in CRS("+init=epsg:4269") : no arguments in initialization list
> st_crs(4269)
Coordinate Reference System:
  EPSG: 4269 
  proj4string: "+proj=longlat +datum=NAD83 +no_defs”

Here is a coordinate transformation from NAD83 CRS to the UTM16 NAD83(2011) projection to show basic GDAL/PROJ6 functionality:
$ cs2cs EPSG:4269 EPSG:6345
35 -84
773798.10 3877156.69 0.00

I had forgotten that I originally reported the issue here https://github.com/edzer/sp/issues/59 which I’ve since closed in testing for this email.

Below is an email exchange that drove this current email. I am unconvinced that Roger is correct. GDAL/PROJ work just fine, as does sf. A user error that he is referring to would cause those not to work/complain. The fact that I can make it work and make it not work should actually be grounds for successful bug identification.

Cal

On Wed, 6 Nov 2019, Cal Abel wrote:

Dr. Bivand,

I attempted to report a bug in rgdal but don't really have any place to put it. I reported it in the OSGeo/gdal git page, but the issue was closed:


I posted:

Almost certainly user blunder - neither **sf** nor **rgdal** install data/share files on source install, always relying on system installs. OP almost certainly neglected to read install notes. If GDAL and PROJ utilities can find their data/share files outside R, not finding them within R could signal R packages built against a different *.so than the one found at runtime.



One update to the above, I tried creating symbolic links to the compiled gdal and proj directories instead of copying the windows folders and it did not work. I also tried various combinations of links/files and was not able to get it to work. The only method that worked was to copy both windows directories.

The temporary fix I have, while it does not produce an error it now produces warnings:"

Do not try any fixes, you are almost certainly in error. How many versions of GDAL and PROJ are on your platform? For source installs, there should be one only, and GDAL should be built with that PROJ.


Warning messages:
1: In CPL_crs_from_epsg(as.integer(x)) :
GDAL Error 1: PROJ: proj_create_from_database: Cannot find proj.db
2: In CPL_crs_from_epsg(as.integer(x)) :
GDAL Error 1: PROJ: proj_create_from_database: Cannot find proj.db

"


Since it cannot find the PROJ proj.db, your installation of PROJ is broken, and copying across from an older version which does not have this key file will obviously fail.

In the future, what is the best avenue for reporting bugs?


Use the R-sig-geo list. The package is not on github and will not be migrated - it is where https://cran.r-project.org/package=rgdal says it is: https://r-forge.r-project.org/projects/rgdal/ under SVN. All corresponndence through R-sig-geo, because others can correct your misapprehensions.

Note that the nabble link is totally different, which you should have known.

Further be aware that your usage pattern (+init=) is no longer valid in GDAL 3 and PROJ 6, and will stop working shortly. Your working example has GDAL 2 and PROJ 5, although this is not the only cause here, I think. It would be to your benefit to read https://proj.org/ and https://gdal.org/, and grasp https://gdal.org/development/rfc/rfc73_proj6_wkt2_srsbarn.html, since this is a work-related query. How you migrate your corporate workflows to GDAL 3 and PROJ 6//7 may be of considerable interest (blog, mailing list post), and I'm spending all my time trying to adapt rgdal for this major change, and do not have patience with wasted time. See https://github.com/r-spatial/sf/issues/1146 and https://github.com/r-spatial/discuss/issues/28.

Roger Bivand



_______________________________________________
R-sig-Geo mailing list
[hidden email]
https://stat.ethz.ch/mailman/listinfo/r-sig-geo

signature.asc (891 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: CRS in rgdal does not function with PROJ 6.2.0

Roger Bivand
Administrator
On Fri, 8 Nov 2019, Cal Abel wrote:

> I have a Ubuntu 18.0.4 with R 3.6.1 system with the following releases
> from OSGeo for GEOS 3.8.0, GDAL 3.0.1, PROJ 6.2.0 I have an application
> that requires using gstat and thus I have to use rgdal as sf
> objects/classes aren’t supported in gstat, only sp. (This is a bit
> frustrating because rpostgis uses sf, so when I import my data it is an
> sf datatype, convert it to sp for gstat, and then back to sf to upload…)

First, note that gstat does support sf and stars class objects, as shown
by https://keen-swartz-3146c4.netlify.com/interpolation.html (a draft
chapter in a WIP-book).

Second, note that unless you are developing software, you should be using
GDAL 2 and PROJ 5. You should not be using GDAL 3 and PROJ 6. For
references, please do read the links I sent privately explaining why all
open source spatial software using GDAL and PROJ is having difficulties
adapting to the speed of chane in their APIs. Elsewhere I believe you
encountered this with regard to libspatialite. Unless you really need to
work on the wild side, stay on the safe side (GDAL2+PROJ5). sf with
GDAL3/PROJ6 is also broken now.

You are working (day-time job) for a power company, and using R and gstat
for free. You need to read more and write less before you impose on
volunteer developers and maintainers whose day jobs do not allocate hours
to handling your queries.


> I thought the issue originally resided in sp, but after reviewing the sp
> source, it comes down to the calls that the sp CRS function makes using
> rgdal::. Below is a summary that shows my problem, sf links with the
> necessary system files and gets the correct CRS, rgdal does not. I think
> the issue is in how proj_api.h is included, specifically with not
> recognizing proj.db and instead relying on proj_def.dat which was
> removed in PROJ 6.1 as I recall. I can confirm that there is only one
> set of OSGeo libraries installed and those are those are the ones that
> I’ve compiled and are in /usr/local.
>
> I can trick rgdal to use the proj_def.dat file by adding gdal/ and proj/
> directories from the latest windows binary *.zip to
> ~/R/x86_64-pc-linux-gnu-library/3.6/rgdal. After doing this, rgdal will
> execute CRS, but complains that it can’t find proj.db Once those files
> are removed it finds proj.db, but then can’t do CRS stuff. Win some,
> loose some I guess…
>
This was an issue faced years ago when PROJ was released without this file
and is completely misleading here.

> After reading through a bit of the source, it seems that rgdal defaults
> to using the deperecated proj4 strings and does not access the newer
> formats in proj.db even though during compilation rgdal finds the db
> with no problem. I am not too familiar with either package, so this is
> conjecture based on my limited testing/research.
>
> As a note, if $PROJ_LIB is specified as an environment variable the
> result is Path to PROJ.4 shared files: /usr/local/share/proj.  But, when
> it is not specified the result is Path to PROJ.4 shared files:
> (autodetected). The behavior of rgdal remains unchanged in either case.
>
>
>> library(sf)
> Linking to GEOS 3.8.0, GDAL 3.0.1, PROJ 6.2.0
>> library(rgdal)
> Loading required package: sp
> rgdal: version: 1.5-1, (SVN revision 879)
> Geospatial Data Abstraction Library extensions to R successfully loaded
> Loaded GDAL runtime: GDAL 3.0.1, released 2019/06/28
> Path to GDAL shared files:
> GDAL binary built with GEOS: TRUE
> Loaded PROJ.4 runtime: Rel. 6.2.0, September 1st, 2019, [PJ_VERSION: 620]
> Path to PROJ.4 shared files: /usr/local/share/proj
> Linking to sp version: 1.3-1
>> CRS("+init=epsg:4269")
> Error in CRS("+init=epsg:4269") : no arguments in initialization list
>> st_crs(4269)
> Coordinate Reference System:
>  EPSG: 4269
>  proj4string: "+proj=longlat +datum=NAD83 +no_defs”
>
> Here is a coordinate transformation from NAD83 CRS to the UTM16 NAD83(2011) projection to show basic GDAL/PROJ6 functionality:
> $ cs2cs EPSG:4269 EPSG:6345
> 35 -84
> 773798.10 3877156.69 0.00
>
> I had forgotten that I originally reported the issue here
> https://github.com/edzer/sp/issues/59 
> <https://github.com/edzer/sp/issues/59> which I’ve since closed in
> testing for this email.
>
> Below is an email exchange that drove this current email. I am
> unconvinced that Roger is correct. GDAL/PROJ work just fine, as does sf.
> A user error that he is referring to would cause those not to
> work/complain. The fact that I can make it work and make it not work
> should actually be grounds for successful bug identification.
>
You did not ask my permission to publish a private exchange on a list
posting. It is normal practice to do so. Another time I might rather
choose not to reply to any private queries, rather than offer advice,
which had you heeded it, would have guided you in the right direction.

Check first, read a lot, pester others less, especially when your job
depends on our offering  goodwill.

Reconfigure your system to use GDAL 2 and PROJ 5, the use whatever you
need.

Roger

> Cal
>
>> On Wed, 6 Nov 2019, Cal Abel wrote:
>>
>> Dr. Bivand,
>>
>> I attempted to report a bug in rgdal but don't really have any place to
>> put it. I reported it in the OSGeo/gdal git page, but the issue was
>> closed:
>>
>> https://github.com/OSGeo/gdal/issues/1988 <https://github.com/OSGeo/gdal/issues/1988>
>> I posted:
>>
>> Almost certainly user blunder - neither **sf** nor **rgdal** install
>> data/share files on source install, always relying on system installs.
>> OP almost certainly neglected to read install notes. If GDAL and PROJ
>> utilities can find their data/share files outside R, not finding them
>> within R could signal R packages built against a different *.so than
>> the one found at runtime.
>>
>>
>>
>> One update to the above, I tried creating symbolic links to the
>> compiled gdal and proj directories instead of copying the windows
>> folders and it did not work. I also tried various combinations of
>> links/files and was not able to get it to work. The only method that
>> worked was to copy both windows directories.
>>
>> The temporary fix I have, while it does not produce an error it now
>> produces warnings:"
>>
>> Do not try any fixes, you are almost certainly in error. How many
>> versions of GDAL and PROJ are on your platform? For source installs,
>> there should be one only, and GDAL should be built with that PROJ.
>>
>>
>> Warning messages:
>> 1: In CPL_crs_from_epsg(as.integer(x)) :
>> GDAL Error 1: PROJ: proj_create_from_database: Cannot find proj.db
>> 2: In CPL_crs_from_epsg(as.integer(x)) :
>> GDAL Error 1: PROJ: proj_create_from_database: Cannot find proj.db
>>
>> "
>>
>>
>> Since it cannot find the PROJ proj.db, your installation of PROJ is
>> broken, and copying across from an older version which does not have
>> this key file will obviously fail.
>>
>> In the future, what is the best avenue for reporting bugs?
>>
>>
>> Use the R-sig-geo list. The package is not on github and will not be
>> migrated - it is where https://cran.r-project.org/package=rgdal 
>> <https://cran.r-project.org/package=rgdal> says it is:
>> https://r-forge.r-project.org/projects/rgdal/ 
>> <https://r-forge.r-project.org/projects/rgdal/> under SVN. All
>> corresponndence through R-sig-geo, because others can correct your
>> misapprehensions.
>>
>> Note that the nabble link is totally different, which you should have
>> known.
>>
>> Further be aware that your usage pattern (+init=) is no longer valid in
>> GDAL 3 and PROJ 6, and will stop working shortly. Your working example
>> has GDAL 2 and PROJ 5, although this is not the only cause here, I
>> think. It would be to your benefit to read https://proj.org/ 
>> <https://proj.org/> and https://gdal.org/ <https://gdal.org/>, and
>> grasp https://gdal.org/development/rfc/rfc73_proj6_wkt2_srsbarn.html 
>> <https://gdal.org/development/rfc/rfc73_proj6_wkt2_srsbarn.html>, since
>> this is a work-related query. How you migrate your corporate workflows
>> to GDAL 3 and PROJ 6//7 may be of considerable interest (blog, mailing
>> list post), and I'm spending all my time trying to adapt rgdal for this
>> major change, and do not have patience with wasted time. See
>> https://github.com/r-spatial/sf/issues/1146 
>> <https://github.com/r-spatial/sf/issues/1146> and
>> https://github.com/r-spatial/discuss/issues/28 
>> <https://github.com/r-spatial/discuss/issues/28>.
>>
>> Roger Bivand
>>
> Cal Abel, PhD
> [hidden email] <mailto:[hidden email]>
>
> www.signalpowerandlight.com <http://www.signalpowerandlight.com/>
>
--
Roger Bivand
Department of Economics, Norwegian School of Economics,
Helleveien 30, N-5045 Bergen, Norway.
voice: +47 55 95 93 55; e-mail: [hidden email]
https://orcid.org/0000-0003-2392-6140
https://scholar.google.no/citations?user=AWeghB0AAAAJ&hl=en
_______________________________________________
R-sig-Geo mailing list
[hidden email]
https://stat.ethz.ch/mailman/listinfo/r-sig-geo
Roger Bivand
Department of Economics
Norwegian School of Economics
Helleveien 30
N-5045 Bergen, Norway