rgdal: About the new gdal3 and proj >=6 condition

classic Classic list List threaded Threaded
14 messages Options
Reply | Threaded
Open this post in threaded view
|

rgdal: About the new gdal3 and proj >=6 condition

Patrick Schratz
The just released rgdal v1.5.8 requires gdal >= 3 when proj >= 6 is
present.
It does not even install the package if this condition is not met but
errors during linking.

I could not find any sources in the web or in the announcement post of
v1.5.8 why this condition was enforced so strictly.
Does it cause unwanted results?
If so, then there is a big issue since the “gdal2 and proj >= 6”
combination has been used by many people in the past already.
I haven’t tracked down for how long this situation was on the market
already.
Also the {sf} package is still installable with this combination and I
am not aware of any warnings/issues that came up due to this so far.

It further causes complete breakage for people relying on the homebrew
package manager on macOS since current versions are gdal v2.2.4 and proj
7.0.1.
The update to gdal3 is blocked since months due to an incompatibility
with `liblas` (https://github.com/libLAS/libLAS/issues/164). Reading the
issue, it looks unclear when this issue will be resolved.
The alternative osgeo4mac tap holds formula that is unstable and
somewhat broken. One cannot link rgdal with the current gdal v3.0.1 from
osgeo4mac.

This incompatibility with homebrew-core formula leads to further issues
for CI tests that rely on the spatial homebrew stack for macOS builds.

All of this is really unfortunate and I am wondering if this was a known
factor when introducing this condition in the new release.
Unfortunately rgdal is not hosted on any Git* provider (there is just
the CRAN mirror with not much information about the recent changes) and
it is unclear/untransparent to me if there is a continuous integration
setup at all for this package.
I hope this is not another case in the series “I do not like operating
system XY and hence I do not test on it” which was seen recently in
another package. That’s what CI is for and the responsibility of
package authors.

Also one should be aware that not every rgdal user is member of this
mailing list (I guess not even 5%) and many people in production will
face this error during install, most of them without any clue what to do
or an idea why this was raised.

In summary it would be great if

- There would be more detailed information on the introduction of the
new condition
- Awareness of state-of-the-art platform related library versions before
making such a release
- Transparency/Existence of a cross-platform build process
- If the incompatibility is just due to some features not being
accessible with this configuration, I suggest to raise a warning during
package load but not prevent the package from installing in the first
place
- A version-based NEWS file would be available. Currently one only sees
a revision-based Changelog:
https://cran.r-project.org/web/packages/rgdal/ChangeLog

I am well aware that I am indirectly criticising (hopefully in a
constructive way) the transparency and release process here, for which I
will probably get a rough response.
However, if that leads to more robustness and transparency in future
release, I am fine with this.

A quick patch release resolving the breakage would be highly
appreciated.

Best, Patrick
        [[alternative HTML version deleted]]

_______________________________________________
R-sig-Geo mailing list
[hidden email]
https://stat.ethz.ch/mailman/listinfo/r-sig-geo
Reply | Threaded
Open this post in threaded view
|

Re: rgdal: About the new gdal3 and proj >=6 condition

Roger Bivand
Administrator
On Fri, 29 May 2020, Patrick Schratz wrote:

> The just released rgdal v1.5.8 requires gdal >= 3 when proj >= 6 is
> present. It does not even install the package if this condition is not
> met but errors during linking.

Please document with the output of 00install.out from R CMD check
rgdal_1.5-8.tar.gz, and pkg-config proj --modversion, gdalinfo --version
with system details.

Yes, PROJ >= 6 makes no sense without GDAL >= 3; GDAL 3 makes no sense
with PROJ < 6. GDAL 3 with PROJ < 6 uses the old proj_api.h API and the
old metadata structure with Proj4 string degradation; GDAL < 3 with PROJ
>= 6 must use the deprecated API and the new metadata structure.

>
> I could not find any sources in the web or in the announcement post of
> v1.5.8 why this condition was enforced so strictly.
> Does it cause unwanted results?

Yes, definitely. GDAL 3 was released as 3, not 2.5, to stress the complete
break between GDAL 2 (with PROJ <= 5) and GDAL >= 3 (with PROJ >= 6) after
GDAL barnraising. So:

           PROJ
           < 6 >= 6
GDAL < 3   OK   NO
     >= 3   NO   OK



> If so, then there is a big issue since the “gdal2 and proj >= 6”
> combination has been used by many people in the past already.

No good precedent, just binary packagers protecting slow downstream
adaptation.

> I haven’t tracked down for how long this situation was on the market
> already.
> Also the {sf} package is still installable with this combination and I
> am not aware of any warnings/issues that came up due to this so far.
>

rgdal is a much older package and has much more older code that needs this
protection. It is possible to accommodate the no-go areas, but I'd really
value patches to R-forge, as I cannot check multiple PROJ/GDAL version
combinations just because someone isn't up to speed.

> It further causes complete breakage for people relying on the homebrew
> package manager on macOS since current versions are gdal v2.2.4 and proj
> 7.0.1.

They should never have gone beyond PROJ 5 if they are stuck on GDAL. PROJ
7 is a very different animal.

> The update to gdal3 is blocked since months due to an incompatibility
> with `liblas` (https://github.com/libLAS/libLAS/issues/164). Reading the
> issue, it looks unclear when this issue will be resolved.
> The alternative osgeo4mac tap holds formula that is unstable and
> somewhat broken. One cannot link rgdal with the current gdal v3.0.1 from
> osgeo4mac.
>
> This incompatibility with homebrew-core formula leads to further issues
> for CI tests that rely on the spatial homebrew stack for macOS builds.
>
> All of this is really unfortunate and I am wondering if this was a known
> factor when introducing this condition in the new release.
> Unfortunately rgdal is not hosted on any Git* provider (there is just
> the CRAN mirror with not much information about the recent changes) and
> it is unclear/untransparent to me if there is a continuous integration
> setup at all for this package.
Do not require me to leave the excellent SVN service on R-forge. Provide
patches there.

> I hope this is not another case in the series “I do not like operating
> system XY and hence I do not test on it” which was seen recently in
> another package. That’s what CI is for and the responsibility of
> package authors.
>

I run and develop on Fedora, I  have no access to OSX, and the current
release was fully checked with ~ 900 revdeps repeatedly since November on
successive older and newer versions of PROJ and GDAL. I blocked new PROJ
with old GDAL recently to simplify development. CI is only as good as the
scripts, I am completely sure that repeated revdep testing and reading the
output is superior.

> Also one should be aware that not every rgdal user is member of this
> mailing list (I guess not even 5%) and many people in production will
> face this error during install, most of them without any clue what to do
> or an idea why this was raised.
>

There has been plenty of information given about the CRS changes. rgdal
could simply have been abandoned by me, and all  those production
volunteers might have fixed things, but I never hear anything at all.

> In summary it would be great if
>
> - There would be more detailed information on the introduction of the
> new condition
> - Awareness of state-of-the-art platform related library versions before
> making such a release
> - Transparency/Existence of a cross-platform build process
> - If the incompatibility is just due to some features not being
> accessible with this configuration, I suggest to raise a warning during
> package load but not prevent the package from installing in the first
> place
> - A version-based NEWS file would be available. Currently one only sees
> a revision-based Changelog:
> https://cran.r-project.org/web/packages/rgdal/ChangeLog
>
> I am well aware that I am indirectly criticising (hopefully in a
> constructive way) the transparency and release process here, for which I
> will probably get a rough response.
> However, if that leads to more robustness and transparency in future
> release, I am fine with this.
>
> A quick patch release resolving the breakage would be highly
> appreciated.
Only with community imput. what you ask is not needed, just extra
complexity. Please provide patches, or accept my invitation to join the
R-forge project and commit your fixes directly. I can see how to do it,
but I don't think it makes sense, and your messsage has not motivated me,
to be honest. I'm prioritizing working with CRAN to iron out reverse
dependency problems.

Roger

>
> Best, Patrick
> [[alternative HTML version deleted]]
>
> _______________________________________________
> R-sig-Geo mailing list
> [hidden email]
> https://stat.ethz.ch/mailman/listinfo/r-sig-geo
>

--
Roger Bivand
Department of Economics, Norwegian School of Economics,
Helleveien 30, N-5045 Bergen, Norway.
voice: +47 55 95 93 55; e-mail: [hidden email]
https://orcid.org/0000-0003-2392-6140
https://scholar.google.no/citations?user=AWeghB0AAAAJ&hl=en
_______________________________________________
R-sig-Geo mailing list
[hidden email]
https://stat.ethz.ch/mailman/listinfo/r-sig-geo
Roger Bivand
Department of Economics
Norwegian School of Economics
Helleveien 30
N-5045 Bergen, Norway
Reply | Threaded
Open this post in threaded view
|

Re: rgdal: About the new gdal3 and proj >=6 condition

Roger Bivand
Administrator
I have put a tarball, built without vignettes with GDAL 2.2.4 and PROJ
7.0.1 that passes CMD check (with warnings for missing vignettes) on:

http://spatial.nhh.no/R/rgdal/gdal_1.5-9.tar.gz

It involves code copying to provide duplicated functions in three
settings:

PROJ < 6 & GDAL < 3
PROJ >= 6 & GDAL >= 3
PROJ >= 6 & GDAL < 3 (your homebrew case)

You need the configure argument:
--configure-args=--with-proj_api=proj_api.h.
Without the argument, you now get directed to use it.

For now I've dropped the configure tests for sqlite3, curl and tiff; works
for me but may not work for you.

Please make the output of:

R CMD check --install-args="--configure-args=--with-proj_api=proj_api.h"
rgdal_1.5-9.tar.gz

available ASAP. I count on an immediate response.

Roger

On Fri, 29 May 2020, Roger Bivand wrote:

> On Fri, 29 May 2020, Patrick Schratz wrote:
>
>>  The just released rgdal v1.5.8 requires gdal >= 3 when proj >= 6 is
>>  present. It does not even install the package if this condition is not met
>>  but errors during linking.
>
> Please document with the output of 00install.out from R CMD check
> rgdal_1.5-8.tar.gz, and pkg-config proj --modversion, gdalinfo --version with
> system details.
>
> Yes, PROJ >= 6 makes no sense without GDAL >= 3; GDAL 3 makes no sense with
> PROJ < 6. GDAL 3 with PROJ < 6 uses the old proj_api.h API and the old
> metadata structure with Proj4 string degradation; GDAL < 3 with PROJ
>> = 6 must use the deprecated API and the new metadata structure.
>
>>
>>  I could not find any sources in the web or in the announcement post of
>>  v1.5.8 why this condition was enforced so strictly.
>>  Does it cause unwanted results?
>
> Yes, definitely. GDAL 3 was released as 3, not 2.5, to stress the complete
> break between GDAL 2 (with PROJ <= 5) and GDAL >= 3 (with PROJ >= 6) after
> GDAL barnraising. So:
>
>           PROJ
>           < 6 >= 6
> GDAL < 3   OK   NO
>    > = 3   NO   OK
>
>
>
>>  If so, then there is a big issue since the “gdal2 and proj >= 6”
>>  combination has been used by many people in the past already.
>
> No good precedent, just binary packagers protecting slow downstream
> adaptation.
>
>>  I haven’t tracked down for how long this situation was on the market
>>  already.
>>  Also the {sf} package is still installable with this combination and I
>>  am not aware of any warnings/issues that came up due to this so far.
>>
>
> rgdal is a much older package and has much more older code that needs this
> protection. It is possible to accommodate the no-go areas, but I'd really
> value patches to R-forge, as I cannot check multiple PROJ/GDAL version
> combinations just because someone isn't up to speed.
>
>>  It further causes complete breakage for people relying on the homebrew
>>  package manager on macOS since current versions are gdal v2.2.4 and proj
>>  7.0.1.
>
> They should never have gone beyond PROJ 5 if they are stuck on GDAL. PROJ 7
> is a very different animal.
>
>>  The update to gdal3 is blocked since months due to an incompatibility
>>  with `liblas` (https://github.com/libLAS/libLAS/issues/164). Reading the
>>  issue, it looks unclear when this issue will be resolved.
>>  The alternative osgeo4mac tap holds formula that is unstable and
>>  somewhat broken. One cannot link rgdal with the current gdal v3.0.1 from
>>  osgeo4mac.
>>
>>  This incompatibility with homebrew-core formula leads to further issues
>>  for CI tests that rely on the spatial homebrew stack for macOS builds.
>>
>>  All of this is really unfortunate and I am wondering if this was a known
>>  factor when introducing this condition in the new release.
>>  Unfortunately rgdal is not hosted on any Git* provider (there is just
>>  the CRAN mirror with not much information about the recent changes) and
>>  it is unclear/untransparent to me if there is a continuous integration
>>  setup at all for this package.
>
> Do not require me to leave the excellent SVN service on R-forge. Provide
> patches there.
>
>>  I hope this is not another case in the series “I do not like operating
>>  system XY and hence I do not test on it” which was seen recently in
>>  another package. That’s what CI is for and the responsibility of
>>  package authors.
>>
>
> I run and develop on Fedora, I  have no access to OSX, and the current
> release was fully checked with ~ 900 revdeps repeatedly since November on
> successive older and newer versions of PROJ and GDAL. I blocked new PROJ with
> old GDAL recently to simplify development. CI is only as good as the scripts,
> I am completely sure that repeated revdep testing and reading the output is
> superior.
>
>>  Also one should be aware that not every rgdal user is member of this
>>  mailing list (I guess not even 5%) and many people in production will
>>  face this error during install, most of them without any clue what to do
>>  or an idea why this was raised.
>>
>
> There has been plenty of information given about the CRS changes. rgdal could
> simply have been abandoned by me, and all  those production volunteers might
> have fixed things, but I never hear anything at all.
>
>>  In summary it would be great if
>>
>>  - There would be more detailed information on the introduction of the
>>  new condition
>>  - Awareness of state-of-the-art platform related library versions before
>>  making such a release
>>  - Transparency/Existence of a cross-platform build process
>>  - If the incompatibility is just due to some features not being
>>  accessible with this configuration, I suggest to raise a warning during
>>  package load but not prevent the package from installing in the first
>>  place
>>  - A version-based NEWS file would be available. Currently one only sees
>>  a revision-based Changelog:
>>  https://cran.r-project.org/web/packages/rgdal/ChangeLog
>>
>>  I am well aware that I am indirectly criticising (hopefully in a
>>  constructive way) the transparency and release process here, for which I
>>  will probably get a rough response.
>>  However, if that leads to more robustness and transparency in future
>>  release, I am fine with this.
>>
>>  A quick patch release resolving the breakage would be highly
>>  appreciated.
>
> Only with community imput. what you ask is not needed, just extra complexity.
> Please provide patches, or accept my invitation to join the R-forge project
> and commit your fixes directly. I can see how to do it, but I don't think it
> makes sense, and your messsage has not motivated me, to be honest. I'm
> prioritizing working with CRAN to iron out reverse dependency problems.
>
> Roger
>
>>
>>  Best, Patrick
>>   [[alternative HTML version deleted]]
>>
>>  _______________________________________________
>>  R-sig-Geo mailing list
>>  [hidden email]
>>  https://stat.ethz.ch/mailman/listinfo/r-sig-geo
>>
>
>
--
Roger Bivand
Department of Economics, Norwegian School of Economics,
Helleveien 30, N-5045 Bergen, Norway.
voice: +47 55 95 93 55; e-mail: [hidden email]
https://orcid.org/0000-0003-2392-6140
https://scholar.google.no/citations?user=AWeghB0AAAAJ&hl=en
_______________________________________________
R-sig-Geo mailing list
[hidden email]
https://stat.ethz.ch/mailman/listinfo/r-sig-geo
Roger Bivand
Department of Economics
Norwegian School of Economics
Helleveien 30
N-5045 Bergen, Norway
Reply | Threaded
Open this post in threaded view
|

Re: rgdal: About the new gdal3 and proj >=6 condition

Patrick Schratz
In reply to this post by Roger Bivand
It’s gdal 2.4.4 instead of 2.2.4 as written before. Should not make a
difference though.

```
pkg-config proj --modversion
7.0.1

gdalinfo --version
GDAL 2.4.4, released 2020/01/08
```

With revision 993 checked out from svn I see

``´
checking for unistd.h... yes
checking proj.h usability... yes
checking proj.h presence... yes
checking for proj.h... yes
checking for proj_context_create in -lproj... yes
checking Using GDAL < 3 with PROJ >= 6... yes
configure: error: GDAL >= 3 should be used with PROJ >= 6, use
--with-proj_api="proj_api.h" for deprecated API
```

> Yes, PROJ >= 6 makes no sense without GDAL >= 3; GDAL 3 makes no sense
> with PROJ < 6. GDAL 3 with PROJ < 6 uses the old proj_api.h API and
> the old metadata structure with Proj4 string degradation; GDAL < 3
> with PROJ

What does “makes no sense” mean here? I am interested whether this
combination produces significant bugs when being used together.
If its just “the user won’t be able to make use of proj7 features
without gdal3” but apart from this everything is fine, then I do not
see **any** reason for this strict installation blocker.
While both gdal and proj are coupled tightly, I would expect assertions
in the upstream libraries to account for such clashes.
It seems unlikely that client packages like rgdal need to prevent
installation if this undesired version mixture of both libraries is
found.

> No good precedent, just binary packagers protecting slow downstream
> adaptation.

Not sure what you mean here exactly. My point is that the gdal2 + proj7
combination exists since March 6 in homebrew. Meaning that since then
users on macOS used this combination.

commit 7234fc2073e5475931238281493ae3367b4dcdf6
Author: Stephan Hügel <[hidden email]>
Date:   Fri Mar 6 12:56:51 2020 +0000

     proj 7.0.0


> rgdal is a much older package and has much more older code that needs
> this protection. It is possible to accommodate the no-go areas, but
> I'd really value patches to R-forge, as I cannot check multiple
> PROJ/GDAL version combinations just because someone isn't up to speed.

I cannot argue about any needed protection here. However, if that is the
case then there was a danger zone of almost 3 months on macOS now.
IF that combination causes unwanted side-effects (which is still not
fully answered).
I am not saying that you should check every gdal/proj combination on
every platform/distribution, especially not on the niche ones.
However, there are three main platforms (with Ubuntu being the main
Linux one) and it is part of the game (imo) for client packages to check
valid installations at least on those three platforms, using their
default package manager.
If that would have happened, we would not be writing here probably
because you would have recognised the issue and maybe enforced  the
update to GDAL3 on macOS or postponed the rgdal update.
It would be great if that kind of coverage could be done in the future.

> Do not require me to leave the excellent SVN service on R-forge.
> Provide patches there.

I am just sharing my user experience, which is probably the view of many
others.
More contributions would be made if contributing would be easier.
R-forge and svn do not make it attractive.
It will always be your choice, no question that.
But if you want to provide a service to the community, sometimes there
is more to that than just code/releases.

> CI is only as good as the scripts, I am completely sure that repeated
> revdep testing and reading the output is superior.

CI and revdep checks are two different things.
CI happens at every commit (in the best scenario) and on multiple
platforms. It aims at package AND platform integrity.
Revdepchecks are on the package side, tackling a completely different
layer of security.
As of today (but actually since at least 5 years) there are no excuses
for not being able to check on all main platforms despite arguing with
“I do not want to do it but rely on my very own local test setup”.

> There has been plenty of information given about the CRS changes.
> rgdal could simply have been abandoned by me, and all those production
> volunteers might have fixed things, but I never hear anything at all.

Not sure what this should tell me/us. If you need help maintaining the
package or porting its infrastructure to the 21st century then there
would be a lot of people willing to help.#
However, replying to (useful) criticism with “I could have simply
abandoned it, be thankful (to me), guys” (that’s my interpretation
of this sentence) does not help anyone.
No one questions your contributions to the r-spatial community.
However, adapting to new (development) changes and making it easier for
others to contribute is also a very valuable contribution that is not
really measurable in any unit.

> Only with community imput. what you ask is not needed, just extra
> complexity. Please provide patches, or accept my invitation to join
> the R-forge project and commit your fixes directly. I can see how to
> do it, but I don't think it makes sense, and your messsage has not
> motivated me, to be honest. I'm prioritizing working with CRAN to iron
> out reverse dependency problems.

Ok, I understood. Sadly I have not enough time fighting to break up old
structures and changing things to the better/making things easier if
there is no real support from the other side.
The r-spatial community is not really a “community” in my eyes but a
conglomerate of three (four) main persons doing it their way, each
completely different, with few motivation to change things.
Each semester students ask me how the relations between the packages and
their authors are and why things are so different.
I am not going to expand on what I tell them usually, but often enough
they come to me with their personal view after some time and all I need
to say is “yes” or “no” (sadly most often “yes”).
What this means is left as an exercise to the reader.

Regards, Patrick

On 29 May 2020, at 14:45, Roger Bivand wrote:

> On Fri, 29 May 2020, Patrick Schratz wrote:
>
>> The just released rgdal v1.5.8 requires gdal >= 3 when proj >= 6 is
>> present. It does not even install the package if this condition is
>> not met but errors during linking.
>
> Please document with the output of 00install.out from R CMD check
> rgdal_1.5-8.tar.gz, and pkg-config proj --modversion, gdalinfo
> --version with system details.
>
> Yes, PROJ >= 6 makes no sense without GDAL >= 3; GDAL 3 makes no sense
> with PROJ < 6. GDAL 3 with PROJ < 6 uses the old proj_api.h API and
> the old metadata structure with Proj4 string degradation; GDAL < 3
> with PROJ
>> = 6 must use the deprecated API and the new metadata structure.
>
>>
>> I could not find any sources in the web or in the announcement post
>> of
>> v1.5.8 why this condition was enforced so strictly.
>> Does it cause unwanted results?
>
> Yes, definitely. GDAL 3 was released as 3, not 2.5, to stress the
> complete break between GDAL 2 (with PROJ <= 5) and GDAL >= 3 (with
> PROJ >= 6) after GDAL barnraising. So:
>
>           PROJ
>           < 6 >= 6 GDAL < 3   OK   NO
>      >= 3   NO   OK
>
>
>
>> If so, then there is a big issue since the “gdal2 and proj >= 6”
>> combination has been used by many people in the past already.
>
> No good precedent, just binary packagers protecting slow downstream
> adaptation.
>
>> I haven’t tracked down for how long this situation was on the
>> market
>> already.
>> Also the {sf} package is still installable with this combination and
>> I
>> am not aware of any warnings/issues that came up due to this so far.
>>
>
> rgdal is a much older package and has much more older code that needs
> this protection. It is possible to accommodate the no-go areas, but
> I'd really value patches to R-forge, as I cannot check multiple
> PROJ/GDAL version combinations just because someone isn't up to speed.
>
>> It further causes complete breakage for people relying on the
>> homebrew
>> package manager on macOS since current versions are gdal v2.2.4 and
>> proj
>> 7.0.1.
>
> They should never have gone beyond PROJ 5 if they are stuck on GDAL.
> PROJ 7 is a very different animal.
>
>> The update to gdal3 is blocked since months due to an incompatibility
>> with `liblas` (https://github.com/libLAS/libLAS/issues/164). Reading
>> the
>> issue, it looks unclear when this issue will be resolved.
>> The alternative osgeo4mac tap holds formula that is unstable and
>> somewhat broken. One cannot link rgdal with the current gdal v3.0.1
>> from
>> osgeo4mac.
>>
>> This incompatibility with homebrew-core formula leads to further
>> issues
>> for CI tests that rely on the spatial homebrew stack for macOS
>> builds.
>>
>> All of this is really unfortunate and I am wondering if this was a
>> known
>> factor when introducing this condition in the new release.
>> Unfortunately rgdal is not hosted on any Git* provider (there is just
>> the CRAN mirror with not much information about the recent changes)
>> and
>> it is unclear/untransparent to me if there is a continuous
>> integration
>> setup at all for this package.
>
> Do not require me to leave the excellent SVN service on R-forge.
> Provide patches there.
>
>> I hope this is not another case in the series “I do not like
>> operating
>> system XY and hence I do not test on it” which was seen recently in
>> another package. That’s what CI is for and the responsibility of
>> package authors.
>>
>
> I run and develop on Fedora, I  have no access to OSX, and the current
> release was fully checked with ~ 900 revdeps repeatedly since November
> on successive older and newer versions of PROJ and GDAL. I blocked new
> PROJ with old GDAL recently to simplify development. CI is only as
> good as the scripts, I am completely sure that repeated revdep testing
> and reading the output is superior.
>
>> Also one should be aware that not every rgdal user is member of this
>> mailing list (I guess not even 5%) and many people in production will
>> face this error during install, most of them without any clue what to
>> do
>> or an idea why this was raised.
>>
>
> There has been plenty of information given about the CRS changes.
> rgdal could simply have been abandoned by me, and all  those
> production volunteers might have fixed things, but I never hear
> anything at all.
>
>> In summary it would be great if
>>
>> - There would be more detailed information on the introduction of the
>> new condition
>> - Awareness of state-of-the-art platform related library versions
>> before
>> making such a release
>> - Transparency/Existence of a cross-platform build process
>> - If the incompatibility is just due to some features not being
>> accessible with this configuration, I suggest to raise a warning
>> during
>> package load but not prevent the package from installing in the first
>> place
>> - A version-based NEWS file would be available. Currently one only
>> sees
>> a revision-based Changelog:
>> https://cran.r-project.org/web/packages/rgdal/ChangeLog
>>
>> I am well aware that I am indirectly criticising (hopefully in a
>> constructive way) the transparency and release process here, for
>> which I
>> will probably get a rough response.
>> However, if that leads to more robustness and transparency in future
>> release, I am fine with this.
>>
>> A quick patch release resolving the breakage would be highly
>> appreciated.
>
> Only with community imput. what you ask is not needed, just extra
> complexity. Please provide patches, or accept my invitation to join
> the R-forge project and commit your fixes directly. I can see how to
> do it, but I don't think it makes sense, and your messsage has not
> motivated me, to be honest. I'm prioritizing working with CRAN to iron
> out reverse dependency problems.
>
> Roger
>
>>
>> Best, Patrick
>> [[alternative HTML version deleted]]
>>
>> _______________________________________________
>> R-sig-Geo mailing list
>> [hidden email]
>> https://stat.ethz.ch/mailman/listinfo/r-sig-geo
>>
>
> --
> Roger Bivand
> Department of Economics, Norwegian School of Economics,
> Helleveien 30, N-5045 Bergen, Norway.
> voice: +47 55 95 93 55; e-mail: [hidden email]
> https://orcid.org/0000-0003-2392-6140
> https://scholar.google.no/citations?user=AWeghB0AAAAJ&hl=en

        [[alternative HTML version deleted]]

_______________________________________________
R-sig-Geo mailing list
[hidden email]
https://stat.ethz.ch/mailman/listinfo/r-sig-geo
Reply | Threaded
Open this post in threaded view
|

Re: rgdal: About the new gdal3 and proj >=6 condition

Patrick Schratz
In reply to this post by Roger Bivand
The URL returns a 404.

FWIW I’ve tried it locally using the latest svn revision but have no
clue where I should find `proj_api.h` (I’ve looked in the complete
proj installation directory).

I see

```
ls /usr/local/Cellar/proj/7.0.1/lib/
libproj.19.dylib  libproj.a         libproj.dylib@    libproj.la*      
pkgconfig/
```

Even if this custom installation might work at some point, making such
adjustments until gdal3 arrives in homebrew will be a major pain, also
for CI builds.
It would still be highly appreciated if this internal assertion would be
removed again.

On 29 May 2020, at 18:37, Roger Bivand wrote:

> I have put a tarball, built without vignettes with GDAL 2.2.4 and PROJ
> 7.0.1 that passes CMD check (with warnings for missing vignettes) on:
>
> http://spatial.nhh.no/R/rgdal/gdal_1.5-9.tar.gz
>
> It involves code copying to provide duplicated functions in three
> settings:
>
> PROJ < 6 & GDAL < 3
> PROJ >= 6 & GDAL >= 3
> PROJ >= 6 & GDAL < 3 (your homebrew case)
>
> You need the configure argument:
> --configure-args=--with-proj_api=proj_api.h.
> Without the argument, you now get directed to use it.
>
> For now I've dropped the configure tests for sqlite3, curl and tiff;
> works for me but may not work for you.
>
> Please make the output of:
>
> R CMD check
> --install-args="--configure-args=--with-proj_api=proj_api.h"
> rgdal_1.5-9.tar.gz
>
> available ASAP. I count on an immediate response.
>
> Roger
>
> On Fri, 29 May 2020, Roger Bivand wrote:
>
>> On Fri, 29 May 2020, Patrick Schratz wrote:
>>
>>>  The just released rgdal v1.5.8 requires gdal >= 3 when proj >= 6 is
>>>  present. It does not even install the package if this condition is
>>> not met
>>>  but errors during linking.
>>
>> Please document with the output of 00install.out from R CMD check
>> rgdal_1.5-8.tar.gz, and pkg-config proj --modversion, gdalinfo
>> --version with system details.
>>
>> Yes, PROJ >= 6 makes no sense without GDAL >= 3; GDAL 3 makes no
>> sense with PROJ < 6. GDAL 3 with PROJ < 6 uses the old proj_api.h API
>> and the old metadata structure with Proj4 string degradation; GDAL <
>> 3 with PROJ
>>> = 6 must use the deprecated API and the new metadata structure.
>>
>>>
>>>  I could not find any sources in the web or in the announcement post
>>> of
>>>  v1.5.8 why this condition was enforced so strictly.
>>>  Does it cause unwanted results?
>>
>> Yes, definitely. GDAL 3 was released as 3, not 2.5, to stress the
>> complete break between GDAL 2 (with PROJ <= 5) and GDAL >= 3 (with
>> PROJ >= 6) after GDAL barnraising. So:
>>
>>           PROJ
>>           < 6 >= 6 GDAL < 3   OK   NO
>>    > = 3   NO   OK
>>
>>
>>
>>>  If so, then there is a big issue since the “gdal2 and proj >=
>>> 6”
>>>  combination has been used by many people in the past already.
>>
>> No good precedent, just binary packagers protecting slow downstream
>> adaptation.
>>
>>>  I haven’t tracked down for how long this situation was on the
>>> market
>>>  already.
>>>  Also the {sf} package is still installable with this combination
>>> and I
>>>  am not aware of any warnings/issues that came up due to this so
>>> far.
>>>
>>
>> rgdal is a much older package and has much more older code that needs
>> this protection. It is possible to accommodate the no-go areas, but
>> I'd really value patches to R-forge, as I cannot check multiple
>> PROJ/GDAL version combinations just because someone isn't up to
>> speed.
>>
>>>  It further causes complete breakage for people relying on the
>>> homebrew
>>>  package manager on macOS since current versions are gdal v2.2.4 and
>>> proj
>>>  7.0.1.
>>
>> They should never have gone beyond PROJ 5 if they are stuck on GDAL.
>> PROJ 7 is a very different animal.
>>
>>>  The update to gdal3 is blocked since months due to an
>>> incompatibility
>>>  with `liblas` (https://github.com/libLAS/libLAS/issues/164).
>>> Reading the
>>>  issue, it looks unclear when this issue will be resolved.
>>>  The alternative osgeo4mac tap holds formula that is unstable and
>>>  somewhat broken. One cannot link rgdal with the current gdal v3.0.1
>>> from
>>>  osgeo4mac.
>>>
>>>  This incompatibility with homebrew-core formula leads to further
>>> issues
>>>  for CI tests that rely on the spatial homebrew stack for macOS
>>> builds.
>>>
>>>  All of this is really unfortunate and I am wondering if this was a
>>> known
>>>  factor when introducing this condition in the new release.
>>>  Unfortunately rgdal is not hosted on any Git* provider (there is
>>> just
>>>  the CRAN mirror with not much information about the recent changes)
>>> and
>>>  it is unclear/untransparent to me if there is a continuous
>>> integration
>>>  setup at all for this package.
>>
>> Do not require me to leave the excellent SVN service on R-forge.
>> Provide patches there.
>>
>>>  I hope this is not another case in the series “I do not like
>>> operating
>>>  system XY and hence I do not test on it” which was seen recently
>>> in
>>>  another package. That’s what CI is for and the responsibility of
>>>  package authors.
>>>
>>
>> I run and develop on Fedora, I  have no access to OSX, and the
>> current release was fully checked with ~ 900 revdeps repeatedly since
>> November on successive older and newer versions of PROJ and GDAL. I
>> blocked new PROJ with old GDAL recently to simplify development. CI
>> is only as good as the scripts, I am completely sure that repeated
>> revdep testing and reading the output is superior.
>>
>>>  Also one should be aware that not every rgdal user is member of
>>> this
>>>  mailing list (I guess not even 5%) and many people in production
>>> will
>>>  face this error during install, most of them without any clue what
>>> to do
>>>  or an idea why this was raised.
>>>
>>
>> There has been plenty of information given about the CRS changes.
>> rgdal could simply have been abandoned by me, and all  those
>> production volunteers might have fixed things, but I never hear
>> anything at all.
>>
>>>  In summary it would be great if
>>>
>>>  - There would be more detailed information on the introduction of
>>> the
>>>  new condition
>>>  - Awareness of state-of-the-art platform related library versions
>>> before
>>>  making such a release
>>>  - Transparency/Existence of a cross-platform build process
>>>  - If the incompatibility is just due to some features not being
>>>  accessible with this configuration, I suggest to raise a warning
>>> during
>>>  package load but not prevent the package from installing in the
>>> first
>>>  place
>>>  - A version-based NEWS file would be available. Currently one only
>>> sees
>>>  a revision-based Changelog:
>>>  https://cran.r-project.org/web/packages/rgdal/ChangeLog
>>>
>>>  I am well aware that I am indirectly criticising (hopefully in a
>>>  constructive way) the transparency and release process here, for
>>> which I
>>>  will probably get a rough response.
>>>  However, if that leads to more robustness and transparency in
>>> future
>>>  release, I am fine with this.
>>>
>>>  A quick patch release resolving the breakage would be highly
>>>  appreciated.
>>
>> Only with community imput. what you ask is not needed, just extra
>> complexity. Please provide patches, or accept my invitation to join
>> the R-forge project and commit your fixes directly. I can see how to
>> do it, but I don't think it makes sense, and your messsage has not
>> motivated me, to be honest. I'm prioritizing working with CRAN to
>> iron out reverse dependency problems.
>>
>> Roger
>>
>>>
>>>  Best, Patrick
>>>   [[alternative HTML version deleted]]
>>>
>>>  _______________________________________________
>>>  R-sig-Geo mailing list
>>>  [hidden email]
>>>  https://stat.ethz.ch/mailman/listinfo/r-sig-geo
>>>
>>
>>
>
> --
> Roger Bivand
> Department of Economics, Norwegian School of Economics,
> Helleveien 30, N-5045 Bergen, Norway.
> voice: +47 55 95 93 55; e-mail: [hidden email]
> https://orcid.org/0000-0003-2392-6140
> https://scholar.google.no/citations?user=AWeghB0AAAAJ&hl=en

        [[alternative HTML version deleted]]

_______________________________________________
R-sig-Geo mailing list
[hidden email]
https://stat.ethz.ch/mailman/listinfo/r-sig-geo
Reply | Threaded
Open this post in threaded view
|

Re: rgdal: About the new gdal3 and proj >=6 condition

Roger Bivand
Administrator
On Fri, 29 May 2020, Patrick Schratz wrote:

> The URL returns a 404.

Do you mean http://spatial.nhh.no/R/rgdal/? Please do say what you mean.
If your browser blocks http: rather than https:, that is your choice. I
imagine that it silently inserts an "s".

>
> FWIW I’ve tried it locally using the latest svn revision but have no clue
> where I should find `proj_api.h` (I’ve looked in the complete proj
> installation directory).

If PROJ 7.0.1 is properly installed, the header will be in
$prefix/include. If your packager put all the headers somewhere else, use
the command argument:

$ ls /usr/local/include/proj*
/usr/local/include/proj_api.h           /usr/local/include/proj.h
/usr/local/include/proj_constants.h
/usr/local/include/proj_symbol_rename.h
/usr/local/include/proj_experimental.h

/usr/local/include/proj:
common.hpp               crs.hpp    metadata.hpp
coordinateoperation.hpp  datum.hpp  nn.hpp
coordinatesystem.hpp     io.hpp     util.hpp

You should not need to know. What does pkg-config proj --cflags say (once
you've set PKG_CONFIG_PATH, of course.

>
> I see
>
> ```
> ls /usr/local/Cellar/proj/7.0.1/lib/
> libproj.19.dylib  libproj.a         libproj.dylib@    libproj.la*
> pkgconfig/
> ```
>

In your earlier post, you negected to say that you used the necessary
configure argument. The error message says that you did not use it. Please
report back when you have done so. Futher, I quote from your message: "It
further causes complete breakage for people relying on the homebrew
package manager on macOS since current versions are gdal v2.2.4 and proj
7.0.1." Please accept that I pay attention to what you write.

> Even if this custom installation might work at some point, making such
> adjustments until gdal3 arrives in homebrew will be a major pain, also for CI
> builds.
> It would still be highly appreciated if this internal assertion would be
> removed again.
>

Getting things right may be painful; GDAL 2 with PROJ 7 is wrong, because
it mixes two incompatible metadata storage systems. Migrating all users to
GDAL >= 3 & PROJ >= 6 is a matter of urgency to avoid silent positional
error of about 125m, or about 4 satellite image cells. See the vignette
here
http://rgdal.r-forge.r-project.org/articles/CRS_projections_transformations.html 
and this r-spatial blog https://www.r-spatial.org/r/2020/03/17/wkt.html.

I am not willing to let silent error creep in because some people find it
inconvenient to avoid the problem. Both sf and sp/rgdal are covering your
backs even if it feels rough. We've already helped dozens of maintainers
of reverse dependencies future-proof their users from the consequences of
changes made in PROJ and GDAL, where we have also contributed to making
the transition easier by exposing helper functions in their APIs, which we
use.

I expect you to be able to CMD check a tarball from the commmand line, no
more than that, to report back, and to try to grasp that these external
software components are really important and completely non-trivial.

Roger

> On 29 May 2020, at 18:37, Roger Bivand wrote:
>
>>  I have put a tarball, built without vignettes with GDAL 2.2.4 and PROJ
>>  7.0.1 that passes CMD check (with warnings for missing vignettes) on:
>>
>>  http://spatial.nhh.no/R/rgdal/gdal_1.5-9.tar.gz
>>
>>  It involves code copying to provide duplicated functions in three
>>  settings:
>>
>>  PROJ < 6 & GDAL < 3
>> PROJ > = 6 & GDAL >= 3
>> PROJ > = 6 & GDAL < 3 (your homebrew case)
>>
>>  You need the configure argument:
>>  --configure-args=--with-proj_api=proj_api.h.
>>  Without the argument, you now get directed to use it.
>>
>>  For now I've dropped the configure tests for sqlite3, curl and tiff; works
>>  for me but may not work for you.
>>
>>  Please make the output of:
>>
>>  R CMD check --install-args="--configure-args=--with-proj_api=proj_api.h"
>>  rgdal_1.5-9.tar.gz
>>
>>  available ASAP. I count on an immediate response.
>>
>>  Roger
>>
>>  On Fri, 29 May 2020, Roger Bivand wrote:
>>
>>>  On Fri, 29 May 2020, Patrick Schratz wrote:
>>>
>>>>   The just released rgdal v1.5.8 requires gdal >= 3 when proj >= 6 is
>>>>   present. It does not even install the package if this condition is
>>>>  not met
>>>>   but errors during linking.
>>>
>>>  Please document with the output of 00install.out from R CMD check
>>>  rgdal_1.5-8.tar.gz, and pkg-config proj --modversion, gdalinfo --version
>>>  with system details.
>>>
>>>  Yes, PROJ >= 6 makes no sense without GDAL >= 3; GDAL 3 makes no sense
>>>  with PROJ < 6. GDAL 3 with PROJ < 6 uses the old proj_api.h API and the
>>>  old metadata structure with Proj4 string degradation; GDAL < 3 with PROJ
>>>>  = 6 must use the deprecated API and the new metadata structure.
>>>
>>>>
>>>>  I could not find any sources in the web or in the announcement post of
>>>>   v1.5.8 why this condition was enforced so strictly.
>>>>   Does it cause unwanted results?
>>>
>>>  Yes, definitely. GDAL 3 was released as 3, not 2.5, to stress the
>>>  complete break between GDAL 2 (with PROJ <= 5) and GDAL >= 3 (with
>>> PROJ > = 6) after GDAL barnraising. So:
>>>
>>>            PROJ
>>>            < 6 >= 6 GDAL < 3   OK   NO
>>>    >  = 3   NO   OK
>>>
>>>
>>>
>>>>  If so, then there is a big issue since the “gdal2 and proj >= 6”
>>>>   combination has been used by many people in the past already.
>>>
>>>  No good precedent, just binary packagers protecting slow downstream
>>>  adaptation.
>>>
>>>>  I haven’t tracked down for how long this situation was on the market
>>>>   already.
>>>>   Also the {sf} package is still installable with this combination
>>>>  and I
>>>>  am not aware of any warnings/issues that came up due to this so far.
>>>>
>>>
>>>  rgdal is a much older package and has much more older code that needs
>>>  this protection. It is possible to accommodate the no-go areas, but I'd
>>>  really value patches to R-forge, as I cannot check multiple PROJ/GDAL
>>>  version combinations just because someone isn't up to speed.
>>>
>>>>  It further causes complete breakage for people relying on the homebrew
>>>>  package manager on macOS since current versions are gdal v2.2.4 and proj
>>>>   7.0.1.
>>>
>>>  They should never have gone beyond PROJ 5 if they are stuck on GDAL. PROJ
>>>  7 is a very different animal.
>>>
>>>>  The update to gdal3 is blocked since months due to an incompatibility
>>>>  with `liblas` (https://github.com/libLAS/libLAS/issues/164). Reading the
>>>>   issue, it looks unclear when this issue will be resolved.
>>>>   The alternative osgeo4mac tap holds formula that is unstable and
>>>>   somewhat broken. One cannot link rgdal with the current gdal v3.0.1
>>>>  from
>>>>   osgeo4mac.
>>>>
>>>>  This incompatibility with homebrew-core formula leads to further issues
>>>>  for CI tests that rely on the spatial homebrew stack for macOS builds.
>>>>
>>>>  All of this is really unfortunate and I am wondering if this was a known
>>>>   factor when introducing this condition in the new release.
>>>>   Unfortunately rgdal is not hosted on any Git* provider (there is
>>>>  just
>>>>  the CRAN mirror with not much information about the recent changes) and
>>>>  it is unclear/untransparent to me if there is a continuous integration
>>>>   setup at all for this package.
>>>
>>>  Do not require me to leave the excellent SVN service on R-forge. Provide
>>>  patches there.
>>>
>>>>  I hope this is not another case in the series “I do not like operating
>>>>  system XY and hence I do not test on it” which was seen recently in
>>>>   another package. That’s what CI is for and the responsibility of
>>>>   package authors.
>>>>
>>>
>>>  I run and develop on Fedora, I  have no access to OSX, and the current
>>>  release was fully checked with ~ 900 revdeps repeatedly since November on
>>>  successive older and newer versions of PROJ and GDAL. I blocked new PROJ
>>>  with old GDAL recently to simplify development. CI is only as good as the
>>>  scripts, I am completely sure that repeated revdep testing and reading
>>>  the output is superior.
>>>
>>>>  Also one should be aware that not every rgdal user is member of this
>>>>  mailing list (I guess not even 5%) and many people in production will
>>>>  face this error during install, most of them without any clue what to do
>>>>   or an idea why this was raised.
>>>>
>>>
>>>  There has been plenty of information given about the CRS changes. rgdal
>>>  could simply have been abandoned by me, and all  those production
>>>  volunteers might have fixed things, but I never hear anything at all.
>>>
>>>>   In summary it would be great if
>>>>
>>>>  - There would be more detailed information on the introduction of the
>>>>   new condition
>>>>   - Awareness of state-of-the-art platform related library versions
>>>>  before
>>>>   making such a release
>>>>   - Transparency/Existence of a cross-platform build process
>>>>   - If the incompatibility is just due to some features not being
>>>>   accessible with this configuration, I suggest to raise a warning
>>>>  during
>>>>  package load but not prevent the package from installing in the first
>>>>   place
>>>>   - A version-based NEWS file would be available. Currently one only
>>>>  sees
>>>>   a revision-based Changelog:
>>>>   https://cran.r-project.org/web/packages/rgdal/ChangeLog
>>>>
>>>>   I am well aware that I am indirectly criticising (hopefully in a
>>>>   constructive way) the transparency and release process here, for
>>>>  which I
>>>>   will probably get a rough response.
>>>>   However, if that leads to more robustness and transparency in
>>>>  future
>>>>   release, I am fine with this.
>>>>
>>>>   A quick patch release resolving the breakage would be highly
>>>>   appreciated.
>>>
>>>  Only with community imput. what you ask is not needed, just extra
>>>  complexity. Please provide patches, or accept my invitation to join the
>>>  R-forge project and commit your fixes directly. I can see how to do it,
>>>  but I don't think it makes sense, and your messsage has not motivated me,
>>>  to be honest. I'm prioritizing working with CRAN to iron out reverse
>>>  dependency problems.
>>>
>>>  Roger
>>>
>>>>
>>>>   Best, Patrick
>>>>    [[alternative HTML version deleted]]
>>>>
>>>>   _______________________________________________
>>>>   R-sig-Geo mailing list
>>>>   [hidden email]
>>>>   https://stat.ethz.ch/mailman/listinfo/r-sig-geo
>>>>
>>>
>>>
>>
>>  --
>>  Roger Bivand
>>  Department of Economics, Norwegian School of Economics,
>>  Helleveien 30, N-5045 Bergen, Norway.
>>  voice: +47 55 95 93 55; e-mail: [hidden email]
>>  https://orcid.org/0000-0003-2392-6140
>>  https://scholar.google.no/citations?user=AWeghB0AAAAJ&hl=en
>
>
--
Roger Bivand
Department of Economics, Norwegian School of Economics,
Helleveien 30, N-5045 Bergen, Norway.
voice: +47 55 95 93 55; e-mail: [hidden email]
https://orcid.org/0000-0003-2392-6140
https://scholar.google.no/citations?user=AWeghB0AAAAJ&hl=en
_______________________________________________
R-sig-Geo mailing list
[hidden email]
https://stat.ethz.ch/mailman/listinfo/r-sig-geo
Roger Bivand
Department of Economics
Norwegian School of Economics
Helleveien 30
N-5045 Bergen, Norway
Reply | Threaded
Open this post in threaded view
|

Re: rgdal: About the new gdal3 and proj >=6 condition

R-sig-geo mailing list
Hi All:

It appears that Anaconda has the requisite versions,  as well as gdal-config and the appropriate include files etc etc,  though I haven’t tried to compile yet,  as I have to be on a different computer. So I have two questions for now:

1.  Other than the rgdal source and setting everything to point to the correct locations,  is there anything else I need to have installed  (I have the compilers).  If I can get it to work,  this might be for R4.0 only,  I don’t keep two versions around,  and I don’t have Catalina.

2.  I assume there are other packages on the Mac that are in a similar situation,  would I be correct in assuming sp and sf?

Thanks,

-Roy


> On May 29, 2020, at 1:44 PM, Roger Bivand <[hidden email]> wrote:
>
> On Fri, 29 May 2020, Patrick Schratz wrote:
>
>> The URL returns a 404.
>
> Do you mean http://spatial.nhh.no/R/rgdal/? <http://spatial.nhh.no/R/rgdal/?> Please do say what you mean. If your browser blocks http: rather than https:, that is your choice. I imagine that it silently inserts an "s".
>
>>
>> FWIW I’ve tried it locally using the latest svn revision but have no clue where I should find `proj_api.h` (I’ve looked in the complete proj installation directory).
>
> If PROJ 7.0.1 is properly installed, the header will be in $prefix/include. If your packager put all the headers somewhere else, use the command argument:
>
> $ ls /usr/local/include/proj*
> /usr/local/include/proj_api.h           /usr/local/include/proj.h
> /usr/local/include/proj_constants.h /usr/local/include/proj_symbol_rename.h
> /usr/local/include/proj_experimental.h
>
> /usr/local/include/proj:
> common.hpp               crs.hpp    metadata.hpp
> coordinateoperation.hpp  datum.hpp  nn.hpp
> coordinatesystem.hpp     io.hpp     util.hpp
>
> You should not need to know. What does pkg-config proj --cflags say (once you've set PKG_CONFIG_PATH, of course.
>
>>
>> I see
>>
>> ```
>> ls /usr/local/Cellar/proj/7.0.1/lib/
>> libproj.19.dylib  libproj.a         libproj.dylib@    libproj.la <http://libproj.la/>* pkgconfig/
>> ```
>>
>
> In your earlier post, you negected to say that you used the necessary configure argument. The error message says that you did not use it. Please report back when you have done so. Futher, I quote from your message: "It further causes complete breakage for people relying on the homebrew package manager on macOS since current versions are gdal v2.2.4 and proj 7.0.1." Please accept that I pay attention to what you write.
>
>> Even if this custom installation might work at some point, making such adjustments until gdal3 arrives in homebrew will be a major pain, also for CI builds.
>> It would still be highly appreciated if this internal assertion would be removed again.
>>
>
> Getting things right may be painful; GDAL 2 with PROJ 7 is wrong, because it mixes two incompatible metadata storage systems. Migrating all users to
> GDAL >= 3 & PROJ >= 6 is a matter of urgency to avoid silent positional error of about 125m, or about 4 satellite image cells. See the vignette here http://rgdal.r-forge.r-project.org/articles/CRS_projections_transformations.html <http://rgdal.r-forge.r-project.org/articles/CRS_projections_transformations.html> and this r-spatial blog https://www.r-spatial.org/r/2020/03/17/wkt.html <https://www.r-spatial.org/r/2020/03/17/wkt.html>.
>
> I am not willing to let silent error creep in because some people find it inconvenient to avoid the problem. Both sf and sp/rgdal are covering your backs even if it feels rough. We've already helped dozens of maintainers of reverse dependencies future-proof their users from the consequences of changes made in PROJ and GDAL, where we have also contributed to making the transition easier by exposing helper functions in their APIs, which we use.
>
> I expect you to be able to CMD check a tarball from the commmand line, no more than that, to report back, and to try to grasp that these external software components are really important and completely non-trivial.
>
> Roger
>
>> On 29 May 2020, at 18:37, Roger Bivand wrote:
>>
>>> I have put a tarball, built without vignettes with GDAL 2.2.4 and PROJ
>>> 7.0.1 that passes CMD check (with warnings for missing vignettes) on:
>>>
>>> http://spatial.nhh.no/R/rgdal/gdal_1.5-9.tar.gz
>>>
>>> It involves code copying to provide duplicated functions in three
>>> settings:
>>>
>>> PROJ < 6 & GDAL < 3
>>> PROJ > = 6 & GDAL >= 3
>>> PROJ > = 6 & GDAL < 3 (your homebrew case)
>>>
>>> You need the configure argument:
>>> --configure-args=--with-proj_api=proj_api.h.
>>> Without the argument, you now get directed to use it.
>>>
>>> For now I've dropped the configure tests for sqlite3, curl and tiff; works
>>> for me but may not work for you.
>>>
>>> Please make the output of:
>>>
>>> R CMD check --install-args="--configure-args=--with-proj_api=proj_api.h"
>>> rgdal_1.5-9.tar.gz
>>>
>>> available ASAP. I count on an immediate response.
>>>
>>> Roger
>>>
>>> On Fri, 29 May 2020, Roger Bivand wrote:
>>>
>>>> On Fri, 29 May 2020, Patrick Schratz wrote:
>>>>
>>>>>  The just released rgdal v1.5.8 requires gdal >= 3 when proj >= 6 is
>>>>>  present. It does not even install the package if this condition is
>>>>> not met
>>>>>  but errors during linking.
>>>>
>>>> Please document with the output of 00install.out from R CMD check
>>>> rgdal_1.5-8.tar.gz, and pkg-config proj --modversion, gdalinfo --version
>>>> with system details.
>>>>
>>>> Yes, PROJ >= 6 makes no sense without GDAL >= 3; GDAL 3 makes no sense
>>>> with PROJ < 6. GDAL 3 with PROJ < 6 uses the old proj_api.h API and the
>>>> old metadata structure with Proj4 string degradation; GDAL < 3 with PROJ
>>>>> = 6 must use the deprecated API and the new metadata structure.
>>>>>
>>>>> I could not find any sources in the web or in the announcement post of
>>>>>  v1.5.8 why this condition was enforced so strictly.
>>>>>  Does it cause unwanted results?
>>>>
>>>> Yes, definitely. GDAL 3 was released as 3, not 2.5, to stress the
>>>> complete break between GDAL 2 (with PROJ <= 5) and GDAL >= 3 (with PROJ > = 6) after GDAL barnraising. So:
>>>>
>>>>           PROJ
>>>>           < 6 >= 6 GDAL < 3   OK   NO
>>>>   >  = 3   NO   OK
>>>>
>>>>> If so, then there is a big issue since the “gdal2 and proj >= 6”
>>>>>  combination has been used by many people in the past already.
>>>>
>>>> No good precedent, just binary packagers protecting slow downstream
>>>> adaptation.
>>>>
>>>>> I haven’t tracked down for how long this situation was on the market
>>>>>  already.
>>>>>  Also the {sf} package is still installable with this combination
>>>>> and I
>>>>> am not aware of any warnings/issues that came up due to this so far.
>>>>
>>>> rgdal is a much older package and has much more older code that needs
>>>> this protection. It is possible to accommodate the no-go areas, but I'd
>>>> really value patches to R-forge, as I cannot check multiple PROJ/GDAL
>>>> version combinations just because someone isn't up to speed.
>>>>
>>>>> It further causes complete breakage for people relying on the homebrew
>>>>> package manager on macOS since current versions are gdal v2.2.4 and proj
>>>>>  7.0.1.
>>>>
>>>> They should never have gone beyond PROJ 5 if they are stuck on GDAL. PROJ
>>>> 7 is a very different animal.
>>>>
>>>>> The update to gdal3 is blocked since months due to an incompatibility
>>>>> with `liblas` (https://github.com/libLAS/libLAS/issues/164). Reading the
>>>>>  issue, it looks unclear when this issue will be resolved.
>>>>>  The alternative osgeo4mac tap holds formula that is unstable and
>>>>>  somewhat broken. One cannot link rgdal with the current gdal v3.0.1
>>>>> from
>>>>>  osgeo4mac.
>>>>>
>>>>> This incompatibility with homebrew-core formula leads to further issues
>>>>> for CI tests that rely on the spatial homebrew stack for macOS builds.
>>>>>
>>>>> All of this is really unfortunate and I am wondering if this was a known
>>>>>  factor when introducing this condition in the new release.
>>>>>  Unfortunately rgdal is not hosted on any Git* provider (there is
>>>>> just
>>>>> the CRAN mirror with not much information about the recent changes) and
>>>>> it is unclear/untransparent to me if there is a continuous integration
>>>>>  setup at all for this package.
>>>>
>>>> Do not require me to leave the excellent SVN service on R-forge. Provide
>>>> patches there.
>>>>
>>>>> I hope this is not another case in the series “I do not like operating
>>>>> system XY and hence I do not test on it” which was seen recently in
>>>>>  another package. That’s what CI is for and the responsibility of
>>>>>  package authors.
>>>>
>>>> I run and develop on Fedora, I  have no access to OSX, and the current
>>>> release was fully checked with ~ 900 revdeps repeatedly since November on
>>>> successive older and newer versions of PROJ and GDAL. I blocked new PROJ
>>>> with old GDAL recently to simplify development. CI is only as good as the
>>>> scripts, I am completely sure that repeated revdep testing and reading
>>>> the output is superior.
>>>>
>>>>> Also one should be aware that not every rgdal user is member of this
>>>>> mailing list (I guess not even 5%) and many people in production will
>>>>> face this error during install, most of them without any clue what to do
>>>>>  or an idea why this was raised.
>>>>
>>>> There has been plenty of information given about the CRS changes. rgdal
>>>> could simply have been abandoned by me, and all  those production
>>>> volunteers might have fixed things, but I never hear anything at all.
>>>>
>>>>>  In summary it would be great if
>>>>>
>>>>> - There would be more detailed information on the introduction of the
>>>>>  new condition
>>>>>  - Awareness of state-of-the-art platform related library versions
>>>>> before
>>>>>  making such a release
>>>>>  - Transparency/Existence of a cross-platform build process
>>>>>  - If the incompatibility is just due to some features not being
>>>>>  accessible with this configuration, I suggest to raise a warning
>>>>> during
>>>>> package load but not prevent the package from installing in the first
>>>>>  place
>>>>>  - A version-based NEWS file would be available. Currently one only
>>>>> sees
>>>>>  a revision-based Changelog:
>>>>>  https://cran.r-project.org/web/packages/rgdal/ChangeLog
>>>>>
>>>>>  I am well aware that I am indirectly criticising (hopefully in a
>>>>>  constructive way) the transparency and release process here, for
>>>>> which I
>>>>>  will probably get a rough response.
>>>>>  However, if that leads to more robustness and transparency in
>>>>> future
>>>>>  release, I am fine with this.
>>>>>
>>>>>  A quick patch release resolving the breakage would be highly
>>>>>  appreciated.
>>>>
>>>> Only with community imput. what you ask is not needed, just extra
>>>> complexity. Please provide patches, or accept my invitation to join the
>>>> R-forge project and commit your fixes directly. I can see how to do it,
>>>> but I don't think it makes sense, and your messsage has not motivated me,
>>>> to be honest. I'm prioritizing working with CRAN to iron out reverse
>>>> dependency problems.
>>>>
>>>> Roger
>>>>>
>>>>>  Best, Patrick
>>>>>   [[alternative HTML version deleted]]
>>>>>
>>>>>  _______________________________________________
>>>>>  R-sig-Geo mailing list
>>>>>  [hidden email]
>>>>>  https://stat.ethz.ch/mailman/listinfo/r-sig-geo
>>>
>>> --
>>> Roger Bivand
>>> Department of Economics, Norwegian School of Economics,
>>> Helleveien 30, N-5045 Bergen, Norway.
>>> voice: +47 55 95 93 55; e-mail: [hidden email]
>>> https://orcid.org/0000-0003-2392-6140
>>> https://scholar.google.no/citations?user=AWeghB0AAAAJ&hl=en
>>
>>
>
> --
> Roger Bivand
> Department of Economics, Norwegian School of Economics,
> Helleveien 30, N-5045 Bergen, Norway.
> voice: +47 55 95 93 55; e-mail: [hidden email] <mailto:[hidden email]>
> https://orcid.org/0000-0003-2392-6140 <https://orcid.org/0000-0003-2392-6140>
> https://scholar.google.no/citations?user=AWeghB0AAAAJ&hl=en_______________________________________________ <https://scholar.google.no/citations?user=AWeghB0AAAAJ&hl=en_______________________________________________>
> R-sig-Geo mailing list
> [hidden email] <mailto:[hidden email]>
> https://stat.ethz.ch/mailman/listinfo/r-sig-geo <https://stat.ethz.ch/mailman/listinfo/r-sig-geo>

        [[alternative HTML version deleted]]

_______________________________________________
R-sig-Geo mailing list
[hidden email]
https://stat.ethz.ch/mailman/listinfo/r-sig-geo
Reply | Threaded
Open this post in threaded view
|

Re: rgdal: About the new gdal3 and proj >=6 condition

chris english-2
rgdal-1.5-9 fixed all my install problems on ubuntu 20.04. My mother has a
needle point pillow at her summer home
which declares "Guests of Guests May Not Bring Guests". Analogously,
'dependencies of dependencies will not
check dependencies' is applicable here. rgdal-1.5-9 facilitated:

> library(rgdal)
Loading required package: sp
rgdal: version: 1.5-9, (SVN revision 991M)
Geospatial Data Abstraction Library extensions to R successfully loaded
Loaded GDAL runtime: GDAL 3.1.0, released 2020/05/03
Path to GDAL shared files: /usr/local/share/gdal
GDAL binary built with GEOS: TRUE
Loaded PROJ runtime: Rel. 7.0.1, May 1st, 2020, [PJ_VERSION: 701]
Path to PROJ shared files:
/root/.local/share/proj:/usr/local/share/proj:/usr/local/share/proj
PROJ CDN enabled:FALSE
Linking to sp version:1.4-1
To mute warnings of possible GDAL/OSR exportToProj4() degradation,
use options("rgdal_show_exportToProj4_warnings"="none") before loading
rgdal.

GEOS version is 3.9.0dev

I still have some work to do on the PostgreSQL end of things, but I am back
to working
spatial. I still received errors unless I started the Install session in
sudo R, which leads me
to think that my /usr/local/(bin,lib,include) are symlinks from
one/some/all installs of GEOS,
PROJ, GDAL, which are not the fault of R developers, but I'd forgotten this
issue and rediscovered
that I pestered Roger and Edzer about it in 2019. So, sudo R.

I'm back to having a working spatial environment. Thank you Roger!

Chris

On Fri, May 29, 2020 at 5:16 PM Roy Mendelssohn - NOAA Federal via
R-sig-Geo <[hidden email]> wrote:

> Hi All:
>
> It appears that Anaconda has the requisite versions,  as well as
> gdal-config and the appropriate include files etc etc,  though I haven’t
> tried to compile yet,  as I have to be on a different computer. So I have
> two questions for now:
>
> 1.  Other than the rgdal source and setting everything to point to the
> correct locations,  is there anything else I need to have installed  (I
> have the compilers).  If I can get it to work,  this might be for R4.0
> only,  I don’t keep two versions around,  and I don’t have Catalina.
>
> 2.  I assume there are other packages on the Mac that are in a similar
> situation,  would I be correct in assuming sp and sf?
>
> Thanks,
>
> -Roy
>
>
> > On May 29, 2020, at 1:44 PM, Roger Bivand <[hidden email]> wrote:
> >
> > On Fri, 29 May 2020, Patrick Schratz wrote:
> >
> >> The URL returns a 404.
> >
> > Do you mean http://spatial.nhh.no/R/rgdal/? <
> http://spatial.nhh.no/R/rgdal/?> Please do say what you mean. If your
> browser blocks http: rather than https:, that is your choice. I imagine
> that it silently inserts an "s".
> >
> >>
> >> FWIW I’ve tried it locally using the latest svn revision but have no
> clue where I should find `proj_api.h` (I’ve looked in the complete proj
> installation directory).
> >
> > If PROJ 7.0.1 is properly installed, the header will be in
> $prefix/include. If your packager put all the headers somewhere else, use
> the command argument:
> >
> > $ ls /usr/local/include/proj*
> > /usr/local/include/proj_api.h           /usr/local/include/proj.h
> > /usr/local/include/proj_constants.h
> /usr/local/include/proj_symbol_rename.h
> > /usr/local/include/proj_experimental.h
> >
> > /usr/local/include/proj:
> > common.hpp               crs.hpp    metadata.hpp
> > coordinateoperation.hpp  datum.hpp  nn.hpp
> > coordinatesystem.hpp     io.hpp     util.hpp
> >
> > You should not need to know. What does pkg-config proj --cflags say
> (once you've set PKG_CONFIG_PATH, of course.
> >
> >>
> >> I see
> >>
> >> ```
> >> ls /usr/local/Cellar/proj/7.0.1/lib/
> >> libproj.19.dylib  libproj.a         libproj.dylib@    libproj.la <
> http://libproj.la/>* pkgconfig/
> >> ```
> >>
> >
> > In your earlier post, you negected to say that you used the necessary
> configure argument. The error message says that you did not use it. Please
> report back when you have done so. Futher, I quote from your message: "It
> further causes complete breakage for people relying on the homebrew package
> manager on macOS since current versions are gdal v2.2.4 and proj 7.0.1."
> Please accept that I pay attention to what you write.
> >
> >> Even if this custom installation might work at some point, making such
> adjustments until gdal3 arrives in homebrew will be a major pain, also for
> CI builds.
> >> It would still be highly appreciated if this internal assertion would
> be removed again.
> >>
> >
> > Getting things right may be painful; GDAL 2 with PROJ 7 is wrong,
> because it mixes two incompatible metadata storage systems. Migrating all
> users to
> > GDAL >= 3 & PROJ >= 6 is a matter of urgency to avoid silent positional
> error of about 125m, or about 4 satellite image cells. See the vignette
> here
> http://rgdal.r-forge.r-project.org/articles/CRS_projections_transformations.html
> <
> http://rgdal.r-forge.r-project.org/articles/CRS_projections_transformations.html>
> and this r-spatial blog https://www.r-spatial.org/r/2020/03/17/wkt.html <
> https://www.r-spatial.org/r/2020/03/17/wkt.html>.
> >
> > I am not willing to let silent error creep in because some people find
> it inconvenient to avoid the problem. Both sf and sp/rgdal are covering
> your backs even if it feels rough. We've already helped dozens of
> maintainers of reverse dependencies future-proof their users from the
> consequences of changes made in PROJ and GDAL, where we have also
> contributed to making the transition easier by exposing helper functions in
> their APIs, which we use.
> >
> > I expect you to be able to CMD check a tarball from the commmand line,
> no more than that, to report back, and to try to grasp that these external
> software components are really important and completely non-trivial.
> >
> > Roger
> >
> >> On 29 May 2020, at 18:37, Roger Bivand wrote:
> >>
> >>> I have put a tarball, built without vignettes with GDAL 2.2.4 and PROJ
> >>> 7.0.1 that passes CMD check (with warnings for missing vignettes) on:
> >>>
> >>> http://spatial.nhh.no/R/rgdal/gdal_1.5-9.tar.gz
> >>>
> >>> It involves code copying to provide duplicated functions in three
> >>> settings:
> >>>
> >>> PROJ < 6 & GDAL < 3
> >>> PROJ > = 6 & GDAL >= 3
> >>> PROJ > = 6 & GDAL < 3 (your homebrew case)
> >>>
> >>> You need the configure argument:
> >>> --configure-args=--with-proj_api=proj_api.h.
> >>> Without the argument, you now get directed to use it.
> >>>
> >>> For now I've dropped the configure tests for sqlite3, curl and tiff;
> works
> >>> for me but may not work for you.
> >>>
> >>> Please make the output of:
> >>>
> >>> R CMD check
> --install-args="--configure-args=--with-proj_api=proj_api.h"
> >>> rgdal_1.5-9.tar.gz
> >>>
> >>> available ASAP. I count on an immediate response.
> >>>
> >>> Roger
> >>>
> >>> On Fri, 29 May 2020, Roger Bivand wrote:
> >>>
> >>>> On Fri, 29 May 2020, Patrick Schratz wrote:
> >>>>
> >>>>>  The just released rgdal v1.5.8 requires gdal >= 3 when proj >= 6 is
> >>>>>  present. It does not even install the package if this condition is
> >>>>> not met
> >>>>>  but errors during linking.
> >>>>
> >>>> Please document with the output of 00install.out from R CMD check
> >>>> rgdal_1.5-8.tar.gz, and pkg-config proj --modversion, gdalinfo
> --version
> >>>> with system details.
> >>>>
> >>>> Yes, PROJ >= 6 makes no sense without GDAL >= 3; GDAL 3 makes no sense
> >>>> with PROJ < 6. GDAL 3 with PROJ < 6 uses the old proj_api.h API and
> the
> >>>> old metadata structure with Proj4 string degradation; GDAL < 3 with
> PROJ
> >>>>> = 6 must use the deprecated API and the new metadata structure.
> >>>>>
> >>>>> I could not find any sources in the web or in the announcement post
> of
> >>>>>  v1.5.8 why this condition was enforced so strictly.
> >>>>>  Does it cause unwanted results?
> >>>>
> >>>> Yes, definitely. GDAL 3 was released as 3, not 2.5, to stress the
> >>>> complete break between GDAL 2 (with PROJ <= 5) and GDAL >= 3 (with
> PROJ > = 6) after GDAL barnraising. So:
> >>>>
> >>>>           PROJ
> >>>>           < 6 >= 6 GDAL < 3   OK   NO
> >>>>   >  = 3   NO   OK
> >>>>
> >>>>> If so, then there is a big issue since the “gdal2 and proj >= 6”
> >>>>>  combination has been used by many people in the past already.
> >>>>
> >>>> No good precedent, just binary packagers protecting slow downstream
> >>>> adaptation.
> >>>>
> >>>>> I haven’t tracked down for how long this situation was on the market
> >>>>>  already.
> >>>>>  Also the {sf} package is still installable with this combination
> >>>>> and I
> >>>>> am not aware of any warnings/issues that came up due to this so far.
> >>>>
> >>>> rgdal is a much older package and has much more older code that needs
> >>>> this protection. It is possible to accommodate the no-go areas, but
> I'd
> >>>> really value patches to R-forge, as I cannot check multiple PROJ/GDAL
> >>>> version combinations just because someone isn't up to speed.
> >>>>
> >>>>> It further causes complete breakage for people relying on the
> homebrew
> >>>>> package manager on macOS since current versions are gdal v2.2.4 and
> proj
> >>>>>  7.0.1.
> >>>>
> >>>> They should never have gone beyond PROJ 5 if they are stuck on GDAL.
> PROJ
> >>>> 7 is a very different animal.
> >>>>
> >>>>> The update to gdal3 is blocked since months due to an incompatibility
> >>>>> with `liblas` (https://github.com/libLAS/libLAS/issues/164).
> Reading the
> >>>>>  issue, it looks unclear when this issue will be resolved.
> >>>>>  The alternative osgeo4mac tap holds formula that is unstable and
> >>>>>  somewhat broken. One cannot link rgdal with the current gdal v3.0.1
> >>>>> from
> >>>>>  osgeo4mac.
> >>>>>
> >>>>> This incompatibility with homebrew-core formula leads to further
> issues
> >>>>> for CI tests that rely on the spatial homebrew stack for macOS
> builds.
> >>>>>
> >>>>> All of this is really unfortunate and I am wondering if this was a
> known
> >>>>>  factor when introducing this condition in the new release.
> >>>>>  Unfortunately rgdal is not hosted on any Git* provider (there is
> >>>>> just
> >>>>> the CRAN mirror with not much information about the recent changes)
> and
> >>>>> it is unclear/untransparent to me if there is a continuous
> integration
> >>>>>  setup at all for this package.
> >>>>
> >>>> Do not require me to leave the excellent SVN service on R-forge.
> Provide
> >>>> patches there.
> >>>>
> >>>>> I hope this is not another case in the series “I do not like
> operating
> >>>>> system XY and hence I do not test on it” which was seen recently in
> >>>>>  another package. That’s what CI is for and the responsibility of
> >>>>>  package authors.
> >>>>
> >>>> I run and develop on Fedora, I  have no access to OSX, and the current
> >>>> release was fully checked with ~ 900 revdeps repeatedly since
> November on
> >>>> successive older and newer versions of PROJ and GDAL. I blocked new
> PROJ
> >>>> with old GDAL recently to simplify development. CI is only as good as
> the
> >>>> scripts, I am completely sure that repeated revdep testing and reading
> >>>> the output is superior.
> >>>>
> >>>>> Also one should be aware that not every rgdal user is member of this
> >>>>> mailing list (I guess not even 5%) and many people in production will
> >>>>> face this error during install, most of them without any clue what
> to do
> >>>>>  or an idea why this was raised.
> >>>>
> >>>> There has been plenty of information given about the CRS changes.
> rgdal
> >>>> could simply have been abandoned by me, and all  those production
> >>>> volunteers might have fixed things, but I never hear anything at all.
> >>>>
> >>>>>  In summary it would be great if
> >>>>>
> >>>>> - There would be more detailed information on the introduction of the
> >>>>>  new condition
> >>>>>  - Awareness of state-of-the-art platform related library versions
> >>>>> before
> >>>>>  making such a release
> >>>>>  - Transparency/Existence of a cross-platform build process
> >>>>>  - If the incompatibility is just due to some features not being
> >>>>>  accessible with this configuration, I suggest to raise a warning
> >>>>> during
> >>>>> package load but not prevent the package from installing in the first
> >>>>>  place
> >>>>>  - A version-based NEWS file would be available. Currently one only
> >>>>> sees
> >>>>>  a revision-based Changelog:
> >>>>>  https://cran.r-project.org/web/packages/rgdal/ChangeLog
> >>>>>
> >>>>>  I am well aware that I am indirectly criticising (hopefully in a
> >>>>>  constructive way) the transparency and release process here, for
> >>>>> which I
> >>>>>  will probably get a rough response.
> >>>>>  However, if that leads to more robustness and transparency in
> >>>>> future
> >>>>>  release, I am fine with this.
> >>>>>
> >>>>>  A quick patch release resolving the breakage would be highly
> >>>>>  appreciated.
> >>>>
> >>>> Only with community imput. what you ask is not needed, just extra
> >>>> complexity. Please provide patches, or accept my invitation to join
> the
> >>>> R-forge project and commit your fixes directly. I can see how to do
> it,
> >>>> but I don't think it makes sense, and your messsage has not motivated
> me,
> >>>> to be honest. I'm prioritizing working with CRAN to iron out reverse
> >>>> dependency problems.
> >>>>
> >>>> Roger
> >>>>>
> >>>>>  Best, Patrick
> >>>>>   [[alternative HTML version deleted]]
> >>>>>
> >>>>>  _______________________________________________
> >>>>>  R-sig-Geo mailing list
> >>>>>  [hidden email]
> >>>>>  https://stat.ethz.ch/mailman/listinfo/r-sig-geo
> >>>
> >>> --
> >>> Roger Bivand
> >>> Department of Economics, Norwegian School of Economics,
> >>> Helleveien 30, N-5045 Bergen, Norway.
> >>> voice: +47 55 95 93 55; e-mail: [hidden email]
> >>> https://orcid.org/0000-0003-2392-6140
> >>> https://scholar.google.no/citations?user=AWeghB0AAAAJ&hl=en
> >>
> >>
> >
> > --
> > Roger Bivand
> > Department of Economics, Norwegian School of Economics,
> > Helleveien 30, N-5045 Bergen, Norway.
> > voice: +47 55 95 93 55; e-mail: [hidden email] <mailto:
> [hidden email]>
> > https://orcid.org/0000-0003-2392-6140 <
> https://orcid.org/0000-0003-2392-6140>
> >
> https://scholar.google.no/citations?user=AWeghB0AAAAJ&hl=en_______________________________________________
> <
> https://scholar.google.no/citations?user=AWeghB0AAAAJ&hl=en_______________________________________________
> >
> > R-sig-Geo mailing list
> > [hidden email] <mailto:[hidden email]>
> > https://stat.ethz.ch/mailman/listinfo/r-sig-geo <
> https://stat.ethz.ch/mailman/listinfo/r-sig-geo>
>
>         [[alternative HTML version deleted]]
>
> _______________________________________________
> R-sig-Geo mailing list
> [hidden email]
> https://stat.ethz.ch/mailman/listinfo/r-sig-geo
>

        [[alternative HTML version deleted]]

_______________________________________________
R-sig-Geo mailing list
[hidden email]
https://stat.ethz.ch/mailman/listinfo/r-sig-geo
Reply | Threaded
Open this post in threaded view
|

Re: rgdal: About the new gdal3 and proj >=6 condition

Roger Bivand
Administrator
In reply to this post by Roger Bivand
Fresh version of rgdal rev 995 on link and R-forge, with more tweaks to
the configure file. Please test.

Roger

On Fri, 29 May 2020, Roger Bivand wrote:

> I have put a tarball, built without vignettes with GDAL 2.2.4 and PROJ 7.0.1
> that passes CMD check (with warnings for missing vignettes) on:
>
> http://spatial.nhh.no/R/rgdal/gdal_1.5-9.tar.gz
>
> It involves code copying to provide duplicated functions in three settings:
>
> PROJ < 6 & GDAL < 3
> PROJ > = 6 & GDAL >= 3
> PROJ > = 6 & GDAL < 3 (your homebrew case)
>
> You need the configure argument: --configure-args=--with-proj_api=proj_api.h.
> Without the argument, you now get directed to use it.
>
> For now I've dropped the configure tests for sqlite3, curl and tiff; works
> for me but may not work for you.
>
> Please make the output of:
>
> R CMD check --install-args="--configure-args=--with-proj_api=proj_api.h"
> rgdal_1.5-9.tar.gz
>
> available ASAP. I count on an immediate response.
>
> Roger
>
> On Fri, 29 May 2020, Roger Bivand wrote:
>
>>  On Fri, 29 May 2020, Patrick Schratz wrote:
>>
>>>   The just released rgdal v1.5.8 requires gdal >= 3 when proj >= 6 is
>>>   present. It does not even install the package if this condition is not
>>>   met
>>>   but errors during linking.
>>
>>  Please document with the output of 00install.out from R CMD check
>>  rgdal_1.5-8.tar.gz, and pkg-config proj --modversion, gdalinfo --version
>>  with system details.
>>
>>  Yes, PROJ >= 6 makes no sense without GDAL >= 3; GDAL 3 makes no sense
>>  with PROJ < 6. GDAL 3 with PROJ < 6 uses the old proj_api.h API and the
>>  old metadata structure with Proj4 string degradation; GDAL < 3 with PROJ
>>>  = 6 must use the deprecated API and the new metadata structure.
>>
>>>
>>>   I could not find any sources in the web or in the announcement post of
>>>   v1.5.8 why this condition was enforced so strictly.
>>>   Does it cause unwanted results?
>>
>>  Yes, definitely. GDAL 3 was released as 3, not 2.5, to stress the complete
>>  break between GDAL 2 (with PROJ <= 5) and GDAL >= 3 (with PROJ >= 6) after
>>  GDAL barnraising. So:
>>
>>            PROJ
>>            < 6 >= 6
>>  GDAL < 3   OK   NO
>>    >  = 3   NO   OK
>>
>>
>>
>>>   If so, then there is a big issue since the “gdal2 and proj >= 6”
>>>   combination has been used by many people in the past already.
>>
>>  No good precedent, just binary packagers protecting slow downstream
>>  adaptation.
>>
>>>   I haven’t tracked down for how long this situation was on the market
>>>   already.
>>>   Also the {sf} package is still installable with this combination and I
>>>   am not aware of any warnings/issues that came up due to this so far.
>>>
>>
>>  rgdal is a much older package and has much more older code that needs this
>>  protection. It is possible to accommodate the no-go areas, but I'd really
>>  value patches to R-forge, as I cannot check multiple PROJ/GDAL version
>>  combinations just because someone isn't up to speed.
>>
>>>   It further causes complete breakage for people relying on the homebrew
>>>   package manager on macOS since current versions are gdal v2.2.4 and proj
>>>   7.0.1.
>>
>>  They should never have gone beyond PROJ 5 if they are stuck on GDAL. PROJ
>>  7 is a very different animal.
>>
>>>   The update to gdal3 is blocked since months due to an incompatibility
>>>   with `liblas` (https://github.com/libLAS/libLAS/issues/164). Reading the
>>>   issue, it looks unclear when this issue will be resolved.
>>>   The alternative osgeo4mac tap holds formula that is unstable and
>>>   somewhat broken. One cannot link rgdal with the current gdal v3.0.1 from
>>>   osgeo4mac.
>>>
>>>   This incompatibility with homebrew-core formula leads to further issues
>>>   for CI tests that rely on the spatial homebrew stack for macOS builds.
>>>
>>>   All of this is really unfortunate and I am wondering if this was a known
>>>   factor when introducing this condition in the new release.
>>>   Unfortunately rgdal is not hosted on any Git* provider (there is just
>>>   the CRAN mirror with not much information about the recent changes) and
>>>   it is unclear/untransparent to me if there is a continuous integration
>>>   setup at all for this package.
>>
>>  Do not require me to leave the excellent SVN service on R-forge. Provide
>>  patches there.
>>
>>>   I hope this is not another case in the series “I do not like operating
>>>   system XY and hence I do not test on it” which was seen recently in
>>>   another package. That’s what CI is for and the responsibility of
>>>   package authors.
>>>
>>
>>  I run and develop on Fedora, I  have no access to OSX, and the current
>>  release was fully checked with ~ 900 revdeps repeatedly since November on
>>  successive older and newer versions of PROJ and GDAL. I blocked new PROJ
>>  with old GDAL recently to simplify development. CI is only as good as the
>>  scripts, I am completely sure that repeated revdep testing and reading the
>>  output is superior.
>>
>>>   Also one should be aware that not every rgdal user is member of this
>>>   mailing list (I guess not even 5%) and many people in production will
>>>   face this error during install, most of them without any clue what to do
>>>   or an idea why this was raised.
>>>
>>
>>  There has been plenty of information given about the CRS changes. rgdal
>>  could simply have been abandoned by me, and all  those production
>>  volunteers might have fixed things, but I never hear anything at all.
>>
>>>   In summary it would be great if
>>>
>>>   - There would be more detailed information on the introduction of the
>>>   new condition
>>>   - Awareness of state-of-the-art platform related library versions before
>>>   making such a release
>>>   - Transparency/Existence of a cross-platform build process
>>>   - If the incompatibility is just due to some features not being
>>>   accessible with this configuration, I suggest to raise a warning during
>>>   package load but not prevent the package from installing in the first
>>>   place
>>>   - A version-based NEWS file would be available. Currently one only sees
>>>   a revision-based Changelog:
>>>   https://cran.r-project.org/web/packages/rgdal/ChangeLog
>>>
>>>   I am well aware that I am indirectly criticising (hopefully in a
>>>   constructive way) the transparency and release process here, for which I
>>>   will probably get a rough response.
>>>   However, if that leads to more robustness and transparency in future
>>>   release, I am fine with this.
>>>
>>>   A quick patch release resolving the breakage would be highly
>>>   appreciated.
>>
>>  Only with community imput. what you ask is not needed, just extra
>>  complexity. Please provide patches, or accept my invitation to join the
>>  R-forge project and commit your fixes directly. I can see how to do it,
>>  but I don't think it makes sense, and your messsage has not motivated me,
>>  to be honest. I'm prioritizing working with CRAN to iron out reverse
>>  dependency problems.
>>
>>  Roger
>>
>>>
>>>   Best, Patrick
>>>    [[alternative HTML version deleted]]
>>>
>>>   _______________________________________________
>>>   R-sig-Geo mailing list
>>>   [hidden email]
>>>   https://stat.ethz.ch/mailman/listinfo/r-sig-geo
>>>
>>
>>
>
>
--
Roger Bivand
Department of Economics, Norwegian School of Economics,
Helleveien 30, N-5045 Bergen, Norway.
voice: +47 55 95 93 55; e-mail: [hidden email]
https://orcid.org/0000-0003-2392-6140
https://scholar.google.no/citations?user=AWeghB0AAAAJ&hl=en
_______________________________________________
R-sig-Geo mailing list
[hidden email]
https://stat.ethz.ch/mailman/listinfo/r-sig-geo
Roger Bivand
Department of Economics
Norwegian School of Economics
Helleveien 30
N-5045 Bergen, Norway
Reply | Threaded
Open this post in threaded view
|

Re: rgdal: About the new gdal3 and proj >=6 condition

Roger Bivand
Administrator
Brian Ripley (thanks!) provided a patch for rgdal's configure.ac, and the
R-forge build and check processes now succeed, so please try to install
either downloading
http://download.r-forge.r-project.org/src/contrib/rgdal_1.5-9.tar.gz and
installing from the command line (after running R CMD check), or
install.packages("rgdal", repos="http://R-Forge.R-project.org").

Remember that if GDAL is < 3, and PROJ >= 6, you need the configure
argument: --configure-args=--with-proj_api=proj_api.h.

Roger

On Sun, 31 May 2020, Roger Bivand wrote:

> Fresh version of rgdal rev 995 on link and R-forge, with more tweaks to the
> configure file. Please test.
>
> Roger
>
> On Fri, 29 May 2020, Roger Bivand wrote:
>
>>  I have put a tarball, built without vignettes with GDAL 2.2.4 and PROJ
>>  7.0.1 that passes CMD check (with warnings for missing vignettes) on:
>>
>>  http://spatial.nhh.no/R/rgdal/gdal_1.5-9.tar.gz
>>
>>  It involves code copying to provide duplicated functions in three
>>  settings:
>>
>>  PROJ < 6 & GDAL < 3
>> PROJ >  = 6 & GDAL >= 3
>> PROJ >  = 6 & GDAL < 3 (your homebrew case)
>>
>>  You need the configure argument:
>>  --configure-args=--with-proj_api=proj_api.h.
>>  Without the argument, you now get directed to use it.
>>
>>  For now I've dropped the configure tests for sqlite3, curl and tiff; works
>>  for me but may not work for you.
>>
>>  Please make the output of:
>>
>>  R CMD check --install-args="--configure-args=--with-proj_api=proj_api.h"
>>  rgdal_1.5-9.tar.gz
>>
>>  available ASAP. I count on an immediate response.
>>
>>  Roger
>>
>>  On Fri, 29 May 2020, Roger Bivand wrote:
>>
>>>   On Fri, 29 May 2020, Patrick Schratz wrote:
>>>
>>>>    The just released rgdal v1.5.8 requires gdal >= 3 when proj >= 6 is
>>>>    present. It does not even install the package if this condition is not
>>>>    met
>>>>    but errors during linking.
>>>
>>>   Please document with the output of 00install.out from R CMD check
>>>   rgdal_1.5-8.tar.gz, and pkg-config proj --modversion, gdalinfo --version
>>>   with system details.
>>>
>>>   Yes, PROJ >= 6 makes no sense without GDAL >= 3; GDAL 3 makes no sense
>>>   with PROJ < 6. GDAL 3 with PROJ < 6 uses the old proj_api.h API and the
>>>   old metadata structure with Proj4 string degradation; GDAL < 3 with PROJ
>>>>   = 6 must use the deprecated API and the new metadata structure.
>>>
>>>>
>>>>    I could not find any sources in the web or in the announcement post of
>>>>    v1.5.8 why this condition was enforced so strictly.
>>>>    Does it cause unwanted results?
>>>
>>>   Yes, definitely. GDAL 3 was released as 3, not 2.5, to stress the
>>>   complete
>>>   break between GDAL 2 (with PROJ <= 5) and GDAL >= 3 (with PROJ >= 6)
>>>   after
>>>   GDAL barnraising. So:
>>>
>>>             PROJ
>>>             < 6 >= 6
>>>   GDAL < 3   OK   NO
>>>    >   = 3   NO   OK
>>>
>>>
>>>
>>>>    If so, then there is a big issue since the “gdal2 and proj >= 6”
>>>>    combination has been used by many people in the past already.
>>>
>>>   No good precedent, just binary packagers protecting slow downstream
>>>   adaptation.
>>>
>>>>    I haven’t tracked down for how long this situation was on the market
>>>>    already.
>>>>    Also the {sf} package is still installable with this combination and I
>>>>    am not aware of any warnings/issues that came up due to this so far.
>>>>
>>>
>>>   rgdal is a much older package and has much more older code that needs
>>>   this
>>>   protection. It is possible to accommodate the no-go areas, but I'd
>>>   really
>>>   value patches to R-forge, as I cannot check multiple PROJ/GDAL version
>>>   combinations just because someone isn't up to speed.
>>>
>>>>    It further causes complete breakage for people relying on the homebrew
>>>>    package manager on macOS since current versions are gdal v2.2.4 and
>>>>    proj
>>>>    7.0.1.
>>>
>>>   They should never have gone beyond PROJ 5 if they are stuck on GDAL.
>>>   PROJ
>>>   7 is a very different animal.
>>>
>>>>    The update to gdal3 is blocked since months due to an incompatibility
>>>>    with `liblas` (https://github.com/libLAS/libLAS/issues/164). Reading
>>>>    the
>>>>    issue, it looks unclear when this issue will be resolved.
>>>>    The alternative osgeo4mac tap holds formula that is unstable and
>>>>    somewhat broken. One cannot link rgdal with the current gdal v3.0.1
>>>>    from
>>>>    osgeo4mac.
>>>>
>>>>    This incompatibility with homebrew-core formula leads to further
>>>>    issues
>>>>    for CI tests that rely on the spatial homebrew stack for macOS builds.
>>>>
>>>>    All of this is really unfortunate and I am wondering if this was a
>>>>    known
>>>>    factor when introducing this condition in the new release.
>>>>    Unfortunately rgdal is not hosted on any Git* provider (there is just
>>>>    the CRAN mirror with not much information about the recent changes)
>>>>    and
>>>>    it is unclear/untransparent to me if there is a continuous integration
>>>>    setup at all for this package.
>>>
>>>   Do not require me to leave the excellent SVN service on R-forge. Provide
>>>   patches there.
>>>
>>>>    I hope this is not another case in the series “I do not like operating
>>>>    system XY and hence I do not test on it” which was seen recently in
>>>>    another package. That’s what CI is for and the responsibility of
>>>>    package authors.
>>>>
>>>
>>>   I run and develop on Fedora, I  have no access to OSX, and the current
>>>   release was fully checked with ~ 900 revdeps repeatedly since November
>>>   on
>>>   successive older and newer versions of PROJ and GDAL. I blocked new PROJ
>>>   with old GDAL recently to simplify development. CI is only as good as
>>>   the
>>>   scripts, I am completely sure that repeated revdep testing and reading
>>>   the
>>>   output is superior.
>>>
>>>>    Also one should be aware that not every rgdal user is member of this
>>>>    mailing list (I guess not even 5%) and many people in production will
>>>>    face this error during install, most of them without any clue what to
>>>>    do
>>>>    or an idea why this was raised.
>>>>
>>>
>>>   There has been plenty of information given about the CRS changes. rgdal
>>>   could simply have been abandoned by me, and all  those production
>>>   volunteers might have fixed things, but I never hear anything at all.
>>>
>>>>    In summary it would be great if
>>>>
>>>>    - There would be more detailed information on the introduction of the
>>>>    new condition
>>>>    - Awareness of state-of-the-art platform related library versions
>>>>    before
>>>>    making such a release
>>>>    - Transparency/Existence of a cross-platform build process
>>>>    - If the incompatibility is just due to some features not being
>>>>    accessible with this configuration, I suggest to raise a warning
>>>>    during
>>>>    package load but not prevent the package from installing in the first
>>>>    place
>>>>    - A version-based NEWS file would be available. Currently one only
>>>>    sees
>>>>    a revision-based Changelog:
>>>>    https://cran.r-project.org/web/packages/rgdal/ChangeLog
>>>>
>>>>    I am well aware that I am indirectly criticising (hopefully in a
>>>>    constructive way) the transparency and release process here, for which
>>>>    I
>>>>    will probably get a rough response.
>>>>    However, if that leads to more robustness and transparency in future
>>>>    release, I am fine with this.
>>>>
>>>>    A quick patch release resolving the breakage would be highly
>>>>    appreciated.
>>>
>>>   Only with community imput. what you ask is not needed, just extra
>>>   complexity. Please provide patches, or accept my invitation to join the
>>>   R-forge project and commit your fixes directly. I can see how to do it,
>>>   but I don't think it makes sense, and your messsage has not motivated
>>>   me,
>>>   to be honest. I'm prioritizing working with CRAN to iron out reverse
>>>   dependency problems.
>>>
>>>   Roger
>>>
>>>>
>>>>    Best, Patrick
>>>>     [[alternative HTML version deleted]]
>>>>
>>>>    _______________________________________________
>>>>    R-sig-Geo mailing list
>>>>    [hidden email]
>>>>    https://stat.ethz.ch/mailman/listinfo/r-sig-geo
>>>>
>>>
>>>
>>
>>
>
>
--
Roger Bivand
Department of Economics, Norwegian School of Economics,
Helleveien 30, N-5045 Bergen, Norway.
voice: +47 55 95 93 55; e-mail: [hidden email]
https://orcid.org/0000-0003-2392-6140
https://scholar.google.no/citations?user=AWeghB0AAAAJ&hl=en
_______________________________________________
R-sig-Geo mailing list
[hidden email]
https://stat.ethz.ch/mailman/listinfo/r-sig-geo
Roger Bivand
Department of Economics
Norwegian School of Economics
Helleveien 30
N-5045 Bergen, Norway
Reply | Threaded
Open this post in threaded view
|

Re: rgdal: About the new gdal3 and proj >=6 condition

robinlovelace
Confirmed: this worked perfectly for me.

Video evidence that this is fixed:
https://asciinema.org/a/zPQFbI6Q01oqUy1HRauCjLUXY

Thanks Roger!

Robin

On Tue, Jun 2, 2020 at 4:00 PM Roger Bivand <[hidden email]> wrote:

> Brian Ripley (thanks!) provided a patch for rgdal's configure.ac, and the
> R-forge build and check processes now succeed, so please try to install
> either downloading
> http://download.r-forge.r-project.org/src/contrib/rgdal_1.5-9.tar.gz and
> installing from the command line (after running R CMD check), or
> install.packages("rgdal", repos="http://R-Forge.R-project.org").
>
> Remember that if GDAL is < 3, and PROJ >= 6, you need the configure
> argument: --configure-args=--with-proj_api=proj_api.h.
>
> Roger
>
> On Sun, 31 May 2020, Roger Bivand wrote:
>
> > Fresh version of rgdal rev 995 on link and R-forge, with more tweaks to
> the
> > configure file. Please test.
> >
> > Roger
> >
> > On Fri, 29 May 2020, Roger Bivand wrote:
> >
> >>  I have put a tarball, built without vignettes with GDAL 2.2.4 and PROJ
> >>  7.0.1 that passes CMD check (with warnings for missing vignettes) on:
> >>
> >>  http://spatial.nhh.no/R/rgdal/gdal_1.5-9.tar.gz
> >>
> >>  It involves code copying to provide duplicated functions in three
> >>  settings:
> >>
> >>  PROJ < 6 & GDAL < 3
> >> PROJ >  = 6 & GDAL >= 3
> >> PROJ >  = 6 & GDAL < 3 (your homebrew case)
> >>
> >>  You need the configure argument:
> >>  --configure-args=--with-proj_api=proj_api.h.
> >>  Without the argument, you now get directed to use it.
> >>
> >>  For now I've dropped the configure tests for sqlite3, curl and tiff;
> works
> >>  for me but may not work for you.
> >>
> >>  Please make the output of:
> >>
> >>  R CMD check
> --install-args="--configure-args=--with-proj_api=proj_api.h"
> >>  rgdal_1.5-9.tar.gz
> >>
> >>  available ASAP. I count on an immediate response.
> >>
> >>  Roger
> >>
> >>  On Fri, 29 May 2020, Roger Bivand wrote:
> >>
> >>>   On Fri, 29 May 2020, Patrick Schratz wrote:
> >>>
> >>>>    The just released rgdal v1.5.8 requires gdal >= 3 when proj >= 6 is
> >>>>    present. It does not even install the package if this condition is
> not
> >>>>    met
> >>>>    but errors during linking.
> >>>
> >>>   Please document with the output of 00install.out from R CMD check
> >>>   rgdal_1.5-8.tar.gz, and pkg-config proj --modversion, gdalinfo
> --version
> >>>   with system details.
> >>>
> >>>   Yes, PROJ >= 6 makes no sense without GDAL >= 3; GDAL 3 makes no
> sense
> >>>   with PROJ < 6. GDAL 3 with PROJ < 6 uses the old proj_api.h API and
> the
> >>>   old metadata structure with Proj4 string degradation; GDAL < 3 with
> PROJ
> >>>>   = 6 must use the deprecated API and the new metadata structure.
> >>>
> >>>>
> >>>>    I could not find any sources in the web or in the announcement
> post of
> >>>>    v1.5.8 why this condition was enforced so strictly.
> >>>>    Does it cause unwanted results?
> >>>
> >>>   Yes, definitely. GDAL 3 was released as 3, not 2.5, to stress the
> >>>   complete
> >>>   break between GDAL 2 (with PROJ <= 5) and GDAL >= 3 (with PROJ >= 6)
> >>>   after
> >>>   GDAL barnraising. So:
> >>>
> >>>             PROJ
> >>>             < 6 >= 6
> >>>   GDAL < 3   OK   NO
> >>>    >   = 3   NO   OK
> >>>
> >>>
> >>>
> >>>>    If so, then there is a big issue since the “gdal2 and proj >= 6”
> >>>>    combination has been used by many people in the past already.
> >>>
> >>>   No good precedent, just binary packagers protecting slow downstream
> >>>   adaptation.
> >>>
> >>>>    I haven’t tracked down for how long this situation was on the
> market
> >>>>    already.
> >>>>    Also the {sf} package is still installable with this combination
> and I
> >>>>    am not aware of any warnings/issues that came up due to this so
> far.
> >>>>
> >>>
> >>>   rgdal is a much older package and has much more older code that needs
> >>>   this
> >>>   protection. It is possible to accommodate the no-go areas, but I'd
> >>>   really
> >>>   value patches to R-forge, as I cannot check multiple PROJ/GDAL
> version
> >>>   combinations just because someone isn't up to speed.
> >>>
> >>>>    It further causes complete breakage for people relying on the
> homebrew
> >>>>    package manager on macOS since current versions are gdal v2.2.4 and
> >>>>    proj
> >>>>    7.0.1.
> >>>
> >>>   They should never have gone beyond PROJ 5 if they are stuck on GDAL.
> >>>   PROJ
> >>>   7 is a very different animal.
> >>>
> >>>>    The update to gdal3 is blocked since months due to an
> incompatibility
> >>>>    with `liblas` (https://github.com/libLAS/libLAS/issues/164).
> Reading
> >>>>    the
> >>>>    issue, it looks unclear when this issue will be resolved.
> >>>>    The alternative osgeo4mac tap holds formula that is unstable and
> >>>>    somewhat broken. One cannot link rgdal with the current gdal v3.0.1
> >>>>    from
> >>>>    osgeo4mac.
> >>>>
> >>>>    This incompatibility with homebrew-core formula leads to further
> >>>>    issues
> >>>>    for CI tests that rely on the spatial homebrew stack for macOS
> builds.
> >>>>
> >>>>    All of this is really unfortunate and I am wondering if this was a
> >>>>    known
> >>>>    factor when introducing this condition in the new release.
> >>>>    Unfortunately rgdal is not hosted on any Git* provider (there is
> just
> >>>>    the CRAN mirror with not much information about the recent changes)
> >>>>    and
> >>>>    it is unclear/untransparent to me if there is a continuous
> integration
> >>>>    setup at all for this package.
> >>>
> >>>   Do not require me to leave the excellent SVN service on R-forge.
> Provide
> >>>   patches there.
> >>>
> >>>>    I hope this is not another case in the series “I do not like
> operating
> >>>>    system XY and hence I do not test on it” which was seen recently in
> >>>>    another package. That’s what CI is for and the responsibility of
> >>>>    package authors.
> >>>>
> >>>
> >>>   I run and develop on Fedora, I  have no access to OSX, and the
> current
> >>>   release was fully checked with ~ 900 revdeps repeatedly since
> November
> >>>   on
> >>>   successive older and newer versions of PROJ and GDAL. I blocked new
> PROJ
> >>>   with old GDAL recently to simplify development. CI is only as good as
> >>>   the
> >>>   scripts, I am completely sure that repeated revdep testing and
> reading
> >>>   the
> >>>   output is superior.
> >>>
> >>>>    Also one should be aware that not every rgdal user is member of
> this
> >>>>    mailing list (I guess not even 5%) and many people in production
> will
> >>>>    face this error during install, most of them without any clue what
> to
> >>>>    do
> >>>>    or an idea why this was raised.
> >>>>
> >>>
> >>>   There has been plenty of information given about the CRS changes.
> rgdal
> >>>   could simply have been abandoned by me, and all  those production
> >>>   volunteers might have fixed things, but I never hear anything at all.
> >>>
> >>>>    In summary it would be great if
> >>>>
> >>>>    - There would be more detailed information on the introduction of
> the
> >>>>    new condition
> >>>>    - Awareness of state-of-the-art platform related library versions
> >>>>    before
> >>>>    making such a release
> >>>>    - Transparency/Existence of a cross-platform build process
> >>>>    - If the incompatibility is just due to some features not being
> >>>>    accessible with this configuration, I suggest to raise a warning
> >>>>    during
> >>>>    package load but not prevent the package from installing in the
> first
> >>>>    place
> >>>>    - A version-based NEWS file would be available. Currently one only
> >>>>    sees
> >>>>    a revision-based Changelog:
> >>>>    https://cran.r-project.org/web/packages/rgdal/ChangeLog
> >>>>
> >>>>    I am well aware that I am indirectly criticising (hopefully in a
> >>>>    constructive way) the transparency and release process here, for
> which
> >>>>    I
> >>>>    will probably get a rough response.
> >>>>    However, if that leads to more robustness and transparency in
> future
> >>>>    release, I am fine with this.
> >>>>
> >>>>    A quick patch release resolving the breakage would be highly
> >>>>    appreciated.
> >>>
> >>>   Only with community imput. what you ask is not needed, just extra
> >>>   complexity. Please provide patches, or accept my invitation to join
> the
> >>>   R-forge project and commit your fixes directly. I can see how to do
> it,
> >>>   but I don't think it makes sense, and your messsage has not motivated
> >>>   me,
> >>>   to be honest. I'm prioritizing working with CRAN to iron out reverse
> >>>   dependency problems.
> >>>
> >>>   Roger
> >>>
> >>>>
> >>>>    Best, Patrick
> >>>>     [[alternative HTML version deleted]]
> >>>>
> >>>>    _______________________________________________
> >>>>    R-sig-Geo mailing list
> >>>>    [hidden email]
> >>>>    https://stat.ethz.ch/mailman/listinfo/r-sig-geo
> >>>>
> >>>
> >>>
> >>
> >>
> >
> >
>
> --
> Roger Bivand
> Department of Economics, Norwegian School of Economics,
> Helleveien 30, N-5045 Bergen, Norway.
> voice: +47 55 95 93 55; e-mail: [hidden email]
> https://orcid.org/0000-0003-2392-6140
> https://scholar.google.no/citations?user=AWeghB0AAAAJ&hl=en
> _______________________________________________
> R-sig-Geo mailing list
> [hidden email]
> https://stat.ethz.ch/mailman/listinfo/r-sig-geo
>

        [[alternative HTML version deleted]]

_______________________________________________
R-sig-Geo mailing list
[hidden email]
https://stat.ethz.ch/mailman/listinfo/r-sig-geo
Reply | Threaded
Open this post in threaded view
|

Re: rgdal: About the new gdal3 and proj >=6 condition

Abdoulaye SARR
Have a problem with needed gdal version on mac

On Tue, Jun 2, 2020 at 4:04 PM Robin Lovelace <[hidden email]> wrote:

> Confirmed: this worked perfectly for me.
>
> Video evidence that this is fixed:
> https://asciinema.org/a/zPQFbI6Q01oqUy1HRauCjLUXY
>
> Thanks Roger!
>
> Robin
>
> On Tue, Jun 2, 2020 at 4:00 PM Roger Bivand <[hidden email]> wrote:
>
> > Brian Ripley (thanks!) provided a patch for rgdal's configure.ac, and
> the
> > R-forge build and check processes now succeed, so please try to install
> > either downloading
> > http://download.r-forge.r-project.org/src/contrib/rgdal_1.5-9.tar.gz and
> > installing from the command line (after running R CMD check), or
> > install.packages("rgdal", repos="http://R-Forge.R-project.org").
> >
> > Remember that if GDAL is < 3, and PROJ >= 6, you need the configure
> > argument: --configure-args=--with-proj_api=proj_api.h.
> >
> > Roger
> >
> > On Sun, 31 May 2020, Roger Bivand wrote:
> >
> > > Fresh version of rgdal rev 995 on link and R-forge, with more tweaks to
> > the
> > > configure file. Please test.
> > >
> > > Roger
> > >
> > > On Fri, 29 May 2020, Roger Bivand wrote:
> > >
> > >>  I have put a tarball, built without vignettes with GDAL 2.2.4 and
> PROJ
> > >>  7.0.1 that passes CMD check (with warnings for missing vignettes) on:
> > >>
> > >>  http://spatial.nhh.no/R/rgdal/gdal_1.5-9.tar.gz
> > >>
> > >>  It involves code copying to provide duplicated functions in three
> > >>  settings:
> > >>
> > >>  PROJ < 6 & GDAL < 3
> > >> PROJ >  = 6 & GDAL >= 3
> > >> PROJ >  = 6 & GDAL < 3 (your homebrew case)
> > >>
> > >>  You need the configure argument:
> > >>  --configure-args=--with-proj_api=proj_api.h.
> > >>  Without the argument, you now get directed to use it.
> > >>
> > >>  For now I've dropped the configure tests for sqlite3, curl and tiff;
> > works
> > >>  for me but may not work for you.
> > >>
> > >>  Please make the output of:
> > >>
> > >>  R CMD check
> > --install-args="--configure-args=--with-proj_api=proj_api.h"
> > >>  rgdal_1.5-9.tar.gz
> > >>
> > >>  available ASAP. I count on an immediate response.
> > >>
> > >>  Roger
> > >>
> > >>  On Fri, 29 May 2020, Roger Bivand wrote:
> > >>
> > >>>   On Fri, 29 May 2020, Patrick Schratz wrote:
> > >>>
> > >>>>    The just released rgdal v1.5.8 requires gdal >= 3 when proj >= 6
> is
> > >>>>    present. It does not even install the package if this condition
> is
> > not
> > >>>>    met
> > >>>>    but errors during linking.
> > >>>
> > >>>   Please document with the output of 00install.out from R CMD check
> > >>>   rgdal_1.5-8.tar.gz, and pkg-config proj --modversion, gdalinfo
> > --version
> > >>>   with system details.
> > >>>
> > >>>   Yes, PROJ >= 6 makes no sense without GDAL >= 3; GDAL 3 makes no
> > sense
> > >>>   with PROJ < 6. GDAL 3 with PROJ < 6 uses the old proj_api.h API and
> > the
> > >>>   old metadata structure with Proj4 string degradation; GDAL < 3 with
> > PROJ
> > >>>>   = 6 must use the deprecated API and the new metadata structure.
> > >>>
> > >>>>
> > >>>>    I could not find any sources in the web or in the announcement
> > post of
> > >>>>    v1.5.8 why this condition was enforced so strictly.
> > >>>>    Does it cause unwanted results?
> > >>>
> > >>>   Yes, definitely. GDAL 3 was released as 3, not 2.5, to stress the
> > >>>   complete
> > >>>   break between GDAL 2 (with PROJ <= 5) and GDAL >= 3 (with PROJ >=
> 6)
> > >>>   after
> > >>>   GDAL barnraising. So:
> > >>>
> > >>>             PROJ
> > >>>             < 6 >= 6
> > >>>   GDAL < 3   OK   NO
> > >>>    >   = 3   NO   OK
> > >>>
> > >>>
> > >>>
> > >>>>    If so, then there is a big issue since the “gdal2 and proj >= 6”
> > >>>>    combination has been used by many people in the past already.
> > >>>
> > >>>   No good precedent, just binary packagers protecting slow downstream
> > >>>   adaptation.
> > >>>
> > >>>>    I haven’t tracked down for how long this situation was on the
> > market
> > >>>>    already.
> > >>>>    Also the {sf} package is still installable with this combination
> > and I
> > >>>>    am not aware of any warnings/issues that came up due to this so
> > far.
> > >>>>
> > >>>
> > >>>   rgdal is a much older package and has much more older code that
> needs
> > >>>   this
> > >>>   protection. It is possible to accommodate the no-go areas, but I'd
> > >>>   really
> > >>>   value patches to R-forge, as I cannot check multiple PROJ/GDAL
> > version
> > >>>   combinations just because someone isn't up to speed.
> > >>>
> > >>>>    It further causes complete breakage for people relying on the
> > homebrew
> > >>>>    package manager on macOS since current versions are gdal v2.2.4
> and
> > >>>>    proj
> > >>>>    7.0.1.
> > >>>
> > >>>   They should never have gone beyond PROJ 5 if they are stuck on
> GDAL.
> > >>>   PROJ
> > >>>   7 is a very different animal.
> > >>>
> > >>>>    The update to gdal3 is blocked since months due to an
> > incompatibility
> > >>>>    with `liblas` (https://github.com/libLAS/libLAS/issues/164).
> > Reading
> > >>>>    the
> > >>>>    issue, it looks unclear when this issue will be resolved.
> > >>>>    The alternative osgeo4mac tap holds formula that is unstable and
> > >>>>    somewhat broken. One cannot link rgdal with the current gdal
> v3.0.1
> > >>>>    from
> > >>>>    osgeo4mac.
> > >>>>
> > >>>>    This incompatibility with homebrew-core formula leads to further
> > >>>>    issues
> > >>>>    for CI tests that rely on the spatial homebrew stack for macOS
> > builds.
> > >>>>
> > >>>>    All of this is really unfortunate and I am wondering if this was
> a
> > >>>>    known
> > >>>>    factor when introducing this condition in the new release.
> > >>>>    Unfortunately rgdal is not hosted on any Git* provider (there is
> > just
> > >>>>    the CRAN mirror with not much information about the recent
> changes)
> > >>>>    and
> > >>>>    it is unclear/untransparent to me if there is a continuous
> > integration
> > >>>>    setup at all for this package.
> > >>>
> > >>>   Do not require me to leave the excellent SVN service on R-forge.
> > Provide
> > >>>   patches there.
> > >>>
> > >>>>    I hope this is not another case in the series “I do not like
> > operating
> > >>>>    system XY and hence I do not test on it” which was seen recently
> in
> > >>>>    another package. That’s what CI is for and the responsibility of
> > >>>>    package authors.
> > >>>>
> > >>>
> > >>>   I run and develop on Fedora, I  have no access to OSX, and the
> > current
> > >>>   release was fully checked with ~ 900 revdeps repeatedly since
> > November
> > >>>   on
> > >>>   successive older and newer versions of PROJ and GDAL. I blocked new
> > PROJ
> > >>>   with old GDAL recently to simplify development. CI is only as good
> as
> > >>>   the
> > >>>   scripts, I am completely sure that repeated revdep testing and
> > reading
> > >>>   the
> > >>>   output is superior.
> > >>>
> > >>>>    Also one should be aware that not every rgdal user is member of
> > this
> > >>>>    mailing list (I guess not even 5%) and many people in production
> > will
> > >>>>    face this error during install, most of them without any clue
> what
> > to
> > >>>>    do
> > >>>>    or an idea why this was raised.
> > >>>>
> > >>>
> > >>>   There has been plenty of information given about the CRS changes.
> > rgdal
> > >>>   could simply have been abandoned by me, and all  those production
> > >>>   volunteers might have fixed things, but I never hear anything at
> all.
> > >>>
> > >>>>    In summary it would be great if
> > >>>>
> > >>>>    - There would be more detailed information on the introduction of
> > the
> > >>>>    new condition
> > >>>>    - Awareness of state-of-the-art platform related library versions
> > >>>>    before
> > >>>>    making such a release
> > >>>>    - Transparency/Existence of a cross-platform build process
> > >>>>    - If the incompatibility is just due to some features not being
> > >>>>    accessible with this configuration, I suggest to raise a warning
> > >>>>    during
> > >>>>    package load but not prevent the package from installing in the
> > first
> > >>>>    place
> > >>>>    - A version-based NEWS file would be available. Currently one
> only
> > >>>>    sees
> > >>>>    a revision-based Changelog:
> > >>>>    https://cran.r-project.org/web/packages/rgdal/ChangeLog
> > >>>>
> > >>>>    I am well aware that I am indirectly criticising (hopefully in a
> > >>>>    constructive way) the transparency and release process here, for
> > which
> > >>>>    I
> > >>>>    will probably get a rough response.
> > >>>>    However, if that leads to more robustness and transparency in
> > future
> > >>>>    release, I am fine with this.
> > >>>>
> > >>>>    A quick patch release resolving the breakage would be highly
> > >>>>    appreciated.
> > >>>
> > >>>   Only with community imput. what you ask is not needed, just extra
> > >>>   complexity. Please provide patches, or accept my invitation to join
> > the
> > >>>   R-forge project and commit your fixes directly. I can see how to do
> > it,
> > >>>   but I don't think it makes sense, and your messsage has not
> motivated
> > >>>   me,
> > >>>   to be honest. I'm prioritizing working with CRAN to iron out
> reverse
> > >>>   dependency problems.
> > >>>
> > >>>   Roger
> > >>>
> > >>>>
> > >>>>    Best, Patrick
> > >>>>     [[alternative HTML version deleted]]
> > >>>>
> > >>>>    _______________________________________________
> > >>>>    R-sig-Geo mailing list
> > >>>>    [hidden email]
> > >>>>    https://stat.ethz.ch/mailman/listinfo/r-sig-geo
> > >>>>
> > >>>
> > >>>
> > >>
> > >>
> > >
> > >
> >
> > --
> > Roger Bivand
> > Department of Economics, Norwegian School of Economics,
> > Helleveien 30, N-5045 Bergen, Norway.
> > voice: +47 55 95 93 55; e-mail: [hidden email]
> > https://orcid.org/0000-0003-2392-6140
> > https://scholar.google.no/citations?user=AWeghB0AAAAJ&hl=en
> > _______________________________________________
> > R-sig-Geo mailing list
> > [hidden email]
> > https://stat.ethz.ch/mailman/listinfo/r-sig-geo
> >
>
>         [[alternative HTML version deleted]]
>
> _______________________________________________
> R-sig-Geo mailing list
> [hidden email]
> https://stat.ethz.ch/mailman/listinfo/r-sig-geo
>

        [[alternative HTML version deleted]]

_______________________________________________
R-sig-Geo mailing list
[hidden email]
https://stat.ethz.ch/mailman/listinfo/r-sig-geo
Reply | Threaded
Open this post in threaded view
|

Re: rgdal: About the new gdal3 and proj >=6 condition

Roger Bivand
Administrator
On Wed, 3 Jun 2020, Abdoulaye Sarr wrote:

> Have a problem with needed gdal version on mac

Please be much more specific. CRAN has macOS binaries of rgdal 1.5-8. Are
you trying to install all the external software dependencies then rgdal
from source? If this is difficult, rely on the CRAN macOS binaries, even
though the PROJ and GDAL versions do not yet support WKT2 CRS
representation. If you have installed PROJ and GDAL successfully, and PROJ
< 6 && GDAL < 3, or PROJ >= 6 && GDAL >= 3, you should be OK. If you have
the pointless PROJ >= 6 && GDAL < 3, you need the 1.5-9 tarball from
R-forge, download locally and run R CMD INSTALL
--configure-args="--with-proj_api=proj_api.h" rgdal_1.5-9.tar.gz.

Please report back on specifics, or simply use CRAN binaries and report
that.

We are actively looking for ways to prevent macOS users getting trapped
into trying to install from source when such a choice is not appropriate.
We have no control over the provenance of the installed software as we
have for Windows, thanks to Jeroen's rwinlib/gdal3 collection.

Roger

>
> On Tue, Jun 2, 2020 at 4:04 PM Robin Lovelace <[hidden email]> wrote:
>
>> Confirmed: this worked perfectly for me.
>>
>> Video evidence that this is fixed:
>> https://asciinema.org/a/zPQFbI6Q01oqUy1HRauCjLUXY
>>
>> Thanks Roger!
>>
>> Robin
>>
>> On Tue, Jun 2, 2020 at 4:00 PM Roger Bivand <[hidden email]> wrote:
>>
>>> Brian Ripley (thanks!) provided a patch for rgdal's configure.ac, and
>> the
>>> R-forge build and check processes now succeed, so please try to install
>>> either downloading
>>> http://download.r-forge.r-project.org/src/contrib/rgdal_1.5-9.tar.gz and
>>> installing from the command line (after running R CMD check), or
>>> install.packages("rgdal", repos="http://R-Forge.R-project.org").
>>>
>>> Remember that if GDAL is < 3, and PROJ >= 6, you need the configure
>>> argument: --configure-args=--with-proj_api=proj_api.h.
>>>
>>> Roger
>>>
>>> On Sun, 31 May 2020, Roger Bivand wrote:
>>>
>>>> Fresh version of rgdal rev 995 on link and R-forge, with more tweaks to
>>> the
>>>> configure file. Please test.
>>>>
>>>> Roger
>>>>
>>>> On Fri, 29 May 2020, Roger Bivand wrote:
>>>>
>>>>>  I have put a tarball, built without vignettes with GDAL 2.2.4 and
>> PROJ
>>>>>  7.0.1 that passes CMD check (with warnings for missing vignettes) on:
>>>>>
>>>>>  http://spatial.nhh.no/R/rgdal/gdal_1.5-9.tar.gz
>>>>>
>>>>>  It involves code copying to provide duplicated functions in three
>>>>>  settings:
>>>>>
>>>>>  PROJ < 6 & GDAL < 3
>>>>> PROJ >  = 6 & GDAL >= 3
>>>>> PROJ >  = 6 & GDAL < 3 (your homebrew case)
>>>>>
>>>>>  You need the configure argument:
>>>>>  --configure-args=--with-proj_api=proj_api.h.
>>>>>  Without the argument, you now get directed to use it.
>>>>>
>>>>>  For now I've dropped the configure tests for sqlite3, curl and tiff;
>>> works
>>>>>  for me but may not work for you.
>>>>>
>>>>>  Please make the output of:
>>>>>
>>>>>  R CMD check
>>> --install-args="--configure-args=--with-proj_api=proj_api.h"
>>>>>  rgdal_1.5-9.tar.gz
>>>>>
>>>>>  available ASAP. I count on an immediate response.
>>>>>
>>>>>  Roger
>>>>>
>>>>>  On Fri, 29 May 2020, Roger Bivand wrote:
>>>>>
>>>>>>   On Fri, 29 May 2020, Patrick Schratz wrote:
>>>>>>
>>>>>>>    The just released rgdal v1.5.8 requires gdal >= 3 when proj >= 6
>> is
>>>>>>>    present. It does not even install the package if this condition
>> is
>>> not
>>>>>>>    met
>>>>>>>    but errors during linking.
>>>>>>
>>>>>>   Please document with the output of 00install.out from R CMD check
>>>>>>   rgdal_1.5-8.tar.gz, and pkg-config proj --modversion, gdalinfo
>>> --version
>>>>>>   with system details.
>>>>>>
>>>>>>   Yes, PROJ >= 6 makes no sense without GDAL >= 3; GDAL 3 makes no
>>> sense
>>>>>>   with PROJ < 6. GDAL 3 with PROJ < 6 uses the old proj_api.h API and
>>> the
>>>>>>   old metadata structure with Proj4 string degradation; GDAL < 3 with
>>> PROJ
>>>>>>>   = 6 must use the deprecated API and the new metadata structure.
>>>>>>
>>>>>>>
>>>>>>>    I could not find any sources in the web or in the announcement
>>> post of
>>>>>>>    v1.5.8 why this condition was enforced so strictly.
>>>>>>>    Does it cause unwanted results?
>>>>>>
>>>>>>   Yes, definitely. GDAL 3 was released as 3, not 2.5, to stress the
>>>>>>   complete
>>>>>>   break between GDAL 2 (with PROJ <= 5) and GDAL >= 3 (with PROJ >=
>> 6)
>>>>>>   after
>>>>>>   GDAL barnraising. So:
>>>>>>
>>>>>>             PROJ
>>>>>>             < 6 >= 6
>>>>>>   GDAL < 3   OK   NO
>>>>>>   >   = 3   NO   OK
>>>>>>
>>>>>>
>>>>>>
>>>>>>>    If so, then there is a big issue since the “gdal2 and proj >= 6”
>>>>>>>    combination has been used by many people in the past already.
>>>>>>
>>>>>>   No good precedent, just binary packagers protecting slow downstream
>>>>>>   adaptation.
>>>>>>
>>>>>>>    I haven’t tracked down for how long this situation was on the
>>> market
>>>>>>>    already.
>>>>>>>    Also the {sf} package is still installable with this combination
>>> and I
>>>>>>>    am not aware of any warnings/issues that came up due to this so
>>> far.
>>>>>>>
>>>>>>
>>>>>>   rgdal is a much older package and has much more older code that
>> needs
>>>>>>   this
>>>>>>   protection. It is possible to accommodate the no-go areas, but I'd
>>>>>>   really
>>>>>>   value patches to R-forge, as I cannot check multiple PROJ/GDAL
>>> version
>>>>>>   combinations just because someone isn't up to speed.
>>>>>>
>>>>>>>    It further causes complete breakage for people relying on the
>>> homebrew
>>>>>>>    package manager on macOS since current versions are gdal v2.2.4
>> and
>>>>>>>    proj
>>>>>>>    7.0.1.
>>>>>>
>>>>>>   They should never have gone beyond PROJ 5 if they are stuck on
>> GDAL.
>>>>>>   PROJ
>>>>>>   7 is a very different animal.
>>>>>>
>>>>>>>    The update to gdal3 is blocked since months due to an
>>> incompatibility
>>>>>>>    with `liblas` (https://github.com/libLAS/libLAS/issues/164).
>>> Reading
>>>>>>>    the
>>>>>>>    issue, it looks unclear when this issue will be resolved.
>>>>>>>    The alternative osgeo4mac tap holds formula that is unstable and
>>>>>>>    somewhat broken. One cannot link rgdal with the current gdal
>> v3.0.1
>>>>>>>    from
>>>>>>>    osgeo4mac.
>>>>>>>
>>>>>>>    This incompatibility with homebrew-core formula leads to further
>>>>>>>    issues
>>>>>>>    for CI tests that rely on the spatial homebrew stack for macOS
>>> builds.
>>>>>>>
>>>>>>>    All of this is really unfortunate and I am wondering if this was
>> a
>>>>>>>    known
>>>>>>>    factor when introducing this condition in the new release.
>>>>>>>    Unfortunately rgdal is not hosted on any Git* provider (there is
>>> just
>>>>>>>    the CRAN mirror with not much information about the recent
>> changes)
>>>>>>>    and
>>>>>>>    it is unclear/untransparent to me if there is a continuous
>>> integration
>>>>>>>    setup at all for this package.
>>>>>>
>>>>>>   Do not require me to leave the excellent SVN service on R-forge.
>>> Provide
>>>>>>   patches there.
>>>>>>
>>>>>>>    I hope this is not another case in the series “I do not like
>>> operating
>>>>>>>    system XY and hence I do not test on it” which was seen recently
>> in
>>>>>>>    another package. That’s what CI is for and the responsibility of
>>>>>>>    package authors.
>>>>>>>
>>>>>>
>>>>>>   I run and develop on Fedora, I  have no access to OSX, and the
>>> current
>>>>>>   release was fully checked with ~ 900 revdeps repeatedly since
>>> November
>>>>>>   on
>>>>>>   successive older and newer versions of PROJ and GDAL. I blocked new
>>> PROJ
>>>>>>   with old GDAL recently to simplify development. CI is only as good
>> as
>>>>>>   the
>>>>>>   scripts, I am completely sure that repeated revdep testing and
>>> reading
>>>>>>   the
>>>>>>   output is superior.
>>>>>>
>>>>>>>    Also one should be aware that not every rgdal user is member of
>>> this
>>>>>>>    mailing list (I guess not even 5%) and many people in production
>>> will
>>>>>>>    face this error during install, most of them without any clue
>> what
>>> to
>>>>>>>    do
>>>>>>>    or an idea why this was raised.
>>>>>>>
>>>>>>
>>>>>>   There has been plenty of information given about the CRS changes.
>>> rgdal
>>>>>>   could simply have been abandoned by me, and all  those production
>>>>>>   volunteers might have fixed things, but I never hear anything at
>> all.
>>>>>>
>>>>>>>    In summary it would be great if
>>>>>>>
>>>>>>>    - There would be more detailed information on the introduction of
>>> the
>>>>>>>    new condition
>>>>>>>    - Awareness of state-of-the-art platform related library versions
>>>>>>>    before
>>>>>>>    making such a release
>>>>>>>    - Transparency/Existence of a cross-platform build process
>>>>>>>    - If the incompatibility is just due to some features not being
>>>>>>>    accessible with this configuration, I suggest to raise a warning
>>>>>>>    during
>>>>>>>    package load but not prevent the package from installing in the
>>> first
>>>>>>>    place
>>>>>>>    - A version-based NEWS file would be available. Currently one
>> only
>>>>>>>    sees
>>>>>>>    a revision-based Changelog:
>>>>>>>    https://cran.r-project.org/web/packages/rgdal/ChangeLog
>>>>>>>
>>>>>>>    I am well aware that I am indirectly criticising (hopefully in a
>>>>>>>    constructive way) the transparency and release process here, for
>>> which
>>>>>>>    I
>>>>>>>    will probably get a rough response.
>>>>>>>    However, if that leads to more robustness and transparency in
>>> future
>>>>>>>    release, I am fine with this.
>>>>>>>
>>>>>>>    A quick patch release resolving the breakage would be highly
>>>>>>>    appreciated.
>>>>>>
>>>>>>   Only with community imput. what you ask is not needed, just extra
>>>>>>   complexity. Please provide patches, or accept my invitation to join
>>> the
>>>>>>   R-forge project and commit your fixes directly. I can see how to do
>>> it,
>>>>>>   but I don't think it makes sense, and your messsage has not
>> motivated
>>>>>>   me,
>>>>>>   to be honest. I'm prioritizing working with CRAN to iron out
>> reverse
>>>>>>   dependency problems.
>>>>>>
>>>>>>   Roger
>>>>>>
>>>>>>>
>>>>>>>    Best, Patrick
>>>>>>>     [[alternative HTML version deleted]]
>>>>>>>
>>>>>>>    _______________________________________________
>>>>>>>    R-sig-Geo mailing list
>>>>>>>    [hidden email]
>>>>>>>    https://stat.ethz.ch/mailman/listinfo/r-sig-geo
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>
>>> --
>>> Roger Bivand
>>> Department of Economics, Norwegian School of Economics,
>>> Helleveien 30, N-5045 Bergen, Norway.
>>> voice: +47 55 95 93 55; e-mail: [hidden email]
>>> https://orcid.org/0000-0003-2392-6140
>>> https://scholar.google.no/citations?user=AWeghB0AAAAJ&hl=en
>>> _______________________________________________
>>> R-sig-Geo mailing list
>>> [hidden email]
>>> https://stat.ethz.ch/mailman/listinfo/r-sig-geo
>>>
>>
>>         [[alternative HTML version deleted]]
>>
>> _______________________________________________
>> R-sig-Geo mailing list
>> [hidden email]
>> https://stat.ethz.ch/mailman/listinfo/r-sig-geo
>>
>
> [[alternative HTML version deleted]]
>
> _______________________________________________
> R-sig-Geo mailing list
> [hidden email]
> https://stat.ethz.ch/mailman/listinfo/r-sig-geo
>
--
Roger Bivand
Department of Economics, Norwegian School of Economics,
Helleveien 30, N-5045 Bergen, Norway.
voice: +47 55 95 93 55; e-mail: [hidden email]
https://orcid.org/0000-0003-2392-6140
https://scholar.google.no/citations?user=AWeghB0AAAAJ&hl=en
_______________________________________________
R-sig-Geo mailing list
[hidden email]
https://stat.ethz.ch/mailman/listinfo/r-sig-geo
Roger Bivand
Department of Economics
Norwegian School of Economics
Helleveien 30
N-5045 Bergen, Norway
Reply | Threaded
Open this post in threaded view
|

Re: rgdal: About the new gdal3 and proj >=6 condition

Abdoulaye SARR
I was trying from source, but now installed the binary.

Thank you

On Wed, Jun 3, 2020 at 3:30 PM Roger Bivand <[hidden email]> wrote:

> On Wed, 3 Jun 2020, Abdoulaye Sarr wrote:
>
> > Have a problem with needed gdal version on mac
>
> Please be much more specific. CRAN has macOS binaries of rgdal 1.5-8. Are
> you trying to install all the external software dependencies then rgdal
> from source? If this is difficult, rely on the CRAN macOS binaries, even
> though the PROJ and GDAL versions do not yet support WKT2 CRS
> representation. If you have installed PROJ and GDAL successfully, and PROJ
> < 6 && GDAL < 3, or PROJ >= 6 && GDAL >= 3, you should be OK. If you have
> the pointless PROJ >= 6 && GDAL < 3, you need the 1.5-9 tarball from
> R-forge, download locally and run R CMD INSTALL
> --configure-args="--with-proj_api=proj_api.h" rgdal_1.5-9.tar.gz.
>
> Please report back on specifics, or simply use CRAN binaries and report
> that.
>
> We are actively looking for ways to prevent macOS users getting trapped
> into trying to install from source when such a choice is not appropriate.
> We have no control over the provenance of the installed software as we
> have for Windows, thanks to Jeroen's rwinlib/gdal3 collection.
>
> Roger
>
> >
> > On Tue, Jun 2, 2020 at 4:04 PM Robin Lovelace <[hidden email]> wrote:
> >
> >> Confirmed: this worked perfectly for me.
> >>
> >> Video evidence that this is fixed:
> >> https://asciinema.org/a/zPQFbI6Q01oqUy1HRauCjLUXY
> >>
> >> Thanks Roger!
> >>
> >> Robin
> >>
> >> On Tue, Jun 2, 2020 at 4:00 PM Roger Bivand <[hidden email]>
> wrote:
> >>
> >>> Brian Ripley (thanks!) provided a patch for rgdal's configure.ac, and
> >> the
> >>> R-forge build and check processes now succeed, so please try to install
> >>> either downloading
> >>> http://download.r-forge.r-project.org/src/contrib/rgdal_1.5-9.tar.gz
> and
> >>> installing from the command line (after running R CMD check), or
> >>> install.packages("rgdal", repos="http://R-Forge.R-project.org").
> >>>
> >>> Remember that if GDAL is < 3, and PROJ >= 6, you need the configure
> >>> argument: --configure-args=--with-proj_api=proj_api.h.
> >>>
> >>> Roger
> >>>
> >>> On Sun, 31 May 2020, Roger Bivand wrote:
> >>>
> >>>> Fresh version of rgdal rev 995 on link and R-forge, with more tweaks
> to
> >>> the
> >>>> configure file. Please test.
> >>>>
> >>>> Roger
> >>>>
> >>>> On Fri, 29 May 2020, Roger Bivand wrote:
> >>>>
> >>>>>  I have put a tarball, built without vignettes with GDAL 2.2.4 and
> >> PROJ
> >>>>>  7.0.1 that passes CMD check (with warnings for missing vignettes)
> on:
> >>>>>
> >>>>>  http://spatial.nhh.no/R/rgdal/gdal_1.5-9.tar.gz
> >>>>>
> >>>>>  It involves code copying to provide duplicated functions in three
> >>>>>  settings:
> >>>>>
> >>>>>  PROJ < 6 & GDAL < 3
> >>>>> PROJ >  = 6 & GDAL >= 3
> >>>>> PROJ >  = 6 & GDAL < 3 (your homebrew case)
> >>>>>
> >>>>>  You need the configure argument:
> >>>>>  --configure-args=--with-proj_api=proj_api.h.
> >>>>>  Without the argument, you now get directed to use it.
> >>>>>
> >>>>>  For now I've dropped the configure tests for sqlite3, curl and tiff;
> >>> works
> >>>>>  for me but may not work for you.
> >>>>>
> >>>>>  Please make the output of:
> >>>>>
> >>>>>  R CMD check
> >>> --install-args="--configure-args=--with-proj_api=proj_api.h"
> >>>>>  rgdal_1.5-9.tar.gz
> >>>>>
> >>>>>  available ASAP. I count on an immediate response.
> >>>>>
> >>>>>  Roger
> >>>>>
> >>>>>  On Fri, 29 May 2020, Roger Bivand wrote:
> >>>>>
> >>>>>>   On Fri, 29 May 2020, Patrick Schratz wrote:
> >>>>>>
> >>>>>>>    The just released rgdal v1.5.8 requires gdal >= 3 when proj >= 6
> >> is
> >>>>>>>    present. It does not even install the package if this condition
> >> is
> >>> not
> >>>>>>>    met
> >>>>>>>    but errors during linking.
> >>>>>>
> >>>>>>   Please document with the output of 00install.out from R CMD check
> >>>>>>   rgdal_1.5-8.tar.gz, and pkg-config proj --modversion, gdalinfo
> >>> --version
> >>>>>>   with system details.
> >>>>>>
> >>>>>>   Yes, PROJ >= 6 makes no sense without GDAL >= 3; GDAL 3 makes no
> >>> sense
> >>>>>>   with PROJ < 6. GDAL 3 with PROJ < 6 uses the old proj_api.h API
> and
> >>> the
> >>>>>>   old metadata structure with Proj4 string degradation; GDAL < 3
> with
> >>> PROJ
> >>>>>>>   = 6 must use the deprecated API and the new metadata structure.
> >>>>>>
> >>>>>>>
> >>>>>>>    I could not find any sources in the web or in the announcement
> >>> post of
> >>>>>>>    v1.5.8 why this condition was enforced so strictly.
> >>>>>>>    Does it cause unwanted results?
> >>>>>>
> >>>>>>   Yes, definitely. GDAL 3 was released as 3, not 2.5, to stress the
> >>>>>>   complete
> >>>>>>   break between GDAL 2 (with PROJ <= 5) and GDAL >= 3 (with PROJ >=
> >> 6)
> >>>>>>   after
> >>>>>>   GDAL barnraising. So:
> >>>>>>
> >>>>>>             PROJ
> >>>>>>             < 6 >= 6
> >>>>>>   GDAL < 3   OK   NO
> >>>>>>   >   = 3   NO   OK
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>>    If so, then there is a big issue since the “gdal2 and proj >= 6”
> >>>>>>>    combination has been used by many people in the past already.
> >>>>>>
> >>>>>>   No good precedent, just binary packagers protecting slow
> downstream
> >>>>>>   adaptation.
> >>>>>>
> >>>>>>>    I haven’t tracked down for how long this situation was on the
> >>> market
> >>>>>>>    already.
> >>>>>>>    Also the {sf} package is still installable with this combination
> >>> and I
> >>>>>>>    am not aware of any warnings/issues that came up due to this so
> >>> far.
> >>>>>>>
> >>>>>>
> >>>>>>   rgdal is a much older package and has much more older code that
> >> needs
> >>>>>>   this
> >>>>>>   protection. It is possible to accommodate the no-go areas, but I'd
> >>>>>>   really
> >>>>>>   value patches to R-forge, as I cannot check multiple PROJ/GDAL
> >>> version
> >>>>>>   combinations just because someone isn't up to speed.
> >>>>>>
> >>>>>>>    It further causes complete breakage for people relying on the
> >>> homebrew
> >>>>>>>    package manager on macOS since current versions are gdal v2.2.4
> >> and
> >>>>>>>    proj
> >>>>>>>    7.0.1.
> >>>>>>
> >>>>>>   They should never have gone beyond PROJ 5 if they are stuck on
> >> GDAL.
> >>>>>>   PROJ
> >>>>>>   7 is a very different animal.
> >>>>>>
> >>>>>>>    The update to gdal3 is blocked since months due to an
> >>> incompatibility
> >>>>>>>    with `liblas` (https://github.com/libLAS/libLAS/issues/164).
> >>> Reading
> >>>>>>>    the
> >>>>>>>    issue, it looks unclear when this issue will be resolved.
> >>>>>>>    The alternative osgeo4mac tap holds formula that is unstable and
> >>>>>>>    somewhat broken. One cannot link rgdal with the current gdal
> >> v3.0.1
> >>>>>>>    from
> >>>>>>>    osgeo4mac.
> >>>>>>>
> >>>>>>>    This incompatibility with homebrew-core formula leads to further
> >>>>>>>    issues
> >>>>>>>    for CI tests that rely on the spatial homebrew stack for macOS
> >>> builds.
> >>>>>>>
> >>>>>>>    All of this is really unfortunate and I am wondering if this was
> >> a
> >>>>>>>    known
> >>>>>>>    factor when introducing this condition in the new release.
> >>>>>>>    Unfortunately rgdal is not hosted on any Git* provider (there is
> >>> just
> >>>>>>>    the CRAN mirror with not much information about the recent
> >> changes)
> >>>>>>>    and
> >>>>>>>    it is unclear/untransparent to me if there is a continuous
> >>> integration
> >>>>>>>    setup at all for this package.
> >>>>>>
> >>>>>>   Do not require me to leave the excellent SVN service on R-forge.
> >>> Provide
> >>>>>>   patches there.
> >>>>>>
> >>>>>>>    I hope this is not another case in the series “I do not like
> >>> operating
> >>>>>>>    system XY and hence I do not test on it” which was seen recently
> >> in
> >>>>>>>    another package. That’s what CI is for and the responsibility of
> >>>>>>>    package authors.
> >>>>>>>
> >>>>>>
> >>>>>>   I run and develop on Fedora, I  have no access to OSX, and the
> >>> current
> >>>>>>   release was fully checked with ~ 900 revdeps repeatedly since
> >>> November
> >>>>>>   on
> >>>>>>   successive older and newer versions of PROJ and GDAL. I blocked
> new
> >>> PROJ
> >>>>>>   with old GDAL recently to simplify development. CI is only as good
> >> as
> >>>>>>   the
> >>>>>>   scripts, I am completely sure that repeated revdep testing and
> >>> reading
> >>>>>>   the
> >>>>>>   output is superior.
> >>>>>>
> >>>>>>>    Also one should be aware that not every rgdal user is member of
> >>> this
> >>>>>>>    mailing list (I guess not even 5%) and many people in production
> >>> will
> >>>>>>>    face this error during install, most of them without any clue
> >> what
> >>> to
> >>>>>>>    do
> >>>>>>>    or an idea why this was raised.
> >>>>>>>
> >>>>>>
> >>>>>>   There has been plenty of information given about the CRS changes.
> >>> rgdal
> >>>>>>   could simply have been abandoned by me, and all  those production
> >>>>>>   volunteers might have fixed things, but I never hear anything at
> >> all.
> >>>>>>
> >>>>>>>    In summary it would be great if
> >>>>>>>
> >>>>>>>    - There would be more detailed information on the introduction
> of
> >>> the
> >>>>>>>    new condition
> >>>>>>>    - Awareness of state-of-the-art platform related library
> versions
> >>>>>>>    before
> >>>>>>>    making such a release
> >>>>>>>    - Transparency/Existence of a cross-platform build process
> >>>>>>>    - If the incompatibility is just due to some features not being
> >>>>>>>    accessible with this configuration, I suggest to raise a warning
> >>>>>>>    during
> >>>>>>>    package load but not prevent the package from installing in the
> >>> first
> >>>>>>>    place
> >>>>>>>    - A version-based NEWS file would be available. Currently one
> >> only
> >>>>>>>    sees
> >>>>>>>    a revision-based Changelog:
> >>>>>>>    https://cran.r-project.org/web/packages/rgdal/ChangeLog
> >>>>>>>
> >>>>>>>    I am well aware that I am indirectly criticising (hopefully in a
> >>>>>>>    constructive way) the transparency and release process here, for
> >>> which
> >>>>>>>    I
> >>>>>>>    will probably get a rough response.
> >>>>>>>    However, if that leads to more robustness and transparency in
> >>> future
> >>>>>>>    release, I am fine with this.
> >>>>>>>
> >>>>>>>    A quick patch release resolving the breakage would be highly
> >>>>>>>    appreciated.
> >>>>>>
> >>>>>>   Only with community imput. what you ask is not needed, just extra
> >>>>>>   complexity. Please provide patches, or accept my invitation to
> join
> >>> the
> >>>>>>   R-forge project and commit your fixes directly. I can see how to
> do
> >>> it,
> >>>>>>   but I don't think it makes sense, and your messsage has not
> >> motivated
> >>>>>>   me,
> >>>>>>   to be honest. I'm prioritizing working with CRAN to iron out
> >> reverse
> >>>>>>   dependency problems.
> >>>>>>
> >>>>>>   Roger
> >>>>>>
> >>>>>>>
> >>>>>>>    Best, Patrick
> >>>>>>>     [[alternative HTML version deleted]]
> >>>>>>>
> >>>>>>>    _______________________________________________
> >>>>>>>    R-sig-Geo mailing list
> >>>>>>>    [hidden email]
> >>>>>>>    https://stat.ethz.ch/mailman/listinfo/r-sig-geo
> >>>>>>>
> >>>>>>
> >>>>>>
> >>>>>
> >>>>>
> >>>>
> >>>>
> >>>
> >>> --
> >>> Roger Bivand
> >>> Department of Economics, Norwegian School of Economics,
> >>> Helleveien 30, N-5045 Bergen, Norway.
> >>> voice: +47 55 95 93 55; e-mail: [hidden email]
> >>> https://orcid.org/0000-0003-2392-6140
> >>> https://scholar.google.no/citations?user=AWeghB0AAAAJ&hl=en
> >>> _______________________________________________
> >>> R-sig-Geo mailing list
> >>> [hidden email]
> >>> https://stat.ethz.ch/mailman/listinfo/r-sig-geo
> >>>
> >>
> >>         [[alternative HTML version deleted]]
> >>
> >> _______________________________________________
> >> R-sig-Geo mailing list
> >> [hidden email]
> >> https://stat.ethz.ch/mailman/listinfo/r-sig-geo
> >>
> >
> >       [[alternative HTML version deleted]]
> >
> > _______________________________________________
> > R-sig-Geo mailing list
> > [hidden email]
> > https://stat.ethz.ch/mailman/listinfo/r-sig-geo
> >
>
> --
> Roger Bivand
> Department of Economics, Norwegian School of Economics,
> Helleveien 30, N-5045 Bergen, Norway.
> voice: +47 55 95 93 55; e-mail: [hidden email]
> https://orcid.org/0000-0003-2392-6140
> https://scholar.google.no/citations?user=AWeghB0AAAAJ&hl=en

        [[alternative HTML version deleted]]

_______________________________________________
R-sig-Geo mailing list
[hidden email]
https://stat.ethz.ch/mailman/listinfo/r-sig-geo