Possible rgdal enhancement request related to null attributes

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
3 messages Options
Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Possible rgdal enhancement request related to null attributes

Alex Mandel-2
Roger et al,

I couldn't find a ticket system for rgdal so this seemed the best place
to post.

We just hit a bug (in QGIS [1]) of sorts due to a change in behavior in
GDAL from 2.1.x to 2.2.x in the way null fields are handled. [2]

Short story if you have a column of null values and you save a geojson
those fields are not saved. So the next time you open up the file, you
have less columns than before. It's great for minifying geojson but
terrible for many other use cases.

I'm not 100% sure this will show up in rgdal (2.2 hasn't been pushed to
my Ubuntu yet so I haven't tested), but there's a good chance it will
depending on how R deals with unset vs null. If someone would like to
test, you can get a sample file from [3], readOGR and then writeOGR (to
geojson) and see if the null columns vanish.

The enhancement requested to deal with this, is to provide the user an
option that allows them to decide when a field is unset vs null. Or
maybe just a clarification on how R represents that data and how a user
could set a field as null instead of unset and vice versa.

Even R. pointed out that this can be controlled with the
OGR_F_IsFieldNull() part of the ogr api.

[1] https://issues.qgis.org/issues/16812
[2] https://trac.osgeo.org/gdal/wiki/rfc67_nullfieldvalues
[3] https://github.com/UCDavisLibrary/ava/issues/270

Thanks,
Alex

_______________________________________________
R-sig-Geo mailing list
[hidden email]
https://stat.ethz.ch/mailman/listinfo/r-sig-geo
Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Possible rgdal enhancement request related to null attributes

Roger Bivand
Administrator
Hi Alex,

GDAL 2.2 is supported in rgdal and in sf - has been for some time. If GDAL
>= 2.2:

https://r-forge.r-project.org/scm/viewvc.php/pkg/src/OGR_write.cpp?root=rgdal&r1=650&r2=660

and

https://r-forge.r-project.org/scm/viewvc.php/pkg/src/ogrsource.cpp?root=rgdal&r1=650&r2=660

This is general, not just GeoJSON, see

https://trac.osgeo.org/gdal/wiki/rfc67_nullfieldvalues

I don't see a good reason for making the user have to choose - could you
suggest one with a use case reprex?

See also the sf code, which is relevant here.

Roger

On Fri, 7 Jul 2017, Alex M wrote:

> Roger et al,
>
> I couldn't find a ticket system for rgdal so this seemed the best place
> to post.
>
> We just hit a bug (in QGIS [1]) of sorts due to a change in behavior in
> GDAL from 2.1.x to 2.2.x in the way null fields are handled. [2]
>
> Short story if you have a column of null values and you save a geojson
> those fields are not saved. So the next time you open up the file, you
> have less columns than before. It's great for minifying geojson but
> terrible for many other use cases.
>
> I'm not 100% sure this will show up in rgdal (2.2 hasn't been pushed to
> my Ubuntu yet so I haven't tested), but there's a good chance it will
> depending on how R deals with unset vs null. If someone would like to
> test, you can get a sample file from [3], readOGR and then writeOGR (to
> geojson) and see if the null columns vanish.
>
> The enhancement requested to deal with this, is to provide the user an
> option that allows them to decide when a field is unset vs null. Or
> maybe just a clarification on how R represents that data and how a user
> could set a field as null instead of unset and vice versa.
>
> Even R. pointed out that this can be controlled with the
> OGR_F_IsFieldNull() part of the ogr api.
>
> [1] https://issues.qgis.org/issues/16812
> [2] https://trac.osgeo.org/gdal/wiki/rfc67_nullfieldvalues
> [3] https://github.com/UCDavisLibrary/ava/issues/270
>
> Thanks,
> Alex
>
> _______________________________________________
> R-sig-Geo mailing list
> [hidden email]
> https://stat.ethz.ch/mailman/listinfo/r-sig-geo
>

--
Roger Bivand
Department of Economics, Norwegian School of Economics,
Helleveien 30, N-5045 Bergen, Norway.
voice: +47 55 95 93 55; e-mail: [hidden email]
Editor-in-Chief of The R Journal, https://journal.r-project.org/index.html
http://orcid.org/0000-0003-2392-6140
https://scholar.google.no/citations?user=AWeghB0AAAAJ&hl=en

_______________________________________________
R-sig-Geo mailing list
[hidden email]
https://stat.ethz.ch/mailman/listinfo/r-sig-geo
Roger Bivand
Department of Economics
Norwegian School of Economics
Helleveien 30
N-5045 Bergen, Norway
Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Possible rgdal enhancement request related to null attributes

Alex Mandel-2
Roger,

It appears you preserved the pre 2.2 behavior which is what I think most people would expect, keep the attribute and set it as Null.

That is not the case with other OGR related tools. I just hadn't had a chance to verify it in R, and had the issue come up in QGIS, so I appreciate you pointing out where in the code you addressed it (I did try to read through that file and didn't find that on my own).

The only reason I could see people wanting to drop nulls, is that certain formats would be more efficient. In the case of GeoJson it could result in a significantly smaller file. Though personally I think that a separate minification tool would be a better place to make that change. Which would mean a user would only need an override option if you had gone with UnsetField as the default.

So at this time I don't see any additional changes needed.

Thanks,
Alex

> On July 10, 2017 at 9:04 AM Roger Bivand <[hidden email]> wrote:
>
>
> Hi Alex,
>
> GDAL 2.2 is supported in rgdal and in sf - has been for some time. If GDAL
> >= 2.2:
>
> https://r-forge.r-project.org/scm/viewvc.php/pkg/src/OGR_write.cpp?root=rgdal&r1=650&r2=660
>
> and
>
> https://r-forge.r-project.org/scm/viewvc.php/pkg/src/ogrsource.cpp?root=rgdal&r1=650&r2=660
>
> This is general, not just GeoJSON, see
>
> https://trac.osgeo.org/gdal/wiki/rfc67_nullfieldvalues
>
> I don't see a good reason for making the user have to choose - could you
> suggest one with a use case reprex?
>
> See also the sf code, which is relevant here.
>
> Roger
>
> On Fri, 7 Jul 2017, Alex M wrote:
>
> > Roger et al,
> >
> > I couldn't find a ticket system for rgdal so this seemed the best place
> > to post.
> >
> > We just hit a bug (in QGIS [1]) of sorts due to a change in behavior in
> > GDAL from 2.1.x to 2.2.x in the way null fields are handled. [2]
> >
> > Short story if you have a column of null values and you save a geojson
> > those fields are not saved. So the next time you open up the file, you
> > have less columns than before. It's great for minifying geojson but
> > terrible for many other use cases.
> >
> > I'm not 100% sure this will show up in rgdal (2.2 hasn't been pushed to
> > my Ubuntu yet so I haven't tested), but there's a good chance it will
> > depending on how R deals with unset vs null. If someone would like to
> > test, you can get a sample file from [3], readOGR and then writeOGR (to
> > geojson) and see if the null columns vanish.
> >
> > The enhancement requested to deal with this, is to provide the user an
> > option that allows them to decide when a field is unset vs null. Or
> > maybe just a clarification on how R represents that data and how a user
> > could set a field as null instead of unset and vice versa.
> >
> > Even R. pointed out that this can be controlled with the
> > OGR_F_IsFieldNull() part of the ogr api.
> >
> > [1] https://issues.qgis.org/issues/16812
> > [2] https://trac.osgeo.org/gdal/wiki/rfc67_nullfieldvalues
> > [3] https://github.com/UCDavisLibrary/ava/issues/270
> >
> > Thanks,
> > Alex
> >
> > _______________________________________________
> > R-sig-Geo mailing list
> > [hidden email]
> > https://stat.ethz.ch/mailman/listinfo/r-sig-geo
> >
>
> --
> Roger Bivand
> Department of Economics, Norwegian School of Economics,
> Helleveien 30, N-5045 Bergen, Norway.
> voice: +47 55 95 93 55; e-mail: [hidden email]
> Editor-in-Chief of The R Journal, https://journal.r-project.org/index.html
> http://orcid.org/0000-0003-2392-6140
> https://scholar.google.no/citations?user=AWeghB0AAAAJ&hl=en
>
> _______________________________________________
> R-sig-Geo mailing list
> [hidden email]
> https://stat.ethz.ch/mailman/listinfo/r-sig-geo

_______________________________________________
R-sig-Geo mailing list
[hidden email]
https://stat.ethz.ch/mailman/listinfo/r-sig-geo
Loading...