-
Notifications
You must be signed in to change notification settings - Fork 2
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Load a GeoPackage dataset #13
Comments
@ErickChacon I can reproduce the issue. Nailed it down to a specific function call. First we load the GDAL geometries: using GeoTables
# auxiliary packages
import ArchGDAL as AG
import Tables
# load layer as a Tables.jl table
data = AG.read("stations.gpkg")
table = AG.getlayer(data, 0)
# get the column with geometries
cols = Tables.columns(table)
geoms = Tables.getcolumn(cols, :geom)
2-element Vector{ArchGDAL.IGeometry{ArchGDAL.wkbPoint}}:
Geometry: POINT (-77.55011 -11.06068)
Geometry: POINT (-77.36839 -11.36014) Second we try to convert to Meshes.jl geometries: # convert GDAL geometries to Meshes.jl geometries
items = GeoTables.geom2meshes.(geoms) but the function It used to work, but the GeoInterface.jl is a never ending effort to convert geometry types, so our function may be broken now. Can you please check if our implementation is up-to-date and is the most efficient? Feel free to submit a PR if you discover what is causing the function call to hang now. |
Also, consider updating to the latest release that I just triggered with the latest version of ArchGDAL.jl. |
Hi @juliohm. Thank for the explanation. This is something that I need so I will take a look to |
Thank you @ErickChacon , this issue must be specific to GDAL because the same function works with Shapefile.jl and GeoJSON.jl geometries. |
HI @juliohm. Yes, this is specific of GDAL but is not clear for me why it does not work. Continuing with your example. We can easily conver the first point with: using Meshes
GI.coordinates(geoms[1]) |> Point However a simple function like the following does not work (it hangs): function topoint(geom)
coords = GI.coordinates(geom)
Point(coords)
end
topoint(geoms[1]) If I splat the coordinates vector, it works without problem. function topoint2(geom)
coords = GI.coordinates(geom)
Point(coords...)
end
topoint2(geoms[1]) I have tried other options like using the |
Thank you @ErickChacon for investigating this further. Can you confirm that the other geometry types from GeoJSON.jl and Shapefile.jl work fine with your topoint function? Our constructors for Point are defined here: https://github.com/JuliaGeometry/Meshes.jl/blob/master/src/points.jl |
I also noticed that this behavior only happens inside a function and that doing it manually with GDAL types on the REPL works. It must be one of the GDAL pointer idiosyncrasies... |
Exactly, it only happens inside the function. |
Worth checking with the maintainers of GDAL.jl and ArchGDAL.jl if there is a better method with GeoInterface.jl to make these conversions. I am not following these developments myself. |
The return value of julia> using ArchGDAL
julia> mypoint = ArchGDAL.createpoint(1.0, 5.0)
Geometry: POINT (1 5)
julia> GeoInterface.coordinates(mypoint)
2-element Vector{Float64}:
1.0
5.0
We usually use |
If you read our comments above you will see that the issue is specific to
GDAL. Whenever we call GI.coordinates from inside a function.
Em qui., 27 de abr. de 2023 11:43, Rafael Schouten ***@***.***>
escreveu:
… The return value of GI.coordinates is just a Vector{Float64} like it
always has been?
julia> using ArchGDAL
julia> mypoint = ArchGDAL.createpoint(1.0, 5.0)
Geometry: POINT (1 5)
julia> GeoInterface.coordinates(mypoint)2-element Vector{Float64}:
1.0
5.0
GeoInterface.coordinates is a pretty stable interface, and not at all
part of our ongoing comparability efforts. So this is probably a Meshes.jl
problem.
We usually use GeoInterface.getpoint in an iteration to do these
conversions efficiently these days, and they work pretty nicely for
everything else, although of course `ArchGDAL.jl will be slower due to the
API overheads.
—
Reply to this email directly, view it on GitHub
<#13 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAZQW3M2YOFCP4LKXWK4RJTXDKA2FANCNFSM6AAAAAAXIDXWLQ>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
julia> f(geom) = GeoInterface.coordinates(geom)
f (generic function with 1 method)
julia> f(mypoint)
2-element Vector{Float64}:
1.0
5.0 ? |
Can you please read the previous comments? The REPL works fine.
Em qui., 27 de abr. de 2023 11:47, Rafael Schouten ***@***.***>
escreveu:
… julia> f(geom) = GeoInterface.coordinates(geom)
f (generic function with 1 method)
julia> f(mypoint)2-element Vector{Float64}:
1.0
5.0
?
—
Reply to this email directly, view it on GitHub
<#13 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAZQW3IB47ATOTC2AHXXK3LXDKBHRANCNFSM6AAAAAAXIDXWLQ>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
I defined a function and ran that |
Or you are saying that this is hanging somewhere else down in the
Meshes.Point constructors?
Em qui., 27 de abr. de 2023 11:50, Júlio Hoffimann <
***@***.***> escreveu:
… Can you please read the previous comments? The REPL works fine.
Em qui., 27 de abr. de 2023 11:47, Rafael Schouten <
***@***.***> escreveu:
> julia> f(geom) = GeoInterface.coordinates(geom)
> f (generic function with 1 method)
>
> julia> f(mypoint)2-element Vector{Float64}:
> 1.0
> 5.0
>
> ?
>
> —
> Reply to this email directly, view it on GitHub
> <#13 (comment)>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AAZQW3IB47ATOTC2AHXXK3LXDKBHRANCNFSM6AAAAAAXIDXWLQ>
> .
> You are receiving this because you were mentioned.Message ID:
> ***@***.***>
>
|
I mean it has to be right? Or at least some combination? Because this works fine everywhere else. |
The other GI.coordinates for Shapefile.jl and GeoJSON.jl return the same
vector type? I find it strange that it only fails with GDAL
Em qui., 27 de abr. de 2023 11:51, Rafael Schouten ***@***.***>
escreveu:
… I mean it has to be right? Or at least some combination?
Because this works fine everywhere else.
—
Reply to this email directly, view it on GitHub
<#13 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAZQW3PFQJHOZTK5CEIHTDTXDKBXBANCNFSM6AAAAAAXIDXWLQ>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
I think it might be a combination of both mypoint = AG.createpoint(1.0, 5.0)
function topoint(geom)
coords = GI.coordinates(geom)
Meshes.Point(coords...)
end
topoint(mypoint) |
But which constructor is hanging? Is it a Julia issue or a low-level issue
in GDAL?
Em qui., 27 de abr. de 2023 12:32, Dr. Erick A. Chacón Montalván <
***@***.***> escreveu:
… I think it might be a combination of both Meshes.jl and ArchGDAL.jl that
does not work. For example, other Meshes constructors for Point work with
ArchGDAL.jl like the following:
mypoint = AG.createpoint(1.0, 5.0)function topoint(geom)
coords = GI.coordinates(geom)
Meshes.Point(coords...)endtopoint(mypoint)
—
Reply to this email directly, view it on GitHub
<#13 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAZQW3L7WB6DAFONMPJVVRLXDKGSHANCNFSM6AAAAAAXIDXWLQ>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
Shapefile.jl will be the same as ArchGDAL.jl.
It cant be the return values causing this unless you are testing with another floating point type in the other packages and thats not triggering the bug. So how are you imagining that Meshes.jl would cause a low-level GDAL error that we don't get without Meshes.jl in the same function? What is the mechanism? memory use? For this one point that shouldn't be an issue. (but yes there are definately more problems with using these C interfaces and its not impossible this is one of them) |
Not sure where because the lack of error trace. I also tried using
|
In particular I am exploring the following contructors https://github.com/JuliaGeometry/Meshes.jl/blob/a0487c6824d6ee9d7389edc25ae937f1e4cf26fd/src/points.jl#L53-L55. The constructor that uses import ArchGDAL as AG
import GeoInterface as GI
using Meshes
mypoint = AG.createpoint(1.0, 5.0)
function topoint1(geom)
coords = GI.coordinates(geom)
Meshes.Point(coords...)
end
function topoint2(geom)
coords = GI.coordinates(geom)
Meshes.Point{2,Float64}(coords)
end
function topoint3(geom)
coords = GI.coordinates(geom)
Meshes.Point(Meshes.Vec(coords))
end
topoint1(mypoint)
topoint2(mypoint)
topoint3(mypoint) |
Yes, in theory they are the same. But the previous functions work without any problem when the
When it hangs, the cpu continue working like getting into a infinite bucle. Maybe it is a method table issue? |
Reducing the problem, I can notice that it happens when calling import ArchGDAL as AG
import GeoInterface as GI
using Meshes
mypoint = AG.createpoint(1.0, 5.0)
function tovec(geom)
coords = GI.coordinates(geom)
Meshes.Vec(coords)
end
tovec(mypoint) However, it works if I am more specific: function tovec2(geom)
coords = GI.coordinates(geom)
Meshes.Vec{2,Float64}(coords)
end
tovec2(mypoint) Might this be related with the following line https://github.com/JuliaGeometry/Meshes.jl/blob/77e90c2efaba40ef5e28a1fa058396bf95994b29/src/vectors.jl#L66, where we make a call to |
Thanks for investigating this further @ErickChacon , can you explain why do you think the call to length could be problematic? Isn't the coordinates vector returned by GI.coordinates a simple Julia Vector? |
For now, the only reason is because the only intermediate step between |
Ah this might actually be a compilation problem? You are calling length on a runtime object and lifting it to the type domain. Much better to clarify the options for the compiler function Vec(coords::AbstractVector{T}) where {T}
if length(coords) == 2
return Vec{2,T}(coords)
else
return Vec{3,T}(coords)
end
# Maybe you have more dimensions to add here...
end Then this is a known small union optimisation problem rather than complex instability. |
@eliascarv can you please double check this? |
My ArchGDAL PR may also reduce the weight from the GDAL side, currently getting points has too much overhead: It may interact with the poor design of Additional to this fix I would call |
Investigating a bit further, I could see that it hangs in the following line https://github.com/JuliaGeometry/Meshes.jl/blob/cb556d34900c64b6f40ac827360039c0fb3549f3/src/vectors.jl#L60, but only when interacting with ArchGDAL. It hangs when calling |
Did you make that change? Also try my ArchGDAL PR for better point performance. But ultimately using Iterating over |
It would help if this "preferred new way" of doing things with GI was implemented somewhere for reference. We could just copy/paste the code that is supposedly fast and more aligned with the latest version of the interface. |
When I say that it hangs when calling
Happy to follow that approach. As @juliohm mentioned, which package would you suggest to check for the geointerface implementation? GeometryBasics.jl? |
I can confirm that using the following constructor solves the problem and passes all test. @juliohm would you like a pull request on function Vec(coords::AbstractVector{T}) where {T}
n = length(coords)
if n == 1
Vec{1,T}(coords)
elseif n == 2
Vec{2,T}(coords)
elseif n == 3
Vec{3,T}(coords)
else
throw(ErrorException("not implemented"))
end
end Of course, it would still be great to update the geointerface for |
@ErickChacon a PR request would be very welcome. The code you shared seems fine, I think we can assume that Meshes.jl only supports up to 3 dimensions currently. |
Maybe LibGEOS.jl is the best to compare to? https://github.com/JuliaGeo/LibGEOS.jl/blob/master/src/geo_interface.jl You may also notice that it accepts all GeoInterface.jl objects in all functions without explicit conversions by users. That could also work in Meshes.jl |
It may be easiest to follow this PR: Something to keep in mind is not all geometry objects in the ecosystem have their dimensions known at compile time. So as with the fix here, if you need D at compile time, always try to find the number of dimensions as early as possible (e.g. on a multypolygon object rather than the points, linestrings or polygons it holds). Use Next is that Then where you dont care about structure you can use |
This has to do with the fact that GI returns a Union{Vector{T}, Vector{Union{T,Nothing}} etc as the resulting type. This is a MWE where Julia apparently has a bug: test(v::AbstractVector{T}) where {T} = NTuple{length(v),T}(v)
test([1,2,3]) # works
test([1,2,nothing]) # hangs Thanks @eliascarv for finding this MWE. |
That's interesting, does seem like some kind of compiler bug. But that return type sucks, we should track down where that comes from. Edit: this is probably an inference problem with 2d/3d geoms from gdal. We should force the type from the first two values. But not using |
@ErickChacon you mentioned that we could fix the issue here by avoiding that constructor, right? Can you prepare a PR here as well to use the splatting operator as you suggested? It would be great if you could create a test set with GDAL geometries making sure that our geom2meshes function is working as expected with all of them. |
@juliohm, yes. I am checking https://github.com/JuliaGeo/LibGEOS.jl/blob/master/src/geo_interface.jl to make the interface similar to it. We will just use function GI.convert(::Type{Point}, ::PointTrait, geom; context=get_global_context())
if GI.is3d(geom)
return Point(GI.x(geom), GI.y(geom), GI.z(geom), context)
else
return Point(GI.x(geom), GI.y(geom), context)
end
end I am still checking other details to have an interface aligned with @rafaqz suggestions. I will start a pull request soon. |
FYI But otherwise that should be generic |
FYI But otherwise that should be generic. Also note Meshes.jl will need a Probably all this code should be in extensions in Meshes.jl anyway at some point, the dependencies on GeoJSON.jl and Shapefile.jl are obsolete now, they have Tables.jl interfaces. |
We don't have plans to add dependencies related to GeoInterface.jl in Meshes.jl, it is not aligned with our philosophy and we don't want to promote use cases where users are fiddling with different geometry types all over the place. We are providing these conversion methods here in GeoTables.jl so that people can do IO, proceed with geometric processing constrained to a single set of geometries in Meshes.jl, and then export the results back to disk. That is the only place where GeoInterface.jl is envisioned in our stack. |
Sure thing, love the strict, clean philosophy! But mostly the idea is that theres no fiddling to be done, interop will be seemless ;) |
Perhaps you are too optimistic about the co-existence of different geometry types in advanced geometric processing pipelines. At some point you will hit function calls that are not implemented in terms of the GeoInterface.jl and the application will just crash. It is as simple as that in my mind, but I would be happy to be proven wrong. Looking forward to the seamless experience! |
Its actually pretty easy to add methods for all functions in a package to accept GeoInterface types. See e.g. my LibGEOS PR. There is very little code used on that interop. And note my comment was in aid of your package getting high level conversion like Meshes.jl does need some minor coordination with GeoInterface.jl for that to work, not even a dependency, but at least a function stub that can have methods added to it here. |
Thank you @ErickChacon for fixing this ❤️ |
I have a problem loading a GeoPackage dataset (https://drive.google.com/file/d/16--kcGrC56zayrK04-Brmfo3ImOwtQ5k/view?usp=sharing). The dataset can be loaded using ArchGDAL.
However, I can not load it using GeoTables.
The previous code does not finish to be executed. I am using the latest version of
GeoTables.jl
.The text was updated successfully, but these errors were encountered: