parallelize distance matrix

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

parallelize distance matrix

aestfd
Hi, I want to parallelize the calculation of a distance matrix, I've tried
but I wasn't succesful, this is what I did with a training data.

My data set is:

ll <- matrix(c(5, 6, 60, 60), ncol=2)

And I use the function *spDistsN1* from the library *"sp"* to obtain a
distance matrix with *apply*:

apply(ll, 1, function(x) spDistsN1(as.matrix(ll), x, longlat = T))

But I want to do it with parallelization, so for that:

library(parallel)
ncore <- detectCores()
cl <- makeCluster(ncore)
clusterEvalQ(cl = cl, expr = c(library(sp)))
parRapply(cl = cl, x = ll, FUN =  function(x) spDistsN1(as.matrix(ll), x,
longlat = T))

It shows the following error:

*Error in checkForRemoteErrors(val) : 4 nodes produced errors; first error:
object 'll' not found*

How do I fix it?
Regards,
Ariel Fuentes

        [[alternative HTML version deleted]]

_______________________________________________
R-sig-Geo mailing list
[hidden email]
https://stat.ethz.ch/mailman/listinfo/r-sig-geo
Reply | Threaded
Open this post in threaded view
|

Re: parallelize distance matrix

Roman Luštrik
Cross posted on SO:
https://stackoverflow.com/questions/51952776/parallelization-apply-to-parrapply

On Tue, Aug 21, 2018 at 8:15 PM Ariel Fuentesdi <[hidden email]>
wrote:

> Hi, I want to parallelize the calculation of a distance matrix, I've tried
> but I wasn't succesful, this is what I did with a training data.
>
> My data set is:
>
> ll <- matrix(c(5, 6, 60, 60), ncol=2)
>
> And I use the function *spDistsN1* from the library *"sp"* to obtain a
> distance matrix with *apply*:
>
> apply(ll, 1, function(x) spDistsN1(as.matrix(ll), x, longlat = T))
>
> But I want to do it with parallelization, so for that:
>
> library(parallel)
> ncore <- detectCores()
> cl <- makeCluster(ncore)
> clusterEvalQ(cl = cl, expr = c(library(sp)))
> parRapply(cl = cl, x = ll, FUN =  function(x) spDistsN1(as.matrix(ll), x,
> longlat = T))
>
> It shows the following error:
>
> *Error in checkForRemoteErrors(val) : 4 nodes produced errors; first error:
> object 'll' not found*
>
> How do I fix it?
> Regards,
> Ariel Fuentes
>
>         [[alternative HTML version deleted]]
>
> _______________________________________________
> R-sig-Geo mailing list
> [hidden email]
> https://stat.ethz.ch/mailman/listinfo/r-sig-geo
>


--
In God we trust, all others bring data.

        [[alternative HTML version deleted]]

_______________________________________________
R-sig-Geo mailing list
[hidden email]
https://stat.ethz.ch/mailman/listinfo/r-sig-geo
Reply | Threaded
Open this post in threaded view
|

Re: parallelize distance matrix

Vijay Lulla
You'll have to do clusterExport(cl, c("ll")) before you call parRapply.
Please see ?`parRapply` and pages 10 and 11 of the parallel::parallel
vignette.
HTH,
Vijay.

On Tue, Aug 21, 2018 at 2:54 PM Roman Luštrik <[hidden email]>
wrote:

> Cross posted on SO:
>
> https://stackoverflow.com/questions/51952776/parallelization-apply-to-parrapply
>
> On Tue, Aug 21, 2018 at 8:15 PM Ariel Fuentesdi <[hidden email]>
> wrote:
>
> > Hi, I want to parallelize the calculation of a distance matrix, I've
> tried
> > but I wasn't succesful, this is what I did with a training data.
> >
> > My data set is:
> >
> > ll <- matrix(c(5, 6, 60, 60), ncol=2)
> >
> > And I use the function *spDistsN1* from the library *"sp"* to obtain a
> > distance matrix with *apply*:
> >
> > apply(ll, 1, function(x) spDistsN1(as.matrix(ll), x, longlat = T))
> >
> > But I want to do it with parallelization, so for that:
> >
> > library(parallel)
> > ncore <- detectCores()
> > cl <- makeCluster(ncore)
> > clusterEvalQ(cl = cl, expr = c(library(sp)))
> > parRapply(cl = cl, x = ll, FUN =  function(x) spDistsN1(as.matrix(ll), x,
> > longlat = T))
> >
> > It shows the following error:
> >
> > *Error in checkForRemoteErrors(val) : 4 nodes produced errors; first
> error:
> > object 'll' not found*
> >
> > How do I fix it?
> > Regards,
> > Ariel Fuentes
> >
> >         [[alternative HTML version deleted]]
> >
> > _______________________________________________
> > R-sig-Geo mailing list
> > [hidden email]
> > https://stat.ethz.ch/mailman/listinfo/r-sig-geo
> >
>
>
> --
> In God we trust, all others bring data.
>
>         [[alternative HTML version deleted]]
>
> _______________________________________________
> R-sig-Geo mailing list
> [hidden email]
> https://stat.ethz.ch/mailman/listinfo/r-sig-geo
>

        [[alternative HTML version deleted]]

_______________________________________________
R-sig-Geo mailing list
[hidden email]
https://stat.ethz.ch/mailman/listinfo/r-sig-geo