spatial 2SLS

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
13 messages Options
Reply | Threaded
Open this post in threaded view
|

spatial 2SLS

Darla Munroe
An embedded and charset-unspecified text was scrubbed...
Name: not available
URL: <https://stat.ethz.ch/pipermail/r-sig-geo/attachments/20041213/26f7be78/attachment.pl>

Reply | Threaded
Open this post in threaded view
|

spatial 2SLS

Luc Anselin
I have some functions that implement this. We used them in last summer's
ICPSR spatial regression course. They are not (yet) very user friendly
and
not up to official R standards yet, but they definitely work.
The functions include:
- standard 2SLS (or IV estimation)
- spatial 2SLS (using WX as instruments for Wy)
- heteroskedastic robust spatial 2SLS (White adjusted variances)
Anyone who wants them and doesn't need a lot of hand holding can e-mail
me directly.
L.


On Dec 13, 2004, at 8:21 AM, Darla Munroe wrote:

> Is the spatial 2SLS developed by Kelejian available in spdep?  If not,
> is it
> available anywhere else at this time?  (besides in SpaceStat).
>
>
>
> Thanks,
>
> Darla Munroe
>
>
> [[alternative HTML version deleted]]
>
> _______________________________________________
> R-sig-Geo mailing list
> R-sig-Geo at stat.math.ethz.ch
> https://stat.ethz.ch/mailman/listinfo/r-sig-geo
>



Reply | Threaded
Open this post in threaded view
|

spatial 2SLS

Mihai Nica
Greetings:

I would like to take advantage of this thread and ask if there is a way to
"enhance" the code in spdep. I have a dataset with 3030 observations and two
independent variables. On the same old computer it runs great in GeoDa, but
fails in R. I assume it is a memory issue, but having two different ways to
check my results would be great.

Thanks,

Mihai Nica
Jackson State University
155 B Parkhurst Dr.
Jackson, MS 39202
601 914 0361

----- Original Message -----
From: "Luc Anselin" <anselin at uiuc.edu>
To: "Darla Munroe" <munroe.9 at osu.edu>
Cc: <r-sig-geo at stat.math.ethz.ch>
Sent: Monday, December 13, 2004 8:36 AM
Subject: Re: [R-sig-Geo] spatial 2SLS


> I have some functions that implement this. We used them in last summer's
> ICPSR spatial regression course. They are not (yet) very user friendly
> and
> not up to official R standards yet, but they definitely work.
> The functions include:
> - standard 2SLS (or IV estimation)
> - spatial 2SLS (using WX as instruments for Wy)
> - heteroskedastic robust spatial 2SLS (White adjusted variances)
> Anyone who wants them and doesn't need a lot of hand holding can e-mail
> me directly.
> L.
>
>
> On Dec 13, 2004, at 8:21 AM, Darla Munroe wrote:
>
> > Is the spatial 2SLS developed by Kelejian available in spdep?  If not,
> > is it
> > available anywhere else at this time?  (besides in SpaceStat).
> >
> >
> >
> > Thanks,
> >
> > Darla Munroe
> >
> >
> > [[alternative HTML version deleted]]
> >
> > _______________________________________________
> > R-sig-Geo mailing list
> > R-sig-Geo at stat.math.ethz.ch
> > https://stat.ethz.ch/mailman/listinfo/r-sig-geo
> >
>
> _______________________________________________
> R-sig-Geo mailing list
> R-sig-Geo at stat.math.ethz.ch
> https://stat.ethz.ch/mailman/listinfo/r-sig-geo
>



Reply | Threaded
Open this post in threaded view
|

spatial 2SLS

Danlin Yu-2
Dear Mihai Nica:

    I think it is a memory issue, too. I ran into the same problem when
using around 3000 observations. Although I used memory.size() to increase
the memroy allocation size, spdep took too long to run.

    The problem, I think, might be due to the fact that spdep's spatial
autoregressive models use the exact method (correct me if I am wrong),
while GeoDa uses an approximatioin method (see the paper "Fast maximum
likelihood estimation of very large spatial autoregressive models: a
characteristic polynomial approach", in Computational Statistics & Data
Analysis 35 (2001) 301-319). If the method could be incorporated in R, it
will be great.

    Danlin

On Mon, 13 Dec 2004, Mihai Nica wrote:

> Greetings:
>
> I would like to take advantage of this thread and ask if there is a way to
> "enhance" the code in spdep. I have a dataset with 3030 observations and two
> independent variables. On the same old computer it runs great in GeoDa, but
> fails in R. I assume it is a memory issue, but having two different ways to
> check my results would be great.
>
> Thanks,
>
> Mihai Nica
> Jackson State University
> 155 B Parkhurst Dr.
> Jackson, MS 39202
> 601 914 0361
>
> ----- Original Message -----
> From: "Luc Anselin" <anselin at uiuc.edu>
> To: "Darla Munroe" <munroe.9 at osu.edu>
> Cc: <r-sig-geo at stat.math.ethz.ch>
> Sent: Monday, December 13, 2004 8:36 AM
> Subject: Re: [R-sig-Geo] spatial 2SLS
>
>
> > I have some functions that implement this. We used them in last summer's
> > ICPSR spatial regression course. They are not (yet) very user friendly
> > and
> > not up to official R standards yet, but they definitely work.
> > The functions include:
> > - standard 2SLS (or IV estimation)
> > - spatial 2SLS (using WX as instruments for Wy)
> > - heteroskedastic robust spatial 2SLS (White adjusted variances)
> > Anyone who wants them and doesn't need a lot of hand holding can e-mail
> > me directly.
> > L.
> >
> >
> > On Dec 13, 2004, at 8:21 AM, Darla Munroe wrote:
> >
> > > Is the spatial 2SLS developed by Kelejian available in spdep?  If not,
> > > is it
> > > available anywhere else at this time?  (besides in SpaceStat).
> > >
> > >
> > >
> > > Thanks,
> > >
> > > Darla Munroe
> > >
> > >
> > > [[alternative HTML version deleted]]
> > >
> > > _______________________________________________
> > > R-sig-Geo mailing list
> > > R-sig-Geo at stat.math.ethz.ch
> > > https://stat.ethz.ch/mailman/listinfo/r-sig-geo
> > >
> >
> > _______________________________________________
> > R-sig-Geo mailing list
> > R-sig-Geo at stat.math.ethz.ch
> > https://stat.ethz.ch/mailman/listinfo/r-sig-geo
> >
>
> _______________________________________________
> R-sig-Geo mailing list
> R-sig-Geo at stat.math.ethz.ch
> https://stat.ethz.ch/mailman/listinfo/r-sig-geo
>

    Sincerely,
    Danlin Yu

----------------------------------
Lecturer, Ph.D. Candidate
Department of Geography
University of Wisconsin, Milwaukee
Tel: (414)229-3943
Fax: (414)229-3981
Email: danlinyu at uwm.edu



Reply | Threaded
Open this post in threaded view
|

Fitting SAR models in spdep (was: spatial 2SLS)

Roger Bivand
Administrator
On Mon, 13 Dec 2004, Danlin Yu wrote:

> Dear Mihai Nica:
>
>     I think it is a memory issue, too. I ran into the same problem when
> using around 3000 observations. Although I used memory.size() to increase
> the memroy allocation size, spdep took too long to run.
>

I think you mean the two SAR fitting functions, lagsarlm() and
errorsarlm()? The default method is "eigen", which does need plenty of
memory with large n (because it both solves the eigenproblem for an nxn
matrix, and later inverts an nxn matrix). There has always been a "sparse"
method using code from Netlib called sparse from UCB, which works on some
platforms, but not others, and has an unresolved memory bug.

I'm pleased to report that Roger Koenker has been kind enough to add a
det() for symmetric (and similar to symmetric) sparse matrices to the
SparseM package. From spdep_0.3-5, spdep depends on SparseM, and
errorsarlm() is now pretty dependable for large n and fast. lagsarlm() is
dependable, but its fitting speed is slowed by fitting models for LR tests
dropping each of the X variables in turn, as well as the specified model,
so that it fits k models, not just 1. The function calls for the lag
models are cheaper than the error models, though.

The restriction to symmetric (and similar to symmetric) sparse matrices is
important to bear in mind. With symmetric neighbour lists (contiguity,
graphe-based and distance bands), and even symmetric inverse distances,
and all weighting styles this is OK (because "W" and "S" style are similar
to symmetric, and the others are symmetric), but absolutely not for
k-nearest-neighbours or for asymmetric general weights (say migration
tables).

I would be very interested in hearing whether this approach is helpful,
and how this scales.

Roger

>     The problem, I think, might be due to the fact that spdep's spatial
> autoregressive models use the exact method (correct me if I am wrong),
> while GeoDa uses an approximatioin method (see the paper "Fast maximum
> likelihood estimation of very large spatial autoregressive models: a
> characteristic polynomial approach", in Computational Statistics & Data
> Analysis 35 (2001) 301-319). If the method could be incorporated in R, it
> will be great.
>
>     Danlin
>
> On Mon, 13 Dec 2004, Mihai Nica wrote:
>
> > Greetings:
> >
> > I would like to take advantage of this thread and ask if there is a way to
> > "enhance" the code in spdep. I have a dataset with 3030 observations and two
> > independent variables. On the same old computer it runs great in GeoDa, but
> > fails in R. I assume it is a memory issue, but having two different ways to
> > check my results would be great.
> >
> > Thanks,
> >
> > Mihai Nica
> > Jackson State University
> > 155 B Parkhurst Dr.
> > Jackson, MS 39202
> > 601 914 0361
> >
> > ----- Original Message -----
> > From: "Luc Anselin" <anselin at uiuc.edu>
> > To: "Darla Munroe" <munroe.9 at osu.edu>
> > Cc: <r-sig-geo at stat.math.ethz.ch>
> > Sent: Monday, December 13, 2004 8:36 AM
> > Subject: Re: [R-sig-Geo] spatial 2SLS
> >
> >
> > > I have some functions that implement this. We used them in last summer's
> > > ICPSR spatial regression course. They are not (yet) very user friendly
> > > and
> > > not up to official R standards yet, but they definitely work.
> > > The functions include:
> > > - standard 2SLS (or IV estimation)
> > > - spatial 2SLS (using WX as instruments for Wy)
> > > - heteroskedastic robust spatial 2SLS (White adjusted variances)
> > > Anyone who wants them and doesn't need a lot of hand holding can e-mail
> > > me directly.
> > > L.
> > >
> > >
> > > On Dec 13, 2004, at 8:21 AM, Darla Munroe wrote:
> > >
> > > > Is the spatial 2SLS developed by Kelejian available in spdep?  If not,
> > > > is it
> > > > available anywhere else at this time?  (besides in SpaceStat).
> > > >
> > > >
> > > >
> > > > Thanks,
> > > >
> > > > Darla Munroe
> > > >
> > > >
> > > > [[alternative HTML version deleted]]
> > > >
> > > > _______________________________________________
> > > > R-sig-Geo mailing list
> > > > R-sig-Geo at stat.math.ethz.ch
> > > > https://stat.ethz.ch/mailman/listinfo/r-sig-geo
> > > >
> > >
> > > _______________________________________________
> > > R-sig-Geo mailing list
> > > R-sig-Geo at stat.math.ethz.ch
> > > https://stat.ethz.ch/mailman/listinfo/r-sig-geo
> > >
> >
> > _______________________________________________
> > R-sig-Geo mailing list
> > R-sig-Geo at stat.math.ethz.ch
> > https://stat.ethz.ch/mailman/listinfo/r-sig-geo
> >
>
>     Sincerely,
>     Danlin Yu
>
> ----------------------------------
> Lecturer, Ph.D. Candidate
> Department of Geography
> University of Wisconsin, Milwaukee
> Tel: (414)229-3943
> Fax: (414)229-3981
> Email: danlinyu at uwm.edu
>
> _______________________________________________
> R-sig-Geo mailing list
> R-sig-Geo at stat.math.ethz.ch
> https://stat.ethz.ch/mailman/listinfo/r-sig-geo
>

--
Roger Bivand
Economic Geography Section, Department of Economics, Norwegian School of
Economics and Business Administration, Breiviksveien 40, N-5045 Bergen,
Norway. voice: +47 55 95 93 55; fax +47 55 95 93 93
e-mail: Roger.Bivand at nhh.no



Roger Bivand
Department of Economics
Norwegian School of Economics
Helleveien 30
N-5045 Bergen, Norway
Reply | Threaded
Open this post in threaded view
|

Fitting SAR models in spdep (was: spatial 2SLS)

Danlin Yu-2
Dear Professor Roger Bivand:

    Yes, I was talking about the two SAR fitting functions. I just did a
very quick run on a dataset with 3437 observations (those are point data,
and I generate the listw object using dnearneigh() - not sure whether the
result is symmetric or asymmetric). However, I tried both "sparse" and
"SparseM" method on errorsarlm(), both failed to go through, the errors
given as follow:

    For "sparse" method:
    "Error in spwdet(sparseweights, rho = rho, debug = debug) :
        Suspicious allocation of memory in sparse functions
         ...bailing out! Save workspace and quit R!"

    For "SparseM" method:
    "Error in .local(x, ...) : Increase tmpmax
     Error in det(chol(x)) : Unable to find the argument "x" in selecting
     a method for function "det" "

    By the way, I am still using R 2.0.0 and when I loaded the libraries,
I got warnings. I will keep trying and keep you posted as I progress.
Thanks.


On Mon, 13 Dec 2004, Roger Bivand wrote:

> On Mon, 13 Dec 2004, Danlin Yu wrote:
>
> > Dear Mihai Nica:
> >
> >     I think it is a memory issue, too. I ran into the same problem when
> > using around 3000 observations. Although I used memory.size() to increase
> > the memroy allocation size, spdep took too long to run.
> >
>
> I think you mean the two SAR fitting functions, lagsarlm() and
> errorsarlm()? The default method is "eigen", which does need plenty of
> memory with large n (because it both solves the eigenproblem for an nxn
> matrix, and later inverts an nxn matrix). There has always been a "sparse"
> method using code from Netlib called sparse from UCB, which works on some
> platforms, but not others, and has an unresolved memory bug.
>
> I'm pleased to report that Roger Koenker has been kind enough to add a
> det() for symmetric (and similar to symmetric) sparse matrices to the
> SparseM package. From spdep_0.3-5, spdep depends on SparseM, and
> errorsarlm() is now pretty dependable for large n and fast. lagsarlm() is
> dependable, but its fitting speed is slowed by fitting models for LR tests
> dropping each of the X variables in turn, as well as the specified model,
> so that it fits k models, not just 1. The function calls for the lag
> models are cheaper than the error models, though.
>
> The restriction to symmetric (and similar to symmetric) sparse matrices is
> important to bear in mind. With symmetric neighbour lists (contiguity,
> graphe-based and distance bands), and even symmetric inverse distances,
> and all weighting styles this is OK (because "W" and "S" style are similar
> to symmetric, and the others are symmetric), but absolutely not for
> k-nearest-neighbours or for asymmetric general weights (say migration
> tables).
>
> I would be very interested in hearing whether this approach is helpful,
> and how this scales.
>
> Roger
>
> >     The problem, I think, might be due to the fact that spdep's spatial
> > autoregressive models use the exact method (correct me if I am wrong),
> > while GeoDa uses an approximatioin method (see the paper "Fast maximum
> > likelihood estimation of very large spatial autoregressive models: a
> > characteristic polynomial approach", in Computational Statistics & Data
> > Analysis 35 (2001) 301-319). If the method could be incorporated in R, it
> > will be great.
> >
> >     Danlin
> >
> > On Mon, 13 Dec 2004, Mihai Nica wrote:
> >
> > > Greetings:
> > >
> > > I would like to take advantage of this thread and ask if there is a way to
> > > "enhance" the code in spdep. I have a dataset with 3030 observations and two
> > > independent variables. On the same old computer it runs great in GeoDa, but
> > > fails in R. I assume it is a memory issue, but having two different ways to
> > > check my results would be great.
> > >
> > > Thanks,
> > >
> > > Mihai Nica
> > > Jackson State University
> > > 155 B Parkhurst Dr.
> > > Jackson, MS 39202
> > > 601 914 0361
> > >
> > > ----- Original Message -----
> > > From: "Luc Anselin" <anselin at uiuc.edu>
> > > To: "Darla Munroe" <munroe.9 at osu.edu>
> > > Cc: <r-sig-geo at stat.math.ethz.ch>
> > > Sent: Monday, December 13, 2004 8:36 AM
> > > Subject: Re: [R-sig-Geo] spatial 2SLS
> > >
> > >
> > > > I have some functions that implement this. We used them in last summer's
> > > > ICPSR spatial regression course. They are not (yet) very user friendly
> > > > and
> > > > not up to official R standards yet, but they definitely work.
> > > > The functions include:
> > > > - standard 2SLS (or IV estimation)
> > > > - spatial 2SLS (using WX as instruments for Wy)
> > > > - heteroskedastic robust spatial 2SLS (White adjusted variances)
> > > > Anyone who wants them and doesn't need a lot of hand holding can e-mail
> > > > me directly.
> > > > L.
> > > >
> > > >
> > > > On Dec 13, 2004, at 8:21 AM, Darla Munroe wrote:
> > > >
> > > > > Is the spatial 2SLS developed by Kelejian available in spdep?  If not,
> > > > > is it
> > > > > available anywhere else at this time?  (besides in SpaceStat).
> > > > >
> > > > >
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Darla Munroe
> > > > >
> > > > >
> > > > > [[alternative HTML version deleted]]
> > > > >
> > > > > _______________________________________________
> > > > > R-sig-Geo mailing list
> > > > > R-sig-Geo at stat.math.ethz.ch
> > > > > https://stat.ethz.ch/mailman/listinfo/r-sig-geo
> > > > >
> > > >
> > > > _______________________________________________
> > > > R-sig-Geo mailing list
> > > > R-sig-Geo at stat.math.ethz.ch
> > > > https://stat.ethz.ch/mailman/listinfo/r-sig-geo
> > > >
> > >
> > > _______________________________________________
> > > R-sig-Geo mailing list
> > > R-sig-Geo at stat.math.ethz.ch
> > > https://stat.ethz.ch/mailman/listinfo/r-sig-geo
> > >
> >
> >     Sincerely,
> >     Danlin Yu
> >
> > ----------------------------------
> > Lecturer, Ph.D. Candidate
> > Department of Geography
> > University of Wisconsin, Milwaukee
> > Tel: (414)229-3943
> > Fax: (414)229-3981
> > Email: danlinyu at uwm.edu
> >
> > _______________________________________________
> > R-sig-Geo mailing list
> > R-sig-Geo at stat.math.ethz.ch
> > https://stat.ethz.ch/mailman/listinfo/r-sig-geo
> >
>
> --
> Roger Bivand
> Economic Geography Section, Department of Economics, Norwegian School of
> Economics and Business Administration, Breiviksveien 40, N-5045 Bergen,
> Norway. voice: +47 55 95 93 55; fax +47 55 95 93 93
> e-mail: Roger.Bivand at nhh.no
>
>

    Sincerely,
    Danlin Yu

----------------------------------
Lecturer, Ph.D. Candidate
Department of Geography
University of Wisconsin, Milwaukee
Tel: (414)229-3943
Fax: (414)229-3981
Email: danlinyu at uwm.edu



Reply | Threaded
Open this post in threaded view
|

Fitting SAR models in spdep (was: spatial 2SLS)

Roger Bivand
Administrator
On Mon, 13 Dec 2004, Danlin Yu wrote:

> Dear Professor Roger Bivand:
>
>     Yes, I was talking about the two SAR fitting functions. I just did a
> very quick run on a dataset with 3437 observations (those are point data,
> and I generate the listw object using dnearneigh() - not sure whether the
> result is symmetric or asymmetric).

How many links are there in the neighbour list? Distance neighbours are
symmetric, but if points are densely placed, there may be parts of your
study area with many neighbours. Use print.nb() to find out how many.

> However, I tried both "sparse" and
> "SparseM" method on errorsarlm(), both failed to go through, the errors
> given as follow:
>
>     For "sparse" method:
>     "Error in spwdet(sparseweights, rho = rho, debug = debug) :
>         Suspicious allocation of memory in sparse functions
>          ...bailing out! Save workspace and quit R!"

Yes, this is the memory bug. It does happen on most/all GCC compiled
versions, but I've seen BSD versions that work - it depends on the memory
allocation code, but it has not been possible to solve. I am planning to
drop "sparse" before long, unless the memory bug is resolved.

>
>     For "SparseM" method:
>     "Error in .local(x, ...) : Increase tmpmax

This is the problem: tmpmax is defined in chol() for "matrix.csr" objects
as only 10 times the number of rows. I think you have more neighbour links
than 10*n, and that means I need to find a way of passing a better value
through. Please let me know, this looks do-able.

Roger

>      Error in det(chol(x)) : Unable to find the argument "x" in selecting
>      a method for function "det" "
>
>     By the way, I am still using R 2.0.0 and when I loaded the libraries,
> I got warnings. I will keep trying and keep you posted as I progress.
> Thanks.
>
>
> On Mon, 13 Dec 2004, Roger Bivand wrote:
>
> > On Mon, 13 Dec 2004, Danlin Yu wrote:
> >
> > > Dear Mihai Nica:
> > >
> > >     I think it is a memory issue, too. I ran into the same problem when
> > > using around 3000 observations. Although I used memory.size() to increase
> > > the memroy allocation size, spdep took too long to run.
> > >
> >
> > I think you mean the two SAR fitting functions, lagsarlm() and
> > errorsarlm()? The default method is "eigen", which does need plenty of
> > memory with large n (because it both solves the eigenproblem for an nxn
> > matrix, and later inverts an nxn matrix). There has always been a "sparse"
> > method using code from Netlib called sparse from UCB, which works on some
> > platforms, but not others, and has an unresolved memory bug.
> >
> > I'm pleased to report that Roger Koenker has been kind enough to add a
> > det() for symmetric (and similar to symmetric) sparse matrices to the
> > SparseM package. From spdep_0.3-5, spdep depends on SparseM, and
> > errorsarlm() is now pretty dependable for large n and fast. lagsarlm() is
> > dependable, but its fitting speed is slowed by fitting models for LR tests
> > dropping each of the X variables in turn, as well as the specified model,
> > so that it fits k models, not just 1. The function calls for the lag
> > models are cheaper than the error models, though.
> >
> > The restriction to symmetric (and similar to symmetric) sparse matrices is
> > important to bear in mind. With symmetric neighbour lists (contiguity,
> > graphe-based and distance bands), and even symmetric inverse distances,
> > and all weighting styles this is OK (because "W" and "S" style are similar
> > to symmetric, and the others are symmetric), but absolutely not for
> > k-nearest-neighbours or for asymmetric general weights (say migration
> > tables).
> >
> > I would be very interested in hearing whether this approach is helpful,
> > and how this scales.
> >
> > Roger
> >
> > >     The problem, I think, might be due to the fact that spdep's spatial
> > > autoregressive models use the exact method (correct me if I am wrong),
> > > while GeoDa uses an approximatioin method (see the paper "Fast maximum
> > > likelihood estimation of very large spatial autoregressive models: a
> > > characteristic polynomial approach", in Computational Statistics & Data
> > > Analysis 35 (2001) 301-319). If the method could be incorporated in R, it
> > > will be great.
> > >
> > >     Danlin
> > >
> > > On Mon, 13 Dec 2004, Mihai Nica wrote:
> > >
> > > > Greetings:
> > > >
> > > > I would like to take advantage of this thread and ask if there is a way to
> > > > "enhance" the code in spdep. I have a dataset with 3030 observations and two
> > > > independent variables. On the same old computer it runs great in GeoDa, but
> > > > fails in R. I assume it is a memory issue, but having two different ways to
> > > > check my results would be great.
> > > >
> > > > Thanks,
> > > >
> > > > Mihai Nica
> > > > Jackson State University
> > > > 155 B Parkhurst Dr.
> > > > Jackson, MS 39202
> > > > 601 914 0361
> > > >
> > > > ----- Original Message -----
> > > > From: "Luc Anselin" <anselin at uiuc.edu>
> > > > To: "Darla Munroe" <munroe.9 at osu.edu>
> > > > Cc: <r-sig-geo at stat.math.ethz.ch>
> > > > Sent: Monday, December 13, 2004 8:36 AM
> > > > Subject: Re: [R-sig-Geo] spatial 2SLS
> > > >
> > > >
> > > > > I have some functions that implement this. We used them in last summer's
> > > > > ICPSR spatial regression course. They are not (yet) very user friendly
> > > > > and
> > > > > not up to official R standards yet, but they definitely work.
> > > > > The functions include:
> > > > > - standard 2SLS (or IV estimation)
> > > > > - spatial 2SLS (using WX as instruments for Wy)
> > > > > - heteroskedastic robust spatial 2SLS (White adjusted variances)
> > > > > Anyone who wants them and doesn't need a lot of hand holding can e-mail
> > > > > me directly.
> > > > > L.
> > > > >
> > > > >
> > > > > On Dec 13, 2004, at 8:21 AM, Darla Munroe wrote:
> > > > >
> > > > > > Is the spatial 2SLS developed by Kelejian available in spdep?  If not,
> > > > > > is it
> > > > > > available anywhere else at this time?  (besides in SpaceStat).
> > > > > >
> > > > > >
> > > > > >
> > > > > > Thanks,
> > > > > >
> > > > > > Darla Munroe
> > > > > >
> > > > > >
> > > > > > [[alternative HTML version deleted]]
> > > > > >
> > > > > > _______________________________________________
> > > > > > R-sig-Geo mailing list
> > > > > > R-sig-Geo at stat.math.ethz.ch
> > > > > > https://stat.ethz.ch/mailman/listinfo/r-sig-geo
> > > > > >
> > > > >
> > > > > _______________________________________________
> > > > > R-sig-Geo mailing list
> > > > > R-sig-Geo at stat.math.ethz.ch
> > > > > https://stat.ethz.ch/mailman/listinfo/r-sig-geo
> > > > >
> > > >
> > > > _______________________________________________
> > > > R-sig-Geo mailing list
> > > > R-sig-Geo at stat.math.ethz.ch
> > > > https://stat.ethz.ch/mailman/listinfo/r-sig-geo
> > > >
> > >
> > >     Sincerely,
> > >     Danlin Yu
> > >
> > > ----------------------------------
> > > Lecturer, Ph.D. Candidate
> > > Department of Geography
> > > University of Wisconsin, Milwaukee
> > > Tel: (414)229-3943
> > > Fax: (414)229-3981
> > > Email: danlinyu at uwm.edu
> > >
> > > _______________________________________________
> > > R-sig-Geo mailing list
> > > R-sig-Geo at stat.math.ethz.ch
> > > https://stat.ethz.ch/mailman/listinfo/r-sig-geo
> > >
> >
> > --
> > Roger Bivand
> > Economic Geography Section, Department of Economics, Norwegian School of
> > Economics and Business Administration, Breiviksveien 40, N-5045 Bergen,
> > Norway. voice: +47 55 95 93 55; fax +47 55 95 93 93
> > e-mail: Roger.Bivand at nhh.no
> >
> >
>
>     Sincerely,
>     Danlin Yu
>
> ----------------------------------
> Lecturer, Ph.D. Candidate
> Department of Geography
> University of Wisconsin, Milwaukee
> Tel: (414)229-3943
> Fax: (414)229-3981
> Email: danlinyu at uwm.edu
> ----------------------------------
>
>

--
Roger Bivand
Economic Geography Section, Department of Economics, Norwegian School of
Economics and Business Administration, Breiviksveien 40, N-5045 Bergen,
Norway. voice: +47 55 95 93 55; fax +47 55 95 93 93
e-mail: Roger.Bivand at nhh.no



Roger Bivand
Department of Economics
Norwegian School of Economics
Helleveien 30
N-5045 Bergen, Norway
Reply | Threaded
Open this post in threaded view
|

Fitting SAR models in spdep (was: spatial 2SLS)

Danlin Yu-2
Dear Professor Roger Bivand:

   Yes, you are right, the total number of links is 686040, much more than
10*n (that is 3437) in the dataset.

   Some of the study area is really dense in data, but some is really
sparse. So the chosen distance ensuring there won't be any non-zero links
yield a large number of links. The actual reports from print.nb follows:

Neighbour list object:
Number of regions: 3437
Number of nonzero links: 686040
Percentage nonzero weights: 5.807515
Average number of links: 199.6043

   Please keep me posted if the chol() problem is solved. Thanks. Danlin

On Mon, 13 Dec 2004, Roger Bivand wrote:

> On Mon, 13 Dec 2004, Danlin Yu wrote:
>
> > Dear Professor Roger Bivand:
> >
> >     Yes, I was talking about the two SAR fitting functions. I just did a
> > very quick run on a dataset with 3437 observations (those are point data,
> > and I generate the listw object using dnearneigh() - not sure whether the
> > result is symmetric or asymmetric).
>
> How many links are there in the neighbour list? Distance neighbours are
> symmetric, but if points are densely placed, there may be parts of your
> study area with many neighbours. Use print.nb() to find out how many.
>
> > However, I tried both "sparse" and
> > "SparseM" method on errorsarlm(), both failed to go through, the errors
> > given as follow:
> >
> >     For "sparse" method:
> >     "Error in spwdet(sparseweights, rho = rho, debug = debug) :
> >         Suspicious allocation of memory in sparse functions
> >          ...bailing out! Save workspace and quit R!"
>
> Yes, this is the memory bug. It does happen on most/all GCC compiled
> versions, but I've seen BSD versions that work - it depends on the memory
> allocation code, but it has not been possible to solve. I am planning to
> drop "sparse" before long, unless the memory bug is resolved.
>
> >
> >     For "SparseM" method:
> >     "Error in .local(x, ...) : Increase tmpmax
>
> This is the problem: tmpmax is defined in chol() for "matrix.csr" objects
> as only 10 times the number of rows. I think you have more neighbour links
> than 10*n, and that means I need to find a way of passing a better value
> through. Please let me know, this looks do-able.
>
> Roger
>
> >      Error in det(chol(x)) : Unable to find the argument "x" in selecting
> >      a method for function "det" "
> >
> >     By the way, I am still using R 2.0.0 and when I loaded the libraries,
> > I got warnings. I will keep trying and keep you posted as I progress.
> > Thanks.
> >
> >
> > On Mon, 13 Dec 2004, Roger Bivand wrote:
> >
> > > On Mon, 13 Dec 2004, Danlin Yu wrote:
> > >
> > > > Dear Mihai Nica:
> > > >
> > > >     I think it is a memory issue, too. I ran into the same problem when
> > > > using around 3000 observations. Although I used memory.size() to increase
> > > > the memroy allocation size, spdep took too long to run.
> > > >
> > >
> > > I think you mean the two SAR fitting functions, lagsarlm() and
> > > errorsarlm()? The default method is "eigen", which does need plenty of
> > > memory with large n (because it both solves the eigenproblem for an nxn
> > > matrix, and later inverts an nxn matrix). There has always been a "sparse"
> > > method using code from Netlib called sparse from UCB, which works on some
> > > platforms, but not others, and has an unresolved memory bug.
> > >
> > > I'm pleased to report that Roger Koenker has been kind enough to add a
> > > det() for symmetric (and similar to symmetric) sparse matrices to the
> > > SparseM package. From spdep_0.3-5, spdep depends on SparseM, and
> > > errorsarlm() is now pretty dependable for large n and fast. lagsarlm() is
> > > dependable, but its fitting speed is slowed by fitting models for LR tests
> > > dropping each of the X variables in turn, as well as the specified model,
> > > so that it fits k models, not just 1. The function calls for the lag
> > > models are cheaper than the error models, though.
> > >
> > > The restriction to symmetric (and similar to symmetric) sparse matrices is
> > > important to bear in mind. With symmetric neighbour lists (contiguity,
> > > graphe-based and distance bands), and even symmetric inverse distances,
> > > and all weighting styles this is OK (because "W" and "S" style are similar
> > > to symmetric, and the others are symmetric), but absolutely not for
> > > k-nearest-neighbours or for asymmetric general weights (say migration
> > > tables).
> > >
> > > I would be very interested in hearing whether this approach is helpful,
> > > and how this scales.
> > >
> > > Roger
> > >
> > > >     The problem, I think, might be due to the fact that spdep's spatial
> > > > autoregressive models use the exact method (correct me if I am wrong),
> > > > while GeoDa uses an approximatioin method (see the paper "Fast maximum
> > > > likelihood estimation of very large spatial autoregressive models: a
> > > > characteristic polynomial approach", in Computational Statistics & Data
> > > > Analysis 35 (2001) 301-319). If the method could be incorporated in R, it
> > > > will be great.
> > > >
> > > >     Danlin
> > > >
> > > > On Mon, 13 Dec 2004, Mihai Nica wrote:
> > > >
> > > > > Greetings:
> > > > >
> > > > > I would like to take advantage of this thread and ask if there is a way to
> > > > > "enhance" the code in spdep. I have a dataset with 3030 observations and two
> > > > > independent variables. On the same old computer it runs great in GeoDa, but
> > > > > fails in R. I assume it is a memory issue, but having two different ways to
> > > > > check my results would be great.
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Mihai Nica
> > > > > Jackson State University
> > > > > 155 B Parkhurst Dr.
> > > > > Jackson, MS 39202
> > > > > 601 914 0361
> > > > >
> > > > > ----- Original Message -----
> > > > > From: "Luc Anselin" <anselin at uiuc.edu>
> > > > > To: "Darla Munroe" <munroe.9 at osu.edu>
> > > > > Cc: <r-sig-geo at stat.math.ethz.ch>
> > > > > Sent: Monday, December 13, 2004 8:36 AM
> > > > > Subject: Re: [R-sig-Geo] spatial 2SLS
> > > > >
> > > > >
> > > > > > I have some functions that implement this. We used them in last summer's
> > > > > > ICPSR spatial regression course. They are not (yet) very user friendly
> > > > > > and
> > > > > > not up to official R standards yet, but they definitely work.
> > > > > > The functions include:
> > > > > > - standard 2SLS (or IV estimation)
> > > > > > - spatial 2SLS (using WX as instruments for Wy)
> > > > > > - heteroskedastic robust spatial 2SLS (White adjusted variances)
> > > > > > Anyone who wants them and doesn't need a lot of hand holding can e-mail
> > > > > > me directly.
> > > > > > L.
> > > > > >
> > > > > >
> > > > > > On Dec 13, 2004, at 8:21 AM, Darla Munroe wrote:
> > > > > >
> > > > > > > Is the spatial 2SLS developed by Kelejian available in spdep?  If not,
> > > > > > > is it
> > > > > > > available anywhere else at this time?  (besides in SpaceStat).
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > Thanks,
> > > > > > >
> > > > > > > Darla Munroe
> > > > > > >
> > > > > > >
> > > > > > > [[alternative HTML version deleted]]
> > > > > > >
> > > > > > > _______________________________________________
> > > > > > > R-sig-Geo mailing list
> > > > > > > R-sig-Geo at stat.math.ethz.ch
> > > > > > > https://stat.ethz.ch/mailman/listinfo/r-sig-geo
> > > > > > >
> > > > > >
> > > > > > _______________________________________________
> > > > > > R-sig-Geo mailing list
> > > > > > R-sig-Geo at stat.math.ethz.ch
> > > > > > https://stat.ethz.ch/mailman/listinfo/r-sig-geo
> > > > > >
> > > > >
> > > > > _______________________________________________
> > > > > R-sig-Geo mailing list
> > > > > R-sig-Geo at stat.math.ethz.ch
> > > > > https://stat.ethz.ch/mailman/listinfo/r-sig-geo
> > > > >
> > > >
> > > >     Sincerely,
> > > >     Danlin Yu
> > > >
> > > > ----------------------------------
> > > > Lecturer, Ph.D. Candidate
> > > > Department of Geography
> > > > University of Wisconsin, Milwaukee
> > > > Tel: (414)229-3943
> > > > Fax: (414)229-3981
> > > > Email: danlinyu at uwm.edu
> > > >
> > > > _______________________________________________
> > > > R-sig-Geo mailing list
> > > > R-sig-Geo at stat.math.ethz.ch
> > > > https://stat.ethz.ch/mailman/listinfo/r-sig-geo
> > > >
> > >
> > > --
> > > Roger Bivand
> > > Economic Geography Section, Department of Economics, Norwegian School of
> > > Economics and Business Administration, Breiviksveien 40, N-5045 Bergen,
> > > Norway. voice: +47 55 95 93 55; fax +47 55 95 93 93
> > > e-mail: Roger.Bivand at nhh.no
> > >
> > >
> >
> >     Sincerely,
> >     Danlin Yu
> >
> > ----------------------------------
> > Lecturer, Ph.D. Candidate
> > Department of Geography
> > University of Wisconsin, Milwaukee
> > Tel: (414)229-3943
> > Fax: (414)229-3981
> > Email: danlinyu at uwm.edu
> > ----------------------------------
> >
> >
>
> --
> Roger Bivand
> Economic Geography Section, Department of Economics, Norwegian School of
> Economics and Business Administration, Breiviksveien 40, N-5045 Bergen,
> Norway. voice: +47 55 95 93 55; fax +47 55 95 93 93
> e-mail: Roger.Bivand at nhh.no
>
> _______________________________________________
> R-sig-Geo mailing list
> R-sig-Geo at stat.math.ethz.ch
> https://stat.ethz.ch/mailman/listinfo/r-sig-geo
>

    Sincerely,
    Danlin Yu

----------------------------------
Lecturer, Ph.D. Candidate
Department of Geography
University of Wisconsin, Milwaukee
Tel: (414)229-3943
Fax: (414)229-3981
Email: danlinyu at uwm.edu



Reply | Threaded
Open this post in threaded view
|

Fitting SAR models in spdep (was: spatial 2SLS)

Roger Bivand
Administrator
On Mon, 13 Dec 2004, Danlin Yu wrote:

> Dear Professor Roger Bivand:
>
>    Yes, you are right, the total number of links is 686040, much more than
> 10*n (that is 3437) in the dataset.
>
>    Some of the study area is really dense in data, but some is really
> sparse. So the chosen distance ensuring there won't be any non-zero links
> yield a large number of links. The actual reports from print.nb follows:
>
> Neighbour list object:
> Number of regions: 3437
> Number of nonzero links: 686040
> Percentage nonzero weights: 5.807515
> Average number of links: 199.6043
>
>    Please keep me posted if the chol() problem is solved. Thanks. Danlin

Please try pre-release version 0.3-6 - source and Windows binary packages
on: http://spatial.nhh.no/R/spdep/

It would be good if this works. I think, though, that a graph-based
neighbour list (SOI graph, for example) would be a more natural choice
than distances, but for testing, this neighbour list is very suitable.

Roger

>
> On Mon, 13 Dec 2004, Roger Bivand wrote:
>
> > On Mon, 13 Dec 2004, Danlin Yu wrote:
> >
> > > Dear Professor Roger Bivand:
> > >
> > >     Yes, I was talking about the two SAR fitting functions. I just did a
> > > very quick run on a dataset with 3437 observations (those are point data,
> > > and I generate the listw object using dnearneigh() - not sure whether the
> > > result is symmetric or asymmetric).
> >
> > How many links are there in the neighbour list? Distance neighbours are
> > symmetric, but if points are densely placed, there may be parts of your
> > study area with many neighbours. Use print.nb() to find out how many.
> >
> > > However, I tried both "sparse" and
> > > "SparseM" method on errorsarlm(), both failed to go through, the errors
> > > given as follow:
> > >
> > >     For "sparse" method:
> > >     "Error in spwdet(sparseweights, rho = rho, debug = debug) :
> > >         Suspicious allocation of memory in sparse functions
> > >          ...bailing out! Save workspace and quit R!"
> >
> > Yes, this is the memory bug. It does happen on most/all GCC compiled
> > versions, but I've seen BSD versions that work - it depends on the memory
> > allocation code, but it has not been possible to solve. I am planning to
> > drop "sparse" before long, unless the memory bug is resolved.
> >
> > >
> > >     For "SparseM" method:
> > >     "Error in .local(x, ...) : Increase tmpmax
> >
> > This is the problem: tmpmax is defined in chol() for "matrix.csr" objects
> > as only 10 times the number of rows. I think you have more neighbour links
> > than 10*n, and that means I need to find a way of passing a better value
> > through. Please let me know, this looks do-able.
> >
> > Roger
> >
> > >      Error in det(chol(x)) : Unable to find the argument "x" in selecting
> > >      a method for function "det" "
> > >
> > >     By the way, I am still using R 2.0.0 and when I loaded the libraries,
> > > I got warnings. I will keep trying and keep you posted as I progress.
> > > Thanks.
> > >
> > >
> > > On Mon, 13 Dec 2004, Roger Bivand wrote:
> > >
> > > > On Mon, 13 Dec 2004, Danlin Yu wrote:
> > > >
> > > > > Dear Mihai Nica:
> > > > >
> > > > >     I think it is a memory issue, too. I ran into the same problem when
> > > > > using around 3000 observations. Although I used memory.size() to increase
> > > > > the memroy allocation size, spdep took too long to run.
> > > > >
> > > >
> > > > I think you mean the two SAR fitting functions, lagsarlm() and
> > > > errorsarlm()? The default method is "eigen", which does need plenty of
> > > > memory with large n (because it both solves the eigenproblem for an nxn
> > > > matrix, and later inverts an nxn matrix). There has always been a "sparse"
> > > > method using code from Netlib called sparse from UCB, which works on some
> > > > platforms, but not others, and has an unresolved memory bug.
> > > >
> > > > I'm pleased to report that Roger Koenker has been kind enough to add a
> > > > det() for symmetric (and similar to symmetric) sparse matrices to the
> > > > SparseM package. From spdep_0.3-5, spdep depends on SparseM, and
> > > > errorsarlm() is now pretty dependable for large n and fast. lagsarlm() is
> > > > dependable, but its fitting speed is slowed by fitting models for LR tests
> > > > dropping each of the X variables in turn, as well as the specified model,
> > > > so that it fits k models, not just 1. The function calls for the lag
> > > > models are cheaper than the error models, though.
> > > >
> > > > The restriction to symmetric (and similar to symmetric) sparse matrices is
> > > > important to bear in mind. With symmetric neighbour lists (contiguity,
> > > > graphe-based and distance bands), and even symmetric inverse distances,
> > > > and all weighting styles this is OK (because "W" and "S" style are similar
> > > > to symmetric, and the others are symmetric), but absolutely not for
> > > > k-nearest-neighbours or for asymmetric general weights (say migration
> > > > tables).
> > > >
> > > > I would be very interested in hearing whether this approach is helpful,
> > > > and how this scales.
> > > >
> > > > Roger
> > > >
> > > > >     The problem, I think, might be due to the fact that spdep's spatial
> > > > > autoregressive models use the exact method (correct me if I am wrong),
> > > > > while GeoDa uses an approximatioin method (see the paper "Fast maximum
> > > > > likelihood estimation of very large spatial autoregressive models: a
> > > > > characteristic polynomial approach", in Computational Statistics & Data
> > > > > Analysis 35 (2001) 301-319). If the method could be incorporated in R, it
> > > > > will be great.
> > > > >
> > > > >     Danlin
> > > > >
> > > > > On Mon, 13 Dec 2004, Mihai Nica wrote:
> > > > >
> > > > > > Greetings:
> > > > > >
> > > > > > I would like to take advantage of this thread and ask if there is a way to
> > > > > > "enhance" the code in spdep. I have a dataset with 3030 observations and two
> > > > > > independent variables. On the same old computer it runs great in GeoDa, but
> > > > > > fails in R. I assume it is a memory issue, but having two different ways to
> > > > > > check my results would be great.
> > > > > >
> > > > > > Thanks,
> > > > > >
> > > > > > Mihai Nica
> > > > > > Jackson State University
> > > > > > 155 B Parkhurst Dr.
> > > > > > Jackson, MS 39202
> > > > > > 601 914 0361
> > > > > >
> > > > > > ----- Original Message -----
> > > > > > From: "Luc Anselin" <anselin at uiuc.edu>
> > > > > > To: "Darla Munroe" <munroe.9 at osu.edu>
> > > > > > Cc: <r-sig-geo at stat.math.ethz.ch>
> > > > > > Sent: Monday, December 13, 2004 8:36 AM
> > > > > > Subject: Re: [R-sig-Geo] spatial 2SLS
> > > > > >
> > > > > >
> > > > > > > I have some functions that implement this. We used them in last summer's
> > > > > > > ICPSR spatial regression course. They are not (yet) very user friendly
> > > > > > > and
> > > > > > > not up to official R standards yet, but they definitely work.
> > > > > > > The functions include:
> > > > > > > - standard 2SLS (or IV estimation)
> > > > > > > - spatial 2SLS (using WX as instruments for Wy)
> > > > > > > - heteroskedastic robust spatial 2SLS (White adjusted variances)
> > > > > > > Anyone who wants them and doesn't need a lot of hand holding can e-mail
> > > > > > > me directly.
> > > > > > > L.
> > > > > > >
> > > > > > >
> > > > > > > On Dec 13, 2004, at 8:21 AM, Darla Munroe wrote:
> > > > > > >
> > > > > > > > Is the spatial 2SLS developed by Kelejian available in spdep?  If not,
> > > > > > > > is it
> > > > > > > > available anywhere else at this time?  (besides in SpaceStat).
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > >
> > > > > > > > Darla Munroe
> > > > > > > >
> > > > > > > >
> > > > > > > > [[alternative HTML version deleted]]
> > > > > > > >
> > > > > > > > _______________________________________________
> > > > > > > > R-sig-Geo mailing list
> > > > > > > > R-sig-Geo at stat.math.ethz.ch
> > > > > > > > https://stat.ethz.ch/mailman/listinfo/r-sig-geo
> > > > > > > >
> > > > > > >
> > > > > > > _______________________________________________
> > > > > > > R-sig-Geo mailing list
> > > > > > > R-sig-Geo at stat.math.ethz.ch
> > > > > > > https://stat.ethz.ch/mailman/listinfo/r-sig-geo
> > > > > > >
> > > > > >
> > > > > > _______________________________________________
> > > > > > R-sig-Geo mailing list
> > > > > > R-sig-Geo at stat.math.ethz.ch
> > > > > > https://stat.ethz.ch/mailman/listinfo/r-sig-geo
> > > > > >
> > > > >
> > > > >     Sincerely,
> > > > >     Danlin Yu
> > > > >
> > > > > ----------------------------------
> > > > > Lecturer, Ph.D. Candidate
> > > > > Department of Geography
> > > > > University of Wisconsin, Milwaukee
> > > > > Tel: (414)229-3943
> > > > > Fax: (414)229-3981
> > > > > Email: danlinyu at uwm.edu
> > > > >
> > > > > _______________________________________________
> > > > > R-sig-Geo mailing list
> > > > > R-sig-Geo at stat.math.ethz.ch
> > > > > https://stat.ethz.ch/mailman/listinfo/r-sig-geo
> > > > >
> > > >
> > > > --
> > > > Roger Bivand
> > > > Economic Geography Section, Department of Economics, Norwegian School of
> > > > Economics and Business Administration, Breiviksveien 40, N-5045 Bergen,
> > > > Norway. voice: +47 55 95 93 55; fax +47 55 95 93 93
> > > > e-mail: Roger.Bivand at nhh.no
> > > >
> > > >
> > >
> > >     Sincerely,
> > >     Danlin Yu
> > >
> > > ----------------------------------
> > > Lecturer, Ph.D. Candidate
> > > Department of Geography
> > > University of Wisconsin, Milwaukee
> > > Tel: (414)229-3943
> > > Fax: (414)229-3981
> > > Email: danlinyu at uwm.edu
> > > ----------------------------------
> > >
> > >
> >
> > --
> > Roger Bivand
> > Economic Geography Section, Department of Economics, Norwegian School of
> > Economics and Business Administration, Breiviksveien 40, N-5045 Bergen,
> > Norway. voice: +47 55 95 93 55; fax +47 55 95 93 93
> > e-mail: Roger.Bivand at nhh.no
> >
> > _______________________________________________
> > R-sig-Geo mailing list
> > R-sig-Geo at stat.math.ethz.ch
> > https://stat.ethz.ch/mailman/listinfo/r-sig-geo
> >
>
>     Sincerely,
>     Danlin Yu
>
> ----------------------------------
> Lecturer, Ph.D. Candidate
> Department of Geography
> University of Wisconsin, Milwaukee
> Tel: (414)229-3943
> Fax: (414)229-3981
> Email: danlinyu at uwm.edu
> ----------------------------------
>
>

--
Roger Bivand
Economic Geography Section, Department of Economics, Norwegian School of
Economics and Business Administration, Breiviksveien 40, N-5045 Bergen,
Norway. voice: +47 55 95 93 55; fax +47 55 95 93 93
e-mail: Roger.Bivand at nhh.no



Roger Bivand
Department of Economics
Norwegian School of Economics
Helleveien 30
N-5045 Bergen, Norway
Reply | Threaded
Open this post in threaded view
|

Fitting SAR models in spdep (was: spatial 2SLS)

Danlin Yu-2
Dear Professor Roger Bivand:

    Yes, it worked! The same set of data (3437), and same neighbor list
(listw), on my computer (a laptop), for the errorsarlm(), using method
"SparseM", it worked within 5 minutes, no error reported. Is the
lagsarlm() the same?

On Mon, 13 Dec 2004, Roger Bivand wrote:

> On Mon, 13 Dec 2004, Danlin Yu wrote:
>
> > Dear Professor Roger Bivand:
> >
> >    Yes, you are right, the total number of links is 686040, much more than
> > 10*n (that is 3437) in the dataset.
> >
> >    Some of the study area is really dense in data, but some is really
> > sparse. So the chosen distance ensuring there won't be any non-zero links
> > yield a large number of links. The actual reports from print.nb follows:
> >
> > Neighbour list object:
> > Number of regions: 3437
> > Number of nonzero links: 686040
> > Percentage nonzero weights: 5.807515
> > Average number of links: 199.6043
> >
> >    Please keep me posted if the chol() problem is solved. Thanks. Danlin
>
> Please try pre-release version 0.3-6 - source and Windows binary packages
> on: http://spatial.nhh.no/R/spdep/
>
> It would be good if this works. I think, though, that a graph-based
> neighbour list (SOI graph, for example) would be a more natural choice
> than distances, but for testing, this neighbour list is very suitable.
>
> Roger
>
> >
> > On Mon, 13 Dec 2004, Roger Bivand wrote:
> >
> > > On Mon, 13 Dec 2004, Danlin Yu wrote:
> > >
> > > > Dear Professor Roger Bivand:
> > > >
> > > >     Yes, I was talking about the two SAR fitting functions. I just did a
> > > > very quick run on a dataset with 3437 observations (those are point data,
> > > > and I generate the listw object using dnearneigh() - not sure whether the
> > > > result is symmetric or asymmetric).
> > >
> > > How many links are there in the neighbour list? Distance neighbours are
> > > symmetric, but if points are densely placed, there may be parts of your
> > > study area with many neighbours. Use print.nb() to find out how many.
> > >
> > > > However, I tried both "sparse" and
> > > > "SparseM" method on errorsarlm(), both failed to go through, the errors
> > > > given as follow:
> > > >
> > > >     For "sparse" method:
> > > >     "Error in spwdet(sparseweights, rho = rho, debug = debug) :
> > > >         Suspicious allocation of memory in sparse functions
> > > >          ...bailing out! Save workspace and quit R!"
> > >
> > > Yes, this is the memory bug. It does happen on most/all GCC compiled
> > > versions, but I've seen BSD versions that work - it depends on the memory
> > > allocation code, but it has not been possible to solve. I am planning to
> > > drop "sparse" before long, unless the memory bug is resolved.
> > >
> > > >
> > > >     For "SparseM" method:
> > > >     "Error in .local(x, ...) : Increase tmpmax
> > >
> > > This is the problem: tmpmax is defined in chol() for "matrix.csr" objects
> > > as only 10 times the number of rows. I think you have more neighbour links
> > > than 10*n, and that means I need to find a way of passing a better value
> > > through. Please let me know, this looks do-able.
> > >
> > > Roger
> > >
> > > >      Error in det(chol(x)) : Unable to find the argument "x" in selecting
> > > >      a method for function "det" "
> > > >
> > > >     By the way, I am still using R 2.0.0 and when I loaded the libraries,
> > > > I got warnings. I will keep trying and keep you posted as I progress.
> > > > Thanks.
> > > >
> > > >
> > > > On Mon, 13 Dec 2004, Roger Bivand wrote:
> > > >
> > > > > On Mon, 13 Dec 2004, Danlin Yu wrote:
> > > > >
> > > > > > Dear Mihai Nica:
> > > > > >
> > > > > >     I think it is a memory issue, too. I ran into the same problem when
> > > > > > using around 3000 observations. Although I used memory.size() to increase
> > > > > > the memroy allocation size, spdep took too long to run.
> > > > > >
> > > > >
> > > > > I think you mean the two SAR fitting functions, lagsarlm() and
> > > > > errorsarlm()? The default method is "eigen", which does need plenty of
> > > > > memory with large n (because it both solves the eigenproblem for an nxn
> > > > > matrix, and later inverts an nxn matrix). There has always been a "sparse"
> > > > > method using code from Netlib called sparse from UCB, which works on some
> > > > > platforms, but not others, and has an unresolved memory bug.
> > > > >
> > > > > I'm pleased to report that Roger Koenker has been kind enough to add a
> > > > > det() for symmetric (and similar to symmetric) sparse matrices to the
> > > > > SparseM package. From spdep_0.3-5, spdep depends on SparseM, and
> > > > > errorsarlm() is now pretty dependable for large n and fast. lagsarlm() is
> > > > > dependable, but its fitting speed is slowed by fitting models for LR tests
> > > > > dropping each of the X variables in turn, as well as the specified model,
> > > > > so that it fits k models, not just 1. The function calls for the lag
> > > > > models are cheaper than the error models, though.
> > > > >
> > > > > The restriction to symmetric (and similar to symmetric) sparse matrices is
> > > > > important to bear in mind. With symmetric neighbour lists (contiguity,
> > > > > graphe-based and distance bands), and even symmetric inverse distances,
> > > > > and all weighting styles this is OK (because "W" and "S" style are similar
> > > > > to symmetric, and the others are symmetric), but absolutely not for
> > > > > k-nearest-neighbours or for asymmetric general weights (say migration
> > > > > tables).
> > > > >
> > > > > I would be very interested in hearing whether this approach is helpful,
> > > > > and how this scales.
> > > > >
> > > > > Roger
> > > > >
> > > > > >     The problem, I think, might be due to the fact that spdep's spatial
> > > > > > autoregressive models use the exact method (correct me if I am wrong),
> > > > > > while GeoDa uses an approximatioin method (see the paper "Fast maximum
> > > > > > likelihood estimation of very large spatial autoregressive models: a
> > > > > > characteristic polynomial approach", in Computational Statistics & Data
> > > > > > Analysis 35 (2001) 301-319). If the method could be incorporated in R, it
> > > > > > will be great.
> > > > > >
> > > > > >     Danlin
> > > > > >
> > > > > > On Mon, 13 Dec 2004, Mihai Nica wrote:
> > > > > >
> > > > > > > Greetings:
> > > > > > >
> > > > > > > I would like to take advantage of this thread and ask if there is a way to
> > > > > > > "enhance" the code in spdep. I have a dataset with 3030 observations and two
> > > > > > > independent variables. On the same old computer it runs great in GeoDa, but
> > > > > > > fails in R. I assume it is a memory issue, but having two different ways to
> > > > > > > check my results would be great.
> > > > > > >
> > > > > > > Thanks,
> > > > > > >
> > > > > > > Mihai Nica
> > > > > > > Jackson State University
> > > > > > > 155 B Parkhurst Dr.
> > > > > > > Jackson, MS 39202
> > > > > > > 601 914 0361
> > > > > > >
> > > > > > > ----- Original Message -----
> > > > > > > From: "Luc Anselin" <anselin at uiuc.edu>
> > > > > > > To: "Darla Munroe" <munroe.9 at osu.edu>
> > > > > > > Cc: <r-sig-geo at stat.math.ethz.ch>
> > > > > > > Sent: Monday, December 13, 2004 8:36 AM
> > > > > > > Subject: Re: [R-sig-Geo] spatial 2SLS
> > > > > > >
> > > > > > >
> > > > > > > > I have some functions that implement this. We used them in last summer's
> > > > > > > > ICPSR spatial regression course. They are not (yet) very user friendly
> > > > > > > > and
> > > > > > > > not up to official R standards yet, but they definitely work.
> > > > > > > > The functions include:
> > > > > > > > - standard 2SLS (or IV estimation)
> > > > > > > > - spatial 2SLS (using WX as instruments for Wy)
> > > > > > > > - heteroskedastic robust spatial 2SLS (White adjusted variances)
> > > > > > > > Anyone who wants them and doesn't need a lot of hand holding can e-mail
> > > > > > > > me directly.
> > > > > > > > L.
> > > > > > > >
> > > > > > > >
> > > > > > > > On Dec 13, 2004, at 8:21 AM, Darla Munroe wrote:
> > > > > > > >
> > > > > > > > > Is the spatial 2SLS developed by Kelejian available in spdep?  If not,
> > > > > > > > > is it
> > > > > > > > > available anywhere else at this time?  (besides in SpaceStat).
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > >
> > > > > > > > > Darla Munroe
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > [[alternative HTML version deleted]]
> > > > > > > > >
> > > > > > > > > _______________________________________________
> > > > > > > > > R-sig-Geo mailing list
> > > > > > > > > R-sig-Geo at stat.math.ethz.ch
> > > > > > > > > https://stat.ethz.ch/mailman/listinfo/r-sig-geo
> > > > > > > > >
> > > > > > > >
> > > > > > > > _______________________________________________
> > > > > > > > R-sig-Geo mailing list
> > > > > > > > R-sig-Geo at stat.math.ethz.ch
> > > > > > > > https://stat.ethz.ch/mailman/listinfo/r-sig-geo
> > > > > > > >
> > > > > > >
> > > > > > > _______________________________________________
> > > > > > > R-sig-Geo mailing list
> > > > > > > R-sig-Geo at stat.math.ethz.ch
> > > > > > > https://stat.ethz.ch/mailman/listinfo/r-sig-geo
> > > > > > >
> > > > > >
> > > > > >     Sincerely,
> > > > > >     Danlin Yu
> > > > > >
> > > > > > ----------------------------------
> > > > > > Lecturer, Ph.D. Candidate
> > > > > > Department of Geography
> > > > > > University of Wisconsin, Milwaukee
> > > > > > Tel: (414)229-3943
> > > > > > Fax: (414)229-3981
> > > > > > Email: danlinyu at uwm.edu
> > > > > >
> > > > > > _______________________________________________
> > > > > > R-sig-Geo mailing list
> > > > > > R-sig-Geo at stat.math.ethz.ch
> > > > > > https://stat.ethz.ch/mailman/listinfo/r-sig-geo
> > > > > >
> > > > >
> > > > > --
> > > > > Roger Bivand
> > > > > Economic Geography Section, Department of Economics, Norwegian School of
> > > > > Economics and Business Administration, Breiviksveien 40, N-5045 Bergen,
> > > > > Norway. voice: +47 55 95 93 55; fax +47 55 95 93 93
> > > > > e-mail: Roger.Bivand at nhh.no
> > > > >
> > > > >
> > > >
> > > >     Sincerely,
> > > >     Danlin Yu
> > > >
> > > > ----------------------------------
> > > > Lecturer, Ph.D. Candidate
> > > > Department of Geography
> > > > University of Wisconsin, Milwaukee
> > > > Tel: (414)229-3943
> > > > Fax: (414)229-3981
> > > > Email: danlinyu at uwm.edu
> > > > ----------------------------------
> > > >
> > > >
> > >
> > > --
> > > Roger Bivand
> > > Economic Geography Section, Department of Economics, Norwegian School of
> > > Economics and Business Administration, Breiviksveien 40, N-5045 Bergen,
> > > Norway. voice: +47 55 95 93 55; fax +47 55 95 93 93
> > > e-mail: Roger.Bivand at nhh.no
> > >
> > > _______________________________________________
> > > R-sig-Geo mailing list
> > > R-sig-Geo at stat.math.ethz.ch
> > > https://stat.ethz.ch/mailman/listinfo/r-sig-geo
> > >
> >
> >     Sincerely,
> >     Danlin Yu
> >
> > ----------------------------------
> > Lecturer, Ph.D. Candidate
> > Department of Geography
> > University of Wisconsin, Milwaukee
> > Tel: (414)229-3943
> > Fax: (414)229-3981
> > Email: danlinyu at uwm.edu
> > ----------------------------------
> >
> >
>
> --
> Roger Bivand
> Economic Geography Section, Department of Economics, Norwegian School of
> Economics and Business Administration, Breiviksveien 40, N-5045 Bergen,
> Norway. voice: +47 55 95 93 55; fax +47 55 95 93 93
> e-mail: Roger.Bivand at nhh.no
>
>

    Sincerely,
    Danlin Yu

----------------------------------
Lecturer, Ph.D. Candidate
Department of Geography
University of Wisconsin, Milwaukee
Tel: (414)229-3943
Fax: (414)229-3981
Email: danlinyu at uwm.edu



Reply | Threaded
Open this post in threaded view
|

Fitting SAR models in spdep (was: spatial 2SLS)

Roger Bivand
Administrator
On Mon, 13 Dec 2004, Danlin Yu wrote:

> Dear Professor Roger Bivand:
>
>     Yes, it worked! The same set of data (3437), and same neighbor list
> (listw), on my computer (a laptop), for the errorsarlm(), using method
> "SparseM", it worked within 5 minutes, no error reported. Is the
> lagsarlm() the same?

Yes, I hope so, the same value is passed to tmpmax, the sum of the number
of neighbour links (the off-diagonal non-zero values) and n (the
diagonal). Five minutes is not fast, but it's not bad, and the results
should be reliable. The big advantage of using SparseM is that Professors
Roger Koenker and Pin Ng are maintaining the package actively, and using
it for plenty of other work, so any problems are very likely to be found
quickly.

If nobody reports serious problems, I'll submit 0.3-6 to CRAN on Tuesday.

Roger

>
> On Mon, 13 Dec 2004, Roger Bivand wrote:
>
> > On Mon, 13 Dec 2004, Danlin Yu wrote:
> >
> > > Dear Professor Roger Bivand:
> > >
> > >    Yes, you are right, the total number of links is 686040, much more than
> > > 10*n (that is 3437) in the dataset.
> > >
> > >    Some of the study area is really dense in data, but some is really
> > > sparse. So the chosen distance ensuring there won't be any non-zero links
> > > yield a large number of links. The actual reports from print.nb follows:
> > >
> > > Neighbour list object:
> > > Number of regions: 3437
> > > Number of nonzero links: 686040
> > > Percentage nonzero weights: 5.807515
> > > Average number of links: 199.6043
> > >
> > >    Please keep me posted if the chol() problem is solved. Thanks. Danlin
> >
> > Please try pre-release version 0.3-6 - source and Windows binary packages
> > on: http://spatial.nhh.no/R/spdep/
> >
> > It would be good if this works. I think, though, that a graph-based
> > neighbour list (SOI graph, for example) would be a more natural choice
> > than distances, but for testing, this neighbour list is very suitable.
> >
> > Roger
> >
> > >
> > > On Mon, 13 Dec 2004, Roger Bivand wrote:
> > >
> > > > On Mon, 13 Dec 2004, Danlin Yu wrote:
> > > >
> > > > > Dear Professor Roger Bivand:
> > > > >
> > > > >     Yes, I was talking about the two SAR fitting functions. I just did a
> > > > > very quick run on a dataset with 3437 observations (those are point data,
> > > > > and I generate the listw object using dnearneigh() - not sure whether the
> > > > > result is symmetric or asymmetric).
> > > >
> > > > How many links are there in the neighbour list? Distance neighbours are
> > > > symmetric, but if points are densely placed, there may be parts of your
> > > > study area with many neighbours. Use print.nb() to find out how many.
> > > >
> > > > > However, I tried both "sparse" and
> > > > > "SparseM" method on errorsarlm(), both failed to go through, the errors
> > > > > given as follow:
> > > > >
> > > > >     For "sparse" method:
> > > > >     "Error in spwdet(sparseweights, rho = rho, debug = debug) :
> > > > >         Suspicious allocation of memory in sparse functions
> > > > >          ...bailing out! Save workspace and quit R!"
> > > >
> > > > Yes, this is the memory bug. It does happen on most/all GCC compiled
> > > > versions, but I've seen BSD versions that work - it depends on the memory
> > > > allocation code, but it has not been possible to solve. I am planning to
> > > > drop "sparse" before long, unless the memory bug is resolved.
> > > >
> > > > >
> > > > >     For "SparseM" method:
> > > > >     "Error in .local(x, ...) : Increase tmpmax
> > > >
> > > > This is the problem: tmpmax is defined in chol() for "matrix.csr" objects
> > > > as only 10 times the number of rows. I think you have more neighbour links
> > > > than 10*n, and that means I need to find a way of passing a better value
> > > > through. Please let me know, this looks do-able.
> > > >
> > > > Roger
> > > >
> > > > >      Error in det(chol(x)) : Unable to find the argument "x" in selecting
> > > > >      a method for function "det" "
> > > > >
> > > > >     By the way, I am still using R 2.0.0 and when I loaded the libraries,
> > > > > I got warnings. I will keep trying and keep you posted as I progress.
> > > > > Thanks.
> > > > >
> > > > >
> > > > > On Mon, 13 Dec 2004, Roger Bivand wrote:
> > > > >
> > > > > > On Mon, 13 Dec 2004, Danlin Yu wrote:
> > > > > >
> > > > > > > Dear Mihai Nica:
> > > > > > >
> > > > > > >     I think it is a memory issue, too. I ran into the same problem when
> > > > > > > using around 3000 observations. Although I used memory.size() to increase
> > > > > > > the memroy allocation size, spdep took too long to run.
> > > > > > >
> > > > > >
> > > > > > I think you mean the two SAR fitting functions, lagsarlm() and
> > > > > > errorsarlm()? The default method is "eigen", which does need plenty of
> > > > > > memory with large n (because it both solves the eigenproblem for an nxn
> > > > > > matrix, and later inverts an nxn matrix). There has always been a "sparse"
> > > > > > method using code from Netlib called sparse from UCB, which works on some
> > > > > > platforms, but not others, and has an unresolved memory bug.
> > > > > >
> > > > > > I'm pleased to report that Roger Koenker has been kind enough to add a
> > > > > > det() for symmetric (and similar to symmetric) sparse matrices to the
> > > > > > SparseM package. From spdep_0.3-5, spdep depends on SparseM, and
> > > > > > errorsarlm() is now pretty dependable for large n and fast. lagsarlm() is
> > > > > > dependable, but its fitting speed is slowed by fitting models for LR tests
> > > > > > dropping each of the X variables in turn, as well as the specified model,
> > > > > > so that it fits k models, not just 1. The function calls for the lag
> > > > > > models are cheaper than the error models, though.
> > > > > >
> > > > > > The restriction to symmetric (and similar to symmetric) sparse matrices is
> > > > > > important to bear in mind. With symmetric neighbour lists (contiguity,
> > > > > > graphe-based and distance bands), and even symmetric inverse distances,
> > > > > > and all weighting styles this is OK (because "W" and "S" style are similar
> > > > > > to symmetric, and the others are symmetric), but absolutely not for
> > > > > > k-nearest-neighbours or for asymmetric general weights (say migration
> > > > > > tables).
> > > > > >
> > > > > > I would be very interested in hearing whether this approach is helpful,
> > > > > > and how this scales.
> > > > > >
> > > > > > Roger
> > > > > >
> > > > > > >     The problem, I think, might be due to the fact that spdep's spatial
> > > > > > > autoregressive models use the exact method (correct me if I am wrong),
> > > > > > > while GeoDa uses an approximatioin method (see the paper "Fast maximum
> > > > > > > likelihood estimation of very large spatial autoregressive models: a
> > > > > > > characteristic polynomial approach", in Computational Statistics & Data
> > > > > > > Analysis 35 (2001) 301-319). If the method could be incorporated in R, it
> > > > > > > will be great.
> > > > > > >
> > > > > > >     Danlin
> > > > > > >
> > > > > > > On Mon, 13 Dec 2004, Mihai Nica wrote:
> > > > > > >
> > > > > > > > Greetings:
> > > > > > > >
> > > > > > > > I would like to take advantage of this thread and ask if there is a way to
> > > > > > > > "enhance" the code in spdep. I have a dataset with 3030 observations and two
> > > > > > > > independent variables. On the same old computer it runs great in GeoDa, but
> > > > > > > > fails in R. I assume it is a memory issue, but having two different ways to
> > > > > > > > check my results would be great.
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > >
> > > > > > > > Mihai Nica
> > > > > > > > Jackson State University
> > > > > > > > 155 B Parkhurst Dr.
> > > > > > > > Jackson, MS 39202
> > > > > > > > 601 914 0361
> > > > > > > >
> > > > > > > > ----- Original Message -----
> > > > > > > > From: "Luc Anselin" <anselin at uiuc.edu>
> > > > > > > > To: "Darla Munroe" <munroe.9 at osu.edu>
> > > > > > > > Cc: <r-sig-geo at stat.math.ethz.ch>
> > > > > > > > Sent: Monday, December 13, 2004 8:36 AM
> > > > > > > > Subject: Re: [R-sig-Geo] spatial 2SLS
> > > > > > > >
> > > > > > > >
> > > > > > > > > I have some functions that implement this. We used them in last summer's
> > > > > > > > > ICPSR spatial regression course. They are not (yet) very user friendly
> > > > > > > > > and
> > > > > > > > > not up to official R standards yet, but they definitely work.
> > > > > > > > > The functions include:
> > > > > > > > > - standard 2SLS (or IV estimation)
> > > > > > > > > - spatial 2SLS (using WX as instruments for Wy)
> > > > > > > > > - heteroskedastic robust spatial 2SLS (White adjusted variances)
> > > > > > > > > Anyone who wants them and doesn't need a lot of hand holding can e-mail
> > > > > > > > > me directly.
> > > > > > > > > L.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Dec 13, 2004, at 8:21 AM, Darla Munroe wrote:
> > > > > > > > >
> > > > > > > > > > Is the spatial 2SLS developed by Kelejian available in spdep?  If not,
> > > > > > > > > > is it
> > > > > > > > > > available anywhere else at this time?  (besides in SpaceStat).
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Thanks,
> > > > > > > > > >
> > > > > > > > > > Darla Munroe
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > [[alternative HTML version deleted]]
> > > > > > > > > >
> > > > > > > > > > _______________________________________________
> > > > > > > > > > R-sig-Geo mailing list
> > > > > > > > > > R-sig-Geo at stat.math.ethz.ch
> > > > > > > > > > https://stat.ethz.ch/mailman/listinfo/r-sig-geo
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > > _______________________________________________
> > > > > > > > > R-sig-Geo mailing list
> > > > > > > > > R-sig-Geo at stat.math.ethz.ch
> > > > > > > > > https://stat.ethz.ch/mailman/listinfo/r-sig-geo
> > > > > > > > >
> > > > > > > >
> > > > > > > > _______________________________________________
> > > > > > > > R-sig-Geo mailing list
> > > > > > > > R-sig-Geo at stat.math.ethz.ch
> > > > > > > > https://stat.ethz.ch/mailman/listinfo/r-sig-geo
> > > > > > > >
> > > > > > >
> > > > > > >     Sincerely,
> > > > > > >     Danlin Yu
> > > > > > >
> > > > > > > ----------------------------------
> > > > > > > Lecturer, Ph.D. Candidate
> > > > > > > Department of Geography
> > > > > > > University of Wisconsin, Milwaukee
> > > > > > > Tel: (414)229-3943
> > > > > > > Fax: (414)229-3981
> > > > > > > Email: danlinyu at uwm.edu
> > > > > > >
> > > > > > > _______________________________________________
> > > > > > > R-sig-Geo mailing list
> > > > > > > R-sig-Geo at stat.math.ethz.ch
> > > > > > > https://stat.ethz.ch/mailman/listinfo/r-sig-geo
> > > > > > >
> > > > > >
> > > > > > --
> > > > > > Roger Bivand
> > > > > > Economic Geography Section, Department of Economics, Norwegian School of
> > > > > > Economics and Business Administration, Breiviksveien 40, N-5045 Bergen,
> > > > > > Norway. voice: +47 55 95 93 55; fax +47 55 95 93 93
> > > > > > e-mail: Roger.Bivand at nhh.no
> > > > > >
> > > > > >
> > > > >
> > > > >     Sincerely,
> > > > >     Danlin Yu
> > > > >
> > > > > ----------------------------------
> > > > > Lecturer, Ph.D. Candidate
> > > > > Department of Geography
> > > > > University of Wisconsin, Milwaukee
> > > > > Tel: (414)229-3943
> > > > > Fax: (414)229-3981
> > > > > Email: danlinyu at uwm.edu
> > > > > ----------------------------------
> > > > >
> > > > >
> > > >
> > > > --
> > > > Roger Bivand
> > > > Economic Geography Section, Department of Economics, Norwegian School of
> > > > Economics and Business Administration, Breiviksveien 40, N-5045 Bergen,
> > > > Norway. voice: +47 55 95 93 55; fax +47 55 95 93 93
> > > > e-mail: Roger.Bivand at nhh.no
> > > >
> > > > _______________________________________________
> > > > R-sig-Geo mailing list
> > > > R-sig-Geo at stat.math.ethz.ch
> > > > https://stat.ethz.ch/mailman/listinfo/r-sig-geo
> > > >
> > >
> > >     Sincerely,
> > >     Danlin Yu
> > >
> > > ----------------------------------
> > > Lecturer, Ph.D. Candidate
> > > Department of Geography
> > > University of Wisconsin, Milwaukee
> > > Tel: (414)229-3943
> > > Fax: (414)229-3981
> > > Email: danlinyu at uwm.edu
> > > ----------------------------------
> > >
> > >
> >
> > --
> > Roger Bivand
> > Economic Geography Section, Department of Economics, Norwegian School of
> > Economics and Business Administration, Breiviksveien 40, N-5045 Bergen,
> > Norway. voice: +47 55 95 93 55; fax +47 55 95 93 93
> > e-mail: Roger.Bivand at nhh.no
> >
> >
>
>     Sincerely,
>     Danlin Yu
>
> ----------------------------------
> Lecturer, Ph.D. Candidate
> Department of Geography
> University of Wisconsin, Milwaukee
> Tel: (414)229-3943
> Fax: (414)229-3981
> Email: danlinyu at uwm.edu
> ----------------------------------
>
>

--
Roger Bivand
Economic Geography Section, Department of Economics, Norwegian School of
Economics and Business Administration, Breiviksveien 40, N-5045 Bergen,
Norway. voice: +47 55 95 93 55; fax +47 55 95 93 93
e-mail: Roger.Bivand at nhh.no



Roger Bivand
Department of Economics
Norwegian School of Economics
Helleveien 30
N-5045 Bergen, Norway
Reply | Threaded
Open this post in threaded view
|

Fitting SAR models in spdep (was: spatial 2SLS)

Danlin Yu-2
Dear Professor Roger Bivand:

    Actually, I just tried lagsarlm() using the same dataset and same
weight list, but it did not go through for two reasons: first, the memory
is still a problem; second, it seems that det() function did not work well
on the lagsarlm(). The actually error message follows:

 Error: cannot allocate vector of size 5386 Kb
 In addition: Warning message:
 Reached total allocation of 446Mb: see help(memory.size)
 Error in det(chol((I - rho * W), tmpmax = tmpmax)) :
    Unable to find the argument "x" in selecting a method for function "det"


On Mon, 13 Dec 2004, Roger Bivand wrote:

> On Mon, 13 Dec 2004, Danlin Yu wrote:
>
> > Dear Professor Roger Bivand:
> >
> >     Yes, it worked! The same set of data (3437), and same neighbor list
> > (listw), on my computer (a laptop), for the errorsarlm(), using method
> > "SparseM", it worked within 5 minutes, no error reported. Is the
> > lagsarlm() the same?
>
> Yes, I hope so, the same value is passed to tmpmax, the sum of the number
> of neighbour links (the off-diagonal non-zero values) and n (the
> diagonal). Five minutes is not fast, but it's not bad, and the results
> should be reliable. The big advantage of using SparseM is that Professors
> Roger Koenker and Pin Ng are maintaining the package actively, and using
> it for plenty of other work, so any problems are very likely to be found
> quickly.
>
> If nobody reports serious problems, I'll submit 0.3-6 to CRAN on Tuesday.
>
> Roger
>
> >
> > On Mon, 13 Dec 2004, Roger Bivand wrote:
> >
> > > On Mon, 13 Dec 2004, Danlin Yu wrote:
> > >
> > > > Dear Professor Roger Bivand:
> > > >
> > > >    Yes, you are right, the total number of links is 686040, much more than
> > > > 10*n (that is 3437) in the dataset.
> > > >
> > > >    Some of the study area is really dense in data, but some is really
> > > > sparse. So the chosen distance ensuring there won't be any non-zero links
> > > > yield a large number of links. The actual reports from print.nb follows:
> > > >
> > > > Neighbour list object:
> > > > Number of regions: 3437
> > > > Number of nonzero links: 686040
> > > > Percentage nonzero weights: 5.807515
> > > > Average number of links: 199.6043
> > > >
> > > >    Please keep me posted if the chol() problem is solved. Thanks. Danlin
> > >
> > > Please try pre-release version 0.3-6 - source and Windows binary packages
> > > on: http://spatial.nhh.no/R/spdep/
> > >
> > > It would be good if this works. I think, though, that a graph-based
> > > neighbour list (SOI graph, for example) would be a more natural choice
> > > than distances, but for testing, this neighbour list is very suitable.
> > >
> > > Roger
> > >
> > > >
> > > > On Mon, 13 Dec 2004, Roger Bivand wrote:
> > > >
> > > > > On Mon, 13 Dec 2004, Danlin Yu wrote:
> > > > >
> > > > > > Dear Professor Roger Bivand:
> > > > > >
> > > > > >     Yes, I was talking about the two SAR fitting functions. I just did a
> > > > > > very quick run on a dataset with 3437 observations (those are point data,
> > > > > > and I generate the listw object using dnearneigh() - not sure whether the
> > > > > > result is symmetric or asymmetric).
> > > > >
> > > > > How many links are there in the neighbour list? Distance neighbours are
> > > > > symmetric, but if points are densely placed, there may be parts of your
> > > > > study area with many neighbours. Use print.nb() to find out how many.
> > > > >
> > > > > > However, I tried both "sparse" and
> > > > > > "SparseM" method on errorsarlm(), both failed to go through, the errors
> > > > > > given as follow:
> > > > > >
> > > > > >     For "sparse" method:
> > > > > >     "Error in spwdet(sparseweights, rho = rho, debug = debug) :
> > > > > >         Suspicious allocation of memory in sparse functions
> > > > > >          ...bailing out! Save workspace and quit R!"
> > > > >
> > > > > Yes, this is the memory bug. It does happen on most/all GCC compiled
> > > > > versions, but I've seen BSD versions that work - it depends on the memory
> > > > > allocation code, but it has not been possible to solve. I am planning to
> > > > > drop "sparse" before long, unless the memory bug is resolved.
> > > > >
> > > > > >
> > > > > >     For "SparseM" method:
> > > > > >     "Error in .local(x, ...) : Increase tmpmax
> > > > >
> > > > > This is the problem: tmpmax is defined in chol() for "matrix.csr" objects
> > > > > as only 10 times the number of rows. I think you have more neighbour links
> > > > > than 10*n, and that means I need to find a way of passing a better value
> > > > > through. Please let me know, this looks do-able.
> > > > >
> > > > > Roger
> > > > >
> > > > > >      Error in det(chol(x)) : Unable to find the argument "x" in selecting
> > > > > >      a method for function "det" "
> > > > > >
> > > > > >     By the way, I am still using R 2.0.0 and when I loaded the libraries,
> > > > > > I got warnings. I will keep trying and keep you posted as I progress.
> > > > > > Thanks.
> > > > > >
> > > > > >
> > > > > > On Mon, 13 Dec 2004, Roger Bivand wrote:
> > > > > >
> > > > > > > On Mon, 13 Dec 2004, Danlin Yu wrote:
> > > > > > >
> > > > > > > > Dear Mihai Nica:
> > > > > > > >
> > > > > > > >     I think it is a memory issue, too. I ran into the same problem when
> > > > > > > > using around 3000 observations. Although I used memory.size() to increase
> > > > > > > > the memroy allocation size, spdep took too long to run.
> > > > > > > >
> > > > > > >
> > > > > > > I think you mean the two SAR fitting functions, lagsarlm() and
> > > > > > > errorsarlm()? The default method is "eigen", which does need plenty of
> > > > > > > memory with large n (because it both solves the eigenproblem for an nxn
> > > > > > > matrix, and later inverts an nxn matrix). There has always been a "sparse"
> > > > > > > method using code from Netlib called sparse from UCB, which works on some
> > > > > > > platforms, but not others, and has an unresolved memory bug.
> > > > > > >
> > > > > > > I'm pleased to report that Roger Koenker has been kind enough to add a
> > > > > > > det() for symmetric (and similar to symmetric) sparse matrices to the
> > > > > > > SparseM package. From spdep_0.3-5, spdep depends on SparseM, and
> > > > > > > errorsarlm() is now pretty dependable for large n and fast. lagsarlm() is
> > > > > > > dependable, but its fitting speed is slowed by fitting models for LR tests
> > > > > > > dropping each of the X variables in turn, as well as the specified model,
> > > > > > > so that it fits k models, not just 1. The function calls for the lag
> > > > > > > models are cheaper than the error models, though.
> > > > > > >
> > > > > > > The restriction to symmetric (and similar to symmetric) sparse matrices is
> > > > > > > important to bear in mind. With symmetric neighbour lists (contiguity,
> > > > > > > graphe-based and distance bands), and even symmetric inverse distances,
> > > > > > > and all weighting styles this is OK (because "W" and "S" style are similar
> > > > > > > to symmetric, and the others are symmetric), but absolutely not for
> > > > > > > k-nearest-neighbours or for asymmetric general weights (say migration
> > > > > > > tables).
> > > > > > >
> > > > > > > I would be very interested in hearing whether this approach is helpful,
> > > > > > > and how this scales.
> > > > > > >
> > > > > > > Roger
> > > > > > >
> > > > > > > >     The problem, I think, might be due to the fact that spdep's spatial
> > > > > > > > autoregressive models use the exact method (correct me if I am wrong),
> > > > > > > > while GeoDa uses an approximatioin method (see the paper "Fast maximum
> > > > > > > > likelihood estimation of very large spatial autoregressive models: a
> > > > > > > > characteristic polynomial approach", in Computational Statistics & Data
> > > > > > > > Analysis 35 (2001) 301-319). If the method could be incorporated in R, it
> > > > > > > > will be great.
> > > > > > > >
> > > > > > > >     Danlin
> > > > > > > >
> > > > > > > > On Mon, 13 Dec 2004, Mihai Nica wrote:
> > > > > > > >
> > > > > > > > > Greetings:
> > > > > > > > >
> > > > > > > > > I would like to take advantage of this thread and ask if there is a way to
> > > > > > > > > "enhance" the code in spdep. I have a dataset with 3030 observations and two
> > > > > > > > > independent variables. On the same old computer it runs great in GeoDa, but
> > > > > > > > > fails in R. I assume it is a memory issue, but having two different ways to
> > > > > > > > > check my results would be great.
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > >
> > > > > > > > > Mihai Nica
> > > > > > > > > Jackson State University
> > > > > > > > > 155 B Parkhurst Dr.
> > > > > > > > > Jackson, MS 39202
> > > > > > > > > 601 914 0361
> > > > > > > > >
> > > > > > > > > ----- Original Message -----
> > > > > > > > > From: "Luc Anselin" <anselin at uiuc.edu>
> > > > > > > > > To: "Darla Munroe" <munroe.9 at osu.edu>
> > > > > > > > > Cc: <r-sig-geo at stat.math.ethz.ch>
> > > > > > > > > Sent: Monday, December 13, 2004 8:36 AM
> > > > > > > > > Subject: Re: [R-sig-Geo] spatial 2SLS
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > > I have some functions that implement this. We used them in last summer's
> > > > > > > > > > ICPSR spatial regression course. They are not (yet) very user friendly
> > > > > > > > > > and
> > > > > > > > > > not up to official R standards yet, but they definitely work.
> > > > > > > > > > The functions include:
> > > > > > > > > > - standard 2SLS (or IV estimation)
> > > > > > > > > > - spatial 2SLS (using WX as instruments for Wy)
> > > > > > > > > > - heteroskedastic robust spatial 2SLS (White adjusted variances)
> > > > > > > > > > Anyone who wants them and doesn't need a lot of hand holding can e-mail
> > > > > > > > > > me directly.
> > > > > > > > > > L.
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > On Dec 13, 2004, at 8:21 AM, Darla Munroe wrote:
> > > > > > > > > >
> > > > > > > > > > > Is the spatial 2SLS developed by Kelejian available in spdep?  If not,
> > > > > > > > > > > is it
> > > > > > > > > > > available anywhere else at this time?  (besides in SpaceStat).
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > Thanks,
> > > > > > > > > > >
> > > > > > > > > > > Darla Munroe
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > [[alternative HTML version deleted]]
> > > > > > > > > > >
> > > > > > > > > > > _______________________________________________
> > > > > > > > > > > R-sig-Geo mailing list
> > > > > > > > > > > R-sig-Geo at stat.math.ethz.ch
> > > > > > > > > > > https://stat.ethz.ch/mailman/listinfo/r-sig-geo
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > _______________________________________________
> > > > > > > > > > R-sig-Geo mailing list
> > > > > > > > > > R-sig-Geo at stat.math.ethz.ch
> > > > > > > > > > https://stat.ethz.ch/mailman/listinfo/r-sig-geo
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > > _______________________________________________
> > > > > > > > > R-sig-Geo mailing list
> > > > > > > > > R-sig-Geo at stat.math.ethz.ch
> > > > > > > > > https://stat.ethz.ch/mailman/listinfo/r-sig-geo
> > > > > > > > >
> > > > > > > >
> > > > > > > >     Sincerely,
> > > > > > > >     Danlin Yu
> > > > > > > >
> > > > > > > > ----------------------------------
> > > > > > > > Lecturer, Ph.D. Candidate
> > > > > > > > Department of Geography
> > > > > > > > University of Wisconsin, Milwaukee
> > > > > > > > Tel: (414)229-3943
> > > > > > > > Fax: (414)229-3981
> > > > > > > > Email: danlinyu at uwm.edu
> > > > > > > >
> > > > > > > > _______________________________________________
> > > > > > > > R-sig-Geo mailing list
> > > > > > > > R-sig-Geo at stat.math.ethz.ch
> > > > > > > > https://stat.ethz.ch/mailman/listinfo/r-sig-geo
> > > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > > Roger Bivand
> > > > > > > Economic Geography Section, Department of Economics, Norwegian School of
> > > > > > > Economics and Business Administration, Breiviksveien 40, N-5045 Bergen,
> > > > > > > Norway. voice: +47 55 95 93 55; fax +47 55 95 93 93
> > > > > > > e-mail: Roger.Bivand at nhh.no
> > > > > > >
> > > > > > >
> > > > > >
> > > > > >     Sincerely,
> > > > > >     Danlin Yu
> > > > > >
> > > > > > ----------------------------------
> > > > > > Lecturer, Ph.D. Candidate
> > > > > > Department of Geography
> > > > > > University of Wisconsin, Milwaukee
> > > > > > Tel: (414)229-3943
> > > > > > Fax: (414)229-3981
> > > > > > Email: danlinyu at uwm.edu
> > > > > > ----------------------------------
> > > > > >
> > > > > >
> > > > >
> > > > > --
> > > > > Roger Bivand
> > > > > Economic Geography Section, Department of Economics, Norwegian School of
> > > > > Economics and Business Administration, Breiviksveien 40, N-5045 Bergen,
> > > > > Norway. voice: +47 55 95 93 55; fax +47 55 95 93 93
> > > > > e-mail: Roger.Bivand at nhh.no
> > > > >
> > > > > _______________________________________________
> > > > > R-sig-Geo mailing list
> > > > > R-sig-Geo at stat.math.ethz.ch
> > > > > https://stat.ethz.ch/mailman/listinfo/r-sig-geo
> > > > >
> > > >
> > > >     Sincerely,
> > > >     Danlin Yu
> > > >
> > > > ----------------------------------
> > > > Lecturer, Ph.D. Candidate
> > > > Department of Geography
> > > > University of Wisconsin, Milwaukee
> > > > Tel: (414)229-3943
> > > > Fax: (414)229-3981
> > > > Email: danlinyu at uwm.edu
> > > > ----------------------------------
> > > >
> > > >
> > >
> > > --
> > > Roger Bivand
> > > Economic Geography Section, Department of Economics, Norwegian School of
> > > Economics and Business Administration, Breiviksveien 40, N-5045 Bergen,
> > > Norway. voice: +47 55 95 93 55; fax +47 55 95 93 93
> > > e-mail: Roger.Bivand at nhh.no
> > >
> > >
> >
> >     Sincerely,
> >     Danlin Yu
> >
> > ----------------------------------
> > Lecturer, Ph.D. Candidate
> > Department of Geography
> > University of Wisconsin, Milwaukee
> > Tel: (414)229-3943
> > Fax: (414)229-3981
> > Email: danlinyu at uwm.edu
> > ----------------------------------
> >
> >
>
> --
> Roger Bivand
> Economic Geography Section, Department of Economics, Norwegian School of
> Economics and Business Administration, Breiviksveien 40, N-5045 Bergen,
> Norway. voice: +47 55 95 93 55; fax +47 55 95 93 93
> e-mail: Roger.Bivand at nhh.no
>
>

    Sincerely,
    Danlin Yu

----------------------------------
Lecturer, Ph.D. Candidate
Department of Geography
University of Wisconsin, Milwaukee
Tel: (414)229-3943
Fax: (414)229-3981
Email: danlinyu at uwm.edu



Reply | Threaded
Open this post in threaded view
|

Fitting SAR models in spdep (was: spatial 2SLS)

Roger Bivand
Administrator
On Mon, 13 Dec 2004, Danlin Yu wrote:

> Dear Professor Roger Bivand:
>
>     Actually, I just tried lagsarlm() using the same dataset and same
> weight list, but it did not go through for two reasons: first, the memory
> is still a problem; second, it seems that det() function did not work well
> on the lagsarlm(). The actually error message follows:

No, because the memory limit was reached, the sparse matrix being passed
to chol() did not exist, provoking the det() error. This is more difficult
to debug without access to your data - could you make them available?
Shall we take this off the list until I find a solution?

Roger

>
>  Error: cannot allocate vector of size 5386 Kb
>  In addition: Warning message:
>  Reached total allocation of 446Mb: see help(memory.size)
>  Error in det(chol((I - rho * W), tmpmax = tmpmax)) :
>     Unable to find the argument "x" in selecting a method for function "det"
>
>
> On Mon, 13 Dec 2004, Roger Bivand wrote:
>
> > On Mon, 13 Dec 2004, Danlin Yu wrote:
> >
> > > Dear Professor Roger Bivand:
> > >
> > >     Yes, it worked! The same set of data (3437), and same neighbor list
> > > (listw), on my computer (a laptop), for the errorsarlm(), using method
> > > "SparseM", it worked within 5 minutes, no error reported. Is the
> > > lagsarlm() the same?
> >
> > Yes, I hope so, the same value is passed to tmpmax, the sum of the number
> > of neighbour links (the off-diagonal non-zero values) and n (the
> > diagonal). Five minutes is not fast, but it's not bad, and the results
> > should be reliable. The big advantage of using SparseM is that Professors
> > Roger Koenker and Pin Ng are maintaining the package actively, and using
> > it for plenty of other work, so any problems are very likely to be found
> > quickly.
> >
> > If nobody reports serious problems, I'll submit 0.3-6 to CRAN on Tuesday.
> >
> > Roger
> >
> > >
> > > On Mon, 13 Dec 2004, Roger Bivand wrote:
> > >
> > > > On Mon, 13 Dec 2004, Danlin Yu wrote:
> > > >
> > > > > Dear Professor Roger Bivand:
> > > > >
> > > > >    Yes, you are right, the total number of links is 686040, much more than
> > > > > 10*n (that is 3437) in the dataset.
> > > > >
> > > > >    Some of the study area is really dense in data, but some is really
> > > > > sparse. So the chosen distance ensuring there won't be any non-zero links
> > > > > yield a large number of links. The actual reports from print.nb follows:
> > > > >
> > > > > Neighbour list object:
> > > > > Number of regions: 3437
> > > > > Number of nonzero links: 686040
> > > > > Percentage nonzero weights: 5.807515
> > > > > Average number of links: 199.6043
> > > > >
> > > > >    Please keep me posted if the chol() problem is solved. Thanks. Danlin
> > > >
> > > > Please try pre-release version 0.3-6 - source and Windows binary packages
> > > > on: http://spatial.nhh.no/R/spdep/
> > > >
> > > > It would be good if this works. I think, though, that a graph-based
> > > > neighbour list (SOI graph, for example) would be a more natural choice
> > > > than distances, but for testing, this neighbour list is very suitable.
> > > >
> > > > Roger
> > > >
> > > > >
> > > > > On Mon, 13 Dec 2004, Roger Bivand wrote:
> > > > >
> > > > > > On Mon, 13 Dec 2004, Danlin Yu wrote:
> > > > > >
> > > > > > > Dear Professor Roger Bivand:
> > > > > > >
> > > > > > >     Yes, I was talking about the two SAR fitting functions. I just did a
> > > > > > > very quick run on a dataset with 3437 observations (those are point data,
> > > > > > > and I generate the listw object using dnearneigh() - not sure whether the
> > > > > > > result is symmetric or asymmetric).
> > > > > >
> > > > > > How many links are there in the neighbour list? Distance neighbours are
> > > > > > symmetric, but if points are densely placed, there may be parts of your
> > > > > > study area with many neighbours. Use print.nb() to find out how many.
> > > > > >
> > > > > > > However, I tried both "sparse" and
> > > > > > > "SparseM" method on errorsarlm(), both failed to go through, the errors
> > > > > > > given as follow:
> > > > > > >
> > > > > > >     For "sparse" method:
> > > > > > >     "Error in spwdet(sparseweights, rho = rho, debug = debug) :
> > > > > > >         Suspicious allocation of memory in sparse functions
> > > > > > >          ...bailing out! Save workspace and quit R!"
> > > > > >
> > > > > > Yes, this is the memory bug. It does happen on most/all GCC compiled
> > > > > > versions, but I've seen BSD versions that work - it depends on the memory
> > > > > > allocation code, but it has not been possible to solve. I am planning to
> > > > > > drop "sparse" before long, unless the memory bug is resolved.
> > > > > >
> > > > > > >
> > > > > > >     For "SparseM" method:
> > > > > > >     "Error in .local(x, ...) : Increase tmpmax
> > > > > >
> > > > > > This is the problem: tmpmax is defined in chol() for "matrix.csr" objects
> > > > > > as only 10 times the number of rows. I think you have more neighbour links
> > > > > > than 10*n, and that means I need to find a way of passing a better value
> > > > > > through. Please let me know, this looks do-able.
> > > > > >
> > > > > > Roger
> > > > > >
> > > > > > >      Error in det(chol(x)) : Unable to find the argument "x" in selecting
> > > > > > >      a method for function "det" "
> > > > > > >
> > > > > > >     By the way, I am still using R 2.0.0 and when I loaded the libraries,
> > > > > > > I got warnings. I will keep trying and keep you posted as I progress.
> > > > > > > Thanks.
> > > > > > >
> > > > > > >
> > > > > > > On Mon, 13 Dec 2004, Roger Bivand wrote:
> > > > > > >
> > > > > > > > On Mon, 13 Dec 2004, Danlin Yu wrote:
> > > > > > > >
> > > > > > > > > Dear Mihai Nica:
> > > > > > > > >
> > > > > > > > >     I think it is a memory issue, too. I ran into the same problem when
> > > > > > > > > using around 3000 observations. Although I used memory.size() to increase
> > > > > > > > > the memroy allocation size, spdep took too long to run.
> > > > > > > > >
> > > > > > > >
> > > > > > > > I think you mean the two SAR fitting functions, lagsarlm() and
> > > > > > > > errorsarlm()? The default method is "eigen", which does need plenty of
> > > > > > > > memory with large n (because it both solves the eigenproblem for an nxn
> > > > > > > > matrix, and later inverts an nxn matrix). There has always been a "sparse"
> > > > > > > > method using code from Netlib called sparse from UCB, which works on some
> > > > > > > > platforms, but not others, and has an unresolved memory bug.
> > > > > > > >
> > > > > > > > I'm pleased to report that Roger Koenker has been kind enough to add a
> > > > > > > > det() for symmetric (and similar to symmetric) sparse matrices to the
> > > > > > > > SparseM package. From spdep_0.3-5, spdep depends on SparseM, and
> > > > > > > > errorsarlm() is now pretty dependable for large n and fast. lagsarlm() is
> > > > > > > > dependable, but its fitting speed is slowed by fitting models for LR tests
> > > > > > > > dropping each of the X variables in turn, as well as the specified model,
> > > > > > > > so that it fits k models, not just 1. The function calls for the lag
> > > > > > > > models are cheaper than the error models, though.
> > > > > > > >
> > > > > > > > The restriction to symmetric (and similar to symmetric) sparse matrices is
> > > > > > > > important to bear in mind. With symmetric neighbour lists (contiguity,
> > > > > > > > graphe-based and distance bands), and even symmetric inverse distances,
> > > > > > > > and all weighting styles this is OK (because "W" and "S" style are similar
> > > > > > > > to symmetric, and the others are symmetric), but absolutely not for
> > > > > > > > k-nearest-neighbours or for asymmetric general weights (say migration
> > > > > > > > tables).
> > > > > > > >
> > > > > > > > I would be very interested in hearing whether this approach is helpful,
> > > > > > > > and how this scales.
> > > > > > > >
> > > > > > > > Roger
> > > > > > > >
> > > > > > > > >     The problem, I think, might be due to the fact that spdep's spatial
> > > > > > > > > autoregressive models use the exact method (correct me if I am wrong),
> > > > > > > > > while GeoDa uses an approximatioin method (see the paper "Fast maximum
> > > > > > > > > likelihood estimation of very large spatial autoregressive models: a
> > > > > > > > > characteristic polynomial approach", in Computational Statistics & Data
> > > > > > > > > Analysis 35 (2001) 301-319). If the method could be incorporated in R, it
> > > > > > > > > will be great.
> > > > > > > > >
> > > > > > > > >     Danlin
> > > > > > > > >
> > > > > > > > > On Mon, 13 Dec 2004, Mihai Nica wrote:
> > > > > > > > >
> > > > > > > > > > Greetings:
> > > > > > > > > >
> > > > > > > > > > I would like to take advantage of this thread and ask if there is a way to
> > > > > > > > > > "enhance" the code in spdep. I have a dataset with 3030 observations and two
> > > > > > > > > > independent variables. On the same old computer it runs great in GeoDa, but
> > > > > > > > > > fails in R. I assume it is a memory issue, but having two different ways to
> > > > > > > > > > check my results would be great.
> > > > > > > > > >
> > > > > > > > > > Thanks,
> > > > > > > > > >
> > > > > > > > > > Mihai Nica
> > > > > > > > > > Jackson State University
> > > > > > > > > > 155 B Parkhurst Dr.
> > > > > > > > > > Jackson, MS 39202
> > > > > > > > > > 601 914 0361
> > > > > > > > > >
> > > > > > > > > > ----- Original Message -----
> > > > > > > > > > From: "Luc Anselin" <anselin at uiuc.edu>
> > > > > > > > > > To: "Darla Munroe" <munroe.9 at osu.edu>
> > > > > > > > > > Cc: <r-sig-geo at stat.math.ethz.ch>
> > > > > > > > > > Sent: Monday, December 13, 2004 8:36 AM
> > > > > > > > > > Subject: Re: [R-sig-Geo] spatial 2SLS
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > > I have some functions that implement this. We used them in last summer's
> > > > > > > > > > > ICPSR spatial regression course. They are not (yet) very user friendly
> > > > > > > > > > > and
> > > > > > > > > > > not up to official R standards yet, but they definitely work.
> > > > > > > > > > > The functions include:
> > > > > > > > > > > - standard 2SLS (or IV estimation)
> > > > > > > > > > > - spatial 2SLS (using WX as instruments for Wy)
> > > > > > > > > > > - heteroskedastic robust spatial 2SLS (White adjusted variances)
> > > > > > > > > > > Anyone who wants them and doesn't need a lot of hand holding can e-mail
> > > > > > > > > > > me directly.
> > > > > > > > > > > L.
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > On Dec 13, 2004, at 8:21 AM, Darla Munroe wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Is the spatial 2SLS developed by Kelejian available in spdep?  If not,
> > > > > > > > > > > > is it
> > > > > > > > > > > > available anywhere else at this time?  (besides in SpaceStat).
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks,
> > > > > > > > > > > >
> > > > > > > > > > > > Darla Munroe
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > [[alternative HTML version deleted]]
> > > > > > > > > > > >
> > > > > > > > > > > > _______________________________________________
> > > > > > > > > > > > R-sig-Geo mailing list
> > > > > > > > > > > > R-sig-Geo at stat.math.ethz.ch
> > > > > > > > > > > > https://stat.ethz.ch/mailman/listinfo/r-sig-geo
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > _______________________________________________
> > > > > > > > > > > R-sig-Geo mailing list
> > > > > > > > > > > R-sig-Geo at stat.math.ethz.ch
> > > > > > > > > > > https://stat.ethz.ch/mailman/listinfo/r-sig-geo
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > _______________________________________________
> > > > > > > > > > R-sig-Geo mailing list
> > > > > > > > > > R-sig-Geo at stat.math.ethz.ch
> > > > > > > > > > https://stat.ethz.ch/mailman/listinfo/r-sig-geo
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > >     Sincerely,
> > > > > > > > >     Danlin Yu
> > > > > > > > >
> > > > > > > > > ----------------------------------
> > > > > > > > > Lecturer, Ph.D. Candidate
> > > > > > > > > Department of Geography
> > > > > > > > > University of Wisconsin, Milwaukee
> > > > > > > > > Tel: (414)229-3943
> > > > > > > > > Fax: (414)229-3981
> > > > > > > > > Email: danlinyu at uwm.edu
> > > > > > > > >
> > > > > > > > > _______________________________________________
> > > > > > > > > R-sig-Geo mailing list
> > > > > > > > > R-sig-Geo at stat.math.ethz.ch
> > > > > > > > > https://stat.ethz.ch/mailman/listinfo/r-sig-geo
> > > > > > > > >
> > > > > > > >
> > > > > > > > --
> > > > > > > > Roger Bivand
> > > > > > > > Economic Geography Section, Department of Economics, Norwegian School of
> > > > > > > > Economics and Business Administration, Breiviksveien 40, N-5045 Bergen,
> > > > > > > > Norway. voice: +47 55 95 93 55; fax +47 55 95 93 93
> > > > > > > > e-mail: Roger.Bivand at nhh.no
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > >     Sincerely,
> > > > > > >     Danlin Yu
> > > > > > >
> > > > > > > ----------------------------------
> > > > > > > Lecturer, Ph.D. Candidate
> > > > > > > Department of Geography
> > > > > > > University of Wisconsin, Milwaukee
> > > > > > > Tel: (414)229-3943
> > > > > > > Fax: (414)229-3981
> > > > > > > Email: danlinyu at uwm.edu
> > > > > > > ----------------------------------
> > > > > > >
> > > > > > >
> > > > > >
> > > > > > --
> > > > > > Roger Bivand
> > > > > > Economic Geography Section, Department of Economics, Norwegian School of
> > > > > > Economics and Business Administration, Breiviksveien 40, N-5045 Bergen,
> > > > > > Norway. voice: +47 55 95 93 55; fax +47 55 95 93 93
> > > > > > e-mail: Roger.Bivand at nhh.no
> > > > > >
> > > > > > _______________________________________________
> > > > > > R-sig-Geo mailing list
> > > > > > R-sig-Geo at stat.math.ethz.ch
> > > > > > https://stat.ethz.ch/mailman/listinfo/r-sig-geo
> > > > > >
> > > > >
> > > > >     Sincerely,
> > > > >     Danlin Yu
> > > > >
> > > > > ----------------------------------
> > > > > Lecturer, Ph.D. Candidate
> > > > > Department of Geography
> > > > > University of Wisconsin, Milwaukee
> > > > > Tel: (414)229-3943
> > > > > Fax: (414)229-3981
> > > > > Email: danlinyu at uwm.edu
> > > > > ----------------------------------
> > > > >
> > > > >
> > > >
> > > > --
> > > > Roger Bivand
> > > > Economic Geography Section, Department of Economics, Norwegian School of
> > > > Economics and Business Administration, Breiviksveien 40, N-5045 Bergen,
> > > > Norway. voice: +47 55 95 93 55; fax +47 55 95 93 93
> > > > e-mail: Roger.Bivand at nhh.no
> > > >
> > > >
> > >
> > >     Sincerely,
> > >     Danlin Yu
> > >
> > > ----------------------------------
> > > Lecturer, Ph.D. Candidate
> > > Department of Geography
> > > University of Wisconsin, Milwaukee
> > > Tel: (414)229-3943
> > > Fax: (414)229-3981
> > > Email: danlinyu at uwm.edu
> > > ----------------------------------
> > >
> > >
> >
> > --
> > Roger Bivand
> > Economic Geography Section, Department of Economics, Norwegian School of
> > Economics and Business Administration, Breiviksveien 40, N-5045 Bergen,
> > Norway. voice: +47 55 95 93 55; fax +47 55 95 93 93
> > e-mail: Roger.Bivand at nhh.no
> >
> >
>
>     Sincerely,
>     Danlin Yu
>
> ----------------------------------
> Lecturer, Ph.D. Candidate
> Department of Geography
> University of Wisconsin, Milwaukee
> Tel: (414)229-3943
> Fax: (414)229-3981
> Email: danlinyu at uwm.edu
> ----------------------------------
>
>

--
Roger Bivand
Economic Geography Section, Department of Economics, Norwegian School of
Economics and Business Administration, Breiviksveien 40, N-5045 Bergen,
Norway. voice: +47 55 95 93 55; fax +47 55 95 93 93
e-mail: Roger.Bivand at nhh.no



Roger Bivand
Department of Economics
Norwegian School of Economics
Helleveien 30
N-5045 Bergen, Norway