Request for help about running a loop and reading .NC files

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Request for help about running a loop and reading .NC files

Bhaskar Mitra
Hello Everyone,

I have written a loop which reads hundreds of .nc files
and extract information from each .nc file and
exports that corresponding information as a csv file.

The loop works fine until it encounters a .nc file which it cannot read
and the loop stops. I would appreciate if anyone can suggest how can I
modify
the loop, whereby the loop will continue to run by bypassing those
particular
files which it cannot read.

In the end I was also hoping to modify the loop such that it
will generate a report which will inform me which
files were not read by the loop.  The codes are given below

Thanks,
Bhaskar Mitra
NAU

#_------------------------------------------------------------------


library(ncdf4)
library(reshape2)
library(dplyr)
library(stringr)


setwd("directory path")

Output <- "directory path"

flist <- list.files(path ="NCFiles/", pattern = "^.*\\.(nc|NC|Nc|Nc)$")

for (i in 1: length(flist))
{

  nc <- nc_open(paste0("NCFiles/",flist[i]))
  mean1 <- ncvar_get(nc,attributes(nc$dim)$names[3])
  nc_close(nc)

  mean_chl_df <- melt(mean1)
  trial.table.df <-as.data.frame(mean_chl_df)

write.csv(trial.table.df,paste0(Output,"/",tools::file_path_sans_ext(flist[i]),".csv"))

}

        [[alternative HTML version deleted]]

_______________________________________________
R-sig-Geo mailing list
[hidden email]
https://stat.ethz.ch/mailman/listinfo/r-sig-geo
Reply | Threaded
Open this post in threaded view
|

Re: [FORGED] Request for help about running a loop and reading .NC files

Rolf Turner

On 24/03/20 10:52 am, Bhaskar Mitra wrote:

> Hello Everyone,
>
> I have written a loop which reads hundreds of .nc files
> and extract information from each .nc file and
> exports that corresponding information as a csv file.
>
> The loop works fine until it encounters a .nc file which it cannot read
> and the loop stops. I would appreciate if anyone can suggest how can I
> modify
> the loop, whereby the loop will continue to run by bypassing those
> particular
> files which it cannot read.
>
> In the end I was also hoping to modify the loop such that it
> will generate a report which will inform me which
> files were not read by the loop.  The codes are given below

Perhaps use try()???

cheers,

Rolf Turner

--
Honorary Research Fellow
Department of Statistics
University of Auckland
Phone: +64-9-373-7599 ext. 88276

_______________________________________________
R-sig-Geo mailing list
[hidden email]
https://stat.ethz.ch/mailman/listinfo/r-sig-geo
Reply | Threaded
Open this post in threaded view
|

Re: Request for help about running a loop and reading .NC files

Ben Tupper
In reply to this post by Bhaskar Mitra
Hi,

Rolf is correct that you might consider enclosing your nc_open(...) with
try(nc_open(...)), and then testing if the value returned by try() inherits
from the "try-error" class. If there is no problem opening the NCDF file,
then it will return an "ncdf4" class object.

Also, you might isolate the file extraction stuff into a function to make
your life easier. Something like this (obviously not tested)...


### START
# extract data from a NCDF file
# @param filename the path and name of the file
# @return either NULL (for an exception) or a data.frame
extract_from_nc <- function(filename){

  nc <- try(nc_open(filename))
  if (inherits(nc, "try-error")){
    # if it isn't a healthy NCDF file you will be here
    x <- NULL
  } else {
    # otherwise you will be here and you can run your extraction
    mean1 <- ...
     ...
    x <- as.data.frame(mean_chl_df)
    write.csv(x, ... other stuff here ...)
    nc_close(nc)
  }
  return(x)
}

flist <- list.files(path ="NCFiles/", pattern = "^.*\\.(nc|NC|Nc|Nc)$",
full.names = TRUE)

xx <- lapply(flist, extract_from_nc)
ix <- sapply(xx, is.null)                    # find the NULLS
xx <- xx[!ix]                                # remove them (if any)
if (length(xx) > 0) x <- do.call(rbind, xx)  # row bind what remains into
one data frame (in case that is helpful)
### END

I hope that helps.

Cheers,
Ben





On Mon, Mar 23, 2020 at 5:53 PM Bhaskar Mitra <[hidden email]>
wrote:

> Hello Everyone,
>
> I have written a loop which reads hundreds of .nc files
> and extract information from each .nc file and
> exports that corresponding information as a csv file.
>
> The loop works fine until it encounters a .nc file which it cannot read
> and the loop stops. I would appreciate if anyone can suggest how can I
> modify
> the loop, whereby the loop will continue to run by bypassing those
> particular
> files which it cannot read.
>
> In the end I was also hoping to modify the loop such that it
> will generate a report which will inform me which
> files were not read by the loop.  The codes are given below
>
> Thanks,
> Bhaskar Mitra
> NAU
>
> #_------------------------------------------------------------------
>
>
> library(ncdf4)
> library(reshape2)
> library(dplyr)
> library(stringr)
>
>
> setwd("directory path")
>
> Output <- "directory path"
>
> flist <- list.files(path ="NCFiles/", pattern = "^.*\\.(nc|NC|Nc|Nc)$")
>
> for (i in 1: length(flist))
> {
>
>   nc <- nc_open(paste0("NCFiles/",flist[i]))
>   mean1 <- ncvar_get(nc,attributes(nc$dim)$names[3])
>   nc_close(nc)
>
>   mean_chl_df <- melt(mean1)
>   trial.table.df <-as.data.frame(mean_chl_df)
>
>
> write.csv(trial.table.df,paste0(Output,"/",tools::file_path_sans_ext(flist[i]),".csv"))
>
> }
>
>         [[alternative HTML version deleted]]
>
> _______________________________________________
> R-sig-Geo mailing list
> [hidden email]
> https://stat.ethz.ch/mailman/listinfo/r-sig-geo
>


--
Ben Tupper
Bigelow Laboratory for Ocean Science
East Boothbay, Maine
http://www.bigelow.org/
https://eco.bigelow.org

        [[alternative HTML version deleted]]

_______________________________________________
R-sig-Geo mailing list
[hidden email]
https://stat.ethz.ch/mailman/listinfo/r-sig-geo
Reply | Threaded
Open this post in threaded view
|

Re: Request for help about running a loop and reading .NC files

Bhaskar Mitra
Hi Ben, Rolf,

Thanks for your help.

Ben : the code works perfectly.

thanks once again,

bhaskar

On Mon, Mar 23, 2020 at 4:17 PM Ben Tupper <[hidden email]> wrote:

> Hi,
>
> Rolf is correct that you might consider enclosing your nc_open(...) with
> try(nc_open(...)), and then testing if the value returned by try() inherits
> from the "try-error" class. If there is no problem opening the NCDF file,
> then it will return an "ncdf4" class object.
>
> Also, you might isolate the file extraction stuff into a function to make
> your life easier. Something like this (obviously not tested)...
>
>
> ### START
> # extract data from a NCDF file
> # @param filename the path and name of the file
> # @return either NULL (for an exception) or a data.frame
> extract_from_nc <- function(filename){
>
>   nc <- try(nc_open(filename))
>   if (inherits(nc, "try-error")){
>     # if it isn't a healthy NCDF file you will be here
>     x <- NULL
>   } else {
>     # otherwise you will be here and you can run your extraction
>     mean1 <- ...
>      ...
>     x <- as.data.frame(mean_chl_df)
>     write.csv(x, ... other stuff here ...)
>     nc_close(nc)
>   }
>   return(x)
> }
>
> flist <- list.files(path ="NCFiles/", pattern = "^.*\\.(nc|NC|Nc|Nc)$",
> full.names = TRUE)
>
> xx <- lapply(flist, extract_from_nc)
> ix <- sapply(xx, is.null)                    # find the NULLS
> xx <- xx[!ix]                                # remove them (if any)
> if (length(xx) > 0) x <- do.call(rbind, xx)  # row bind what remains into
> one data frame (in case that is helpful)
> ### END
>
> I hope that helps.
>
> Cheers,
> Ben
>
>
>
>
>
> On Mon, Mar 23, 2020 at 5:53 PM Bhaskar Mitra <[hidden email]>
> wrote:
>
>> Hello Everyone,
>>
>> I have written a loop which reads hundreds of .nc files
>> and extract information from each .nc file and
>> exports that corresponding information as a csv file.
>>
>> The loop works fine until it encounters a .nc file which it cannot read
>> and the loop stops. I would appreciate if anyone can suggest how can I
>> modify
>> the loop, whereby the loop will continue to run by bypassing those
>> particular
>> files which it cannot read.
>>
>> In the end I was also hoping to modify the loop such that it
>> will generate a report which will inform me which
>> files were not read by the loop.  The codes are given below
>>
>> Thanks,
>> Bhaskar Mitra
>> NAU
>>
>> #_------------------------------------------------------------------
>>
>>
>> library(ncdf4)
>> library(reshape2)
>> library(dplyr)
>> library(stringr)
>>
>>
>> setwd("directory path")
>>
>> Output <- "directory path"
>>
>> flist <- list.files(path ="NCFiles/", pattern = "^.*\\.(nc|NC|Nc|Nc)$")
>>
>> for (i in 1: length(flist))
>> {
>>
>>   nc <- nc_open(paste0("NCFiles/",flist[i]))
>>   mean1 <- ncvar_get(nc,attributes(nc$dim)$names[3])
>>   nc_close(nc)
>>
>>   mean_chl_df <- melt(mean1)
>>   trial.table.df <-as.data.frame(mean_chl_df)
>>
>>
>> write.csv(trial.table.df,paste0(Output,"/",tools::file_path_sans_ext(flist[i]),".csv"))
>>
>> }
>>
>>         [[alternative HTML version deleted]]
>>
>> _______________________________________________
>> R-sig-Geo mailing list
>> [hidden email]
>> https://stat.ethz.ch/mailman/listinfo/r-sig-geo
>>
>
>
> --
> Ben Tupper
> Bigelow Laboratory for Ocean Science
> East Boothbay, Maine
> http://www.bigelow.org/
> https://eco.bigelow.org
>
>

        [[alternative HTML version deleted]]

_______________________________________________
R-sig-Geo mailing list
[hidden email]
https://stat.ethz.ch/mailman/listinfo/r-sig-geo