Bug using sits_apply

[etiennelalechere]

Hi,

I would like to report the following error returned by the sits_apply function:

<error/purrr_error_indexed>

Error in purrr::map():

ℹ In index: 1.

Caused by error in .check_remote_errors():

! 76 nodes produced errors; first error: gdal_utils warp: an error occured

---

Backtrace:

▆

1. ├─sits::sits_apply(...)

2. ├─sits:::sits_apply.raster_cube(...)

3. │ └─sits:::.jobs_map_parallel_dfr(...)

4. │ └─sits:::.jobs_map_parallel(jobs, fn, ..., progress = progress)

5. │ ├─base::unlist(...)

6. │ └─purrr::map(...)

7. │ └─purrr:::map_("list", .x, .f, ..., .progress = .progress)

8. │ ├─purrr:::with_indexed_errors(...)

9. │ │ └─base::withCallingHandlers(...)

10. │ ├─purrr:::call_with_cleanup(...)

11. │ └─sits (local) .f(.x[[i]], ...)

12. │ └─sits:::.parallel_map(round, fn, ..., progress = progress)

13. │ └─sits:::.parallel_cluster_apply(x, fn, ..., pb = pb)

14. │ └─parallel (local) .check_remote_errors(val)

15. │ └─base::stop(...)

16. └─base::.handleSimpleError(...)

17. └─purrr (local) h(simpleError(msg, call))

18. └─cli::cli_abort(...)

19. └─rlang::abort(...)

The bug can be reproduced from the following example:

Define roi

roi <- c(
lon_min = 5.558055, lat_min = 45.845646,
lon_max = 5.784589, lat_max = 46.152492
)

create a local directory to store the files

local_dir <- "./../../../../Documents/Traitements/Datacubes/data/outputs/2029_11_21_test_sits_release"
start_dates <- c("1984-06-01", "1985-06-01", "1986-06-01", "1987-06-01", "1988-06-01", "1989-06-01", "1990-06-01", "1991-06-01", "1992-06-01", "1993-06-01", "1994-06-01", "1995-06-01", "1996-06-01", "1997-06-01", "1998-06-01", "1999-06-01",
"2000-06-01", "2001-06-01", "2002-06-01", "2003-06-01", "2004-06-01", "2005-06-01", "2006-06-01", "2007-06-01", "2008-06-01", "2009-06-01", "2010-06-01", "2011-06-01", "2012-06-01", "2013-06-01", "2014-06-01", "2015-06-01", "2016-06-01", "2017-06-01",
"2018-06-01", "2019-06-01", "2020-06-01", "2021-06-01", "2022-06-01", "2023-06-01")
end_dates <- c("1984-08-31", "1985-08-31", "1986-08-31", "1987-08-31", "1988-08-31", "1989-08-31", "1990-08-31", "1991-08-31", "1992-08-31", "1993-08-31", "1994-08-31", "1995-08-31", "1996-08-31", "1997-08-31", "1998-08-31", "1999-08-31",
"2000-08-31", "2001-08-31", "2002-08-31", "2003-08-31", "2004-08-31", "2005-08-31", "2006-08-31", "2007-08-31", "2008-08-31", "2009-08-31", "2010-08-31", "2011-08-31", "2012-08-31", "2013-08-31", "2014-08-31", "2015-08-31", "2016-08-31", "2017-08-31",
"2018-08-31", "2019-08-31", "2020-08-31", "2021-08-31", "2022-08-31", "2023-08-31")

Build a cube for each year and save it locally

length(start_dates)
for (i in 1:length(start_dates)){
print(i)
MPC_cube_year <- sits_cube(
source = "MPC",
collection = "LANDSAT-C2-L2",
roi = roi,
bands = c("RED", "NIR08", "CLOUD"),
start_date = start_dates[i],
end_date = end_dates[i]
)

copy the yearly cube to a local directory

MPC_cube_year <- sits_cube_copy(
cube = MPC_cube_year,
output_dir = local_dir
# multicores = 16
)
Sys.sleep(10) # H of sits package maintainer: avoid corrupted files at this step will avoid further bug
}

create a cube from local files

MPC_cube <- sits_cube(
source = "MPC",
collection = "LANDSAT-C2-L2",
data_dir = paste(local_dir,"./MPC_cube",sep="")
)

Time line

summary(MPC_cube)
print(MPC_cube$file_info[[1]] , n = 100)
sits_timeline(MPC_cube)
length(sits_timeline(MPC_cube)$197028)
length(sits_timeline(MPC_cube)$196028)
length(sits_timeline(MPC_cube)$195028)

Remove tile with only 5 images

str(MPC_cube)
MPC_cube <- MPC_cube[c(1,2),]

Regularize the cube to 1 years intervals

MPC_cube_reg <- sits_regularize(
cube = MPC_cube,
output_dir = paste(local_dir,"./MPC_cube_reg",sep=""),
res = 30,
period = "P1Y",
multicores = 1,
progress = T
)
summary(MPC_cube_reg)
print(MPC_cube_reg$file_info[[1]] , n = 100) # the cloud band is lost
sits_timeline(MPC_cube_reg)

Plot a color composite of one date of the cube

plot(MPC_cube_reg,
red = "RED", blue = "RED", green = "NIR08",
date = "1984-01-01",
tile = 196028
)

Retrieve the regular cube from local files

MPC_cube_reg <- sits_cube(
source = "MPC",
collection = "LANDSAT-C2-L2",
data_dir = "./../../../../Documents/Traitements/Datacubes/data/outputs/2029_10_29_test_sits_sleep/MPC_cube_reg"
)
summary(MPC_cube_reg)

Vegetation index (on regularized cubes only)

MPC_cube_NDVI <- sits_apply(MPC_cube_reg,
NDVI = (NIR08 - RED) / (NIR08 + RED),
normalized = T,
output_dir = paste(local_dir,"./MPC_cube_NDVI",sep=""),
# multicores = 2,
progress = T
) # Here's the error reported at the begining of this message

I used the last version of the package (v1.5.2) installed yesterday from: devtools::install_github("e-sensing/sits@dev"). By the way, thank you for this recent update of the package and for your previous feedbacks!

Do you have any ideas to solve this issue when using the sits_apply function?

Cheers,

Etienne

[OldLipe]

Hi @etiennelalechere , thank you for reporting the issue.

I tried to reproduce your error with the dev version but was unable to replicate it.

I believe the error may be related to corrupted images generated in the SITS previous version. If possible, please try generating the cubes again with the dev version.

[gilbertocamara]

Hi @etiennelalechere To obtain the development version ("dev") please use the command:

> devtools::install_github("e-sensing/sits@dev")

Please run your script again with the new version and inform us.

[etiennelalechere]

Hi, yes I had already followed this recommandation before I reported this bug.

The command devtools::install_github("e-sensing/sits@dev") was used before the script lead to the reported bug.

This command installed the v1.5.2 of the package as indicated above.