v0.13.1.post1

same as v0.13.1, including tests for conda-forge pipeline

v0.13.1

pyaerocom v0.13.1 2022.05 release

What's Changed

• Make aeroval cope with full NaN obs time series (solve #639:) by @jgriesfeller in #653
• update conda environment and package manifest by @avaldebe in #655

Full Changelog: v0.13.0...v0.13.1

v0.13.0

pyaerocom v0.13.0 2022.05 release

What's Changed

• some iris 3.1.0 adjustments by @jgriesfeller in #493
• bump version to next dev version by @jgriesfeller in #497
• Add AeroVal config documentation example by @jgliss in #495
• make JSON files smaller by limiting the precision by @jgriesfeller in #499
• build best practices by @avaldebe in #500
• readthedocs setup by @avaldebe in #510
• added some missing minimum values to make the aeroval analysis work with default ranges by @jgriesfeller in #503
• Treat warnings as errors by @avaldebe in #462
• Fix #488 (multiyear support for diurnal processing) by @hansbrenna in #502
• Get tutorials to work again by @dulte in #514
• Changes tutorial link by @dulte in #513
• make aeroval work for the case when the model data variable has been aliased automatically by @jgriesfeller in #517
• Emep mulitple years by @dulte in #451
• Add lint job to CI by @avaldebe in #397
• Create CONTRIBUTING.md by @hansbrenna in #511
• replace deprecated np.float by no.float64 by @avaldebe in #524
• fix ungriddedata.UngriddedData.from_stationdata with single station by @Ovewh in #536
• Split tests with/without exceptions by @avaldebe in #526
• make the AERONET reading more fault tolerable; add possibility to read gzipped files by @jgriesfeller in #538
• use plain dicts by @avaldebe in #530
• one logger per module by @avaldebe in #535
• use importlib.resources/metadata instead of pyaeorocom.dir/version by @avaldebe in #531
• test against python 3.10 on CI by @avaldebe in #542
• clean up entry points by @avaldebe in #540
• AeronetSDAv3 reading: provide od550gtaer data as od550dust data by @jgriesfeller in #545
• update conda CI jobs by @avaldebe in #548
• Pin black version by @avaldebe in #578
• Fix logging configuration by @avaldebe in #575
• Changes sspm10 to be ssf+ssc by @dulte in #579
• improve filter/ignore warnings by @avaldebe in #552
• Exclude cartopy 0.20 from pip installs by @avaldebe in #586
• missing ini files by @avaldebe in #590
• make anaconda CI work with iris 3.2 as well by @jgriesfeller in #589
• Docs551 by @lewisblake in #598
• correct EBAS data reading by @jgriesfeller in #595
• Fairmode Beta*MQI by @lewisblake in #607
• cartopy broken CI jobs by @avaldebe in #608
• add Fairmode statistic to aeroval output by @avaldebe in #613
• Missing INI files by @avaldebe in #617
• Fix minor fairmode bug by @lewisblake in #614
• EEA Reading: make the file search patterns more specific to not find unwanted files by @jgriesfeller in #612
• Obs only by @dulte in #593
• pip install on machines with old/new versions of Proj by @avaldebe in #620
• Typer-based CLI by @lewisblake in #624
• Add vmrno2 by @WillemVanCaspel in #631
• Fix o3max by @dulte in #627
• Remove extra c from concsspm10 by @dulte in #633
• change URL to test new pyaerocom webserver with CI by @jgriesfeller in #626
• Change pm25 UrRV to 0.36 by @lewisblake in #646
• Adds list of regions instead of just the best by @dulte in #582
• add angstrom exponent filtered AERONET AOD for dust evaluation by @Ovewh in #621
• enable variable ratpm10pm25 via aeroval by @jgriesfeller in #641
• Update README.rst by @AugustinMortier in #649

New Contributors

• @Ovewh made their first contribution in #536
• @lewisblake made their first contribution in #598
• @WillemVanCaspel made their first contribution in #631

Full Changelog: v0.12.1...v0.13.0

Version 0.12.1

Same as v0.12.0, just some updates for shipping and testing to ensure conda-forge build succeeds (see #490).

v0.12.0

pyaerocom version 0.12.0 (release summary)

This release comes with many new features, major improvements and a more stable API. Please see the individual points below for major changes.

The notes below summarise the major updates since the release of version 0.10.0. This includes the changes that were done in the recent release of version 0.11.0, which is the last version that officially supports the former "Aerocom Evaluation" tools, e.g. used extensively in CAMS61 project.

Major updates

NEW AeroVal tools

The web processing tools have seen major improvements and the backend has a new API, which can be found in the aeroval sub-package. Compared to the former AeroCom Evaluation tools (web sup-package in former releases), the new tools are a lot more flexible and support, for instance:

• Processing of time series of statistical parameters.
• Option to calculate seasonal statistics.
• Temporal and spatial correlation in heatmap display.
• Combination of different observation datasets into "superobservations".
• All this is packed into a new modular and intuitive API.

Visit AeroVal: https://aeroval.met.no/
Follow AeroVal on twitter: https://twitter.com/AeroVal_MetNo

NOTE: web sub-package is deprecated

• All code under pyaerocom/web is deprecated and this sub-package does not exist anymore in v0.12.0. Relevant code has been shipped and redesigned into the code basis for the new AeroVal tools (pyaerocom/aeroval subpackage).
• Processing tools for old trends interface are deprecated. Instead trends visualisation is included in the new AeroVal interface and can be processed via the new tools in the aeroval subpackage.
• Former CLIs for web tools (pyaeroeval, pyaerotrends) are deprecated. However, the simple pyaerocom (pya) CLI is still available.

New observation networks

• Added support for reading of EEA air pollution data
• Added support for reading of AirNow air pollution data
• Added support for reading of MarcoPolo air pollution data

Further updates and improvements

• Improvements in EBAS reader, e.g.
  • Evaluation of individual timestamps to resolve ts_type of NASA Ames file.
  • Improved logic for automatic column selection.
  • Automatic conversion of vmr to conc and vice versa (e.g. vmro3 to conco3)
• Evaluation of wet deposition and precipitation data.
• Revised, more powerful and intuitive co-location routines.
  • E.g. "per-station" time co-location
• Improved and more robust unit conversions
  • E.g. Automatic detection and correction of implicit deposition units to
    explicit units, both for model and obs, centralised in units_helpers.
    py module.
• Improved robustness and performance of EMEP reader.
• Support reading and co-location of AeroCom model climatology files (year
  9999 in filename). See here
  for details.
• New features in TsType class (e.g.infer ts_type from total seconds)
• Improved and more clearly defined API.
  • E.g. Harmonisation of ungridded reading API
• Installation with all requirements via PyPi
• Many bug fixes
• Improved test coverage.

1st release candidate for v0.12.0

This is a pre-release for v0.12.0, release summary and changelog will follow.

v0.11.0

Las release to support the old aerocom evaluation tools!

Practice release for new release managers.

Version used for 2021 EMEP report (trends analysis)

NOTE: this is not an official release but a pre-release intended for reproducibility of the trends calculation done for the 2021 EMEP report. This corresponds to a development branch for release of pyaerocom version 0.12.0.

The code used to compute the trends for the EMEP report can be found here.

Version 0.10.1: Minor updates and bug fixes

This is just a minor release that provides minor updates. That is, it includes

• quick fixes for #319 and #322 ,
• more restrictive with respect to dependencies, due to recent updates in pandas and iris, and
• dependency netcdf4 was removed
• the official documentation was moved from pyaerocom.met.no to ReadTheDocs (https://pyaerocom.readthedocs.io)

Version 0.10.0

pyaerocom version 0.10.0

This release comes with many new features, major improvements and a more stable API. Please see the individual points below for major changes.

New feature modules

Modules that contain new features. See here for a list of all file modifications between version 0.8.0 and 0.10.0.

• pyaerocom/combine_vardata_ungridded.py: Colocation of ungridded data.
• pyaerocom/helpers_landsea_masks.py: Access and helpers for land-sea mask filtering.
• pyaerocom/io/read_ghost.py: reading routine for GHOST dataset.
• pyaerocom/io/read_mscw_ctm.py: Reading interface for EMEP data.
• pyaerocom/molmasses.py: helpers related to access of molecular masses for species.
• pyaerocom/trends_engine.py: Interface for computing trends using the method by Mortier et al., 2020.
• pyaerocom/web/helpers_evaluation_iface.py: helper methods for conversion of ColocatedData to json files for Aerocom Evaluation interface.
• pyaerocom/web/helpers_trends_iface.py: helper methods for conversion of ColocatedData to json files for Aerocom trends interface.
• pyaerocom/web/utils.py: High-level methods based on results from standard Aerocom analysis. Currently, this contains a method compute_model_average_and_diversity which can be used to compute ensemble median or mean modeldata (e.g. used to compute AEROCOM-MEDIAN and MEAN in Gliss et al., 2020).
• pyaerocom/scripts/cli.py: simple command line interface (currently very limited).
• pyaerocom/scripts/highlevel_utils.py: high-level functions used in CLI.
• pyaerocom/testdata_access.py: helpers related to initialization and access of pyaerocom testdata.

Updates related to supported observation data-sets and naming conventions

• Support for reading GHOST data.
• Support for reading of EMEP model data.

Below, relevant changes applied to already existing code are summarized.

Reading of data (io sub-package)

Reading of gridded data

• ReadGridded class

  • Implement logic to apply constraint reading (i.e. read AOD where AE<0.5, read scattering where RH<40%, etc.). See input arg. constraints in ReadGridded.read_var and associated new class methods listed below.
  • Improve flexibility related to multiple vert_code matches using new method get_vert_code in Variable class.
  • Improve logic associated with resolving variable names for edge cases that include usage of alias variables and auxiliary reading of derived variables.
  • Add new default auxiliary / derived variables: sc550dryaer, concox, vmrox, fmf550aer.
  • New attrs. (incl. @Property decorators): vars_provided, registered_var_patterns.
  • New methods: reinit, check_constraint_valid, apply_read_constraint, __repr__.

• New: Reading interface and support for EMEP file conventions (class ReadMscvCtm).

Reading of ungridded data

• ReadUngridded

  • Add option only_cached in read method, to only read cached data e.g. when working offline.
  • ReadUngridded can post compute variables with merge method combine.
  • New attrs. (incl. @Property decorators): data_dir, post_compute
  • New methods: read_dataset_post, get_vars_supported

• NEW: Reading interface for GHOST data (class ReadGhost).

• read_ebas.oy (ReadEbas and ReadEbasOptions)

  • Introduced more reading options (e.g. related to shifting of wavelengths and assumed Angstrom Exponents used for shifting).
  • NOTE: Before, data at wavelengths (wvl) within a tolerance of +/- 50nm were considered around the output wvl (e.g. 500-600nm for sc550aer) and were used as is (i.e. without converting the values to 550nm). Now, the column with wvl closest to the output wvl is used and the measurement data is shifted to the output wvl (e.g. 550nm) assuming a typical Angstrom Exponent (AE=1.5 for extinction and scattering, AAE=1 for absorption, note also upcoming changes for next major release version 0.11.0).
  • Remove filelog in ReadEbas.
  • Resolve some issues related to old variable names and aliases.
  • Implement unit check in column selection method _find_best_data_column.
  • Add framework to output data.
  • Added attrs. in ReadEbas (incl. @Property methods): sqlite_database_file, file_dir.

• ReadUngriddedBase (template metaclass for ungridded reading)

  • allow to specify dataset_path (less dependency on server access, easier to work with local datasets)
  • New attrs. (incl. @Property decorators): data_id
  • New methods (implemented): var_supported

Further updates in io sub-package

• cachehandler_ungridded.py (CacheHandlerUngridded class)

  • Modified to handle custom filenames (e.g. used in new method UngriddedData.save_as).
  • New method delete_all_cache_files
  • New option force_use_outdated

• Add method get_ungridded_reader in io/utils.py.

Data objects

• GriddedData

  • Can be converted to xarray.
  • More robust and flexible time-series extraction, e.g., constraints can be applied during temporal resampling (i.e. hierarchical min_num_obs or how).
  • Improving automatic retrieval of lowest layer for profile data using CF attr. "positive".
  • Method mean now uses area weighted mean by default.
  • New methods: years_avail, split_years, mean_at_coords, filter_altitude, filter_region, apply_region_mask, aerocom_savename, to_xarray, area_weighted_mean

• UngriddedData

  • Can be filtered by country names
  • Can now be saved as pickled objects
  • Create UngriddedData from StationData object(s)
  • Colocation of UngriddedData is now possible (relevant code is in new module combine_vardata_ungridded.py)
  • Support for wildcards in station data conversion methods.
  • New attrs. (incl. decorators): last_meta_idx, nonunique_station_names, countries_available
  • New methods: from_station_data (static method), check_set_country, check_convert_var_units, filter_altitude, filter_region, apply_region_mask, colocate_vardata, save_as, from_cache (static method).

• ColocatedData

  • Implement method to compute regional time-series.
  • Support automatic assignments of countries for each site.
  • Support computation of area weighted statistics in calc_statistics.
  • New attributes: has_time_dim, has_latlon_dims, countries_available, country_codes_available, area_weights
  • New methods: get_country_codes, calc_area_weights, flatten_latlondim_station_name, stack, unstack, check_set_countries, filter_altitude, apply_country_filter, set_zeros_nan, apply_region_mask, filter_region, get_regional_timeseries

• StationData

  • Computation of climatological time-series
  • Improved handling of metadata access (get_meta) and merging of metadata
  • resample_timeseries was renamed to resample_time (but old version still works.)
  • new method StationData.copy

Colocation of data

Low-level colocation routines (colocation.py)

• Outliers in gridded/gridded colocation are now removed in original resolution.
• Gridded/gridded colocation now re-grids to the lowest of both resolutions.
• Add option resample_how, which can also be applied hierarchical, like min_num_obs (e.g. used to resample O3max).
• resample_how option in high level colocation routines.
• Option to use obs climatology for gridded / ungridded colocation.
• New helper method correct_model_stp_coldata in colocation.py which applies STP correction to a colocated data object containing obs at STP based on station altitude and temperature derived from ERA Interim data (BETA feature only working for 2010 data and with access to METNo infrastructure).
• Some bug fixes for certain edge cases.

High-level colocation routines (colocation_auto.py)

Affects classes ColocationSetup and Colocator

• Support new EMEP reading routine.
• Model and obsdata directories can be specified explicitly.
• Option model_to_stp (BETA feature which will not work in most cases, see above).
• New attrs. obs_add_meta, resample_how.
• New methods (Colocator): read_model_data, read_ungridded

Filtering of data

• Implement filtering of binary masks for GriddedData, UngriddedData and ColocatedData
• Harmonize API of spatial filtering in data classes (i.e. method filter_region that can handle rectangular and binary region masks)
• Automatic access to HTAP binary land-sea masks
• Handling of binary and rectangular regions in Filter class.

Other updates (in top-level modules)

• config.py (Config class)

  • Major improvements and API changes, e.g. related to automatic setup and adding new data search directories and ungridded observations.
  • In particular, attrs. BASEDIR, MODELBASEDIR, OBSBASEDIR are deprecated.
  • Instead, methods add_data_search_dir and add_ungridded_obs can be used to register data locations.

• geodesy.py: new methods calc_latlon_dists, find_coord_indices_within_distance, get_country_info_coords.

• New methods in helpers.py, the most relevant ones are extract_latlon_dataarray, make_dummy_cube_latlon, numpy_to_cube, sort_ts_types, calc_climatology.

• New methods in mathutils.py: weighted_sum, sum, weighted_mean, weighted_cov, weighted_corr, corr (which were implemented in calc_statistics), vmrx_to_concx, concx_to_vmrx.

• New class AerocomDataID in **...