Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add static typing support #1302

Merged
merged 58 commits into from
Jul 8, 2024
Merged
Show file tree
Hide file tree
Changes from 54 commits
Commits
Show all changes
58 commits
Select commit Hold shift + click to select a range
68938a6
initial stub file
headtr1ck Dec 13, 2023
dcf4cb6
add always_mask to private_atts
headtr1ck Dec 14, 2023
a0484b3
fix typehints
headtr1ck Dec 14, 2023
6324875
remove _netCDF4 from reprs
headtr1ck Dec 14, 2023
7d8efdb
add linebreaks
headtr1ck Dec 14, 2023
a389d06
update gitignore
headtr1ck Dec 14, 2023
28f37e4
add stubtest workflow co-authored by https://github.com/Woefie
headtr1ck Dec 15, 2023
ca7215a
fix repr replacement
headtr1ck Dec 15, 2023
f965598
add _netCDF4 stubs
headtr1ck Dec 15, 2023
d9f7ffd
add entry to changelog
headtr1ck Dec 15, 2023
5cb9aa3
submodule update
headtr1ck Jan 17, 2024
02a8c4e
Merge branch 'master' into typing
headtr1ck Jan 17, 2024
55455da
add stubtest to existing workflow
headtr1ck Jan 17, 2024
7888be2
Merge branch 'master' into typing
headtr1ck Jan 24, 2024
aa33352
Merge branch 'Unidata:master' into typing
headtr1ck Jun 7, 2024
d4c6ace
install mypy in workflow
headtr1ck Jun 7, 2024
8de2c1b
add mypy check of tests
headtr1ck Jun 7, 2024
ca26429
ignore imports with missing stubs
headtr1ck Jun 7, 2024
3f93b49
fix some types found by tests
headtr1ck Jun 7, 2024
b9089df
fix examples so mypy passes
headtr1ck Jun 7, 2024
92d2626
add mypy test for examples
headtr1ck Jun 7, 2024
17616b0
fix last error in examples
headtr1ck Jun 7, 2024
4bdf621
fix type of chunksizes
headtr1ck Jun 7, 2024
5da5b82
remove some redundant funcs on MFDataset
headtr1ck Jun 7, 2024
f433aab
make variable generic
headtr1ck Jun 7, 2024
60580bf
add overloads for createVariable to return generics
headtr1ck Jun 7, 2024
b616b93
fix mpi_example
headtr1ck Jun 19, 2024
8011349
Merge branch 'master' into typing
headtr1ck Jun 19, 2024
08a6a69
fix changelog
headtr1ck Jun 19, 2024
e8ee7a8
add stubs to manifest
headtr1ck Jun 20, 2024
a192f68
tiny docs fix
headtr1ck Jun 20, 2024
c531ba8
fix return type of tocdl
headtr1ck Jun 20, 2024
f819d44
sort imports
headtr1ck Jun 20, 2024
558a31f
fix some more types
headtr1ck Jun 20, 2024
1974c70
add forgotten attrs for MFDataset
headtr1ck Jun 20, 2024
b884d69
fix some more types
headtr1ck Jun 20, 2024
826a5a9
install types-setuptools in CI
headtr1ck Jun 21, 2024
cb4d011
import dunder vars in _netCDF4
headtr1ck Jun 21, 2024
64140de
remove dunder methods from allowlist
headtr1ck Jun 21, 2024
89584e7
import __has_parallel_support__ and __has_ncfilter__ in __init__
headtr1ck Jun 21, 2024
b1b9ecb
move dtype_is_complex to _netCDF4
headtr1ck Jun 21, 2024
dc8dcb6
apply some changes from review
headtr1ck Jun 26, 2024
f151aef
more suggestions from review
headtr1ck Jun 26, 2024
3094ed8
remove str and repr stubs
headtr1ck Jun 26, 2024
2671af9
fix typing of grp args
headtr1ck Jun 26, 2024
f628ff3
remove all ._netCDF4 from all reprs
headtr1ck Jun 26, 2024
7576a37
improve typing of fill_value
headtr1ck Jun 26, 2024
6d8a734
use GetSetItemKey type
headtr1ck Jun 26, 2024
3f3df1b
fix dtype of EnumType
headtr1ck Jun 26, 2024
d774ae4
fix docstring arg names of CompoundType
headtr1ck Jun 26, 2024
053e049
remove redundand properties
headtr1ck Jun 26, 2024
d0e9bd4
str as valid dtype
headtr1ck Jun 26, 2024
db03592
add some properties and overloads
headtr1ck Jun 26, 2024
b0f08be
Merge branch 'master' into typing
headtr1ck Jun 26, 2024
9704822
Update MANIFEST.in
headtr1ck Jun 27, 2024
fe92b19
fix typing of tutorial
headtr1ck Jun 27, 2024
58ba81b
!fixup a192f68ec706fb8bb78e7c6aec533bdc7d2b900e
headtr1ck Jun 28, 2024
1644caa
!fixup a192f68ec706fb8bb78e7c6aec533bdc7d2b900e
headtr1ck Jun 28, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
33 changes: 33 additions & 0 deletions .github/stubtest-allowlist
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
netCDF4.AccessModeOptions
netCDF4.CompressionLevelOptions
netCDF4.CompressionOptions
netCDF4.DatatypeOptions
netCDF4.DimensionsOptions
netCDF4.DiskFormatOptions
netCDF4.EndianOptions
netCDF4.FormatOptions
netCDF4.QuantizeOptions
netCDF4.CalendarOptions
netCDF4.ellipsis
netCDF4.DateTimeArray
netCDF4.FiltersDict
netCDF4.SzipInfo
netCDF4.BloscInfo
netCDF4.BoolInt
netCDF4.GetSetItemKey
netCDF4.T_Datatype
netCDF4.T_DatatypeNC
netCDF4.Dataset.__dealloc
netCDF4.Dimension.__reduce_cython__
netCDF4.Dimension.__setstate_cython__
netCDF4.Variable.auto_complex
netCDF4._netCDF4.Dataset.__dealloc
netCDF4._netCDF4.Dimension.__reduce_cython__
netCDF4._netCDF4.Dimension.__setstate_cython__
netCDF4._netCDF4.NC_DISKLESS
netCDF4._netCDF4.NC_PERSIST
netCDF4._netCDF4.Variable.auto_complex
netCDF4._netCDF4.__reduce_cython__
netCDF4._netCDF4.__setstate_cython__
netCDF4._netCDF4.__test__
netCDF4.utils.bytes
9 changes: 8 additions & 1 deletion .github/workflows/build_master.yml
Original file line number Diff line number Diff line change
Expand Up @@ -47,13 +47,14 @@ jobs:
- name: Install python dependencies via pip
run: |
python -m pip install --upgrade pip
pip install numpy cython cftime pytest twine wheel check-manifest mpi4py
pip install numpy cython cftime pytest twine wheel check-manifest mpi4py mypy types-setuptools

- name: Install netcdf4-python
run: |
export PATH=${NETCDF_DIR}/bin:${PATH}
export NETCDF_PLUGIN_DIR=${{ github.workspace }}/netcdf-c/plugins/plugindir
python setup.py install

- name: Test
run: |
export PATH=${NETCDF_DIR}/bin:${PATH}
Expand All @@ -78,3 +79,9 @@ jobs:
else
echo "hdf5 compressed mpi test passed!"
fi

- name: Stubtest
run: |
stubtest netCDF4 --allowlist .github/stubtest-allowlist --mypy-config-file=pyproject.toml
mypy test
mypy examples
7 changes: 5 additions & 2 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -2,8 +2,11 @@ build/
*.pyc
dist/
*.egg-info/
netCDF4/_netCDF4.c
netCDF4/*.so
__pycache__
.mypy_cache
src/netCDF4/*.c
src/netCDF4/*.so
src/netCDF4/*.pyd
include/constants.pyx
include/parallel_support_imports.pxi
netcdftime/_netcdftime.c
Expand Down
4 changes: 4 additions & 0 deletions Changelog
Original file line number Diff line number Diff line change
@@ -1,3 +1,7 @@
version 1.7.2 (tag v1.7.2rel)
===============================
* add static type hints (PR #1302)

version 1.7.1 (tag v1.7.1rel)
===============================
* include nc_complex source code from v0.2.0 tag (instead of using submodule).
Expand Down
2 changes: 2 additions & 0 deletions MANIFEST.in
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@ include src/netCDF4/_netCDF4.pyx
exclude src/netCDF4/_netCDF4.c
include src/netCDF4/utils.py
include src/netCDF4/plugins/empty.txt
include src/netCDF4/py.typed
include include/netCDF4.pxi
include include/mpi-compat.h
include include/membuf.pyx
Expand All @@ -23,6 +24,7 @@ include include/no_parallel_support_imports.pxi.in
include include/parallel_support_imports.pxi.in
include *.md
include *.py
include *.pyi
include *.release
include *.sh
include LICENSE
4 changes: 2 additions & 2 deletions docs/index.html
Original file line number Diff line number Diff line change
Expand Up @@ -118,8 +118,8 @@ <h2 id="creatingopeningclosing-a-netcdf-file">Creating/Opening/Closing a netCDF
If the file is open for write access (<code>mode='w', 'r+'</code> or <code>'a'</code>), you may
write any type of data including new dimensions, groups, variables and
attributes.
netCDF files come in five flavors (<code>NETCDF3_CLASSIC,
NETCDF3_64BIT_OFFSET, NETCDF3_64BIT_DATA, NETCDF4_CLASSIC&lt;code&gt;, and &lt;/code&gt;NETCDF4</code>).
netCDF files come in five flavors (<code>NETCDF3_CLASSIC</code>,
<code>NETCDF3_64BIT_OFFSET</code>, <code>NETCDF3_64BIT_DATA</code>, <code>NETCDF4_CLASSIC</code>, and <code>NETCDF4</code>).
headtr1ck marked this conversation as resolved.
Show resolved Hide resolved
<code>NETCDF3_CLASSIC</code> was the original netcdf binary format, and was limited
to file sizes less than 2 Gb. <code>NETCDF3_64BIT_OFFSET</code> was introduced
in version 3.6.0 of the library, and extended the original binary format
Expand Down
2 changes: 1 addition & 1 deletion examples/bench_compress.py
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@
ntrials = 10
sys.stdout.write('reading and writing a %s by %s by %s by %s random array ..\n'%(n1dim,n2dim,n3dim,n4dim))
sys.stdout.write('(average of %s trials)\n' % ntrials)
array = netCDF4.utils._quantize(uniform(size=(n1dim,n2dim,n3dim,n4dim)),4)
array = netCDF4.utils._quantize(uniform(size=(n1dim,n2dim,n3dim,n4dim)),4) # type: ignore


def write_netcdf(filename,zlib=False,shuffle=False,complevel=6):
Expand Down
24 changes: 19 additions & 5 deletions examples/mpi_example.py
Original file line number Diff line number Diff line change
@@ -1,36 +1,50 @@
# to run: mpirun -np 4 python mpi_example.py
import sys
from typing import Literal
from mpi4py import MPI
import numpy as np
from netCDF4 import Dataset

format: Literal[
'NETCDF4',
'NETCDF4_CLASSIC',
'NETCDF3_CLASSIC',
'NETCDF3_64BIT_OFFSET',
'NETCDF3_64BIT_DATA'
]
if len(sys.argv) == 2:
format = sys.argv[1]
format = sys.argv[1] # type: ignore
else:
format = 'NETCDF4_CLASSIC'

rank = MPI.COMM_WORLD.rank # The process ID (integer 0-3 for 4-process run)
if rank == 0:
print('Creating file with format {}'.format(format))
print('Creating file with format {}'.format(format))
nc = Dataset('parallel_test.nc', 'w', parallel=True, comm=MPI.COMM_WORLD,
info=MPI.Info(),format=format)
info=MPI.Info(), format=format)
# below should work also - MPI_COMM_WORLD and MPI_INFO_NULL will be used.
#nc = Dataset('parallel_test.nc', 'w', parallel=True)
d = nc.createDimension('dim',4)
v = nc.createVariable('var', np.int32, 'dim')
v[rank] = rank

# switch to collective mode, rewrite the data.
v.set_collective(True)
v[rank] = rank
nc.close()

# reopen the file read-only, check the data
nc = Dataset('parallel_test.nc', parallel=True, comm=MPI.COMM_WORLD,
info=MPI.Info())
info=MPI.Info())
assert rank==nc['var'][rank]
nc.close()

# reopen the file in append mode, modify the data on the last rank.
nc = Dataset('parallel_test.nc', 'a',parallel=True, comm=MPI.COMM_WORLD,
info=MPI.Info())
info=MPI.Info())
if rank == 3: v[rank] = 2*rank
nc.close()

# reopen the file read-only again, check the data.
# leave out the comm and info kwargs to check that the defaults
# (MPI_COMM_WORLD and MPI_INFO_NULL) work.
Expand Down
4 changes: 2 additions & 2 deletions examples/test_stringarr.py
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
from netCDF4 import Dataset, stringtochar, chartostring
import random, numpy
from typing import Final

# test utilities for converting arrays of fixed-length strings
# to arrays of characters (with an extra dimension), and vice-versa.
Expand All @@ -16,7 +17,7 @@


FILE_NAME = 'tst_stringarr.nc'
FILE_FORMAT = 'NETCDF4_CLASSIC'
FILE_FORMAT: Final = 'NETCDF4_CLASSIC'
chars = '1234567890aabcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ'

nc = Dataset(FILE_NAME,'w',format=FILE_FORMAT)
Expand All @@ -26,7 +27,6 @@
nc.createDimension('nchar',nchar)
v = nc.createVariable('strings','S1',('n1','n2','nchar'))
for nrec in range(nrecs):
data = []
data = numpy.empty((n2,),'S'+repr(nchar))
# fill data with random nchar character strings
for n in range(n2):
Expand Down
2 changes: 1 addition & 1 deletion examples/threaded_read.py
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@
nc.close()

# Queue them up
items = queue.Queue()
items: queue.Queue = queue.Queue()
for data,fname in zip(datal,fnames):
items.put(fname)

Expand Down
Loading
Loading