Awdb - active forecast point webservice

[jdduh]

We currently don't have a feature layer webservice for NWCC active forecasts. This layer will eventually replace the "stations USGS ACTIVE" service on NRCS AGOL and be used as the "master" AOI list (i.e., replacing Master AOI List Feature Service).

The current master AOI list is based on the stations USGS ACTIVE layer. The attribute denoting whether a USGS station is an active NWCC forecast location was provided by NWCC as a static layer. The attribute is not updated routinely.

The key attribute for identifying an active forecast is the "endDate". When the endData value is beyond the current date value (usually with a value of 2100-01-01 00:00:00) then the stations is an active forecast. The logic for creating a NWCC active forecasts is:

1. Use the active USGS stations to retrieve the stationtriplet values for each station
2. Use the getStationMetadata method to retrieve the endDate values for each station
3. Keep the stations whose endDate is later then present date

For more information on AWDB webservices, see: https://www.nrcs.usda.gov/wps/portal/wcc/home/dataAccessHelp/webService/webServiceReference

Please refer to #4 for additional requirements.

[beautah]

@jdduh - this layer should NOT replace the active USGS layer, rather supplement it, I might suggest calling it "stations FCST ACTIVE" on agol - but you should really keep both since we store observed data for all the USGS network stations. You SHOULD base the AOI generation on the FCST stations though.

[beautah]

I'll add a step #4 as well:

4. Use the get getForecastPoints method to cross reference which ACTIVE USGS network stations are indeed FCST points - the "stationTriplet" is returned from this method and can be used to further filter the ACTIVE USGS STATIONS to ACTIVE FORECAST STATIONS

[lbross]

Just to clarify, does the getForecastPoints method return ALL forecast points or just ACTIVE forecast points? If just ACTIVE, I could skip checking the endDate values.

[beautah]

it returns active forecast points only, but you also need an active USGS layer as well, so you really are going to need both anyways. The getForecastPoints only returns triplets and some other forecast related metadata too, so its important to join the data from step 2 with the triplets from step 4. (also note you should probably use getStationMetadataMultiple rather than the single station method in step 2)

[lbross]

The BAGIS-Pro reporting tool also needs to query a winter start and end month at the basin level. We currently maintain this in the PSU Master AOI List Feature Service. This new forecast point service will need to also query the PSU master service to get the most current values from these fields. BAGIS-Pro also needs the HUC2 value. Ensure that this is in the new webservice.

[jdduh]

NWCC suggests that use getForecastPoints (using wildcard) to get a list of all active forecast points, then use the getStationMetadata to get other forecast point attributes.

[jdduh]

The new stations_FCST_Active webservice will be disconnected from the USGS station database due to different stationTriplet ID values are used at NWCC. The basinarea data, which come from USGS, won't be available in the new webservice.

[lbross]

I've been experimenting with the getForecastPoints() api but can't find a combination of arguments to get ALL of the stationTriplets. Any suggestions?

[beautah]

@lbross - it appears the service is broken atm, you should be able to leave all the optional fields blank and get them all... but it appears that it only returns them when you give it stationIds. I'll look into the service, get that working again. Our new REST API can return forecast points and associated metadata, but you need to filter them

https://wcc.sc.egov.usda.gov/awdbRestApi/swagger-ui.html#/Station%20Metadata/getStations

this url pulls all the sites that could be a forecast point, if the object in the returned array has the "forecastPoint" key in it you know its a forecast point, so that's how you can filter. Here's the url to use for that (note that I'm pulling all USGS and BOR network sites as they are the networks that have forecast points): https://wcc.sc.egov.usda.gov/awdbRestApi/services/v1/stations?activeOnly=true&returnForecastPointMetadata=true&returnReservoirMetadata=false&returnStationElements=false&stationTriplets=*:*:USGS,*:*:BOR

in python you'd have something like:

import requests
url = "https://wcc.sc.egov.usda.gov/awdbRestApi/services/v1/stations?activeOnly=true&returnForecastPointMetadata=true&returnReservoirMetadata=false&returnStationElements=false&stationTriplets=*:*:USGS,*:*:BOR"
req = requests.get(url)
sites = req.json()
forecast_pnts = f = [i for i in sites if i.get("forecastPoint", False)]

I would totally understand if you want to keep all this in the SOAP webservice, so I'll try to remember to let you know when that functionality is restored. Its 100% a bug.

[lbross]

Thanks @beautah. I'm just starting to familiarize myself with the current script to see what the best approach might be for populating this new service. I was unaware that some of the forecast stations could also be in the BOR network. My initial thought was to make a copy of the stations from the USGS_ACTIVE service and then to delete stations that aren't in the collection returned by getForecastPoints. This allowed me to re-use much of our existing code. But if these stations need to be merged with USGS_BOR, this might not be the best approach.

[beautah]

Yeah, if you hold out a little while you can just make a blank request to the getForecastPoints method (the logicalAnd arg IS required however, the rest can be ignored), that'll be the best way, no need to pass actual IDs. It might take us a week or two to get the fix deployed however

…

On Thu, Jan 4, 2024 at 2:32 PM Lesley ***@***.***> wrote: Thanks @beautah <https://github.com/beautah>. I'm just starting to familiarize myself with the current script to see what the best approach might be for populating this new service. I was unaware that some of the forecast stations could also be in the BOR network. My initial thought was to make a copy of the stations from the USGS_ACTIVE service and then to delete stations that aren't in the collection returned by getForecastPoints. This allowed me to re-use much of our existing code. But if these stations need to be merged with USGS_BOR, this might not be the best approach. — Reply to this email directly, view it on GitHub <#5 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABO6OL5ZKRCZFSQTHCRJTEDYM4NVVAVCNFSM6AAAAAAUWZFYYGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQNZXG44DQNBZHA> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[jdduh]

The PSU team has decided to use only the station information from AWDB based on the fact that the station ID values can be independently assigned in AWDB and cannot be correctly linked to USGS & BOR records. The current stations_USGS_ACTIVE webservice is derived from AWDB, but the PSU script also uses USGS api to get the basinarea data from USGS based on the station ID. This part of the script will be (has been?) deactivated.

If NWCC's active forecast station is always a subset of the stations_USGS_ACTIVE and that NWCC has resolved the station ID conflicts when pulling data from USGS & BOR, then Lesley's solution would be reasonable.

[lbross]

I'm still not clear if we need to merge stations_USGS_ACTIVE and stations_BOR_ACTIVE before checking for the awdb_id in the collection returned from getForecastPoints? Or if manipulating stations_USGS_ACTIVE can provide the full set of forecast stations? And it's no problem on our end to wait for the fix to be deployed.

[beautah]

There is no reason to use the triplets from USGS_ACTIVE or BOR_ACTIVE. Once the SOAP method works again you can simply get the full list of forecast triplets from the getForecastPoints by not using the filters at all (but logicalAnd does need to be set to True). Once you have the list of forecast triplets you can get use that to pull the metadata from USGS_ACTIVE and BOR_ACTIVE to make a layer. @jdduh 's comment that USGS_ACTIVE represents all the forecasts is not correct (there are a couple BOR as well), and furthermore you cannot assume the station conflicts don't exist, they absolutely do and will never not exist, ignore the fact these networks are called BOR and USGS, they should really be called RES and GAGE or something like that so to summarize, you can get the triplets from getForecastPoints once we get it fixed (a couple weeks I hope), then you can use that list of triplets to make a FORECAST_ACTIVE layer by including the metadata from USGS_ACTIVE and BOR_ACTIVE for that list of forecast triplets. You basically will join a subset of BOR and USGS stations that have been identified as forecast points.

…

On Thu, Jan 4, 2024 at 3:23 PM Lesley ***@***.***> wrote: I'm still not clear if we need to merge stations_USGS_ACTIVE and stations_BOR_ACTIVE before checking for the awdb_id in the collection returned from getForecastPoints? Or if manipulating stations_USGS_ACTIVE can provide the full set of forecast stations? And it's no problem on our end to wait for the fix to be deployed. — Reply to this email directly, view it on GitHub <#5 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABO6OL7UBNQDR4J4T5B7YTLYM4TVXAVCNFSM6AAAAAAUWZFYYGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQNZXHA2DANBQHA> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[beautah]

Try the getForecastPoints method again, I discovered that it suddenly works again without any fix on our end. Obviously there is an issue and we'll look into it, but I checked last night and it was just magically working. B

…

On Thu, Jan 4, 2024, 4:58 PM Beau Uriona ***@***.***> wrote: There is no reason to use the triplets from USGS_ACTIVE or BOR_ACTIVE. Once the SOAP method works again you can simply get the full list of forecast triplets from the getForecastPoints by not using the filters at all (but logicalAnd does need to be set to True). Once you have the list of forecast triplets you can get use that to pull the metadata from USGS_ACTIVE and BOR_ACTIVE to make a layer. @jdduh 's comment that USGS_ACTIVE represents all the forecasts is not correct (there are a couple BOR as well), and furthermore you cannot assume the station conflicts don't exist, they absolutely do and will never not exist, ignore the fact these networks are called BOR and USGS, they should really be called RES and GAGE or something like that so to summarize, you can get the triplets from getForecastPoints once we get it fixed (a couple weeks I hope), then you can use that list of triplets to make a FORECAST_ACTIVE layer by including the metadata from USGS_ACTIVE and BOR_ACTIVE for that list of forecast triplets. You basically will join a subset of BOR and USGS stations that have been identified as forecast points. On Thu, Jan 4, 2024 at 3:23 PM Lesley ***@***.***> wrote: > I'm still not clear if we need to merge stations_USGS_ACTIVE and > stations_BOR_ACTIVE before checking for the awdb_id in the collection > returned from getForecastPoints? Or if manipulating stations_USGS_ACTIVE > can provide the full set of forecast stations? And it's no problem on our > end to wait for the fix to be deployed. > > — > Reply to this email directly, view it on GitHub > <#5 (comment)>, > or unsubscribe > <https://github.com/notifications/unsubscribe-auth/ABO6OL7UBNQDR4J4T5B7YTLYM4TVXAVCNFSM6AAAAAAUWZFYYGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQNZXHA2DANBQHA> > . > You are receiving this because you were mentioned.Message ID: > ***@***.***> >

[lbross]

Glad that getForecastPoints is "fixed". Will a station ever be in both the BOR and USGS networks or are the two mutually exclusive?

[beautah]

since triplets are unique that's impossible, the network is in the triplet, so you could have a USGS and BOR site in the same physical location but they would be separate forecast points (one would forecast streamflow and the other lake rise) It appears that the getForecastPoints is not actually fixed too btw, we have a pair of load balanced servers and it appears one of them is failing while the other is not, so you have a 50/50 chance of it working. I'd hold off coding against it for now.. although in the meantime you could simply pass a list of all 50 states into the stateCd field and get a complete set, that arg seems to be working.

…

On Fri, Jan 5, 2024 at 11:26 AM Lesley ***@***.***> wrote: Glad that getForecastPoints is "fixed". Will a station ever be in both the BOR and USGS networks or are the two mutually exclusive? — Reply to this email directly, view it on GitHub <#5 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABO6OL2GCE5N45B2QBGR3D3YNBAV3AVCNFSM6AAAAAAUWZFYYGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQNZZGA4TEMBZGU> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[beautah]

the SOAP method is working again, it was a simple fix. You should be able to query the forecast points without including filters

[lbross]

Reference syntax to set huc2 from forecast aoi master list:

1. reclass(!L0master_fcst_list_fs.huc2!)
2. def reclass(huc2):
   if (huc2 != None):
   return huc2
   else:
   return None

[lbross]

The new service has been deployed on basins and the update script ran successfully. I would like to see it run for a couple of weeks before updating the clients. Action items on the client side:

• Update \vb-bagis\src\BAGIS_Reference_Maps.lyr to use new service as source for 'Gauge Stations' layer
• Update \bagis-pro\Settings\bagis.json to use new service url for gaugeStation parameter; Also used by bagis-pro
• Update \bagis-pro\Settings\bagis-pro.json to use new service url for MasterAoiList parameter. Note: We query the MasterAoiList when an AOI is set to get the current winter start/end month values. We don't save these values locally. The huc2 value is queried when the forecast station data is updated and is saved in the pourpoint FC. An AOI is set when a report is generated. Update: Eliminate MasterAoiList parameter and use gaugeStation parameter everywhere since it's the same service.

Note that BAGIS V3 users will have to use the 'Default Settings' button to retrieve these new settings when they are deployed.

Can I remove this warning from BAGIS V3? It shouldn't take too long and is confusing. It happens when you load the BAGIS settings form.

[lbross]

Regarding the verbal request to generate a separate file when stations are added/removed. I could add a 'createdDate' to the service fairly easily. This would at least allow us to find the new ones easily...

[jdduh]

That's a good solution. With the same logic, I can check, for example, the bagis_note value for newly added stations. Let's ignore the request to keep track of the station updates in the script for now.

Just to confirm the removal of checking basinarea attribute in BAGIS V3.

[lbross]

So if bagis_note is null, it is a newly added station? That should work. There is only one station in the service currently where bagis_note is null: 06102500:MT:USGS. I will work on updating the BAGIS clients to use the new service.

[lbross]

I updated gaugeStation parameter in the bagis settings to point to the FCST ACTIVE service. This will affect BAGIS v3. I updated BAGIS Pro to use this parameter in lieu of the MasterAoiList parameter that we are sunsetting. I posted a new version of the BAGIS-Pro addIn to the basins FTP server. Please download and test. Once we confirm that this works, I will remove the MasterAoiList parameter from the parameter list.

FYI, the new service doesn't have the awdb_id field so this field will no longer be updated going forward. I still need to make the changes to BAGIS v3.

[lbross]

An update: I only needed to update the two .json configuration files on the server to repoint the gauge station layer to the new FCST ACTIVE service. The BAGIS_Reference_Maps.lyr didn't contain any references to the gauge stations so no changes were needed. I have removed the warnings about the missing basinarea field from BAGIS v3. To wrap this up I am waiting for @jduh to

1. Test the latest version of the BAGIS-Pro addIn from the basins FTP server. Note that the previous version might not work well because the field names are different in the new service
2. Update to the Default Settings in BAGIS v3. This should reset the gauge stations layer to FCST ACTIVE service. You will still get warnings about the basinarea field, but the AOI creation process should still work. It seemed to work for me. Once we know there are no additional changes required, I will publish a new version of this addIn.

[jdduh]

Confirm both features are working. Thanks!

[lbross]

Great! I just posted a new addIn to BAGIS v3 that eliminates the warning messages. And I have removed the MasterAoiList parameter from the configuration settings since BAGIS-Pro is now using the gaugeStation parameter which is pointed to the new FCST ACTIVE service.

[jdduh]

It seems that the stations_FCST_ACTIVE webservice hasn't been updated since May 2024. Also, the service is now shown as a geodatabase, instead of a feature layer. BAGIS Pro's Forecast Station Data tool seems to be working ok. It didn't give any error message the last time I used it.

[lbross]

The stations_FCST_ACTIVE webservice should be in the awdb folder. I'm not sure how it got deleted. I will look into this.

Something strange is going on here. I'm not sure how to proceed. We need to republish the stations_FCST_ACTIVE feature service. I can't update an existing service because it is missing from the choices in the UI. When I try to publish it as a new service, it already exists.

I tried to delete the service definition from AGOL and got this warning message:

Any suggestions? You probably have the same privileges in AGOL.

[jdduh]

I think several webservices got corrupted the last time I tried to publish the stations_FCST_ACTIVE_polygons_statistics_HU_16. ArcGIS Pro failed to publish the said service on NRCS AGOL. Afterwards, I started to notice some of the data contents do not match their names (for example, some polygon feature layers became point feature layers). I am going to delete the webservices that I know we don't need. Feel free to clean up NRCS AGOL if you see something that we don't need there. Hopefully this will tidy up the account.

[jdduh]

I have removed all the PRISM imageservices and several other files from the NRCS AGOL. Please don't delete the 4 items in the basin_analysis_reports folder for now.

[lbross]

AGOL won't let me share the feature service stations_FCST_ACTIVE. I tried renaming the service in the sd definition that still appeared in the awdb folder. That didn't work. I tried deleting the sd definition file and that still doesn't work. Can you think of any options besides opening a case with ESRI or publishing the service under a different name? BAGIS-Pro should be okay with a different name because it uses the item id in the parameter file. I will have to make some changes to the Python script that updates the service because it searches by name.

[jdduh]

We might need to delete stations_FCST_ACTIVE and republish it. I have a copy of the data that I downloaded on Oct 10, 2024 that I can use to republish the service. Let me know when I can try this approach.
Moving forward, we might need to keep a backup webservice of stations_FCST_ACTIVE so that we can recover the data if the service is corrupted again. I will also check the access privilege of the feature layer just to make sure that only nwcc_nrcs can modify it.

[lbross]

Do you mean delete and re-add the stations_FCST_ACTIVE map from the ArcGIS Pro project? I can't find a way to delete the service from ArcGIS Online. You can try the approach any time. Let me know if you think it worked and I will try running the update script.

[jdduh]

stations_FCST_ACTIVE is republished. Everything looks normal. AGOL now has a recycle bin feature. We need to delete a webservice permanently to actually delete it. Otherwise, AGOL won't allow the publishing of the webservice with the same name even if the webservice is in the recycle bin. If your script uses the Delete function to remove a webservice, then you need to look for the permanent delete option/parameter to prevent issues.

[lbross]

So to clarify ... even though I renamed the sd definition and deleted it, you had to delete it out of the recycle bin before it was actually gone? The script does not use a Delete function. I don't think the script caused this problem. It's been running for months with no issues. I'll check now to make sure that it still works.

[jdduh]

AGOL recycle bin is a new feature. It was released in June 2024! What a coincidence! https://www.esri.com/arcgis-blog/products/arcgis-online/announcements/recycle-bin-what-you-need-to-know/

API Changes
Changes to the existing APIs have been made to accommodate recycle bin. These include:

returning recycle bin support and status in portal calls
returning recycle bin items in item and content calls
allowing delete calls to bypass the recycle bin via the permanent delete call
Additionally, new endpoints allow restoring items and checking whether an item can be recycled or restored.

It is important to note that the default behavior of delete APIs has changed so that supported items are recycled rather than permanently deleted. Previous behavior can be replicated by passing a permanentDelete call as detailed in the doc pages below.

[lbross]

Any chance you could move it to the awdb folder with the other awdb services? This is where I look to check every 2 weeks to make sure the services updated? I also don't know if the script needs it there.

[jdduh]

Done!

[lbross]

I'm getting this error now when the Python script tries to publish the updated service. Any ideas?
https://support.esri.com/en-us/knowledge-base/error-user-cant-overwrite-this-service-using-this-data--000026287

It looks like some wires are still crossed. When I try to update the SD, it is finding this:
Found SD: stations_FCST_ACTIVE_polygons_firestats, ID: 0faecfd92e0641afbcf93508c645339d n Uploading and overwriting…

It doesn't work because the names don't match. I'm searching for stations_FCST_ACTIVE. Maybe I need to modify my search? Or are some wires still crossed?

[jdduh]

I will remove the firestats services. I think the SDs are all corrupted/crossed. Here is the URL for the stations_FCST_ACTIVE, based on the ESRI page that you shared.

https://services.arcgis.com/SXbDpmb7xQkk44JV/arcgis/rest/services/stations_FCST_ACTIVE/FeatureServer

[jdduh]

I will try not to use stations_FCST_ACTIVE as the prefix to any other feature layer services. Maybe AGOL is confused by the long, identical prefix. Please give it a try. I removed the firestats feature layer service.

[lbross]

It looks like that fixed the problem. Here is an article from ESRI on their fuzzy search and how the api differs from the interactive search on AGOL. https://developers.arcgis.com/python/latest/guide/accessing-and-creating-content/. I tried to see if I could query by id, which would always be unique, but couldn't get it to work.
If we really need to use that prefix for a feature layer service, it looks like there are options to examine the returned collection and be more precise. But for now things are good. Thanks for your help with this.