TADA_CreateParamRef() and TADA_CreateParamUseRef() #555

wokenny13 · 2024-12-13T22:49:25Z

First 2 reference files draft functions have been pushed through to test.

I am still working on the Mod 3 vignettes, however the R document contains a detailed explanation of what the two functions' goals are, and can be reviewed in the meantime while I continue to work on the Mod 3 vignettes.

Check to see if using argument input, excel = TRUE, creates the myfileRef spreadsheet in your downloads folder path.
Test out different argument inputs and test on other datasets.
Test out any warning/error messages from running the functions due to any invalid inputs to ensure no additional bugs.
Test out the general usability and user interface of the 2 functions.

First 2 reference files draft functions have been pushed through.

wokenny13 · 2024-12-16T14:13:52Z

working on addressing some check issues that were found

TADA_CreateParamRef and TADA_CreateParamUseRef .RD files

removal of some intermediate objects

Update - only print df if there are failing URLs

documentation suggestions

hillarymarler

Comments about requested changes are in-line or discussed in our working session call.

hillarymarler · 2024-12-16T19:29:18Z

R/CriteriaInputs.R

+  if (n > 100) {
+    message(paste0("There are ", n, " unique TADA.ComparableDataIdentifier names in your TADA data frame.
+    This may result in slow runtime for TADA_CreateParamRef() if you are generating an excel spreadsheet.
+    Consider filtering your .data TADA dataframe to a smaller subset of TADA.ComparableDataIdentifier first."))


Might belong in documentation instead of in this message, but maybe some suggestions on how to create smaller subset of TADA.ComparableDataIdentifier would be helpful for users. For example, suggest working through characteristicType groups when setting up these ref files for the first time.

hillarymarler · 2024-12-16T19:44:16Z

R/CriteriaInputs.R

+#' with the corresponding unique list of TADA/WQP Characteristic Names/TADA.ComparableDataIdentifier. 
+#' 
+#' If an ATTAINS parameter name is not listed as a prior domain value for your org from prior 
+#' ATTAINS assessment cycles, users should consider contacting the ATTAINS team to add this to the domain list. 


This is not correct. The ATTAINS team can't add a parameter to an org. Only the org can do that by associating that parameter with a particular assessment unit/designated use. Any org can use any parameter in ATTAINS (even if it was requested by a different org). Users should only contact the ATTAINS team to add a parameter name if there is not an acceptable/appropriate parameter name in ATTAINS.

As far as ATTAINS is concerned, there is no issue with an org using any ATTAINS-accepted parameter for the first time. So the ATTAINS team does not need to do anything in this situation.

hillarymarler · 2024-12-16T19:49:11Z

R/CriteriaInputs.R

+#' Otherwise, users can still proceed by overriding the data validation by value pasting. 
+#' Users will be warned in the ATTAINS.FlagParameterName column if they choose to include an
+#' ATTAINS.ParameterName that was not named in prior ATTAINS assessment cycles as: 
+#' 'Suspect: parameter name is not found as a prior parameter name for this organization'


I think we should have a way to flag differentiate between "This is an accepted ATTAINS parameter, that has not been previously used by this organization" and "This is not an accepted ATTAINS parameter, contact the ATTAINS team to inquire about adding it" as from an ATTAINS perspective these are two very different scenarios.

Is there still an option for the validation in Excel to allow ALL ATTAINS parameters, not just the ones from a specific org? This may be an important option for users, especially for orgs (tribes? territories?) that in the process of growing their assessment programs and may not yet have many assessments completed. In these cases, they might rather be able to access the whole possible list of parameters from ATTAINS rather than be limited to the handful that may have been listed as causes for a small number of assessments.

They may also find that newer parameter names may be more appropriate than ones that have been used in the past if their have been changes to assessment methodologies or more specific parameter names have been added (for example: specific aquatic plant species vs. more general plant parameter or a specific cyanotoxin vs. a more general algal parameter). So while I think using previous causes as a starting point for the parameter crosswalk is helpful and makes the process less overwhelming, we shouldn't make it too difficult for users to apply other ATTAINS parameter names to their crosswalks if they want or need to.

hillarymarler · 2024-12-16T20:08:02Z

R/CriteriaInputs.R

+#' 
+#' If an ATTAINS parameter name is not listed as a prior domain value for your org from prior 
+#' ATTAINS assessment cycles, users should consider contacting the ATTAINS team to add this to the domain list. 
+#' Otherwise, users can still proceed by overriding the data validation by value pasting. 


If the only way to override the data validation is by pasting (not just typing in a different entry) could we provide a sheet in the Excel file that lists all the current ATTAINS parameter names (without filtering them by organization)?

hillarymarler · 2024-12-16T20:36:23Z

R/CriteriaInputs.R

+#' if there is trouble locating the file. The file will be named "myfileRef.xlsx". 
+#' The excel spreadsheet will highlight the cells in which users should input information. Users may need to
+#' insert additional rows to capture certain ATTAINS.ParameterName that correspond with multiple TADA.CharacteristicName.
+#' Example: If your organization defines pH as "pH, High" and "pH, Low" that correspond to the TADA.CharacteristicName 'PH'.


I think this is an example of the opposite scenario than initially described as this is multiple ATTAINS parameter names to one TADA.CharacteristicName

hillarymarler · 2024-12-17T14:55:10Z

R/CriteriaInputs.R

+#' # After running TADA_CreateParamRef() users will provide this as a function input. First example is only for the org Utah
+#' paramUseRef_UT <- TADA_CreateParamUseRef(Data_Nutrients_UT, paramRef = paramRef_UT3, org_names = c("Utah"), excel = FALSE)
+#'
+#' # Users can include the EPA304a standards by itself or compared to their org(s)


I like this example to show both org/304a and just 304a - really useful

hillarymarler · 2024-12-17T14:55:52Z

R/CriteriaInputs.R

+  }
+
+  if (is.null(org_names)) {
+    print("No organization name provided, users should provide a list of ATTAINS domain organization state or tribal name that pertains to their dataframe. Attempting to pull in organization names found in the TADA data frame.")


Inconsistencies in printed message format. List function name first or not?

hillarymarler · 2024-12-17T14:56:51Z

R/CriteriaInputs.R

+  }
+
+  if (sum(!is.na(paramRef$ATTAINS.ParameterName)) == 0) {
+    stop("No values were found in ATTAINS.ParameterName. Please ensure that you have inputted all field values of interest in the ATTAINS.ParameterName column generated from TADA_CreateParamRef() function")


State more explicitly that the function cannot continue without some ATTAINS.ParameterName inputs?

I might want to change this to a warning. I am considering the case in which a user is only interested in the EPA304a standards. In this scenario, there is not a need for a user to provide a crosswalk between ATTAINS.ParameterName and TADA.ComparableDataIdentifier as it will only leverage the EPA304aPollutantName.

hillarymarler · 2024-12-17T14:58:55Z

R/CriteriaInputs.R

+  }
+
+  if (sum(is.na(paramRef$ATTAINS.ParameterName)) > 1) {
+    print("NAs were found in ATTAINS.ParameterName. Please ensure that you have inputted all field values of interest in the ATTAINS.ParameterName column generated from TADA_CreateParamRef() function")


In addition to NA could a "No corresponding ATTAINS.ParameterName" option be added to the ATTAINS.ParameterName drop-down list? Then in situations where a user has thoroughly reviewed the param list, they would not get this message if they had reviewed and provided some input for each entry in the ATTAINS.ParameterName field

hillarymarler · 2024-12-17T15:00:14Z

R/CriteriaInputs.R

+  }
+
+  if (sum(is.na(paramRef$EPA304A.PollutantName)) > 1 && org_names == "EPA304a") {
+    print("NAs were found in EPA304A.PollutantName. Please ensure that you have inputted all field values of interest in the EPA304A.PollutantName column generated from TADA_CreateParamRef() function if you are interested in using the 304a recommended standards")


Do users input this? Or is it done automatically? or a combination? Might be worth clarifying in the message and documentation that not all 304a pollutant names can be automatically crosswalked, so user review/input is required.

This is a good question. I had these lines included in past development to consider allowing users to also make edits to the EPA304a.PollutantName. But was unsure if we wanted to give this option to allow these values to be changed?

Making it clear that 'not all 304a pollutant names can be automatically crosswalked, so user review/input is required' will be nice to include if we want to proceed with allowing users to make further edits to the efforts we made with this crosswalk if an org feels one is more appropriate.

Added a tab for org_name filtered paramter and use names from ATTAINS Modified return values of ATTAINS.FlagParameterName

wokenny13 · 2024-12-19T01:29:16Z

@cristinamullin @hillarymarler There are still some edits and formatting I would like to make, but I think this is in a good spot to review now. I will be out of the office tomorrow but will be back on Friday the 20th.

hillarymarler · 2024-12-23T14:04:43Z

R/TADARefTables.R

+#' TADA_CreateParamUseRef() as the basis for the pulling in prior ATTAINS
+#' parameter names and use name by organization name. This helps to filter
+#' selections of drop down values and creating parameter and use combination
+#' summaries that will need to be defined.


I think you could remove "that will need to be defined"

hillarymarler · 2024-12-23T14:06:08Z

R/TADARefTables.R

+#' ATTAINS Parameter and Use Name by Organization Reference Key
+#'
+#' Function downloads and returns the newest available ATTAINS domain values
+#' reference dataframe summarized by parameter, use and organization.


Add clarifying language about parameters (listed as cause by org in previous assessments)

hillarymarler · 2024-12-23T14:06:42Z

R/TADARefTables.R

+#' TADA_CreateParamUseRef() as the basis for the pulling in EPA304a recommended
+#' pollutant name and use_name for assessment under the CWA.
+#'
+#' (Currently only numeric priority characteristic in TADA are the focus.)


Add info on where to find list of TADA priority chars?

…/USEPA/EPATADA into Create-Module-3-Reference-Tables

Cleaned the updating process for this ref table. Will need to make further edits to reflect changes throughout .R function dependencies.

Reflects some updates with the CST ref table. CreateParamUseRef() now contains a column for Include or Exclude.

Added a column to the CST file.

Provided additional details and formatting fixes.

Mod 3 notes, additional comments, and a minor fix in removing duplicate rows for dataframe output in CreateParamRef()

CreateAUIDRef() is in development and was not meant to be included in the prior commit

TADA_CreateParamRef() and TADA_CreateParamUseRef()

33429a3

First 2 reference files draft functions have been pushed through.

wokenny13 requested review from cristinamullin and hillarymarler December 13, 2024 22:49

wokenny13 and others added 2 commits December 13, 2024 17:53

pushed devtool document()

7e5c6dd

Merge branch 'develop' into Create-Module-3-Reference-Tables

785e4c7

hillarymarler and others added 10 commits December 16, 2024 09:28

Update WORDLIST

359d20e

Update R documentation for errors

df513ad

TADA_CreateParamRef and TADA_CreateParamUseRef .RD files

updates

2816e3e

arg input name misspell fix and removal of intermediate objects

c9d4ad2

removal of some intermediate objects

Update CriteriaInputs.R

67da081

Update CriteriaInputs.R

fb47bfd

Changed order of creating ATTAINS_param_all variable

c513fb7

Update test-URLChecker.R

75be2ec

Update - only print df if there are failing URLs

Update CriteriaInputs.R

8514702

Update CriteriaInputs.R

797230f

documentation suggestions

hillarymarler requested changes Dec 17, 2024

View reviewed changes

wokenny13 added 2 commits December 17, 2024 16:20

Updated ATTAINS.FlagParameterName

b328058

Added a tab for org_name filtered paramter and use names from ATTAINS Modified return values of ATTAINS.FlagParameterName

Mod 3 vignette testing and updates

14a7f8c

wokenny13 added 2 commits December 20, 2024 11:34

Update documentation and excel formula logic

5359692

Speed up loops within both functions and additional edits

ab32395

hillarymarler reviewed Dec 23, 2024

View reviewed changes

wokenny13 and others added 5 commits December 23, 2024 16:43

Additional comments and further updates

4461bfa

Addressed some ref documentation

44f9246

Merge branch 'develop' into Create-Module-3-Reference-Tables

38912fd

run spelling check and update wordlist

45df19d

Merge branch 'Create-Module-3-Reference-Tables' of https://github.com…

22c0f68

…/USEPA/EPATADA into Create-Module-3-Reference-Tables

wokenny13 and others added 12 commits December 24, 2024 17:33

Update CST EPA 304a Ref table

7b69b29

Cleaned the updating process for this ref table. Will need to make further edits to reflect changes throughout .R function dependencies.

CreateParamUseRef() and CST ref

e6b7357

Reflects some updates with the CST ref table. CreateParamUseRef() now contains a column for Include or Exclude.

further updates to reflect dependencies on CST ref file changes

661a63e

Added a column to the CST file.

Format updates, text updates, mod 3 vignette updates

6cc04b1

Provided additional details and formatting fixes.

Merge branch 'develop' into Create-Module-3-Reference-Tables

b4c3a71

Update CriteriaInputs.R

98c22b9

update mod 3 vignette

17f59e7

mod 3 vignette update

20e0ddc

global variables message fix

bb694df

Updates, global variable message handling, and text fixes

42c06f3

Notes and minor fixes

8970ec5

Mod 3 notes, additional comments, and a minor fix in removing duplicate rows for dataframe output in CreateParamRef()

Removed function CreateAUIDRef()

7396485

CreateAUIDRef() is in development and was not meant to be included in the prior commit

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

TADA_CreateParamRef() and TADA_CreateParamUseRef() #555

TADA_CreateParamRef() and TADA_CreateParamUseRef() #555

wokenny13 commented Dec 13, 2024

wokenny13 commented Dec 16, 2024

hillarymarler left a comment

hillarymarler Dec 16, 2024

hillarymarler Dec 16, 2024

hillarymarler Dec 16, 2024

hillarymarler Dec 16, 2024

hillarymarler Dec 16, 2024

hillarymarler Dec 17, 2024

hillarymarler Dec 17, 2024

hillarymarler Dec 17, 2024

wokenny13 Dec 17, 2024

hillarymarler Dec 17, 2024

hillarymarler Dec 17, 2024

wokenny13 Dec 17, 2024

wokenny13 commented Dec 19, 2024

hillarymarler Dec 23, 2024

hillarymarler Dec 23, 2024

hillarymarler Dec 23, 2024

TADA_CreateParamRef() and TADA_CreateParamUseRef() #555

Are you sure you want to change the base?

TADA_CreateParamRef() and TADA_CreateParamUseRef() #555

Conversation

wokenny13 commented Dec 13, 2024

wokenny13 commented Dec 16, 2024

hillarymarler left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

wokenny13 commented Dec 19, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment