WEBVTT 199 00:00:00.000 --> 00:00:03.934 Our first speaker today will be Felicia who 200 00:00:03.934 --> 00:00:05.934 is coming from EPA's office of research and development, and Felicia has 201 00:00:05.934 --> 00:00:09.000 been on the previous seminars in the series, but I will remind everybody 202 00:00:09.000 --> 00:00:11.000 that she is the director of the characterization support Center, 203 00:00:11.000 --> 00:00:15.000 which is responsible for investigations in the 204 00:00:15.000 --> 00:00:18.000 development and maintenance of ProUCL. 205 00:00:18.000 --> 00:00:22.000 Street is the reason pours superfine and technology liaison for the office 206 00:00:22.000 --> 00:00:26.000 of research and development. Felicia joined EPA back 207 00:00:26.000 --> 00:00:31.934 in 1985. She has also worked as an RPM and 208 00:00:31.934 --> 00:00:36.934 has over 20 years here at EPA OIG. We will move over to our 209 00:00:36.934 --> 00:00:41.934 primary speakers, Travis from Neptune company as well as Carson, also 210 00:00:41.934 --> 00:00:46.934 from Neptune. Travis is a graduate from 211 00:00:46.934 --> 00:00:50.934 the University of Colorado, and he basically has a 212 00:00:50.934 --> 00:00:56.934 Masters in statistics and have strong experience with working with statistical 213 00:00:56.934 --> 00:01:01.934 methods. This will tieback to his 214 00:01:01.934 --> 00:01:05.934 resource economics and has given him a lot of opportunities to work 215 00:01:05.934 --> 00:01:10.000 with us to to models. Carson has 25 years of 216 00:01:10.000 --> 00:01:15.000 experience working on multidisciplinary teams in 217 00:01:15.000 --> 00:01:20.000 environmental finance. For research was on the evaluation of sampling 218 00:01:20.000 --> 00:01:25.000 design for monitoring for soil using 219 00:01:25.000 --> 00:01:28.000 composite samples. So with those brief introduction, let me call 220 00:01:28.000 --> 00:01:29.934 up our opening materials and invite Felicia to join us for some kickoff 221 00:01:29.934 --> 00:01:32.934 remarks 222 00:01:32.934 --> 00:01:37.934 >> Thank you. I want to welcome everybody to this 223 00:01:37.934 --> 00:01:43.934 series of seminars in the utilization of ProUCL. Many 224 00:01:43.934 --> 00:01:46.934 may know, my name is Felicia, and I am the director 225 00:01:46.934 --> 00:01:54.934 of EPA's office of research and development site 226 00:01:54.934 --> 00:02:00.934 support center. We are responsible for creating and maintaining 227 00:02:00.934 --> 00:02:04.934 ProUCL. ProUCL is a statistical software package 228 00:02:04.934 --> 00:02:08.000 tool free on the website for the analysis of environmental data sets 229 00:02:08.000 --> 00:02:13.000 with and without observations. It provides statistical method and 230 00:02:13.000 --> 00:02:19.000 graphical tools to address many environmental 231 00:02:19.000 --> 00:02:22.000 sampling and statistical issues. It is not intended 232 00:02:22.000 --> 00:02:24.000 to address every situation, nor is it the only option for them by 233 00:02:24.000 --> 00:02:29.067 mental statistics. It was developed to be easy for many issues encountered 234 00:02:29.067 --> 00:02:32.000 when evaluating data. 235 00:02:32.000 --> 00:02:36.000 This third seminar will focus on using ProUCL for background analysis. 236 00:02:36.000 --> 00:02:40.000 The worst and second seminars in the series 237 00:02:40.000 --> 00:02:43.000 are already available for viewing online. If you were not able to 238 00:02:43.000 --> 00:02:49.000 attend or want to use specific parts your own phase, we hope you'll find 239 00:02:49.000 --> 00:02:51.000 this to be a useful tool and the seminars assist 240 00:02:51.000 --> 00:02:56.000 human capabilities and navigating the program. As previously noted, 241 00:02:56.000 --> 00:03:00.000 we will be responding to all questions asked here during the session or 242 00:03:00.000 --> 00:03:05.000 after documenting those answers. Additionally, 243 00:03:05.000 --> 00:03:11.067 sometimes limited external responses to those who use it. If one 244 00:03:11.067 --> 00:03:16.067 has questions or issues, we do try to 245 00:03:16.067 --> 00:03:19.067 answer those as much as we possibly can. On the website, you will find 246 00:03:19.067 --> 00:03:20.067 my name and contact information. 247 00:03:20.067 --> 00:03:25.067 I recommend contacting me via email with as much detail 248 00:03:25.067 --> 00:03:31.000 as possible. We will try to respond to your questions as promptly as 249 00:03:31.000 --> 00:03:37.000 we can. I will turn the seminar over to Travis 250 00:03:37.000 --> 00:03:41.000 with Neptune to start the instruction on background 251 00:03:41.000 --> 00:03:46.000 level calculations. 252 00:03:46.000 --> 00:03:51.000 >> Thanks Jean and Felicia. Like they mentioned, welcome back to 253 00:03:51.000 --> 00:03:54.000 our third ProUCL utilization 2020 presentation, 254 00:03:54.000 --> 00:03:59.000 which is fun to say three times fast. Today we are going 255 00:03:59.000 --> 00:04:05.000 to talk about that ground threshold values as well as finally talking 256 00:04:05.000 --> 00:04:14.067 about UCL. Everybody's favorite topic. So as 257 00:04:14.067 --> 00:04:17.067 we are leading into some fairly heavy statistical 258 00:04:17.067 --> 00:04:19.067 things, I want to make sure we take a second to recap what we talked 259 00:04:19.067 --> 00:04:21.067 about an hour first couple presentations. 260 00:04:21.067 --> 00:04:24.067 We are going to be using some of that in again today. Our first presentation, 261 00:04:24.067 --> 00:04:29.067 we talked about getting familiar with 262 00:04:29.067 --> 00:04:32.000 the basics of ProUCL as a whole, as well as starting to look at 263 00:04:32.000 --> 00:04:38.000 some functionality that might not have been known to veterans, such 264 00:04:38.000 --> 00:04:43.000 as loading grahams and other tools, as well as how 265 00:04:43.000 --> 00:04:49.000 to handle the Acer 266 00:04:49.000 --> 00:04:52.000 of hypothesis testing, which is not going to be touch on today, 267 00:04:52.000 --> 00:04:55.000 but it will be referenced and we need to be doing that and a lot 268 00:04:55.000 --> 00:04:57.000 of situations. We also discussed some of the ways that ProUCL handles 269 00:04:57.000 --> 00:05:03.000 nondetectable and missing values, which will be coming up 270 00:05:03.000 --> 00:05:05.000 again today, and additionally, we touched 271 00:05:05.000 --> 00:05:08.067 on outliers and their treatments, which is another thing that we are 272 00:05:08.067 --> 00:05:12.067 going to talk about as some of our estimates can be really sensitive 273 00:05:12.067 --> 00:05:16.067 to those outliers. So 274 00:05:16.067 --> 00:05:21.067 in our second presentation, we talked about trend analysis. As Jean reminded 275 00:05:21.067 --> 00:05:25.067 us earlier today, we discussed nonparametric methods 276 00:05:25.067 --> 00:05:29.067 of testing as well as parametric ones, 277 00:05:29.067 --> 00:05:35.000 namely, that was OLS regression and subsequent 278 00:05:35.000 --> 00:05:40.000 residual analysis to check the assumptions of that OLS regression. We also 279 00:05:40.000 --> 00:05:44.000 touched on why not to text can be so tricky to deal with 280 00:05:44.000 --> 00:05:46.000 in certain trend analysis situations. 281 00:05:46.000 --> 00:05:51.000 We will be touching on that a bit more today as well. So let's go 282 00:05:51.000 --> 00:05:56.000 ahead and dig on him, so as I mentioned today, we 283 00:05:56.000 --> 00:06:01.000 will be focused on background threshold values, but we will go ahead and 284 00:06:01.000 --> 00:06:06.067 start off with vocabulary and definitions, followed up by 285 00:06:06.067 --> 00:06:11.067 some discussions about the different background threshold 286 00:06:11.067 --> 00:06:15.067 value options that we have at our disposal. And then finally, 287 00:06:15.067 --> 00:06:20.067 we will get to talk about UCL's 288 00:06:20.067 --> 00:06:24.067 limits, and covering certain situations in which users should maybe feel 289 00:06:24.067 --> 00:06:28.067 a little apprehensive about blindly trusting certain outputs 290 00:06:28.067 --> 00:06:33.000 without at least 291 00:06:33.000 --> 00:06:36.000 a little bit of training. And so just like our last presentation, 292 00:06:36.000 --> 00:06:42.000 as Jean mentioned, we will be taking a couple knowledge check 293 00:06:42.000 --> 00:06:46.000 breaks, and time to answer some questions in the middle, 294 00:06:46.000 --> 00:06:50.000 and at the end of the presentation, and also like 295 00:06:50.000 --> 00:06:53.000 the last ones, I know we have not sent them out yet, but we will be 296 00:06:53.000 --> 00:06:56.000 getting a full set of answers to questions that folks asked that 297 00:06:56.000 --> 00:07:02.000 we do not have time to get to within the next couple 298 00:07:02.000 --> 00:07:05.000 weeks here. Cool. Just as a reminder for this presentation, 299 00:07:05.000 --> 00:07:08.067 a lot of the material that I am going to be bringing to you is actually 300 00:07:08.067 --> 00:07:12.067 located in the ProUCL user and tech guide. Those are going to be 301 00:07:12.067 --> 00:07:17.067 included with your ProUCL download, 302 00:07:17.067 --> 00:07:19.067 which is awesome. The technical guide especially is going to have 303 00:07:19.067 --> 00:07:24.067 a lot more detailed information on some of the 304 00:07:24.067 --> 00:07:29.067 statistical guidance that we cannot really get into today just because 305 00:07:29.067 --> 00:07:34.000 of time constraints. And the user guide 306 00:07:34.000 --> 00:07:38.000 is going to be great for more of 307 00:07:38.000 --> 00:07:42.000 how do I go about doing this even if I do not care about the nitty-gritty 308 00:07:42.000 --> 00:07:46.000 of each of the test? But I am going to be trying to hit the highlights 309 00:07:46.000 --> 00:07:49.000 as we go through today. 310 00:07:49.000 --> 00:07:55.000 And so Jean, if I hand screen sharing capabilities. 311 00:07:55.000 --> 00:07:59.000 >> Travis pulled up the screen share, I will 312 00:07:59.000 --> 00:08:03.000 remind everyone that you will have additional controls at the top of 313 00:08:03.000 --> 00:08:06.000 those windows to either maximize his screen, the slides, or to change 314 00:08:06.000 --> 00:08:12.000 how you are viewing his slide within that particular window. 315 00:08:12.000 --> 00:08:16.000 >> Cool. So just like last time, I will be 316 00:08:16.000 --> 00:08:21.000 leaving the standard set up here with the panel on. Just 317 00:08:21.000 --> 00:08:27.000 so that I make sure I am not making any silly mistakes as we go 318 00:08:27.000 --> 00:08:32.934 through. And so let's go ahead and just get our data loaded 319 00:08:32.934 --> 00:08:35.934 first. So the first one we will be looking at is going to 320 00:08:35.934 --> 00:08:39.934 be the Superfund data set, that was included in your ProUCL 321 00:08:39.934 --> 00:08:44.934 download. The data set is going to get us familiar with some of 322 00:08:44.934 --> 00:08:47.934 the different distributional options. 323 00:08:47.934 --> 00:08:53.934 For you tales and whatnot, but it does not have 324 00:08:53.934 --> 00:08:57.934 any non-detects in it, so we are going to get to those. So let's 325 00:08:57.934 --> 00:09:03.934 go ahead and pull that up. And then so this is going to be 326 00:09:03.934 --> 00:09:09.000 inside of where your folder is, and inside of data there. We are 327 00:09:09.000 --> 00:09:16.000 going to click Superfund. 328 00:09:16.000 --> 00:09:23.000 There we go. And so we we are going to see here 329 00:09:23.000 --> 00:09:29.934 is in this dataset, we have got a bunch of different analytes 330 00:09:29.934 --> 00:09:32.934 with about 24 different observations for each of them. Again, note non-detects, 331 00:09:32.934 --> 00:09:35.934 but it is nice because our sample size is pretty solid for everything 332 00:09:35.934 --> 00:09:37.934 we are going to be looking at today. 333 00:09:37.934 --> 00:09:43.934 So then the second one we are going to be playing with is that 334 00:09:43.934 --> 00:09:48.934 blank status file, which is again in the download, 335 00:09:48.934 --> 00:09:53.934 and we are going to be using this one really just as an example that 336 00:09:53.934 --> 00:09:58.934 includes non-detects for both our background threshold values 337 00:09:58.934 --> 00:10:04.934 and UCL's today. We will go ahead and open up that 338 00:10:04.934 --> 00:10:14.000 as well. There we go. 339 00:10:14.000 --> 00:10:17.000 So as we see here, visited is that is going to be about as small 340 00:10:17.000 --> 00:10:23.000 as we can get for some of our sample size restrictions. As there is 12 341 00:10:23.000 --> 00:10:27.000 total observations, I know there are 14 rows, but we have 342 00:10:27.000 --> 00:10:32.934 two missing, so 12. and there is only eight actual non-detects, so 343 00:10:32.934 --> 00:10:38.934 we are limited on sample size there. 344 00:10:38.934 --> 00:10:41.934 And so coming back away from the screen 345 00:10:41.934 --> 00:10:44.934 sharing right now, we can keep showing it, but nothing will be happening 346 00:10:44.934 --> 00:10:47.934 for a minute, first we are going to discuss some of the different 347 00:10:47.934 --> 00:10:53.934 background threshold value options that we have available 348 00:10:53.934 --> 00:10:56.934 to us, and as we get into that, I wanted to make sure everybody 349 00:10:56.934 --> 00:10:59.934 is on the same page with some definitions. 350 00:10:59.934 --> 00:11:02.934 It you are super, it will be a lot of review, 351 00:11:02.934 --> 00:11:08.000 but it is important stuff so that people are quoting stats that 352 00:11:08.000 --> 00:11:12.000 make sense. So namely, we want to make sure that everybody understands 353 00:11:12.000 --> 00:11:17.000 the definitions for confidence as well 354 00:11:17.000 --> 00:11:23.000 as coverage when it comes to our limits. So first for 355 00:11:23.000 --> 00:11:28.000 confidence, as we can see on this slide is that we are referring 356 00:11:28.000 --> 00:11:35.934 to our beliefs that whatever level of competence percent of 357 00:11:35.934 --> 00:11:39.934 the time, something that we are interested in is going to happen, 358 00:11:39.934 --> 00:11:45.934 so for ample, 95% confidence, as I am sure 359 00:11:45.934 --> 00:11:51.934 everybody remembers, saying that 95% confidence interval is 360 00:11:51.934 --> 00:11:54.934 saying that if we took a series of samples and generated confidence 361 00:11:54.934 --> 00:11:59.934 intervals up teen times, we believe the population mean would be within 362 00:11:59.934 --> 00:12:05.934 that generated confidence interval, 95% of 363 00:12:05.934 --> 00:12:11.000 the time. That is given that all those samples 364 00:12:11.000 --> 00:12:16.000 were coming from the same 365 00:12:16.000 --> 00:12:18.000 distribution. That is going to give us that rough estimate of where 366 00:12:18.000 --> 00:12:21.000 we believe our population mean lives. 367 00:12:21.000 --> 00:12:25.000 However, it is not directly just saying the probability of the actual 368 00:12:25.000 --> 00:12:30.000 population mean is between these points, 95% of the time with 369 00:12:30.000 --> 00:12:35.000 an a frequent frame. Is some wording in there 370 00:12:35.000 --> 00:12:42.000 that is kind of annoying. So our next definition is going to 371 00:12:42.000 --> 00:12:46.000 be of coverage. That can be thought of as the percent of 372 00:12:46.000 --> 00:12:52.000 future samples that should fall below a given 373 00:12:52.000 --> 00:12:56.000 percentile, so if we are looking at a coverage of.5 374 00:12:56.000 --> 00:12:59.000 or 50. we are expecting to see half of our future samples all below 375 00:12:59.000 --> 00:13:03.000 whatever estimate it is that we are making. So if we combine 376 00:13:03.000 --> 00:13:09.067 those concepts, we are going to have what we need to start 377 00:13:09.067 --> 00:13:13.067 looking at something like a 95 you tell, where what we are saying 378 00:13:13.067 --> 00:13:17.067 is that we are building a confidence interval around the 95th 379 00:13:17.067 --> 00:13:21.067 percentile of given future data sets, and saying 380 00:13:21.067 --> 00:13:26.067 that we believe 95% of the time that the 95th percentile will fall 381 00:13:26.067 --> 00:13:33.000 below our 95 UTL. If anyone still has 382 00:13:33.000 --> 00:13:38.000 questions on that verbiage or that definition, 383 00:13:38.000 --> 00:13:43.000 feel free to stare a bit more at the definitions in the tech guide 384 00:13:43.000 --> 00:13:47.000 they are there. Or feel free to 385 00:13:47.000 --> 00:13:50.000 ask anything specific in the Q&A box and we will try and get to that 386 00:13:50.000 --> 00:13:54.000 or you know, as always, [ Indiscernible ] with those definitions in mind, 387 00:13:54.000 --> 00:13:59.000 let's look at the four different background threshold 388 00:13:59.000 --> 00:14:04.000 value options available to us. And when 389 00:14:04.000 --> 00:14:07.067 we might actually be using them. 390 00:14:07.067 --> 00:14:13.067 So first off, of these background values, not 391 00:14:13.067 --> 00:14:19.067 to exceed values, we are going to be best using them when 392 00:14:19.067 --> 00:14:24.067 we are really strong on it like, future site sample size. So 393 00:14:24.067 --> 00:14:28.067 to build these background threshold values, we want to 394 00:14:28.067 --> 00:14:32.000 have a minimum of about 10 samples in our background data set, but 395 00:14:32.000 --> 00:14:39.000 they should generally be only used 396 00:14:39.000 --> 00:14:42.000 to compare against a relatively small number of future site samples. 397 00:14:42.000 --> 00:14:46.000 So you know, like somewhere like 1-6. but as soon as you get anywhere 398 00:14:46.000 --> 00:14:50.000 up near 10 or so, as far as the future site samples, not the 399 00:14:50.000 --> 00:14:55.000 background wounds, we should instead be comparing them using some of 400 00:14:55.000 --> 00:14:58.000 the hypothesis testing methods that we discussed 401 00:14:58.000 --> 00:15:03.000 in our first presentation. So you know, 402 00:15:03.000 --> 00:15:07.067 those test, all that stuff is what we are going to want to be using 403 00:15:07.067 --> 00:15:12.067 when we had really adequate 404 00:15:12.067 --> 00:15:16.067 site sample data, but if we are hamstrung, we only got you know, 405 00:15:16.067 --> 00:15:19.067 five or six site temples, these not to exceed background 406 00:15:19.067 --> 00:15:25.067 threshold value markers are going to be 407 00:15:25.067 --> 00:15:26.067 useful. Cool. 408 00:15:26.067 --> 00:15:34.000 I just want to reiterate that, that like, it is for when you are 409 00:15:34.000 --> 00:15:37.000 struggling on site samples, we are not trying to cut corners, just 410 00:15:37.000 --> 00:15:41.000 if you only have a few, these will be nice for creating not 411 00:15:41.000 --> 00:15:47.000 to exceed values, but the more samples you use, the more chance of a false 412 00:15:47.000 --> 00:15:51.000 positive that you are going to get just due to the nature 413 00:15:51.000 --> 00:15:55.000 of the statistics. So with that in mind, the first of our background 414 00:15:55.000 --> 00:15:58.000 threshold value options that we are going to be looking at is just 415 00:15:58.000 --> 00:16:01.000 using a high percentile of the data. 416 00:16:01.000 --> 00:16:05.000 So these are going to be a simple measure, just looking at 417 00:16:05.000 --> 00:16:12.067 how high or low and of value might fall. They are nice because 418 00:16:12.067 --> 00:16:16.067 they help ignore at least a little bit of the digital 419 00:16:16.067 --> 00:16:21.067 outliers, and all that jazz on either end of 420 00:16:21.067 --> 00:16:25.067 the scale. Whereas, a lot of our other statistics are going to be 421 00:16:25.067 --> 00:16:27.067 just exceptionally susceptible to those, not to say this is not, but 422 00:16:27.067 --> 00:16:30.000 just to a slightly lesser degree. 423 00:16:30.000 --> 00:16:33.000 So just as a note, if you will remember your 424 00:16:33.000 --> 00:16:37.000 first presentation when we look at hypothesis testing stuff as 425 00:16:37.000 --> 00:16:43.000 well, we can also be using these high percentiles as a 426 00:16:43.000 --> 00:16:47.000 potential point of comparison for you know, the 427 00:16:47.000 --> 00:16:51.000 percentile test or proportion test or anything within that framework. 428 00:16:51.000 --> 00:16:56.000 So next, we have got a UPL or 429 00:16:56.000 --> 00:16:59.000 just our upper production limit. 430 00:16:59.000 --> 00:17:05.000 This is going to be what we use to generate a level 431 00:17:05.000 --> 00:17:07.067 of competence to say for the next independent 432 00:17:07.067 --> 00:17:11.067 sample from our population, we expect that that value should be below 433 00:17:11.067 --> 00:17:17.067 this point. And so if we actually look at just a graph given 434 00:17:17.067 --> 00:17:22.067 from our last presentation, we will see that these are shown on 435 00:17:22.067 --> 00:17:28.067 some of the trend analysis graphs and per UCL 436 00:17:28.067 --> 00:17:31.000 as guidelines surrounding the regression, so obviously those 437 00:17:31.000 --> 00:17:36.000 are going to be wider generally than the standard 438 00:17:36.000 --> 00:17:41.000 confidence interval on mean, the green lines on there, as we are 439 00:17:41.000 --> 00:17:46.000 saying we are going a single sample as opposed 440 00:17:46.000 --> 00:17:49.000 to averaging over a series of values, like we would when we 441 00:17:49.000 --> 00:17:53.000 are interpreting standard confidence intervals. 442 00:17:53.000 --> 00:17:58.000 So basically on that graph, just at any given point, if you took 443 00:17:58.000 --> 00:18:02.000 a given sample, you expect it 95% of the time within those 444 00:18:02.000 --> 00:18:04.000 redlines. We took a bunch of examples, you would expect the 445 00:18:04.000 --> 00:18:10.000 mean between those redlines 95% of the time. Just an 446 00:18:10.000 --> 00:18:15.000 understanding check. So next up is going to be a pretty common 447 00:18:15.000 --> 00:18:20.000 one, that is going to be the upper tolerance limit, 448 00:18:20.000 --> 00:18:26.000 which can give us a solid not to exceed value 449 00:18:26.000 --> 00:18:29.934 for comparing site to background data sets. This 450 00:18:29.934 --> 00:18:32.934 one is going to be used as an example a couple times, and we talked about 451 00:18:32.934 --> 00:18:38.934 it a minute ago with that 95 you TL again, just saying we are 452 00:18:38.934 --> 00:18:43.934 95% certain that the 95th percentile of the data would fall below 95 453 00:18:43.934 --> 00:18:47.934 UTL value. It sometimes shows it as 454 00:18:47.934 --> 00:18:52.934 a not to exceed value as attend to be greater than the maximum 455 00:18:52.934 --> 00:18:57.934 of background data set, so it is going to give us a nice solid level. 456 00:18:57.934 --> 00:19:02.934 It is going to be fairly hard to get a false 457 00:19:02.934 --> 00:19:07.000 positive on. But this is again going to start being really sensitive 458 00:19:07.000 --> 00:19:13.000 to outliers in SKU in the background 459 00:19:13.000 --> 00:19:16.000 data, so just be aware. 460 00:19:16.000 --> 00:19:21.000 And finally, we have a bit 461 00:19:21.000 --> 00:19:24.000 of a niche that can be useful in these situations, this is going 462 00:19:24.000 --> 00:19:26.000 to be the upper simultaneous limit. 463 00:19:26.000 --> 00:19:30.934 This provides us with a value such that we believe all future samples 464 00:19:30.934 --> 00:19:33.934 will fall below it. 465 00:19:33.934 --> 00:19:35.934 95% of the time. 466 00:19:35.934 --> 00:19:38.934 So in essence, what we are going to be doing is making a 467 00:19:38.934 --> 00:19:43.934 slightly harder to violate version of that 95 you TL 468 00:19:43.934 --> 00:19:48.934 and sweet are basically just moving the coverage from 95% to 100% of 469 00:19:48.934 --> 00:19:54.934 the data. This method is going to be nice if you are in a funky 470 00:19:54.934 --> 00:20:01.934 situation where you have decided to use a not to exceed value, but 471 00:20:01.934 --> 00:20:07.000 you actually have more than that, like samples insight to compare 472 00:20:07.000 --> 00:20:11.000 to, and you have decided for some reason, hypothesis 473 00:20:11.000 --> 00:20:16.000 testing is not the way to go for your site. As it is a lot 474 00:20:16.000 --> 00:20:22.000 more resistant to that false positive rate. Going 475 00:20:22.000 --> 00:20:26.000 up as your number of site samples increases, and again, though, 476 00:20:26.000 --> 00:20:29.934 I want to stress that generally you are going to want to use hypothesis 477 00:20:29.934 --> 00:20:38.934 testing if your site sample size is large 478 00:20:38.934 --> 00:20:41.934 enough. And like I said, the reason this is 479 00:20:41.934 --> 00:20:43.934 going to work it decently though, even if you do have a little bit 480 00:20:43.934 --> 00:20:46.934 higher ever sample size, is that false positive rate does not go 481 00:20:46.934 --> 00:20:51.934 up nearly as much like it does with a UTL or what have you, 482 00:20:51.934 --> 00:20:58.934 which you know, it makes sense because with the UCL, if you take 483 00:20:58.934 --> 00:21:03.934 more and more samples, eventually, one is going to be 484 00:21:03.934 --> 00:21:08.000 above that found where you were bounding at at the 95th percentile 485 00:21:08.000 --> 00:21:12.000 of the data, whereas you know, with the USL, you are saying I never 486 00:21:12.000 --> 00:21:16.000 expect anything to ever be above this. Wanting to be careful with 487 00:21:16.000 --> 00:21:23.000 here, though, is that while all of these can 488 00:21:23.000 --> 00:21:27.000 be, like I said, sensitive to outliers, USL can be like, extremely sensitive. 489 00:21:27.000 --> 00:21:31.934 So with that in mind, there are specific 490 00:21:31.934 --> 00:21:35.934 use cases for it, but in order for them to be reliable, the data actually 491 00:21:35.934 --> 00:21:40.934 needs to be really quite tidy and free 492 00:21:40.934 --> 00:21:44.934 of outliers, which if we remember from our first presentation can 493 00:21:44.934 --> 00:21:48.934 only be done 494 00:21:48.934 --> 00:21:51.934 you know, if we are trying to remove outliers, if we have a very good 495 00:21:51.934 --> 00:21:55.934 reason to explain those outlier points do not really belong in the 496 00:21:55.934 --> 00:21:59.934 actual data population, and in addition to that, 497 00:21:59.934 --> 00:22:03.934 weirdly, actually having more than about 20-ish back 498 00:22:03.934 --> 00:22:13.000 rent observations can make this a bit unreliable as 499 00:22:13.000 --> 00:22:16.000 well. So if this happens to match up with your data, 500 00:22:16.000 --> 00:22:19.000 it could be a useful tool, but sometimes it is not going to be the right 501 00:22:19.000 --> 00:22:22.000 tool for the job. And again, not to beat a dead horse, but if you 502 00:22:22.000 --> 00:22:25.000 are concerned, not sure which one to use, not sure what that, make 503 00:22:25.000 --> 00:22:31.000 sure to consult a statistician because as for which 504 00:22:31.000 --> 00:22:33.000 of these background threshold values, or 505 00:22:33.000 --> 00:22:37.000 not to exceed options is going to be the most relevant for your site, 506 00:22:37.000 --> 00:22:38.000 I can't give you a definitive answer. 507 00:22:38.000 --> 00:22:43.000 You know, depending on site characteristics as well as the analytics you are 508 00:22:43.000 --> 00:22:45.000 looking for and whatever different things you are 509 00:22:45.000 --> 00:22:50.000 trying to be doing, different ones might be preferable in 510 00:22:50.000 --> 00:22:54.000 different situations. So let's go ahead and actually do's death and 511 00:22:54.000 --> 00:23:00.000 ProUCL. We are going to first create the background threshold 512 00:23:00.000 --> 00:23:05.000 values for the Superfund data. So we 513 00:23:05.000 --> 00:23:11.067 are going to get that select did, and so when we go up here 514 00:23:11.067 --> 00:23:14.067 to the upper limits, we are going to see that ProUCL 515 00:23:14.067 --> 00:23:19.067 is smart enough to just not display the option 516 00:23:19.067 --> 00:23:23.067 to use not tax since there are not any in a data set, but when we click 517 00:23:23.067 --> 00:23:27.067 down and do this on the TCE stop here, we can see that it 518 00:23:27.067 --> 00:23:33.000 does give us that choice. For right now, 519 00:23:33.000 --> 00:23:39.000 Superfund. So when we click all here, we are going to hit 520 00:23:39.000 --> 00:23:43.000 [ Indiscernible ] and we will see in the options, we have a couple 521 00:23:43.000 --> 00:23:47.000 different things we can take a look at. I will 522 00:23:47.000 --> 00:23:56.000 go ahead and leave the coverage values at 95%, just to keep with 523 00:23:56.000 --> 00:23:59.000 the standard here, as well as keeping our valued at one, so this value 524 00:23:59.000 --> 00:24:03.000 is going to be adjusted based on how many site samples we are comparing 525 00:24:03.000 --> 00:24:09.067 in a specific future data set. It can 526 00:24:09.067 --> 00:24:13.067 really inflate the UPL values especially as it increases. 527 00:24:13.067 --> 00:24:20.067 So for right now, we are going to stick with 528 00:24:20.067 --> 00:24:22.067 just comparing to one observation. 529 00:24:22.067 --> 00:24:23.067 So just so you know, it is all there. 530 00:24:23.067 --> 00:24:27.067 The bootstraps, we will leave that at 2000. A reasonable thing to do, 531 00:24:27.067 --> 00:24:30.000 and as far as the grouping column, it is there, but we do not have 532 00:24:30.000 --> 00:24:34.000 anything to be grouping by this time, so we will leave that blank. 533 00:24:34.000 --> 00:24:37.000 The first thing we are going to want to remember 534 00:24:37.000 --> 00:24:42.000 when we are looking at this here is we are treating our data 535 00:24:42.000 --> 00:24:48.000 as in this case, a set of background data, so what we are doing here 536 00:24:48.000 --> 00:24:53.000 is we will be generating these estimates compare some future set 537 00:24:53.000 --> 00:24:56.000 of data too, so we will want to check that our distributions in 538 00:24:56.000 --> 00:25:01.000 the data, sorry, we will want to check what distributions our data 539 00:25:01.000 --> 00:25:10.067 seems to be following in this, so I know 540 00:25:10.067 --> 00:25:12.067 we gave a bit more of a specific run-through on goodness 541 00:25:12.067 --> 00:25:15.067 of fit for our first presentation, but in the interest of time, and 542 00:25:15.067 --> 00:25:19.067 not saying the same thing over and over, we will just trust these recommendations 543 00:25:19.067 --> 00:25:24.067 that are presented by Pro UCL using some of these goodness of 544 00:25:24.067 --> 00:25:30.000 fit tests. So for our data, we will see that stuff, they 545 00:25:30.000 --> 00:25:35.000 don't appear normal, at 546 00:25:35.000 --> 00:25:38.000 the 95% confidence level, so we are going to go ahead and mix that 547 00:25:38.000 --> 00:25:44.000 option. But we will see that it appears to 548 00:25:44.000 --> 00:25:48.000 follow the lognormal distributions. So 549 00:25:48.000 --> 00:25:51.000 if you ever run into a case such as this, generally, you 550 00:25:51.000 --> 00:25:57.000 are going to want to rely on the gamma choices over the 551 00:25:57.000 --> 00:25:59.000 normal choices. Again, a general statement. Mainly because 552 00:25:59.000 --> 00:26:03.000 lognormal distributions have a tendency to produce 553 00:26:03.000 --> 00:26:09.067 rather unstable values as far as limit estimations go. I tend to 554 00:26:09.067 --> 00:26:15.067 over and under estimate reliable estimates depending on 555 00:26:15.067 --> 00:26:19.067 the situation, where as animals tend to be a bit 556 00:26:19.067 --> 00:26:23.067 less sensitive to the weird SKU or funkiness going on in the data, 557 00:26:23.067 --> 00:26:29.067 and we are seeing that nice positive nature. 558 00:26:29.067 --> 00:26:35.000 So if we go down here, we 559 00:26:35.000 --> 00:26:41.000 will see the nicely gathered up options, so we can 560 00:26:41.000 --> 00:26:46.000 see a few things, basically, the background shoot value options 561 00:26:46.000 --> 00:26:50.000 that are presented to you guys are going to be generally increasing 562 00:26:50.000 --> 00:26:56.000 in order of size. We are going to have our 563 00:26:56.000 --> 00:27:01.000 95th percentile right down here, our UTL 564 00:27:01.000 --> 00:27:07.067 and then our options, and then our USL 565 00:27:07.067 --> 00:27:12.067 as well. So just by the way, I will 566 00:27:12.067 --> 00:27:15.067 actually create a little box plot for us just so that we can kind 567 00:27:15.067 --> 00:27:21.067 of see what is going on in the data visually so that we can compare 568 00:27:21.067 --> 00:27:33.000 these nicely. Going to create that. There 569 00:27:33.000 --> 00:27:42.000 we go. 570 00:27:42.000 --> 00:27:46.000 All right. When we are looking at this, we can see first, 571 00:27:46.000 --> 00:27:50.000 there is definitely three values that 572 00:27:50.000 --> 00:27:56.000 are a little concerning on the outlier sense of things. 573 00:27:56.000 --> 00:28:03.000 But when we look at everything compared to 574 00:28:03.000 --> 00:28:07.000 our different options, we can see that the 95th percentile is going 575 00:28:07.000 --> 00:28:12.000 to be right below that first possible outlier, whereas, 576 00:28:12.000 --> 00:28:16.000 then our UTL is going to be just about that, not much different, 577 00:28:16.000 --> 00:28:20.000 but our UTL is going to be like way up 578 00:28:20.000 --> 00:28:26.000 here, so our UTL is going to be just 579 00:28:26.000 --> 00:28:29.934 below the very maps of the data, or maybe just about it if you are 580 00:28:29.934 --> 00:28:35.934 using the other option, and then the USL is going to be 581 00:28:35.934 --> 00:28:41.934 solidly above the max of the data set. So as you can see, 582 00:28:41.934 --> 00:28:43.934 again, like, it is going to matter that those are 583 00:28:43.934 --> 00:28:48.934 there, and we are not going to talk about it directly right now, but 584 00:28:48.934 --> 00:28:51.934 we will get to that later. 585 00:28:51.934 --> 00:28:54.934 Bum bum bum bum bum. Cool. 586 00:28:54.934 --> 00:28:58.934 So with that in mind, let's go ahead, and I just want to show 587 00:28:58.934 --> 00:29:04.934 you that at the bottom of your ProUCL output for background 588 00:29:04.934 --> 00:29:10.000 threshold values, it is going to be letting us know down here that 589 00:29:10.000 --> 00:29:13.000 the USL can tend to give us those rather 590 00:29:13.000 --> 00:29:19.000 high estimates of a background threshold value. Again, it is 591 00:29:19.000 --> 00:29:25.000 the value that we never expect to see another value 592 00:29:25.000 --> 00:29:29.934 above it in any future data. And that high estimate can definitely 593 00:29:29.934 --> 00:29:34.934 be exacerbated by having you know, possible outliers such as these 594 00:29:34.934 --> 00:29:37.934 three points or any possible contamination involvement 595 00:29:37.934 --> 00:29:42.934 in the background data 596 00:29:42.934 --> 00:29:46.934 set, or even starting with that, you know, background sample size 597 00:29:46.934 --> 00:29:49.934 exceeding roughly 20. as I mentioned. So you know, and just 598 00:29:49.934 --> 00:29:52.934 as a general point, and we have mentioned it before, but for most 599 00:29:52.934 --> 00:29:56.934 of these limits, we are going to want to have a background sample 600 00:29:56.934 --> 00:30:00.934 size of around 10 or so, so 601 00:30:00.934 --> 00:30:07.000 we are doing good here since we had about 24. Cool. 602 00:30:07.000 --> 00:30:10.000 So next, we are going to go through this 603 00:30:10.000 --> 00:30:12.000 same progression, but taking a look at the different options with non-detects 604 00:30:12.000 --> 00:30:18.000 included, so let's go ahead and flip back to our data here, 605 00:30:18.000 --> 00:30:22.000 and so again, this one is, you know, 606 00:30:22.000 --> 00:30:27.000 included in your downloads, easily available for you, but we will see 607 00:30:27.000 --> 00:30:30.934 that actually of 12 observations that we have, 608 00:30:30.934 --> 00:30:34.934 only eight are detects. We are going to be 609 00:30:34.934 --> 00:30:39.934 right on the cusp of acceptable sample size 610 00:30:39.934 --> 00:30:45.934 here, or just below it, but that is okay for right now. 611 00:30:45.934 --> 00:30:49.934 Also, we do see that we nicely have 612 00:30:49.934 --> 00:30:53.934 only one not detect level, so all these guys that are not attacked 613 00:30:53.934 --> 00:31:01.934 seem to only be a 614 00:31:01.934 --> 00:31:04.934 0.68 value, so that is nice. I think our estimates are going to be able 615 00:31:04.934 --> 00:31:07.000 to handle it a little bit better than if we had a whole bunch of 616 00:31:07.000 --> 00:31:11.000 weirdness going on. Just like I mentioned, obviously, these boxes 617 00:31:11.000 --> 00:31:16.000 here are just going to be treated as missing values and disregarded 618 00:31:16.000 --> 00:31:21.000 by ProUCL. So let's go ahead and 619 00:31:21.000 --> 00:31:26.000 run our background test as well as create that second 620 00:31:26.000 --> 00:31:30.934 box but just so that we can compare these in the same way that we did 621 00:31:30.934 --> 00:31:35.934 our Superfund data. Here we go with 622 00:31:35.934 --> 00:31:38.934 non-detects. 623 00:31:38.934 --> 00:31:44.934 And then off and then TCE and we are going to leave these 624 00:31:44.934 --> 00:31:48.934 options just the same as we had before. We will create 625 00:31:48.934 --> 00:31:54.934 that, and we will go back again, graph, box plot 626 00:31:54.934 --> 00:32:00.934 with non-detects, TCE and we are just going to adjust 627 00:32:00.934 --> 00:32:13.000 the options here again. 628 00:32:13.000 --> 00:32:19.000 Okay there we go. So when 629 00:32:19.000 --> 00:32:25.000 we take a look at this output here, we are going 630 00:32:25.000 --> 00:32:30.000 to see that first off, the output is telling 631 00:32:30.000 --> 00:32:34.000 us that it appears the data 632 00:32:34.000 --> 00:32:38.000 is approximately normal, so this is going to come about because as 633 00:32:38.000 --> 00:32:43.000 we can see one of the goodness of fit tests, it 634 00:32:43.000 --> 00:32:49.000 thinks the data is normal and the other one thinks it is 635 00:32:49.000 --> 00:32:50.000 not normal. 636 00:32:50.000 --> 00:32:54.000 We are going to definitely take these with a grain of salt because 637 00:32:54.000 --> 00:32:59.000 we only have that sample size of eight detected values come up but 638 00:32:59.000 --> 00:33:05.000 since I looked down here and the that both of the 639 00:33:05.000 --> 00:33:09.067 gamma goodness test think that it is gamma 640 00:33:09.067 --> 00:33:14.067 distributed, I am probably going to go ahead and generally lean this 641 00:33:14.067 --> 00:33:19.067 way a little bit easier. With that being said, 642 00:33:19.067 --> 00:33:25.067 we want to make sure and be careful with the gamma ROS generated values 643 00:33:25.067 --> 00:33:30.000 for our non-detects. 644 00:33:30.000 --> 00:33:32.000 Again, just due to the small sample size and a fairly high percentage 645 00:33:32.000 --> 00:33:36.000 of non-detects, and instead, you might want to look down a little 646 00:33:36.000 --> 00:33:42.000 further here, so these guys maybe not ideal, 647 00:33:42.000 --> 00:33:48.000 but instead, use some of these Kaplan-Meier generated values. And 648 00:33:48.000 --> 00:33:52.000 so again here, we are going to go ahead and see 649 00:33:52.000 --> 00:33:56.000 that generally increasing trend from percentile 650 00:33:56.000 --> 00:34:02.000 to UPL to 95-95 UPL, and USL, however, in this case, 651 00:34:02.000 --> 00:34:11.067 our USL is actually going to be a little bit lower than our 652 00:34:11.067 --> 00:34:14.067 UTL. My guess without getting super into the stats of it is this is 653 00:34:14.067 --> 00:34:19.067 an artifact heading into a really low sample size, and again, which 654 00:34:19.067 --> 00:34:23.067 one of these values is going to be right for 655 00:34:23.067 --> 00:34:28.067 your situation, it's something you're going to need to either seek guidance 656 00:34:28.067 --> 00:34:32.000 on for your site, or have discussions as to what might be the 657 00:34:32.000 --> 00:34:38.000 right case in your case by case basis. So just to touch back on 658 00:34:38.000 --> 00:34:43.000 that Meyer versus the gamma aggression on orders statistics 659 00:34:43.000 --> 00:34:49.000 stuff, if we take a comparison between our gamma ROS, and the 660 00:34:49.000 --> 00:34:52.000 Meyer imputed values for you know, background 661 00:34:52.000 --> 00:34:59.000 threshold values, it is going to seem like this is a little more 662 00:34:59.000 --> 00:35:04.000 reasonable, so like with our 95-95 UTO, Meyer has it at about you 663 00:35:04.000 --> 00:35:10.067 know, 11 1/2. maybe 12. something, which these are around 10 as 664 00:35:10.067 --> 00:35:14.067 a top with a possible outlier, so you know, just a little 665 00:35:14.067 --> 00:35:18.067 bit above the possible outliers that you know, 666 00:35:18.067 --> 00:35:21.067 in this case let's pretend they are totally related 667 00:35:21.067 --> 00:35:25.067 to the distribution and it is wider than we expect versus if we looked 668 00:35:25.067 --> 00:35:30.000 up here at this gamma 669 00:35:30.000 --> 00:35:34.000 ROS, it is going to be sending them out here at you know, 20 and 670 00:35:34.000 --> 00:35:38.000 28. which just seems rather unreasonable considering we have not seen anything 671 00:35:38.000 --> 00:35:44.000 even half that size. So you know, maybe not 672 00:35:44.000 --> 00:35:46.000 ideal. 673 00:35:46.000 --> 00:35:50.000 Cool. So I think that is going to cover what we are talking about 674 00:35:50.000 --> 00:35:55.000 as far as threshold values, and that will take us to our 675 00:35:55.000 --> 00:35:58.000 first question break. I know that was a lot to digest, but hopefully 676 00:35:58.000 --> 00:36:03.000 you are all down for another one of our quick quiz games, questions 677 00:36:03.000 --> 00:36:09.067 and then we can get back into the action. Jean, if 678 00:36:09.067 --> 00:36:16.067 you want to throw up those first questions. 679 00:36:16.067 --> 00:36:19.067 >> We are going to start our first quiz. If you played along, you will 680 00:36:19.067 --> 00:36:25.067 notice there is a new window at the top. You will see a question 681 00:36:25.067 --> 00:36:27.067 of here and you will have a series of options you can select multiple 682 00:36:27.067 --> 00:36:30.000 choices. You have a limited amount of time. The quicker you get, the 683 00:36:30.000 --> 00:36:33.000 more points you get. If your data appears only to follow gamma and 684 00:36:33.000 --> 00:36:38.000 lognormal distribution, which ProUCL option should you generally prefer? 685 00:36:38.000 --> 00:36:47.000 Gamma, lognormal, normal or nonparametric? 686 00:36:47.000 --> 00:36:53.000 Our next question on 687 00:36:53.000 --> 00:37:01.000 the screen. Which BTV option is the most sensitive 688 00:37:01.000 --> 00:37:09.067 to outliers? The UPL, you TL, or USL? 689 00:37:09.067 --> 00:37:12.067 For those of you that may be on a mobile device, if you are unable 690 00:37:12.067 --> 00:37:16.067 to see those on the screen, you can always respond using the Q&A 691 00:37:16.067 --> 00:37:22.067 window. It looks like we went through and a couple of people have gotten 692 00:37:22.067 --> 00:37:35.000 those concepts pretty well. 693 00:37:35.000 --> 00:37:38.000 All right. I will switch as back, but we will take time for our break. 694 00:37:38.000 --> 00:37:41.000 Remember there is a window in the lower right corner of your screen 695 00:37:41.000 --> 00:37:44.000 where you can type in questions and comments and I am going to take 696 00:37:44.000 --> 00:37:47.000 this first one now and we will try to get through a couple more before 697 00:37:47.000 --> 00:37:49.000 we go back to the session. One of your attendees noted that when you 698 00:37:49.000 --> 00:37:52.000 were discussing the background threshold value options is beginning, you 699 00:37:52.000 --> 00:37:54.000 mentioned that when you get more than 10 samples, you should be using 700 00:37:54.000 --> 00:38:00.000 a hypothesis testing. Are you referring to 701 00:38:00.000 --> 00:38:03.000 more than 10 background samples or are you 702 00:38:03.000 --> 00:38:05.000 saying that if your background is based on less than 10 samples, your 703 00:38:05.000 --> 00:38:09.000 data set for comparison has more than 10 samples? 704 00:38:09.000 --> 00:38:12.000 >> What I was saying there is that you want at least 10 samples 705 00:38:12.000 --> 00:38:18.000 in your background data set to be able to generate reliable 706 00:38:18.000 --> 00:38:21.000 background threshold value estimates. 707 00:38:21.000 --> 00:38:25.000 However, if you have more than about 10 site 708 00:38:25.000 --> 00:38:30.934 table data, then you should be using those in 709 00:38:30.934 --> 00:38:35.934 a hypothesis testing fence and comparing those. And not just using a not 710 00:38:35.934 --> 00:38:38.934 to exceed value, but both of those what kind 711 00:38:38.934 --> 00:38:47.934 of rely on having a reasonable 10 sample size 712 00:38:47.934 --> 00:38:49.934 background to be able to do that, or sorry, let me rephrase, 713 00:38:49.934 --> 00:38:54.934 to do hypothesis, you want at least 10 in both. To do just 714 00:38:54.934 --> 00:38:57.934 these background hold value, you know, not to exceed situations, 715 00:38:57.934 --> 00:39:01.934 you still need 10 in the background, but if you have less than that in 716 00:39:01.934 --> 00:39:05.934 the site sample, this is owing to be sort of a 717 00:39:05.934 --> 00:39:07.000 stock measure. 718 00:39:07.000 --> 00:39:12.000 >> Okay. And other attendees noted they were instructed to not remove 719 00:39:12.000 --> 00:39:13.000 outliers when computing BTV's. 720 00:39:13.000 --> 00:39:18.000 They wanted to know if you agree with 721 00:39:18.000 --> 00:39:20.000 that approach. 722 00:39:20.000 --> 00:39:24.000 >> I do not know that I have a list Pacific answer as far as guidance. 723 00:39:24.000 --> 00:39:28.000 From a purely statistical 724 00:39:28.000 --> 00:39:32.934 standpoint, I would say the same idea that we have said before, as 725 00:39:32.934 --> 00:39:36.934 if you do make the choice to remove 726 00:39:36.934 --> 00:39:38.934 something for a very legitimate reason, you still need to make sure 727 00:39:38.934 --> 00:39:43.934 to include these statistics presented with and without those 728 00:39:43.934 --> 00:39:47.934 outliers. If you are running into a situation where guidance specifically 729 00:39:47.934 --> 00:39:53.934 says do not remove it, then obviously, don't remove 730 00:39:53.934 --> 00:39:58.934 it, but if it is open to interpretation, like I said, you 731 00:39:58.934 --> 00:40:02.934 need to make sure and include both options. So that whoever you are 732 00:40:02.934 --> 00:40:12.000 showing it to can see, yeah, can see either way with 733 00:40:12.000 --> 00:40:14.000 and without. 734 00:40:14.000 --> 00:40:18.000 >> Okay. We have had a couple people chiming in with comments on regulations 735 00:40:18.000 --> 00:40:20.000 in different approaches and different states, just so that they 736 00:40:20.000 --> 00:40:23.000 wanted to share that some regulators are actually hiring outlier removals 737 00:40:23.000 --> 00:40:27.000 from background but not from the eidetic, so there may be 738 00:40:27.000 --> 00:40:29.934 some regulatory policies influencing what they are doing with their data 739 00:40:29.934 --> 00:40:31.934 sets as well. 740 00:40:31.934 --> 00:40:34.934 >> That is why I am trying to tear clear of the regulatory things because 741 00:40:34.934 --> 00:40:39.934 I know that they definitely vary from place to place and time to 742 00:40:39.934 --> 00:40:43.934 time, so I am just trying to keep it to the 743 00:40:43.934 --> 00:40:47.934 general statistical thought process on it. 744 00:40:47.934 --> 00:40:52.934 >> Okay. I will try to squeeze in one more question here. Can you 745 00:40:52.934 --> 00:40:57.934 describe the difference between WH and HW gamma estimates? 746 00:40:57.934 --> 00:41:02.934 >> Yes. So that WH and H&W actually is just happening 747 00:41:02.934 --> 00:41:11.000 to be two weirdly similar names, so it is Wilson and 748 00:41:11.000 --> 00:41:16.000 hot comes test, which is also impossible to say five times fast. So they 749 00:41:16.000 --> 00:41:20.000 are just two different statistical testing methods, 750 00:41:20.000 --> 00:41:26.000 both are I believe what outlined in the technical guide, but 751 00:41:26.000 --> 00:41:29.934 as far as going into the nitty-gritty of makes 752 00:41:29.934 --> 00:41:33.934 each one different, I feel like that would probably be on the 753 00:41:33.934 --> 00:41:34.934 purview of the call. 754 00:41:34.934 --> 00:41:40.934 >> Okay. I think in the interest of time, we will pause with bastions, 755 00:41:40.934 --> 00:41:43.934 but we will come back to them, so why don't we carry on 756 00:41:43.934 --> 00:41:47.934 with the presentation and I will remind the audience to keep the 757 00:41:47.934 --> 00:41:49.934 questions and comments and we will pause for another break in a little 758 00:41:49.934 --> 00:41:52.934 while. 759 00:41:52.934 --> 00:41:55.934 >> Sounds great. All right. So the time has come. The moment you have 760 00:41:55.934 --> 00:42:00.934 all been waiting to have presentations for, finally talking 761 00:42:00.934 --> 00:42:07.000 about UTL's. Anyway, I would like to say that I assume in this 762 00:42:07.000 --> 00:42:13.000 area that most users are the most familiar 763 00:42:13.000 --> 00:42:17.000 with it, as it is you know, probably the one that gets the most 764 00:42:17.000 --> 00:42:22.000 use within ECL. That being said, it 765 00:42:22.000 --> 00:42:25.000 is important to make sure that everyone is generating their output correctly, 766 00:42:25.000 --> 00:42:31.000 and then interpreting that output in a 767 00:42:31.000 --> 00:42:32.000 meaningful way. 768 00:42:32.000 --> 00:42:35.000 So as we dig into them, I want to make sure again that everyone is 769 00:42:35.000 --> 00:42:39.000 on the same page with UCL, similar to what I talked about with the 770 00:42:39.000 --> 00:42:44.000 UTL's, since quoting random statistics when you do not know what they mean 771 00:42:44.000 --> 00:42:47.000 is pretty not helpful. 772 00:42:47.000 --> 00:42:51.000 So for UCL's, similar to our easy of, we are establishing that 773 00:42:51.000 --> 00:42:58.000 level of confidence, usually 95%, whatever. And 774 00:42:58.000 --> 00:43:02.000 instead of setting a level of coverage at say that 95th percentile, 775 00:43:02.000 --> 00:43:07.067 which we did with our 95-95 UTL, we are focusing on 776 00:43:07.067 --> 00:43:11.067 the mean. There were, the estimates are going to be inherently 777 00:43:11.067 --> 00:43:17.067 less extreme than our UTL estimates giving the 778 00:43:17.067 --> 00:43:21.067 same data set. So just to reiterate, using the same language that we 779 00:43:21.067 --> 00:43:25.067 did with UTL's, we are saying that 95% UCL is 780 00:43:25.067 --> 00:43:30.000 our estimate that given a set of sample data 781 00:43:30.000 --> 00:43:36.000 from our population, we expect the population mean to fall below 782 00:43:36.000 --> 00:43:41.000 that 95% UCL value, 95% of the times that we generated it using 783 00:43:41.000 --> 00:43:44.000 various data set. And so because of that 784 00:43:44.000 --> 00:43:48.000 interpretation, it makes sense that we should be 785 00:43:48.000 --> 00:43:54.000 using UCL's as more of a comparison of distributional tendencies to 786 00:43:54.000 --> 00:43:58.000 a given level rather than what we did with BTB's, you tail, whatever, 787 00:43:58.000 --> 00:44:02.000 where we were looking for really anything 788 00:44:02.000 --> 00:44:06.067 of value that we can collect that happened to be above that generated 789 00:44:06.067 --> 00:44:12.067 value. So for us to have a reliable look at these distributional 790 00:44:12.067 --> 00:44:14.067 tendencies to create our UCL's, we will want to 791 00:44:14.067 --> 00:44:20.067 make sure we have a bare minimum of about 8-10 samples 792 00:44:20.067 --> 00:44:26.067 for our parametric UCL, and at least 15 samples for our nonparametric, 793 00:44:26.067 --> 00:44:32.000 you know, bootstrap type UCL's. And so this time, we 794 00:44:32.000 --> 00:44:36.000 will just pop back in and reuse our Superfund 795 00:44:36.000 --> 00:44:42.000 data set. This time, we are going to treat it like this is site data 796 00:44:42.000 --> 00:44:46.000 being generated as opposed to background data, you 797 00:44:46.000 --> 00:44:51.000 know, just to keep you on your toes here. So cool, 798 00:44:51.000 --> 00:44:58.000 so just like most of the others statistical tests we have been 799 00:44:58.000 --> 00:45:04.000 mentioning, UCL can also be sensitive to outliers, so again, we are going 800 00:45:04.000 --> 00:45:10.067 to go ahead and have this old box plot of 801 00:45:10.067 --> 00:45:16.067 here. And for this first pass, I will keep the potential 802 00:45:16.067 --> 00:45:25.067 outliers in and load up our UCL estimates as well. We can 803 00:45:25.067 --> 00:45:34.000 go back, UPL, UPC's, all 804 00:45:34.000 --> 00:45:39.000 -- for our options, we are going to leave confidence level at 805 00:45:39.000 --> 00:45:45.000 0.95 and bootstrap operations at 2000. that 806 00:45:45.000 --> 00:45:54.000 is fine. There we go. 807 00:45:54.000 --> 00:45:57.000 Cool. 808 00:45:57.000 --> 00:46:01.000 So just to be clear here, while we are reusing 809 00:46:01.000 --> 00:46:05.000 this data set for the purposes of this presentation, we are not 810 00:46:05.000 --> 00:46:10.067 building the not to exceed values, but in 811 00:46:10.067 --> 00:46:16.067 dead just calculating the 95% UCL of the data that we have 812 00:46:16.067 --> 00:46:20.067 present here. They are not interchangeable statistics. We 813 00:46:20.067 --> 00:46:24.067 are measuring different things to be used in 814 00:46:24.067 --> 00:46:30.000 different ways. But when we take a look at it, thankfully, we 815 00:46:30.000 --> 00:46:32.434 do see it is consistent in his 816 00:46:32.434 --> 00:46:36.000 estimation of which distributions the data 817 00:46:36.000 --> 00:46:39.000 seems to come from. We can see that it is 818 00:46:39.000 --> 00:46:47.000 not normal, but it appeared it could come from the gamma or 819 00:46:47.000 --> 00:46:52.000 lognormal distribution. So in this case, ProUCL has done a nice job 820 00:46:52.000 --> 00:46:58.000 of giving us just our general 95% adjusted gamma UCL 821 00:46:58.000 --> 00:47:04.000 options as the recommended UCL choice, 822 00:47:04.000 --> 00:47:08.067 which is the one I would totally take in this case, like not a problem, 823 00:47:08.067 --> 00:47:14.067 just got, you nailed it. As we can see our 824 00:47:14.067 --> 00:47:21.067 adjusted gamma UCL is at that 171 value, which puts it right 825 00:47:21.067 --> 00:47:26.067 about here, like really possibly outside of that 826 00:47:26.067 --> 00:47:31.000 whisker, and pretty high in relation to most of our data. So let's go 827 00:47:31.000 --> 00:47:36.000 ahead and just to illustrate the point that I have 828 00:47:36.000 --> 00:47:40.000 been making of the sensitivity to outliers, we 829 00:47:40.000 --> 00:47:44.000 are going to go back and operate under the assumption 830 00:47:44.000 --> 00:47:48.000 that we went through our whole process that we detailed in our first presentation 831 00:47:48.000 --> 00:47:54.000 as to when and why it might possibly be reasonable to deem it necessary 832 00:47:54.000 --> 00:47:58.000 to remove some outliers from our data and came to the conclusion 833 00:47:58.000 --> 00:48:02.000 that we in fact should, they don't represent 834 00:48:02.000 --> 00:48:06.000 the data at all, they were from laboratory errors, stuff that 835 00:48:06.000 --> 00:48:11.000 has nothing to do with the data we actually 836 00:48:11.000 --> 00:48:15.000 collect did. Cool, so we are going to go back to our Superfund data 837 00:48:15.000 --> 00:48:21.000 here and we are going to go ahead and remove that 838 00:48:21.000 --> 00:48:25.000 530 value, the 440 value and the 340 value and also to be clear, 839 00:48:25.000 --> 00:48:30.934 if you are doing something like this in your actual data, do 840 00:48:30.934 --> 00:48:33.934 not do what I did, we are doing that for presentations 841 00:48:33.934 --> 00:48:37.934 sake. You want to eat them in the data, but flagged them somehow or 842 00:48:37.934 --> 00:48:42.934 create a separate data set without it, but don't just proudly go 843 00:48:42.934 --> 00:48:48.934 deleting stuff. Cool. So let's go ahead 844 00:48:48.934 --> 00:48:57.934 and reload the UCL's without 845 00:48:57.934 --> 00:49:02.934 that there. So first off, we will see that removing these values now 846 00:49:02.934 --> 00:49:09.000 get us to a place where both 847 00:49:09.000 --> 00:49:15.000 the tests for the normal distribution seemed to believe that it 848 00:49:15.000 --> 00:49:21.000 is normal. Neat stuff, then UCL is giving us 849 00:49:21.000 --> 00:49:26.000 this 95% at 84. which puts us much more in 850 00:49:26.000 --> 00:49:31.934 line with something we might expect to see from 851 00:49:31.934 --> 00:49:36.934 this data. However, if we really just wanted 852 00:49:36.934 --> 00:49:42.934 to have a nice apples to apples comparison here, if we looked 853 00:49:42.934 --> 00:49:47.934 back at the gamma adjusted 854 00:49:47.934 --> 00:49:51.934 95% UCL, we will see it at 94. so they are different, 855 00:49:51.934 --> 00:49:55.934 but really within the same ballpark and just removing those three values, 856 00:49:55.934 --> 00:50:01.934 you know, drop the UCL by basically half. Definitely 857 00:50:01.934 --> 00:50:08.000 sensitive, and while I did not show it to 858 00:50:08.000 --> 00:50:13.000 you with UTL's and that kind of stuff, that same level of change 859 00:50:13.000 --> 00:50:17.000 is there as well. Just wanted to physically 860 00:50:17.000 --> 00:50:20.000 actually show you that level of change of complaint in this presentation 861 00:50:20.000 --> 00:50:25.000 because it is pretty drastic, so really making sure your data is 862 00:50:25.000 --> 00:50:29.934 what it should be is right. You know, if you have 863 00:50:29.934 --> 00:50:33.934 high values that are really not representative of your 864 00:50:33.934 --> 00:50:38.934 data, you know, maybe it is right to take them out, but sadly saying 865 00:50:38.934 --> 00:50:42.934 that value is high is definitely not a good enough reason to just 866 00:50:42.934 --> 00:50:48.934 discard that data point on its own. So I hope I have illustrated 867 00:50:48.934 --> 00:50:52.934 that point and up, but I am sure that okay, 868 00:50:52.934 --> 00:50:58.934 should I always use the recommended UCL questions? It is definitely 869 00:50:58.934 --> 00:51:04.934 one that is going to pop up and in a very general sense, 870 00:51:04.934 --> 00:51:11.000 most of the time, yeah, 871 00:51:11.000 --> 00:51:14.000 whatever it puts out is going to generally be a good idea. However, 872 00:51:14.000 --> 00:51:18.000 there is a few cases where it is not quite. Certainly in the case 873 00:51:18.000 --> 00:51:21.000 where you have lognormal only situation or data 874 00:51:21.000 --> 00:51:28.000 that is highly skewed and unruly, it might not 875 00:51:28.000 --> 00:51:30.934 necessarily be the case. So let's just go ahead and take a second 876 00:51:30.934 --> 00:51:34.934 to go through some of those hypotheticals for different UCL out for 877 00:51:34.934 --> 00:51:40.934 our data. So first off, if you're data followed the normal 878 00:51:40.934 --> 00:51:45.934 argument distribution, you are going to take whatever 879 00:51:45.934 --> 00:51:50.934 ProUCL recommends. You should always take a second to double check that 880 00:51:50.934 --> 00:51:55.934 ProUCL has not tried to pull a fast one 881 00:51:55.934 --> 00:52:03.934 on you and recommends the 97.5. or 99% 882 00:52:03.934 --> 00:52:08.000 UCL, which can happen when your data is, you know, particularly 883 00:52:08.000 --> 00:52:10.000 skewed or unruly. These are not necessarily 884 00:52:10.000 --> 00:52:12.000 bad distributional estimates as they are going to provide you with 885 00:52:12.000 --> 00:52:17.000 a more conservative estimate for the UCL, which could be warranted 886 00:52:17.000 --> 00:52:22.000 if you have unruly 887 00:52:22.000 --> 00:52:27.000 data. However, if what you are really needing to report is that 888 00:52:27.000 --> 00:52:33.000 95% UCL, those are still going to be returned in the output. 889 00:52:33.000 --> 00:52:38.000 Now is your data returns that it is only lognormal, 890 00:52:38.000 --> 00:52:42.000 so, let's go down here and say, only lognormal 891 00:52:42.000 --> 00:52:47.000 and that is it, you should definitely take a 892 00:52:47.000 --> 00:52:51.000 second look at it, so as I mentioned before, lognormal distribution contend 893 00:52:51.000 --> 00:52:54.000 to lead to those unstable UCL's 894 00:52:54.000 --> 00:53:00.000 in these situations, and especially if you see that the H UCL is returned 895 00:53:00.000 --> 00:53:05.000 as the recommended UCL from 896 00:53:05.000 --> 00:53:10.067 ProUCL. You definitely got a red flag, so in those situations, I 897 00:53:10.067 --> 00:53:15.067 tend to prefer to rotate to a 898 00:53:15.067 --> 00:53:23.067 nonparametric UCL estimate, so like I said, it is 899 00:53:23.067 --> 00:53:25.067 is lognormal, gamma, whatever, just maybe the gamma, you are good. If 900 00:53:25.067 --> 00:53:30.000 it is lognormal only, instead looking down at 901 00:53:30.000 --> 00:53:36.000 the parametric distribution, or the parametric estimates 902 00:53:36.000 --> 00:53:41.000 of UCL's, it might be a better choice. You need to make 903 00:53:41.000 --> 00:53:46.000 sure you have that increased minimum sample size, though for some of 904 00:53:46.000 --> 00:53:51.000 the bootstrap things, like I said, you know, roughly 15. 905 00:53:51.000 --> 00:53:57.000 Inside of these, my personal preference is that I generally happen 906 00:53:57.000 --> 00:54:03.000 to like the bootstrap method. Corrected and 907 00:54:03.000 --> 00:54:07.067 accelerated, if you are curious what that stands for, 908 00:54:07.067 --> 00:54:11.067 as a replacement for those adjusted UTL's. The other 909 00:54:11.067 --> 00:54:17.067 nonparametric options have their merits as well. 910 00:54:17.067 --> 00:54:21.067 I mean, the standard ones are you know, I almost exactly 911 00:54:21.067 --> 00:54:27.067 the same as the UCL in this situation. But I have had reasonably 912 00:54:27.067 --> 00:54:31.000 decent success with the BCA ones, so that is what I 913 00:54:31.000 --> 00:54:35.000 look at. But as I have said 1 million 914 00:54:35.000 --> 00:54:39.000 times before, if you have any questions or preoccupations about what might 915 00:54:39.000 --> 00:54:48.000 be the best option, consult the data station. And 916 00:54:48.000 --> 00:54:51.000 as for dealing with UCL's and data sets with not a text, we will 917 00:54:51.000 --> 00:54:57.000 go ahead and again swing on back to our TCE data 918 00:54:57.000 --> 00:55:03.000 set here. And let's go ahead and pull that guy up. So 919 00:55:03.000 --> 00:55:09.067 with non-detects, TCE, there we go. 920 00:55:09.067 --> 00:55:13.067 And so again, as we go through here, we are going to see 921 00:55:13.067 --> 00:55:18.067 that our data is consistent and follows that approximate normal 922 00:55:18.067 --> 00:55:22.067 and also gamma, and so this is an example 923 00:55:22.067 --> 00:55:27.067 where you may need to pay a bit of attention to what ProUCL is 924 00:55:27.067 --> 00:55:35.000 outputting. It is going to recommend this Meyer UCL here inside of the 925 00:55:35.000 --> 00:55:40.000 normal distribution, but if we scroll all the way down, it is going 926 00:55:40.000 --> 00:55:46.000 to also recommend that if you have an output such as this 927 00:55:46.000 --> 00:55:52.000 where it has an approximate and is 928 00:55:52.000 --> 00:55:56.000 actually fitting, in the case it is saying it is approximate normal, 929 00:55:56.000 --> 00:56:01.000 but it also has those of the fifth testing positive for gamma, 930 00:56:01.000 --> 00:56:09.067 you might want to use a gamma UCL. Again, we are going to 931 00:56:09.067 --> 00:56:12.067 take a bit of a grain of salt on these because of the rather low 932 00:56:12.067 --> 00:56:19.067 sample size with the eight detected samples, 933 00:56:19.067 --> 00:56:23.067 so whichever one you choose here, there is going to be a reason for 934 00:56:23.067 --> 00:56:27.067 interpretation, but just like before, whichever one 935 00:56:27.067 --> 00:56:32.000 you are choosing, the ROS imputed values 936 00:56:32.000 --> 00:56:36.000 are going to be a little bit suspect because of that really small sample 937 00:56:36.000 --> 00:56:40.000 size and relatively high not detect percentage, so we are going 938 00:56:40.000 --> 00:56:46.000 to want to take a look at instead the 939 00:56:46.000 --> 00:56:54.000 Kaplan-Meier estimate based predictions. 940 00:56:54.000 --> 00:56:59.000 And also, just when we are in here, similar to before, we will see that 941 00:56:59.000 --> 00:57:05.000 ProUCL does give us a nice note here inside 942 00:57:05.000 --> 00:57:08.067 of gamma that if our sample size is over 943 00:57:08.067 --> 00:57:15.067 50. we will be using this, approximate UCL, and then if it is below 50. 944 00:57:15.067 --> 00:57:20.067 we will want to take a look at this gamma adjusted 945 00:57:20.067 --> 00:57:28.067 Kaplan-Meier UCL for just when we have that smaller sample 946 00:57:28.067 --> 00:57:31.000 size. So since I feel like a through a lot of abbreviations and definitions 947 00:57:31.000 --> 00:57:36.000 that you guys, I just want to make sure that everybody got 948 00:57:36.000 --> 00:57:38.000 a quick recap and ended up on the same page as far as understanding 949 00:57:38.000 --> 00:57:43.000 the limits we just generated, so this 950 00:57:43.000 --> 00:57:48.000 table here, whenever it is up is going to give you a little bit, 951 00:57:48.000 --> 00:57:51.000 just that little quick one-liner of the meanings 952 00:57:51.000 --> 00:57:56.000 of what we are talking about, so standard UCL is going to 953 00:57:56.000 --> 00:58:04.000 always be referring to competence on the mean of the data 954 00:58:04.000 --> 00:58:08.000 set while our UTS will be basing confidence on some you know, specified 955 00:58:08.000 --> 00:58:14.000 percentile of the data, and UPL being the next observation is 956 00:58:14.000 --> 00:58:18.000 going to be able to this.95% of the time, and finally USL be using 957 00:58:18.000 --> 00:58:23.000 that niche specific case where our backgrounds that is you know, tidy, 958 00:58:23.000 --> 00:58:25.000 free of outliers, and the background sample 959 00:58:25.000 --> 00:58:29.000 size is roughly between 10 and 20 that can work as a 960 00:58:29.000 --> 00:58:31.934 not to exceed value, even if we are testing a large number of samples 961 00:58:31.934 --> 00:58:36.934 against it, and for some reason have decided we are just not 962 00:58:36.934 --> 00:58:41.934 applying hypothesis testing for some reason. 963 00:58:41.934 --> 00:58:45.934 Cool. So with all that in mind, let's dive into the second quiz 964 00:58:45.934 --> 00:58:50.934 of the day. 965 00:58:50.934 --> 00:58:54.934 >> Okay. I had my general is a staged. I should have 966 00:58:54.934 --> 00:59:00.934 thought up. I am going to pull up our second quiz here on the screen, 967 00:59:00.934 --> 00:59:01.934 and you will go ahead and get started. 968 00:59:01.934 --> 00:59:07.000 The faster you answer the question correctly, the more points you will 969 00:59:07.000 --> 00:59:12.000 get. Okay. What does the approximate 970 00:59:12.000 --> 00:59:25.934 note mean and ProUCL output for families? 971 00:59:25.934 --> 00:59:41.934 We are queuing up for our second question. The data 972 00:59:41.934 --> 00:59:44.934 is lognormal only in the H UCL is returned by Pro UCL, 973 00:59:44.934 --> 00:59:50.934 what is a reasonable alternative to the 974 00:59:50.934 --> 00:59:57.934 UCL option? 975 00:59:57.934 --> 01:00:00.934 Well done everyone who played along. 976 01:00:00.934 --> 01:00:07.000 Let me go back to our split screen view 977 01:00:07.000 --> 01:00:11.000 and Travis, shall we pause for questions here? 978 01:00:11.000 --> 01:00:15.000 >> We can. I am happy to give a couple little final remarks before 979 01:00:15.000 --> 01:00:21.000 we dive into questions because I know those 980 01:00:21.000 --> 01:00:24.000 tended to expand out a little bit. 981 01:00:24.000 --> 01:00:27.000 >> Carry-on. 982 01:00:27.000 --> 01:00:30.934 >> Before we wrapped up and got to those, I wanted to make sure 983 01:00:30.934 --> 01:00:37.934 that we went over a couple final things. First off, I mentioned 984 01:00:37.934 --> 01:00:40.934 a few times that we want to spend time thinking about you know, what 985 01:00:40.934 --> 01:00:45.934 hypothesis testing method, if we can get to it is really going to 986 01:00:45.934 --> 01:00:49.934 be the most useful for a site-specific problem as there is 987 01:00:49.934 --> 01:00:56.934 really no several bullet, and so we also really want to, when we 988 01:00:56.934 --> 01:01:00.934 are doing anything like that, double check our sample size assumptions 989 01:01:00.934 --> 01:01:04.934 so that we are not using 990 01:01:04.934 --> 01:01:08.000 unreasonable data for our goals, and you know, piggybacking on that's. 991 01:01:08.000 --> 01:01:13.000 We really need to scour the data to make sure that we do consider 992 01:01:13.000 --> 01:01:16.000 any outliers carefully, and you know, whether or not to include 993 01:01:16.000 --> 01:01:20.000 them, whether or not we are going to remove them, and if you know, 994 01:01:20.000 --> 01:01:26.000 we have taken that step of let's not include this, then we need to 995 01:01:26.000 --> 01:01:29.934 provide our analysis both with and without those data points in question, 996 01:01:29.934 --> 01:01:33.934 and then while doing that, remember the most 997 01:01:33.934 --> 01:01:38.934 important thing is that documenting all of your analysis 998 01:01:38.934 --> 01:01:42.934 ups and decisions so that you have good traceability in their. If 999 01:01:42.934 --> 01:01:45.934 you hit a point where you are not totally sure what you should be 1000 01:01:45.934 --> 01:01:48.934 doing, as always, consult a statistician. 1001 01:01:48.934 --> 01:01:52.934 I think that is very much the wrap up, but I am 1002 01:01:52.934 --> 01:01:58.934 happy to look at some questions now if we want 1003 01:01:58.934 --> 01:02:00.934 to. 1004 01:02:00.934 --> 01:02:02.934 >> Fantastic. I am going to start reading through questions that came 1005 01:02:02.934 --> 01:02:07.000 in earlier, but I will remind the audience, you can continue to 1006 01:02:07.000 --> 01:02:11.000 submit them. This first question asked what reason is therefore removing 1007 01:02:11.000 --> 01:02:13.000 an outlier besides showing lab error? 1008 01:02:13.000 --> 01:02:18.000 It's just mathematical reasoning or logic? 1009 01:02:18.000 --> 01:02:22.000 >> So we talked about that a fair but in our first presentation. The 1010 01:02:22.000 --> 01:02:25.000 short answer is pretty much know. 1011 01:02:25.000 --> 01:02:31.000 Just saying that looks really far away. It's not 1012 01:02:31.000 --> 01:02:37.000 good enough. Definitely look at that first hesitation if you want 1013 01:02:37.000 --> 01:02:41.000 even more specific stuff, but lab errors or something that can 1014 01:02:41.000 --> 01:02:47.000 be shown to not have anything to 1015 01:02:47.000 --> 01:02:50.000 do with the population we are looking at, those are going to be 1016 01:02:50.000 --> 01:02:55.000 the things that could possibly be a reason to take it out. As 1017 01:02:55.000 --> 01:02:58.000 I said, include it both ways if you do, but just seeing 1018 01:02:58.000 --> 01:03:02.000 something like that looks really high does not necessarily mean that 1019 01:03:02.000 --> 01:03:07.067 you should take it out. You might have hit a hotspot, there is a lot 1020 01:03:07.067 --> 01:03:10.067 of reasons that it should be kept in. 1021 01:03:10.067 --> 01:03:14.067 >> So I just want, I'm feeling a 1022 01:03:14.067 --> 01:03:19.067 little bit like you know, sometimes the designated background carrier, 1023 01:03:19.067 --> 01:03:23.067 you do not know all the history about the area, 1024 01:03:23.067 --> 01:03:29.067 so it may be a hot but from the previous usage, so basically, 1025 01:03:29.067 --> 01:03:33.000 outliers can give you a lot of information, additional 1026 01:03:33.000 --> 01:03:39.000 information about a size, so don't just go by statistical tests, shows 1027 01:03:39.000 --> 01:03:43.000 me it is a high-value. I will disregard 1028 01:03:43.000 --> 01:03:47.000 it. If you dig a little bit, so 1029 01:03:47.000 --> 01:03:50.000 the possible history on the side, the other option is when it does 1030 01:03:50.000 --> 01:03:55.000 not make sense to remove the outlier, that it may be a different geology, 1031 01:03:55.000 --> 01:04:01.000 so it is a smaller area with 1032 01:04:01.000 --> 01:04:07.067 a different geology, and it can create, it 1033 01:04:07.067 --> 01:04:13.067 can translate into a higher value. So 1034 01:04:13.067 --> 01:04:19.067 do not be too quick to remove outliers because they 1035 01:04:19.067 --> 01:04:24.067 tell you a 1036 01:04:24.067 --> 01:04:26.067 lot. 1037 01:04:26.067 --> 01:04:27.067 >> Excellent. 1038 01:04:27.067 --> 01:04:30.000 >> Okay. And then a lot of the questions on outliers. So somebody wanted 1039 01:04:30.000 --> 01:04:34.000 to know why not just run an outlier test, which is available and ProUCL 1040 01:04:34.000 --> 01:04:38.000 to confirm the presence of the outliers, 1041 01:04:38.000 --> 01:04:43.000 >> So the ProUCL test to confirm the 1042 01:04:43.000 --> 01:04:48.000 presence of possible outliers is doing to assume normality at 1043 01:04:48.000 --> 01:04:54.000 the data, assuming removal of the outliers, I believe, 1044 01:04:54.000 --> 01:04:56.000 and so first of all, you are not going to have 1045 01:04:56.000 --> 01:05:01.000 that, so it is not always going to be reliable and even if that 1046 01:05:01.000 --> 01:05:07.067 is true, there could be situations where you are just, you 1047 01:05:07.067 --> 01:05:11.067 are hitting those hotspot that you know, might 1048 01:05:11.067 --> 01:05:15.067 not be part of your distribution, but make, or the distribution you 1049 01:05:15.067 --> 01:05:19.067 think you are looking at, but the site just might be a little 1050 01:05:19.067 --> 01:05:26.067 more heterogeneous than you thought it was. So 1051 01:05:26.067 --> 01:05:29.067 definitely looking at those and using that as a rationale on top 1052 01:05:29.067 --> 01:05:33.000 of some of the stuff you were looking at to possibly remove outliers is 1053 01:05:33.000 --> 01:05:38.000 good, but just seeing that by itself, I would be 1054 01:05:38.000 --> 01:05:43.000 hesitant to remove something, just solely based on seeing that one 1055 01:05:43.000 --> 01:05:44.000 test. 1056 01:05:44.000 --> 01:05:49.000 >> Okay. And then so in addition to doing that outlier test you 1057 01:05:49.000 --> 01:05:52.000 would recommend visualizing the data on a plot before removing those 1058 01:05:52.000 --> 01:05:58.000 outliers?? That for sure, as well as far as the digging into why 1059 01:05:58.000 --> 01:06:03.000 you might have seen those values that are very different 1060 01:06:03.000 --> 01:06:06.067 from the main portion of your data. 1061 01:06:06.067 --> 01:06:10.067 So like as we said, looking into like, was there a possible 1062 01:06:10.067 --> 01:06:14.067 lab error, was there a sampling error, extreme dilution 1063 01:06:14.067 --> 01:06:18.067 that went into it, you know, yeah, or some 1064 01:06:18.067 --> 01:06:21.067 weird natural occurrence, but, yeah, just purely looking at a number 1065 01:06:21.067 --> 01:06:29.067 and just purely looking at the statistical test 1066 01:06:29.067 --> 01:06:33.000 is doing, sorry, doing the statistical test is good, definitely do that, 1067 01:06:33.000 --> 01:06:37.000 but just by itself, I would have a hard time rationalizing 1068 01:06:37.000 --> 01:06:41.000 that as a good enough reason to remove a piece 1069 01:06:41.000 --> 01:06:42.000 of data. 1070 01:06:42.000 --> 01:06:51.000 >> We had someone who wanted to know where is the recommended 1071 01:06:51.000 --> 01:06:52.000 UCL shop? 1072 01:06:52.000 --> 01:06:57.000 >> Yeah yeah yeah, sorry, so, if you just scroll down to the bottom 1073 01:06:57.000 --> 01:07:01.000 here, it is going to say that suggested UCL to use. 1074 01:07:01.000 --> 01:07:07.067 Also, within the output up here, the one that is a blue highlighted 1075 01:07:07.067 --> 01:07:13.067 is the recommended 1076 01:07:13.067 --> 01:07:16.067 one to use, and sometimes it will give you a couple, usually 1077 01:07:16.067 --> 01:07:20.067 it will just give you one, but, yeah, if you are looking for your 1078 01:07:20.067 --> 01:07:23.067 bare minimum, just, which one does it say I should use, 1079 01:07:23.067 --> 01:07:25.300 if you scroll down to the bottom in the suggested UCL to use area, 1080 01:07:25.300 --> 01:07:26.267 it is going to give you that. 1081 01:07:26.267 --> 01:07:29.067 >> Okay. 1082 01:07:29.067 --> 01:07:36.000 >> Another area of questions here, someone noted there is also an option 1083 01:07:36.000 --> 01:07:39.000 for nonparametric in the UT health, they wanted to know how does the 1084 01:07:39.000 --> 01:07:45.000 presence of the non-detects affect the nonparametric ROS UTO value 1085 01:07:45.000 --> 01:07:51.000 that is generated by ProUCL and the output, is this 1086 01:07:51.000 --> 01:07:55.000 nonparametric UTL options incident outliers? 1087 01:07:55.000 --> 01:08:01.000 >> Yeah, so 1088 01:08:01.000 --> 01:08:07.000 yeah. So, I would say, I would say anything 1089 01:08:07.000 --> 01:08:12.000 with ROS stuff 1090 01:08:12.000 --> 01:08:16.000 is going to, I would think it would be sensitive to outliers. As 1091 01:08:16.000 --> 01:08:21.000 far as the bum bum bum bum bum, this is not going to show it because 1092 01:08:21.000 --> 01:08:26.000 it already has, Dang, that is annoying. 1093 01:08:26.000 --> 01:08:28.000 You see if I can show this better. 1094 01:08:28.000 --> 01:08:33.934 There we go. So the nonparametric stuff, I believe 1095 01:08:33.934 --> 01:08:37.934 with non-detects included is going to have a bit 1096 01:08:37.934 --> 01:08:42.934 harder time showing them here. I do not remember off the top of my 1097 01:08:42.934 --> 01:08:46.934 head if it always does or not, it looks like in this, not because 1098 01:08:46.934 --> 01:08:52.934 it is showing that it is to tuck did normal. 1099 01:08:52.934 --> 01:08:55.934 I apologize. 1100 01:08:55.934 --> 01:08:58.934 What was the question again? I got lost. 1101 01:08:58.934 --> 01:09:04.934 >> Sure, so let me, so 1102 01:09:04.934 --> 01:09:07.000 they are noting the option for the nonparametric, they wanted to know 1103 01:09:07.000 --> 01:09:12.000 how the presence of the non-detects FX the nonparametric UTL value generated 1104 01:09:12.000 --> 01:09:16.000 by ProUCL, and then their second question, 1105 01:09:16.000 --> 01:09:21.000 is this nonparametric UTL sensitive to outliers? 1106 01:09:21.000 --> 01:09:24.000 >> Yeah. I think -- 1107 01:09:24.000 --> 01:09:29.934 >> One problem is ROS. ROS does not behave 1108 01:09:29.934 --> 01:09:34.934 normally, so be careful because there is some 1109 01:09:34.934 --> 01:09:40.934 rounding and can generate a lot of numbers 1110 01:09:40.934 --> 01:09:46.934 of the same values, which one leads to a good 1111 01:09:46.934 --> 01:09:51.934 conclusion. It is on our list to improve. 1112 01:09:51.934 --> 01:09:55.934 >> Someone add that to the list of stuff that we 1113 01:09:55.934 --> 01:09:58.934 want better. 1114 01:09:58.934 --> 01:10:02.934 >> So be careful. 1115 01:10:02.934 --> 01:10:04.934 >> Okay. All right. 1116 01:10:04.934 --> 01:10:10.000 Someone wanted to know, is there any way to calculate lower confidence 1117 01:10:10.000 --> 01:10:13.000 limits and ProUCL? 1118 01:10:13.000 --> 01:10:18.000 >> So sort of is the answer. If you look in the ProUCL 1119 01:10:18.000 --> 01:10:26.000 user guys in the UCL chapter, there is 1120 01:10:26.000 --> 01:10:30.934 a note about how to do it. It is not super direct, but it is the 1121 01:10:30.934 --> 01:10:37.934 way is, off the top of my head, it is taking some of the estimates 1122 01:10:37.934 --> 01:10:39.934 you generate out and then you are going to flip and subtract them, 1123 01:10:39.934 --> 01:10:43.100 but it is explained slightly inside the user 1124 01:10:43.100 --> 01:10:43.767 guide. 1125 01:10:43.767 --> 01:10:47.934 >> Okay. 1126 01:10:47.934 --> 01:10:51.934 >> Okay. This is a tubular application question. 1127 01:10:51.934 --> 01:10:57.934 In this particular situation, the user wants to know that if a 1128 01:10:57.934 --> 01:11:00.934 particular 95% UCL for soil shows a value above an action limit 1129 01:11:00.934 --> 01:11:04.934 and you you back late this hotspot and then rerun 1130 01:11:04.934 --> 01:11:09.000 the stitches, they were wondering in that scenario, is it appropriate 1131 01:11:09.000 --> 01:11:13.000 to look at the same UCL distribution in the 1132 01:11:13.000 --> 01:11:14.000 reanalysis, particularly if the reanalysis wants to use a different 1133 01:11:14.000 --> 01:11:17.000 distribution? 1134 01:11:17.000 --> 01:11:22.000 >> I would say use the distribution that your data 1135 01:11:22.000 --> 01:11:26.000 is actually showing, so if you excavate an area, removed 1136 01:11:26.000 --> 01:11:29.934 that soil, that data point is no longer 1137 01:11:29.934 --> 01:11:35.934 valid, and you reran it, I would say look at the stuff your current 1138 01:11:35.934 --> 01:11:38.934 data is showing. When I was showing that comparison of the 1139 01:11:38.934 --> 01:11:43.934 gamma ones, that was more just to show, you know, that that estimate 1140 01:11:43.934 --> 01:11:46.934 actually changed. 1141 01:11:46.934 --> 01:11:50.934 But had I removed those values, you know, again for 1142 01:11:50.934 --> 01:11:57.934 legitimate reasons, and that reanalysis popped up that it was normal, I 1143 01:11:57.934 --> 01:12:01.934 would use that normal estimated UCL, I would take the recommended 1144 01:12:01.934 --> 01:12:05.934 one and not be stuck on the fact that it said before that it was 1145 01:12:05.934 --> 01:12:11.000 gamma, I would take that it said currently it 1146 01:12:11.000 --> 01:12:12.000 is normal. 1147 01:12:12.000 --> 01:12:19.000 >> Okay. What about the situation where multiple UCL's are recommended, 1148 01:12:19.000 --> 01:12:24.000 which one should you select or how should you make 1149 01:12:24.000 --> 01:12:25.000 that determination? 1150 01:12:25.000 --> 01:12:29.067 >> So again, as long as we follow the rules I have outlined before 1151 01:12:29.067 --> 01:12:34.000 of you know, generally lean to the ones where the fit test are strongest 1152 01:12:34.000 --> 01:12:38.000 to say they believe it is distribution, leaning slightly 1153 01:12:38.000 --> 01:12:43.000 away from the lognormal ones, law.doc, if we kind of filter it by those, 1154 01:12:43.000 --> 01:12:48.000 we still have multiples to look at, they are all just methods 1155 01:12:48.000 --> 01:12:56.000 of creating them. I would kind of take each of them with 1156 01:12:56.000 --> 01:12:59.000 a potential grain of salt, if like most of them are about the same, 1157 01:12:59.000 --> 01:13:02.000 and one is wildly off in the distance, maybe take a look at why that test 1158 01:13:02.000 --> 01:13:06.067 is producing such a different answer and see if it is something that 1159 01:13:06.067 --> 01:13:12.067 matters for yours right. But as to the answer of 1160 01:13:12.067 --> 01:13:16.067 which one should I always choose, kind of like I have said, 1161 01:13:16.067 --> 01:13:22.067 there is no silver bullet, but you know, if your data is normal, 1162 01:13:22.067 --> 01:13:28.067 I mean, taking the like, UCL is 1163 01:13:28.067 --> 01:13:36.000 a standard thing, you know, there is merit to all 1164 01:13:36.000 --> 01:13:38.000 of them. 1165 01:13:38.000 --> 01:13:40.000 Providing all the different ones and then using the one that seems 1166 01:13:40.000 --> 01:13:44.000 like the most common standard UCL is not an unreasonable 1167 01:13:44.000 --> 01:13:47.000 approach. 1168 01:13:47.000 --> 01:13:50.000 >> Okay. 1169 01:13:50.000 --> 01:13:57.000 Okay. So how about this one, a number of people have commented 1170 01:13:57.000 --> 01:14:00.000 on the Chevy method or the output in they have been 1171 01:14:00.000 --> 01:14:02.000 told not to use it. Do you have any comments on that? 1172 01:14:02.000 --> 01:14:08.067 >> So yeah, sorry, that is another one that generally, like if I saw 1173 01:14:08.067 --> 01:14:14.067 the UCL being presented, I would lean towards 1174 01:14:14.067 --> 01:14:17.067 the T UCL pressure. It is a more common 1175 01:14:17.067 --> 01:14:23.067 standard one that is going to have a more 1176 01:14:23.067 --> 01:14:26.067 robust answer that is not going to get wiggled around much by weirdness 1177 01:14:26.067 --> 01:14:31.000 in the data. Sorry. I should have included that, the Chevy one, I 1178 01:14:31.000 --> 01:14:35.000 personally would not say it is quite as unreliable as the UCL, 1179 01:14:35.000 --> 01:14:41.000 but it is something that is definitely questionable. 1180 01:14:41.000 --> 01:14:45.000 >> Okay. When you are calculating 1181 01:14:45.000 --> 01:14:49.000 BTV values, would you change the assumed value to something other 1182 01:14:49.000 --> 01:14:50.000 than one? 1183 01:14:50.000 --> 01:14:56.000 >> Yeah. So, generally speaking, I will have it out one. 1184 01:14:56.000 --> 01:15:02.000 That week and there, like 1185 01:15:02.000 --> 01:15:06.067 I said, it is going to be for if you are comparing multiple 1186 01:15:06.067 --> 01:15:10.067 values to it as opposed to one at a 1187 01:15:10.067 --> 01:15:15.067 time. But since you are trying to keep your sample 1188 01:15:15.067 --> 01:15:19.067 size low, or your site sample size low if you are comparing background 1189 01:15:19.067 --> 01:15:28.067 threshold values, you can move it up if you are looking 1190 01:15:28.067 --> 01:15:33.000 at those, but you are only going to see much of a difference as far 1191 01:15:33.000 --> 01:15:39.000 as the EPL change as to what ProUCL back at you. If you 1192 01:15:39.000 --> 01:15:43.000 are using you TL's or USL, I do not believe it is going 1193 01:15:43.000 --> 01:15:48.000 to change much from that one-6 range, but the UPL 1194 01:15:48.000 --> 01:15:50.000 will change. 1195 01:15:50.000 --> 01:15:52.000 >> Okay. I think we will take a couple more questions before we 1196 01:15:52.000 --> 01:15:54.000 go through some of our final comments. 1197 01:15:54.000 --> 01:15:59.000 And attendee noted that the gamma you TL's always give 1198 01:15:59.000 --> 01:16:02.000 more significant digits than another distribution. They wanted to know 1199 01:16:02.000 --> 01:16:07.067 if it was okay to round up these values to less 1200 01:16:07.067 --> 01:16:10.067 significant digits that are more in line with what a normal distribution 1201 01:16:10.067 --> 01:16:13.067 or other distribution would be providing them. 1202 01:16:13.067 --> 01:16:19.067 >> I mean, I feel like producing less significant 1203 01:16:19.067 --> 01:16:24.067 digits is always a safer option than using more 1204 01:16:24.067 --> 01:16:27.067 significant digits. I do not off the top of my head and roughly how 1205 01:16:27.067 --> 01:16:31.000 these are coded in here, so I don't know if 1206 01:16:31.000 --> 01:16:37.000 the significant digit calculations were taken into consideration 1207 01:16:37.000 --> 01:16:42.000 when producing these, so I can't directly answer that, but 1208 01:16:42.000 --> 01:16:46.000 I guess I would say as a general rule of thumb, if 1209 01:16:46.000 --> 01:16:50.000 you are providing things with a small amount of 1210 01:16:50.000 --> 01:16:53.000 significant digits, probably use a smaller number of significant 1211 01:16:53.000 --> 01:16:56.000 digits when you are coming out with your answer 1212 01:16:56.000 --> 01:16:57.000 as well. 1213 01:16:57.000 --> 01:17:02.000 >> Not more rounding than your data has. Yeah. 1214 01:17:02.000 --> 01:17:05.000 >> Okay. 1215 01:17:05.000 --> 01:17:12.067 Let's take my question here, and 1216 01:17:12.067 --> 01:17:15.067 I know that there are many more left in the queue. If we do not 1217 01:17:15.067 --> 01:17:19.067 take your question here, it is not disappearing. We just did not have 1218 01:17:19.067 --> 01:17:21.067 a chance to get to it, but we will ask about other questions that we 1219 01:17:21.067 --> 01:17:24.067 may not have addressed before we close things out. So one more question 1220 01:17:24.067 --> 01:17:27.067 here. This is a comment that one of the attendees said they learned 1221 01:17:27.067 --> 01:17:31.000 that using you TL test and can lead to false negatives, as pacifically 1222 01:17:31.000 --> 01:17:34.000 they will run into a situation were cited a below the 1223 01:17:34.000 --> 01:17:37.000 UTL will be identified as background when really they are not. They 1224 01:17:37.000 --> 01:17:41.000 oftentimes will then request mean testing to support UTL 1225 01:17:41.000 --> 01:17:44.000 testing, and they are wondering if this is necessary. 1226 01:17:44.000 --> 01:17:50.000 >> So that is what I was saying as far as, if we 1227 01:17:50.000 --> 01:17:56.000 have enough site sample data, we prefer to 1228 01:17:56.000 --> 01:17:59.000 do hypothesis testing, use that, you know, like 1229 01:17:59.000 --> 01:18:03.000 if they are normal, whatever, like, and really compare if the background 1230 01:18:03.000 --> 01:18:07.000 distribution is the same as the site distribution, and that is going 1231 01:18:07.000 --> 01:18:11.000 to be a much more rigorous solid approach. This is, 1232 01:18:11.000 --> 01:18:17.000 I hesitate to use the word stopgap, that kind of, if you do not have 1233 01:18:17.000 --> 01:18:21.000 that capability to take those examples for some reason, 1234 01:18:21.000 --> 01:18:29.000 using these not to exceed values will, it is a way to 1235 01:18:29.000 --> 01:18:33.934 do it if you really can't get to that hypothesis 1236 01:18:33.934 --> 01:18:34.934 testing level. 1237 01:18:34.934 --> 01:18:36.934 >> Okay. All right. As noted, I know there are still many other 1238 01:18:36.934 --> 01:18:40.934 questions left in the queue, but let's get out of this official 1239 01:18:40.934 --> 01:18:44.934 Q&A period. The questions are not disappearing, we will go through 1240 01:18:44.934 --> 01:18:48.934 some final remarks, so I will turn off the screen sharing, and 1241 01:18:48.934 --> 01:18:52.934 see if we have any final closing thoughts and touch upon what people 1242 01:18:52.934 --> 01:18:58.934 can do if they have additional questions that we did not 1243 01:18:58.934 --> 01:19:04.934 address today. 1244 01:19:04.934 --> 01:19:07.000 >> So Felicia, if you are on the phone, you should be able to come 1245 01:19:07.000 --> 01:19:10.000 off of mute. 1246 01:19:10.000 --> 01:19:12.000 >> I am here. 1247 01:19:12.000 --> 01:19:16.000 >> I wanted to tell everyone we appreciate you for attending the 1248 01:19:16.000 --> 01:19:21.000 class is. As we close out, I want to let everybody 1249 01:19:21.000 --> 01:19:26.000 know one of the reasons we ask, what would you like to see ProUCL 1250 01:19:26.000 --> 01:19:29.000 do in this final course, is that we 1251 01:19:29.000 --> 01:19:34.934 are working on an update that we hope to have next year 1252 01:19:34.934 --> 01:19:37.934 in 2021. and so we are looking at some of the, you 1253 01:19:37.934 --> 01:19:42.934 know, just improving the program as much as we can. We are moving 1254 01:19:42.934 --> 01:19:48.934 it to you know, the latest Windows 1255 01:19:48.934 --> 01:19:51.934 platform. Windows 10. it was done on a Windows 8 basis, so we are 1256 01:19:51.934 --> 01:19:54.934 trying to make sure it is going to be usable for the foreseeable 1257 01:19:54.934 --> 01:19:57.934 future, and so if you do have some additional comments that you would 1258 01:19:57.934 --> 01:20:02.934 like to get to me about the program, you can send them to me 1259 01:20:02.934 --> 01:20:07.000 via email. My email address is on the website, and so with that, like 1260 01:20:07.000 --> 01:20:11.000 I said, we will be finishing up these questions for 1261 01:20:11.000 --> 01:20:15.000 those that we did not get to during the course, and we thank you 1262 01:20:15.000 --> 01:20:20.000 very much for attending these seminars. 1263 01:20:20.000 --> 01:20:23.000 Thank you. 1264 01:20:23.000 --> 01:20:26.000 >> Thank you so very much. Let me walk through just a 1265 01:20:26.000 --> 01:20:29.934 few final reminders before we close things out today, so I know a number 1266 01:20:29.934 --> 01:20:34.934 of you had asked about additional questions are what you should do 1267 01:20:34.934 --> 01:20:36.934 if you had other input is not addressed, contact 1268 01:20:36.934 --> 01:20:44.934 information is available for our instructors 1269 01:20:44.934 --> 01:20:46.934 and organizers. They will also be working with me to comb through 1270 01:20:46.934 --> 01:20:49.934 the questions that we still had left to try to directly get answers 1271 01:20:49.934 --> 01:20:55.934 to you. I will ask each of you just a few quick reminders, if you 1272 01:20:55.934 --> 01:20:57.934 wanted copies of the presentation materials, we have 1273 01:20:57.934 --> 01:21:02.934 is available to download. The data sets that they used, the super funds 1274 01:21:02.934 --> 01:21:07.000 and the TCE not detect one, those were all part 1275 01:21:07.000 --> 01:21:11.000 of the ProUCL download, but we have them available to access from this 1276 01:21:11.000 --> 01:21:15.000 seminar links. If you did not see it earlier, hit refresh on your 1277 01:21:15.000 --> 01:21:21.000 browser. As noted, I would ask you to take a moment to 1278 01:21:21.000 --> 01:21:25.000 fill out our forms to let us know what you thought of today's session. 1279 01:21:25.000 --> 01:21:27.000 I often get asked for participation certificates and we will provide 1280 01:21:27.000 --> 01:21:32.934 you with one of those as soon as you 1281 01:21:32.934 --> 01:21:35.934 submit feedback for the session. Please be sure 1282 01:21:35.934 --> 01:21:38.934 to correctly enter your email address and check the box at the bottom 1283 01:21:38.934 --> 01:21:40.934 of the form, which certifies you were there for the whole session 1284 01:21:40.934 --> 01:21:42.934 or you replayed the entire recording. 1285 01:21:42.934 --> 01:21:44.934 As long as you put your email address incorrectly, once you submit the 1286 01:21:44.934 --> 01:21:50.934 feedback form, you will have access to one of 1287 01:21:50.934 --> 01:21:53.934 these certificate that you can either save or print out for your own records. 1288 01:21:53.934 --> 01:21:55.934 Sometimes that you wind up getting stuck in your spam folders, you 1289 01:21:55.934 --> 01:21:58.934 may need to look closely there, but we can reissue them if they 1290 01:21:58.934 --> 01:22:01.934 do not arrive I will also let you know that if you shared the room 1291 01:22:01.934 --> 01:22:05.934 with others, for example, if you registered and invited 1292 01:22:05.934 --> 01:22:09.000 others to join, each person in the room and thought the feedback form 1293 01:22:09.000 --> 01:22:13.000 on their own to get their own certificate, even if they did not register. I 1294 01:22:13.000 --> 01:22:16.000 would ask those of you who are joining us, please get the link to the feedback 1295 01:22:16.000 --> 01:22:22.000 form, you will see that under related URLs, and be sure to 1296 01:22:22.000 --> 01:22:26.000 share that. Today's session was recorded, you will automatically 1297 01:22:26.000 --> 01:22:31.000 get an email. You can play back on demand. 1298 01:22:31.000 --> 01:22:36.000 If you had a hard time keeping up with the demos, 1299 01:22:36.000 --> 01:22:39.000 please note the recorded version, you composite, started, fast-forward, 1300 01:22:39.000 --> 01:22:43.000 rewind and play it back at your own speed to follow along with the 1301 01:22:43.000 --> 01:22:46.000 examples. With that, I will echo Felicia's Yankee for presenters 1302 01:22:46.000 --> 01:22:50.000 for joining us for today's session and sharing their expertise, and 1303 01:22:50.000 --> 01:22:53.000 then I also wanted to thank all of you 1304 01:22:53.000 --> 01:22:55.000 who joined us on today's broadcast. 1305 01:22:55.000 --> 01:22:58.000 We had nearly 450 life participants today, we hope that you found this 1306 01:22:58.000 --> 01:23:00.000 to be a valuable expenditure of your time and that you will join 1307 01:23:00.000 --> 01:23:06.067 us on a future webinar. Will go ahead and conclude 1308 01:23:06.067 --> 01:23:07.267 today's broadcast. 1309 01:23:07.267 --> 01:23:08.200 >> Thank you. 1310 01:23:08.200 --> 01:23:10.067 >> Thank you. 1311 01:23:10.067 --> 01:23:13.067 >> Thank you. 1312 01:23:13.067 --> 01:23:18.067 >> Thank you.