So at this point I'm going to turn it over to my co moderator.
Joshua Williams, who's joining us again from EPA, Office of Superfund remediation and technology innovation, and I will let him provide some opening remarks before we start into our technical presentation.
So Josh, feel free to unmute and take it away.
Thank you very much. Good afternoon everyone who's joining us from East Coast, and everyone else. Thank you very much for taking time out of your day to join us for this webinar.
Before we begin, I'd like to go over a little bit about the EPA SME partnership that we have and kind of our objectives. So our goal is to create an open space for informational exchange between public and private sectors on common design and construction
issues encountered at hazardous waste sites, and these webinars and the BI annual symposiums offer a forum for practitioners to talk about their experiences lessons learned and techniques in design and construction issues.
And also, this, these webinars offer an opportunity to enhance communication networks and promote dialogue, amongst professionals in the industry.
And as you can see at the bottom of the slide here there is the DCHWSE symposium.
Starting on March 30 going to April 1 in Philadelphia, and we also just that the registration that went live for the DCHWS West symposium which was dropped into the chat box, and that would be from October 25 to October 27th in Denver, Colorado, and in
addition to that we have different seminar series.
Pretty much So speaking of webinars we had several this year in January. in March. May we have to July this month, which I will talk about a little bit more.
And we have one coming up in September as well.
In process we have different collaborative construction management trainings going on, they're working on and feel free to visit the website www.dchws.org.
To get more information about what is offered.
So speaking of the upcoming webinars that we have later on today, we have a webinar about increasing treatment certainty while controlling remediation costs based studies using hydraulic fracturing that will be at 330.
Eastern time today.
And then in September. We have a webinar on 10 years of optimization of environmental restoration program, dr d facility in North Carolina that will be on September 15.
Two to 3pm Eastern, and we have the registration
link down below to go to clue and to register for it. It opens my podcast
on to the main event, I would like to introduce our speakers today we have ma Hey, he is an estimating manager at Jacobs, and ch to him. Greg is a licensed professional engineer in California, with a BS in civil engineering from University of California
Berkeley. He previously worked for 13 years as a project engineer and project manager for a heavy civil contractor specializing in deep excavations tunnels, utilities, across country pipelines.
Yes, there's 20 years experience with the HMM slice Jacobs, as manager of estimating for the Environmental Services Group, as manager who's responsible for global estimating services, including proposed development and bidding strategy for costume verticals
Price fix, and unit price contracts for hazardous waste, and continue to waste clean up munitions clean up nuclear water wastewater treatment plants utilities federal facilities, and construction management projects.
He's also served as the construction manager for the San Francisco open Bay Bridge skyway in Los Angeles red line Tom project.
He's worked for seven years as an estimating manager for Ohm remediation covering the Western States through the Pacific area. And as a chief estimator for Bechtel national three years.
He's a co author of a C's recommended practice one of seven r dash 19 cost estimate classification system as Applied Engineering procurement construction environmental remediation industries.
In addition, today we'll have Steve foster speaking with us today.
Steve is an ASP Certified Professional estimator, and he has a BS in construction management from the Washington State University is impressive 36 years of experience in construction men with including estimating risk management, project controls and
project management is 10 years experience as a construction superintendent and electrician, working on commercial high rise buildings, hospitals, refineries pulp and paper operations.
Steve served as the manager of estimating at the following nuclear sites, and for Washington Carver and Canada, as well as the Harwell when for nuclear sites in the UK is supported and trained enough upon partners at the ITV our nuclear site in France.
That's a little bit about our presenters. I will now turn it back to gene, so we can learn a little bit more about you guys with some polls.
All right, thank you so let me call up a few polls here on the screen since we've learned a bit about our speakers our speakers wanted to be sure to learn a bit about you before launching into their presentation.
So I'm going to launch on the screen now a series of questions or poll questions and so this very first one that I will launch on the screen now is asking what is your job title or function.
Now we do recognize that some of you may wear many hats I know I have done more than one job function over the years to my job so we've allowed you to check off any of those roles that apply.
and I'll give you all just a few more moments, it looks like about a quarter of you have voted, and then we'll go ahead and end that poll here in just a second and then talk a little bit about who's joining us right now in our session today so we've,
We've got nearly two thirds of the audience voting.
Oh, they're racing in here.
So we're gonna go ahead and close this out here I think we've collected the responses will get. And it looks like overwhelmingly. Most of the audience is in a project management role with about 68% of you responding.
A large percentage of you have listed other, but we also have a significant number who serve as design engineers cost engineers, and then also some owners or owner representatives So, project management is a big job function for our audience today.
So that is excellent let's move on to our next question here. And so the very next question is going to ask how do you currently quantify risk. So in this case we've given you a series of multiple choice options and our speakers are simply interested
in how you currently quantify risk at your particular job. Is it a percentage based on value or type of project as a percentage or value provided by subject matter experts are you doing some sort of a risk analysis tool, perhaps you're using Monte Carlo
analysis. So we just want to have you check off options, and it's looking like we've got about half of the audience responding here. And I know that this will help our presenters as they deliver their talk later if they have a sense of your background
on our topic today.
Great looks like the answers are starting to slow down so I'm going to go ahead and close this poll and let's take a look so overwhelmingly, there is no formal analysis that there's some sort of percentage based on a value or type of a project but it's
not formal. And then, some of you also use a risk analysis tool or spreadsheet about a third of the audience were selected that option and then some of you are using some value that's been provided by subject matter expert with a small percentage using
Monte Carlo analysis so again thank you for that.
Let's ask our next question, shall we and our very next question will ask how often do you perform risk analysis I'm going to launch that poll now.
And I'll come up with some.
Next time I'm going to have some better hold music to play, get a good theme song while we're learning about you so that we can be entertained.
Alright I'm seeing these answers coming in now about half of you have responded and a great big thank you we can't see who's voting how so please know your responses will be anonymous, but this will help us gauge your background to help deliver the talk
here in just a moment.
All right, looks like about three quarters of you have responded, so I will go ahead and close this one down. And let's take a look at those results. So, the overwhelming winner here about 43% of you, sometimes perform risk analysis, and then a lot of
you are using it at the majority of your projects with some of you doing it all and then some of you not doing it at all. So that should help a little bit later.
Alright so let's stop and we'll take our last question here to get background. And I'm going to go ahead and launch this particular question asking how many years of experience do you have with risk analysis.
So we're really looking to see if you are brand new to this field, or if you are a seasoned professional and you've been doing this for some time. So I'm starting to see those answers come in, and if you happen to be sharing your participation spot with
a group of you I'd ask you to average your work experience amongst yourselves if you can to get your rep value.
All right.
Excellent. So I'm starting to see those answers slow down with again about three quarters of you haven't responded, so let's go ahead and end this and take a look at those results.
So we have a very experienced audience here about two thirds of them have five years or more experience with risk analysis. And then about a quarter are brand new to it with less than two years and then the rest to responded or somewhere with three to
five years. So, that is excellent. So I'm going to stop sharing those results. And you can go ahead and click the X in the upper right corner to close that particular box so that the poll results will go away on your screen if they haven't on your own.
And I think we have covered enough to learn a bit about you I want to thank you all for doing that. And at this point, I'll go ahead and turn it over to our presenters to officially launch into their presentation.
So I'm going to put their videos up on the screen here so that you can see them as they get started. I'm going to remove myself but let's make sure you get to at least see your two speakers before they begin.
And then I will allow them to just say hello if they'd like and we can turn off their videos, and then have them share their screen so Stephen and Greg if you want to come off of mute and just say hello quickly to the audience before you get started and
then I'll let you know when we've got your slides up to begin.
Well, thanks gene and Josh.
That's a good soundtrack I believe we can hear you. You sound great Greg and I've got good video. All right, and Stephen Would you like to unmute and say hello to the audience.
Yeah.
Appreciate the opportunity to talk today and welcome everybody and regardless what time zone you're in whether you're just waking up the top isn't the case but yeah.
Looking forward to the presentation hope it is a benefit to you. All right. Fantastic. Well, with that gentlemen you both sound loud and clear and I can see the slides perfectly on your end Stephen so I'll go ahead and we'll turn the video feeds off now
that we've had a chance to see the two of you, and I will turn it over to you, gentlemen to begin your presentation so feel free to get started.
Okay, Thanks Jane and little bit worried when I hear that people are passionate about this topic so we'll see, we'll see how it goes.
So Steve Let's start with the first slide, and again, thanks everyone for attending here
in Raleigh involved Steve's finding the right control.
Yeah. So here are the topics, and then Jacobs we always start our meetings with the safety topic. So we'll do a real quick one here before we get into the rest of the items, and this will be tire safety.
A lot of us have not been driving very much during the past year. And I think it's a good time to just go out and walk around your cars and check your tire pressures to check the tread where and see if the tires are showing anywhere.
Or, where did a misalignment.
And just the age that the tires tires don't last forever, and they're one of the most important things in your vehicle It's just tire pressure and in contact area.
So, check your tires.
Okay, risk.
What is we're asking why do you manage it, they'll be our first topic in saying that 68% of the audience, or in pm roles we probably don't need to talk about that much identification and quantification of risk during progressive phases of design will
provide examples of how to mitigate risk through contract language or add cost will provide examples of how risks are estimated assigned and weighed the probability and the impact risk.
And we'll show you the multiple risk components and how they're analyzed in how you can put together a cost strategy
right next slide, Steve.
Okay, and we could skip to the next one.
And again, we'll go through these pretty quick, seeing the audience has quite a bit experience here.
What is risk, it's an uncertain event or condition that if it occurs has a negative threat or positive opportunity to effect on a project objectives.
When a risk event occurs the negative effects, typically results in a negative financial schedule or project objective and concerning a project objective a good example of that might be if we're designing a building.
And there's an easement running through the footprint of the building. We believe we're going to get access to that easement or the ability to build over it, but if we don't get that ability to build over that easement, and it reduces the footprint of
the building, that's affected our project objectives. So that's what we mean by project objectives were due to a risk we weren't able to build what we want.
So, we'll move on.
The risk analysis process, provides a basis for evaluation and treatment of potential events and contingency development. So in other words, you'll have a plan in place.
When you start your project.
The three forms of risk are managed within a risk register discreet risk uncertainty, which is the former risk that is based on the accuracy of the estimate, so it's the unknown unknowns, and just unknown events, and we'll talk about these three later
on. Once identified their three elements of any risk event that then and the treatment or mitigation and risk maybe from an internal or external events so, Steve.
Next slide.
And this is just a graphic that shows the components of project risk.
The uncertainty risk should go down as scope and design progresses, during the course of the project.
The unknown risk, however, may not change it all in the discrete risk, we should be able to, as the scope progresses, we should be able to cost those items fairly easily.
Steve. Next slide.
And, as you all know we manage risk to minimize the threat are the opportunity, identified the risk events, and and
identify the risk events that will affect our costs and schedule.
We use mitigation to produce the following desired outcomes to improve the cost and skilled of control, to protect the project from financial and schedule losses, it allows senior management to make informed decisions on budget threats and opportunities.
And during the proposal and bid stage, that is the time when we should put the estimated costs, the risk into a project cost estimate and budget, and we don't want to wait, That's the best time to do it and get it started.
And, and the risk register to track what we're doing with risk items.
Can you Steve next item.
And I think most of you have seen this trumpet chart in it shows the accuracy of cost estimates as design progresses, from preliminary to final design.
The risk levels should also move from left to right, it should go down as, as we learn more about the project.
During every design during every design phase and submitted, there should be a discussion of risk, and you should have a risk register where you document the risk event, and the mitigation at each stage
on some projects especially environmental projects the cost of the risk elements, really want to crease, and we'll show some examples of that later. So, that truly is what makes environmental worked a little bit different, the risk may not change and,
in some cases, increase as we go.
Next slide, Steve.
Here's a list of 18 different topics that can create risk in will put a lot of these into the risk registers that we use.
And I think a lot of you
are participating, mainly due to what's happened over the past year with the coven 19.
And we've seen costs skyrocket on our construction projects and we've been doing a lot of surveys with contractors and owners and what we found is, for example, safety and productivity.
Most contractors are adding anywhere from 17 to 30% to their labor costs due to productivity issues and compass international in fact just published a document on that.
And as you know, the trades, specifically saved the mechanical contractors and electrical contractors they're adding close to 30 to 35% to their labor costs to to productivity issues material costs have gone way up.
we, again, we'll have some more information on that later lumber and steel are at all time highs and in that's if you could get the materials, right now, numbers just hard to get.
And the suppliers are representing that with higher prices in under procurement. A lot of our contractors are having issues.
Due to supply chain costs and schedule issues the vendors and manufacturers are having an even delivery. Believe it or not, there is a shortage of truck drivers to deliver materials.
Under subcontracting fewer contractors are bidding work so we're seeing a lot of projects with only say three bidders or less, many cases only one bitter.
And as I work. I used to work for our contractor, you typically know when you're the only bidder on a project and contractors will take advantage of that and I don't blame them.
So, stable now join us with the discussion on the basis of basics of risk assessment
is the cricket Jean Tommy so Thanks, Greg.
Yeah.
I wanted to start out by talking about some of the elements of the risk assessment process.
We have on the key elements side of the house. There we have a what we call a risk breakdown structure. And really what this is, this is a way that we consistently document, the risks that we apply to a project, and we we monitor your those risk occurred
Do they not occur, provides a source for checklist, which is a good idea when you're developing a risk register. An example of a risk Breakdown Structure could include, this is actually from a project that I worked on the there was a group that we collect
the risk and project and program management, another bucket that was technology and engineering, third bucket was environmental and regulatory issues.
Another one was construction, then organization, and then programmatic which are external risk to the project. So these are important elements that you want to start with the very beginning, when did develop your, your risk process and your risk register
to keep them organized and help build that database.
And then once we move past that we then get into the qualitative analysis aspect of it. And this is more than what I call the talkative and recording stage of it, we,
we determine we get together as a group we develop a workshop and the workshops can be depending on the size of project. I go from an hour to I've had them up last upwards of a week, but it's a brainstorming session we discussed the, the various risk
that may impact the project both internal and external.
And then we determine we give each risk a name and an ID, we determine the gift so if this event happens, then, which is the impact portion of it what are the impacts of that risk.
And then we start out with a mitigation strategy on thing. So that's the next part of it and then we once we have all that down and discussed. Then we move into the quantitative analysis aspect of and this is where you get more into the ranking the scoring
cost development or scheduled impacts the risk. And here was actually score the risk and I'll show you an example that in a minute.
Using a scoring matrix. And with that matrix will determine the probability and the impacts, we developed three point estimates probabilities, all this kind of good stuff which you'll see more, and we run the Monte Carlo simulation.
And then once we run the Monte Carlo simulation, it will give us outputs, such as density charts, both in the density chart with a stack of histogram type of view and then also a cumulative curve on thing.
It'll provide us providers sensitivity analysis, and then also give us a range of confidence intervals that we use to actually determine our contingency value and, and just take a moment here, because I've to my experience I've found that a number of
people don't really know when I what people mean when they're talking about confidence interval and that's the confidence percentage you hear the term 80% confidence on the project.
And what we're saying there is there's an 80% probability that you will be able to achieve your objectives, within that range that you're showing on the output.
There's only a 20% chance of failure. Now, the output will give you a range of numbers so it's very important to assess and understand your risk appetite how much you're willing to take on board.
And then on the right side we've got responses over here in each event will call them events because some people perceive risk as a threat. Well, the risk event may also be an opportunity, but what we want to do and this ties into everyone on the left
side is that we want to look at that event and do we, we have determined we want to avoid it. Do you want to mitigate it the lower the cost of the impact on the project.
Do we want to trash that risk transfer may be that we get a firm quote from subcontractors so we've expressed that risk away from us to the contractor.
We can just accept it, which then we manage the risk and, you know, work it from there. We also have opportunities which are, you know, we can exploit we can enhance the opportunity, such as schedule improvement price improvement productivity improvement,
a number of ways to do that now which you'll see on both of them is the share, that's actually on both of them because we want to under threat, we want to add a minimum at least be able to share the threat to kick some butt off fast, and also with opportunities
you know we might want to share that with the, with other contractors with the owner the client, everything so these are the key elements of you, that you involve in you develop risk.
Now, move on to the next slide here, we now look more at the risk register the whole documented itself. And with the risk register which is a standard industry practice.
We wanted to identify and describe the risk events, you're wanting to develop an action plan for those events, and whether that be mitigation as I mentioned just a minute ago, you know, avoid transfer reduce except so on so forth.
And then we want to be able to estimate the, the cost of that event or that risk.
So, we, we want to be careful on how we develop this and it gets refined and managed to this risk register. So there's two types of risk bear determination, and those are determinant methods and probabilistic determinant is more, just as each probably
all know is just basically a fixed number that you determine be a formula or subject matter expert probabilistic is this this is the statistical approach to developing that result.
And, again, walk you through that here in a few minutes.
Okay.
You know science in America, cost and schedule impacts to risk items to help establish a risk reserved aka your cost contingency or an allowance.
We have the project delivery stage which helps incorporate the cost in the project budget is Greg talked about earlier.
And then the risk register prepared during the solicitation period is your starting point. This can also be, if you're developing a program for instance you may not be building a project but you're actually developing a project.
So we want to start this this process early in design, and let it evolve through the process of design and scope maturation.
Lastly, we have the risk register itself which we want to keep and maintain a bit through the course the project which is very critical. In fact, almost can't emphasize that enough because then manage your contingency.
Okay. So, with that, their questions.
All right, we actually did have a few questions that came in and I think it might be a nice time to start about to start into on some of these. So one of the attendees asked what about opportunities risks that can benefit the project rather than the more
traditional negative risks that we've been thinking about can you comment on those opportunities. It's when we talked about just a minute ago.
It's nice to be able to you know, we're trying to look for that that optimum price on a project and one of the things we want to be able to do is save money on the project or be able to advance the project so we want to with the opportunities will develop
those figure out a cost of what that opportunity could save the project both cost and schedule and share that with the client. And then we move that back over into the budget so I hope that answers the question.
Excellent.
Yeah, Jane I can actually give a good example of that, on an environmental project.
We often are doing groundwater treatment projects where we're injecting some sort of substrate in that materials reacting with what's in the ground or groundwater and treating the material.
In some cases, we haven't finished all our studies and an opportunity would be that we would be able to lower the mixer adjust the mix to wear what we're injecting the cost of that material might be lower.
So at the time of the estimate we may carry $1 value of what that injected material might cost. but in the risk register we would list that there's an opportunity that the chemistry might might allow us to go to a lower cost option, and is, you know,
it follows not just with injection projects but just about every project you, you're always looking at those opportunities, and you should list them, and they may be low probability but it's always good to have them on your risk register as a possible
opportunity so the team is always looking for it.
Okay. Excellent. I'll remind the audience that that q amp a window is available to send in your questions you can send them in privately as well as anonymously.
I'm going to jump into a grouping of questions that have come in and the best way I can summarize this, it's people are expressing problems and costs when you're talking about risk.
So one of the attendees noted that it can be difficult to associate costs to risks in some cases, and they're looking for examples, or references that you may use when you're trying to associate a cost with a risk because in their case they're encountering
suggestions from their support contractors that additional fees get added into allocate costs and the risk register and they really can't they themselves don't know how to validate those costs that are being tied to the risk.
Great don't take the first shot that you want me to jump in. Yeah, I would say this.
I'm a true believer and you just have to do the hard work sometimes and really digging in and do a cost estimate for that specific risk item.
If. but as far as, say, a document, or a table that would help you understand what the risk might be within the EPA cost estimating guidelines in its Boyd's data now it's back.
I think it was published in, the year 2000, they do have a table in there where they talk about different types of remedial projects and they give you a list of contingency that they recommend for those types of projects and even the, the contingency
percentages, don't truly represent the risk percentage that you want to put into your budget in at least gives you an idea of the, of what the industry is seen as far as what they believe is the true contingency safe, or an excavation of a hotspot versus
groundwater treatment, you know, versus some, you know, in situ soil treatment.
So I don't know if that helps the audience but you know that is one source.
I think the other.
Another way to look at it is rely on your subject matter experts. That's what we do at Jacobs we're going to show a risk register later and how we use it and how we really rely on their subject matter experts to give us your opinion on the, you know,
on a, on each risk item, what their judgment is on the probability of it happening in the total cost of it.
Okay.
Okay. There's, there's another area that you can look for, you know, I mentioned earlier about, you know, developing history of your risk. Now this obviously will take time but this is the importance of recording risk and their outcomes.
As you go through in your monitor your risk register and you've applied a risk and you know, put a cost to it. You run all the scenarios in the accuracy of the risk actually occurs.
And you can actually record that cost and over time, you can develop an average or mean cost for that type of risk, the other. The advantage of doing probabilistic estimating and deterministic to a certain point is that you can apply a three point estimate
so there's a lot of uncertainty, or subjectivity to the cost, you may not have it nailed down but there's validity to putting your best guests in there and then rapper range bound at the lower high, or even if you don't have the best guest given range,
low and a high range and you can run that to the Monte Carlo simulation and reply confidence levels to that and I'll show you later how that's that's done.
All right, excellent. I think in the interest of time, we'll go ahead and move on. There are a few other questions just getting into some of the specifics of some of the tools and Monte Carlo so let's, let's go ahead and carry on, we'll just do a quick
time check that it's about 142 and I'll turn it back to you, Steve to carry on. Okay. Yeah, thank you. Yeah, I'll try to set this up a little bit.
As we talked earlier you know we talked about risk and the risk register the process that we want to go through is illustrated in this pyramid on the left side here.
And when we start the project and beginning we want to go through the planning process which won't have read everything here but it's basically how you develop your, your strategy to developing recording and managing risk to the left of the project.
Then you go through that workshop session in talking with subject matter experts other project.
People historical data, so on so forth. And your identify your risk and record those in your register them go through the qualitative, as we talked about earlier.
And we record all those, the, the actual risk, the, the triggers on it, the impacts, and any kind of residuals, you might want to have on the, the risk.
And then we go through the quantitative risk analysis process which is where we actually do the ranking. The three point estimate probabilities we run the Monte Carlo simulation and run the program see what the results are.
That moves us on up into the risk response planning phase of it which, once we see the results, then we have to dive in and see how can we mitigate or exploit the, the event to take advantage of it.
And finally, once you've gone through that, then we move into the risk monitoring and control phase the project which is where you go through monthly meetings updates.
The site owners to the risk and know and manage the process through.
So, with that one. I talked about a scoring matrix and this is an example of a scoring matrix that we've used and this could actually be combined into one matrix, but here I'm showing the threats and opportunities.
And what we've done is, we, you want to measure the impact of a of an event, and also the probability of the event. So we've looked at various aspects of an event that, you know, could occur be safety costs related schedule environmental quality technology
reputational.
Consider and then we sort of score, those are described what they are, ranging from a very low impact very high impact on the project and this is good recorded here because it gives the team.
A quick document that they can look at to see how they want to score that, and then they want to look at the probability and here we've got a five by five matrix so we got five levels of probability, starting from zero to 10% probability which gives a
very low probability, all the way up to 70 to 100% probability which means very high. So if we look at a event for instance and we say that it's a high risk, you know the the impact is going to be hiring the project.
And there's a 50% chance of that occurring. That's going to score is a medium level risk. Now when you go through and you actually start managing risk on the project.
It's the high in the medium that you want to put the primary focus on, and then monitor and track the low risk to the process. So this is the tool that you use in your start of your qualitative or quantitative process of the risk analysis to help you
rank those, those events.
So next, see if I can get the singer to turn that over to great let him go through deterministic approach to risk development.
And so this is a short form that we use at Jacobs to look at risk and what allows us to use, essentially, a short form and not go to the Monte Carlo, on, on a lot of our projects, is we do 300 to 400 estimates a year.
And these are pretty much recurrent type projects with similar risk items, and our teams have extensive experience with the sites and the different technologies that we're using.
So it's Excel based, and hopefully you can see this on your screen or if you've downloaded it you could read the small print.
But for example, in the orange grouping, we described the rest the type, including location the person who identified the risk.
The risk category, who it will impact, whether or not it's a risk or opportunity. And we also rank the priority based on the risk matrix that Steve just showed
within the blue grouping, it's the mitigation area.
And we said the description of the action, the person responsible completion date or target completion date, and the status of whether or not it's open or closed.
Now, even if it's closed we leave it under risk register once once an item gets on the risk register you should never delete it just keep it there and you never know when you're going to want to come back to it.
In the green area is where we do the risk allowance. So the probabilities usually looked at in agreed to by the team and especially the again the subject matter expert helps us with the probability of a risk occurring.
And then the estimator works on the items that are the best case most likely and worst case and these are the cost of the risk event, not the cost of the project but just the risk event.
And then in the orange column.
Those cells takes the probability multiplied against the distribution of the best case most likely and worst case and develops to value and the total cost that we would put into the project is right under the word risk there Stephen in the Whitesell
further down in the green.
Go up.
Anyway, it's the 643,000 in this case.
And that is the risk tool that we use at Jacobs for the majority of our projects.
Steve's going to show the Monte Carlo system, little bit later. We're going to go into the next slide, which is mitigation through contract language or cost reserves.
And during contract preparation or negotiations is the best time to mitigate risk contractors and owners should clearly agree on scope, you know the quantities contaminant cleanup levels.
Those should really be in writing before the contract is signed a couple examples recently of what's happened during the past year and coded and things such as material pricing.
A good example is one of our clients said a large pipeline project with the large quantity of plastic pipe.
The contract is worried about the rise and plastic costs, it was really jumping and spiking, and they talked to the owner about issues such as putting in risk for that event.
After interviewing the contractors what we found, where the pipe suppliers were offering a 14 day guarantee on the price, we convince the owner to guarantee to award the work within 14 days have been day.
And because of that, the contractors were able to minimize the risk. and they pass that on to the owner.
So that's a good example of, you know, keeping communications open understanding the market, and working with your contractors.
Another good example is just simply extending the period of performance if you can.
Most of the contractors with coven issues. What they worry about the most is having the project shut down and really the way the courts have looked at it.
In most cases, contractors are not getting any compensation for last days due to coven, they're getting schedule extensions but no compensation. So, if you can at least give them extended periods of performance that would greatly help them.
We currently perform many projects under negotiated guaranteed maximum price contract where the wondering contractors share risk.
An example is that is where the contractor assumes the risk for labor and productivity, but the owner will accept the risk of material price and so again it's communications and being able to negotiate and put things down on paper before the contract
starts
costs reserves are best determined by performing estimates to represent the best case, most likely and worst case example is we had a ground water treatment plant project and sending California power costs was large component.
The risk was the cost of power during peak periods where the cost goes from nine cents a kilowatt hour to $4 a kilowatt hour.
The trigger was when the temperature seated 91 degrees.
We researched the historical records, and we found that average peak days was 27 to 28 days per year, with the exception of 2020, which had 40 days.
This gave us a good read on the most likely and, in, worst case scenarios and we're able to price that work based on that data. So again, it gets back to just doing the hard work and spending some time in evaluating what the true risk is and putting it
down on paper and coming to a consensus on where you want to be
the last topic here contract is really aren't taking risks, at all.
An example is, if they're bidding on a project and they received three electrical bids. They might name too low better but they'll actually use the second bitters proposed the Mount, and what they're saying and what we're seeing as it passes on up to
us is, there is a huge spread now between low bitter, and high bidder on these projects we recently had a water treatment plant project where the low bidder was at around 80 million in the hybrid or was it 110, and there are only three vendors, so that's
what we're saying.
Again the contractors that are adding anywhere from 17 to, you know, 30 35% labor to covert issues there there is a spike in material pricing Steve's going to show a slide on that later.
And this is happening at all levels starts with the vendors and manufacturers and they pass it to their subs, or to their contractors and those subcontractors pass it on to us.
So stevia. The next slide.
This is the Monte Carlo process.
Okay. Thanks. Great. So now we'll move from the deterministic to a Monte Carlo.
The process. And when we undertake a Monte Carlo process is.
It's more of a senior manager or project management decision to make that as Greg talked about a minute ago. There's a deterministic method which he used on projects, we're very familiar with him a lot of history on to other projects that are more open
to and subject to risks that we're not familiar with that we don't typically encounter.
And they usually happen on larger, more complex projects. So those are the kinds of projects and decision makers we want to have to move into the Monte Carlo process.
So the next step is I call it. So when you do that, the two software that are typically used and there's a number of them out there but the most popular seem to be at risk.
And then another one by Oracle called crystal ball. The information that I'm going to share for the down here is done through at risk.
And again, the Monte Carlo process is the historical approach, as opposed to a determinant approach.
So, when I go through the Monte Carlo process, I'm just going to sort of walk you through the steps are that we do, we define the risk events, as I talked about earlier those the IF THEN treatment.
Things we discussed, you know if this happens, then the impact is this, and this is how we're going to treat it. So, once we go through and we've listed the, the risk and everything.
Then we probably want to go move into the quantitative aspect of it where we look at the risk scoring and we earlier walk through that risk scoring matrix where we look at the probability and impacts.
So we want to very carefully evaluate and determine what that is because that does have an influence on the result of the Monte Carlo simulation,
determine the appropriate distribution variable. Now when I say distribution variable, those are actually curves, we use distribution curves are can be put curves triangular curves, normal curve there's a bright and I'll show you some examples here in
the next slide. So we pick one that is most suitable for what we, the event requires, then we want to evaluate the variable range which is the low and higher what we call three point estimate.
So you want to have either via the estimate and This usually happens, if you know what the cost history of the if it's a discrete risk event that may or may not happen.
We may be able to assign a point cost to it. But then we want to look on a new easiest number of percentage like if we have a, an event that say $1,000 event.
Okay, and we're going to say that okay the low side it's 80% of that high it's 120% of that so we get a point are most likely cost of the thousand paid low in that $800 and a high of 1200 thing.
So, that determines our range, and then we want to take a look, if it's a discrete risk, as opposed to the uncertainty of an estimate and again I'll talk about this but we want to look at the probability of that event occurring in real life, some, you
Some if we have a history of the event, then we can look back and say okay we've had 100. This event has happened. We've recorded it 100 times. And it's only happened 20 times out of that hundred so we got 20% probability of occurrence.
Otherwise, you have to sort of look at whatever history you can find, you know, in the on the web or wherever you can find subject matter experts, so on and so forth.
Then, next one, take a look through this risk register everyone make correlations between events.
And this is often overlooked in the risk process but if you have example, say, fuel costs increase. Okay, well that fuel increase is going to cause it's going to impact.
You know equipment operation costs, and it could in cost utility costs, so there's a number that it can tie it. So you want to correlate those in the correlation.
You know it's either positive or negative correlation, one being a strong negative one being very weak correlation.
So you go through and you're ranked and there's a scoring matrix in the program, you do that with.
Once we've done that we review it and we're happy with what we've got. We run the simulation and tip be a bigger projects you want that between 5000 10,000 I usually run 10,000 because it gives me a very strong sampling of those events and how many times
they will appear under that curve that I want to look at the results of that, and select my confidence percentage and I mentioned earlier, you know, like 80% confidence means that there's 80% chance that that it's going to fall within that range of cost
them all that results in the cost impact.
So the next we want to look at potential residual and scheduling packs. Now, this is particularly useful in a discrete event. And when I say residual.
The, the risk event is has got a description it's got a scope that we evaluate, but that could be knocked on offense, or it could develop minor risk, you know, in addition to that.
So, what I typically do is I will assign a percentage of my most likely cost as a residual so that in the event that some smaller events happen that I'm not really aware of right now, I've got some money in there to cover that schedule impacts are basically
a hotel Oh the impact on the project. If this rent, through this event does occur. Does it have a schedule impact, and if it does, what is the cost impact of that schedule event so you may have innovation that says okay I've got the most likely have of
of 10 days low, high, you know, 12 on the, on the highest side. And that's going to cost, $100, per day. Okay, well I want to kick that pens whatever day is most likely, and add that to my, the cost of my risk event so at the end of the day will have
the cost of the risk will have any residual risk and will have any scheduled impacts that comprise the whole event event.
And then, once we're done with that we want to make sure that we documented and what to say document, we want to be able to show any calculations historical data commentary, whatever we can find this kind of support the basis of that risk evaluation.
And then lastly we use what we call a risk and opportunity information sheet or rice for short. And what this does, for every item event that we record on the risk register.
We have a backup sheet that defines that risk or they vent in much more detail, and actually tracks, so that y'all. If you've got that might be sitting out there in the contemporary bucket and we need to move money from contingency into the project be
control for something. It's all recorded and managed on this sheet so you on the, if I look at layers, the top layer is the risk register and then underneath that is a book of rice documents that actually give a reader or the project much more information
about those risk individually.
Okay. So, moving to the next one. I talked about distribution curve so there's there's a discrete curve. And there's distributor distributor of curves.
So what we have here is we have the newly and I'll show you where this is applied. Later on, but this is used exclusively in risky events where it's a discrete risk that may or may not happen.
So what this does is it the don't think it does really is determined through the random sampling, one of the curve, yes or no.
I'm Stephen, Stephen I'm gonna hear. Real quickly, I think we may have missed the last few seconds of what you were just saying so if you could go back about two sentences and then repeat what you just said, Okay, sure.
I just don't the top here Can you hear me OK now.
Yes, I can. Okay one make sure this is that issue is having some things cut it out.
Okay, so the the various distributions we've got here discreet and distributed curves. The newly function is primarily for yes no advance, will the risk occur, yes or no, it's the same thing as a coin flip.
We have the poison curve which is not shown on here and that's basically for events that may happen multiple times that project.
And then there's this as I've described them too much but the ones that we use most typically are perk triangular, and uniform. These two are very similar nature is just the, the shape of the curve that is different.
And that's really more of a user's choice on thing. The one thing that that gives you is longer tails on the left and the right as opposed to the triangular uniform.
I mentioned earlier that if you read on no point estimate, you assign uniform distribution. So, there you define a low and the high range not worry about the midpoint.
Here you assign low medium high on these two. So these are typical distributions that we use and we use them we look out for each event, the risk register.
Okay, so now we've, We've would just say we've run the simulation.
And this is where you get some really good information.
We have the competence range, which is actually it's on the left hand side of the curve here because you know bisque is zero to 100, and also appear. And what this is telling us is that around the sort of gore that over here expand this right here in
this example is the simulation and reason is a, an estimate here, $8.66 million is the value of the estimate. So when you ran the, the Monte Carlo simulation, each of these columns here show where these values that we defined appear.
So I'm saying here that we've got a point six $6 million estimate, and according to everything that we provide in our uncertainty, part of the not the discrete risk but the uncertainty evaluation of the estimate, we're saying we've got a 30.6 probability
of occurring, or reaching achieving that project for that value.
So, I'm saying well okay I want an 80% probability of that happening. So what that's telling us is that for 80% confidence. It's going to be $9.175 million, and that contingency value is this range right in here.
And if you'd look over here to the right on this statistics table that it produces, you'll see that, you know, we have the, the, the left axis here, which is 8.6, the right, which is the 9.175.
Okay, that's this value right here and that is our contingency value based on 80% probability of the Monte Carlo simulation. And you'll see here in a minute that we can select various competence levels and the range of that.
So, moving the next slide, we have in this current be overlaid on the density chart, but we're looking at the same information here.
Okay, and this is again 10,000 iterations. And over here on this side we have the confidence intervals, and what is displaying right now is 25 to 99%.
So when I said, 80% confidence, we're looking at this right here.
And if you read that from the chart, it's the same thing as going on from planning here. Following the line over and dropping straight down and you'll see the range of values and our estimate right there.
So that's how you read the, the curve on this thing. But you can see the usefulness here of using the confidence levels. If we want to day say it's 70% confidence.
Then we're going to drop down to 9.053 as opposed to the 9.175 day. So, this is where management has to look and understand the risk event, and determine what their appetite for risk acceptance on the project.
So,
catch up with the, we talked earlier about, you know the impacts of things and this is a, an example here of looking at commodities, we can see that from last year even as recent as is November of 20, you know prices were at a very low rate, but then
as the spike and co bid hit and really got bad through the November December January timeframe, we can see the increase on that. And in particular, you'll see in the example here I'll just say steel mill products and we'll see that structural steel okay
that time pretty dramatically to so that's a risk that we would want to evaluate.
When we're going through our uncertainty, or our discrete risk events. So this could be both the distinct example of how you know things like commodities, or labor or numerous other things, you know, can happen in a very short period of time and have
a substantial impact on the project.
So, moving on to the, the next one here.
You know, we ran the Monte Carlo simulation for the uncertainty aspect of the estimate no words, we tested the.
Our competence in the estimate itself. So I'm showing different charts are in there both turnover charts. The one on the left here is a contribution to variance.
So, the percentages defined here are how much they contribute to the overall impact, the total impact of all the risk or the elements of the uncertain nation to the estimate, and you see over here that we have a number of items, you know, direct labor
is the biggest impact. So that's contributing the most basically 60% of the impact on contingency is wrapped around labor so we want to put our focus on mitigation strategies on labor that would be the first five Good how are we going to improve labor
costs and what are the impacts of that. And I mentioned on the curve earlier steel. Well, still turns out to be number two here that's got a 14.7.
You know chance, or contribution to this thing. And when you look at the right hand turn you have a trigger. This shows you the variants of the main values little red line right here.
And that's the main value of the estimate of the $8.8.
million, which is the being guy as opposed to the actual determined discrete value. So, it shows you the range of impacts the labor could have steel subcontracts so on so forth.
So, the recommendation here is when you get into this level of hearing to start evaluating the risk register, you want to focus on the top five, eight, or however you determine are the important events that are showing here.
And these are the ones you want to really seriously look at in mitigating and lowering the cost of the project.
So, next time move into the discrete or the Monte Carlo simulation where we have looking at, you know, our risky bets. Now this is an example of a risk register that that I use and actually used for this one.
Well I don't show it. This is where the scoring matrix comes in and this is one of the things that you also look at when you rank and score your risk so for instance if, if I had four or five out of here that were high red and a couple of yellow.
Those are the ones that I want to focus on because I'm considering those subjectively to be about high risk things I need to focus on.
So, once we determine that we move over here for each event.
You know, we've got a most likely cost here which we either historical data, estimated however we come up with that number that we document. We then develop the low rings, and the hirings.
And when we run the Monte Carlo simulation.
We will take, and it'll turn out and mean value and that's what we're seeing highlighted in blue here it's just outputs the main day.
Then we move to the right here we look at the probability. So when you run the Monte Carlo simulation, and I talked about the binary distribution which is the discrete function on the thing.
We're going to put in our probability of occurrence on this thing in the form percentage. So we're seeing here that we got 60% and 50%, but pretty high.
Rest of a fairly low, but when I run 10,000 simulations, out of all of these all nine of these risks, only two of them appeared. So when I develop my contingency cost or my risk cost for this thing.
It's the result of my P value which I call p value which is your confidence level, and we've talked about 80% all the time, or all the way through this.
So, we got a 60% probability of this happening, it did happen. We're going to apply the 80% confidence level, which is going to push it up a little bit higher than the main value in give us a $40,078 by the same thing down here for 58 now becomes for
83 80% competence. We some those up and that is our discrete risk, where the risk race or total for those. And that gets added on top of the uncertainty risk of, you'll see that the others do not appear.
And I've lower I've managed to lower most all of them here at second one I lowered from 650 to 500 and so on so forth, ran the same thing, I kept the same a chance to view the the probabilities because I was able to mitigate some of the the causes of
it. So come up with the same occurrences on here. I still find 80% but just see now, I dropped from 523 to 466 that mitigation strategy. So this is part of that.
Once you run your first simulation, you go back. Pick the results that evaluated. Develop your strategies and how you can realistically, you know, improve your cost on it.
So, this is the second aspect of a risk evaluation or project.
And with the, the discrete risks that we were just talking there you'll see that on the pre mitigation risk number two was by far the biggest one. It shows the of the median value here, the range that that risk would could move to.
Okay, we move down, and you know it ranks them by the value in the, the frequency of how they occur in the simulation. So we want to focus here on this one, and then we want to look at this, this, this, evaluating them.
Now, after I've done my risk analysis running a simulation. We see similar results but you'll see that the the values details are lower went from 1.9 million here to 1.7 million here, and also lowered this so we pulled the curve or the tornado chart,
towards the center. One thing I noticed though, once we did the evaluation. It really ranked them risk number five over here has now dropped off the chart here and been replaced by risk number six.
So, it's going to score them and rank them and this helps you see how well you know what impacts. You've your mitigation strategies have had. Okay.
So, with that, moving to the
final output of a risk. This shows the comparison of a Monte Carlo simulation and what I've done here is I've, I've simulated I'll just say a class three estimate, you know, to a class to estimate.
So, to keep things level I didn't change the base estimate cabinet 8.66 which is solid here on the curves.
I also kept 80% confidence level on my contingency. And then I, you know, the resultant of the contingency. So you'll see that, you know when I ran the first simulation, and I pointed out in the slide if you recall the 515,000, the difference between
the estimate, and the 80% confidence level. So, we have this value here which results in 6% contingency. And from the risk register that top number that I showed you in the slide here that 523 here.
Okay, is what we're seeing down here see add the two together for a total of contingency value of $1.38 million, dollars, which gets added to your base for total project I have $9.6 million.
Now after we've mitigated we can see that keeping things constant here.
For various reasons, design has improved scope has matured. We've got better quotes in the estimate, but either way we've been able to lower that uncertainty the estimate from 515,000 now down to $343,000.
By the same token, when we went through and we look at the the discrete risk mitigation strategies were able to lower that down to 466,000 so we went from 65% contingency on discrete risk.
Overall we dropped her from 12% to 9%, resulting in about a $228,000 savings on our bottom line cost of the project.
So, one thing I was, I was going to show you that I'm just going to move up to is this right here is is a valuable tool because when you run the simulation, you can actually play scenarios and here you can type in what, you know, the value here or you
can say I want 10% here, we can just pull this thing along your tail with your cost needs to be, and the resulting competence level just by looking at the percentages here.
So this is a very good tool when you're evaluating risk and how much you want to just or look at your, your contingency cost.
It's one point that I because I didn't mention it earlier.
Okay. Wrapping things up.
We've got, you know, we want every project for every project is subject to uncertainty and risk, and certainly being the quality estimate risk being events that may or may not happen in that project.
We want to begin the risk identification process as early as possible. If it's a project during the solicitation phase or program during the early development phase of the project.
We there's two type of risk there's internal risk that we can control that we can manage.
Then there's external risk, some of those are things that we cannot like we can't control interest rates, we can't control external funding to the project we can't control coven.
So these are examples of programmatic or external risk that we would apply in the risk register.
When the evaluating the project continuously we want to consider as I've talked. We both talked about through here, uncertainty discrete risk, and the unknowns.
Now the unknowns is always sort of a mystery one but sometimes people use them as allowances, because they just the management wants to have a number in there for unknowns because maybe the contracts issue, don't have the opportunity to to adjust that
but really if you're managing the risk register and the risk project through the course of the project. Then, risks are going to drop off the good appear.
Your new ones may materialize. And this is where the unknown risk, come into play.
I might add also that it's good for the owner to hold their contingency outside and that's part of this programmatic risk that project can hold the risk which is the uncertainty and internal risk.
The owner may decide to have a contingency value also for maybe additional scope changes or to handle some of these other outside events that because interest rates and so on so forth.
So, just wanted to mention that calculated contents he is near his statistical determinant methods.
As I said earlier begin to mitigation strategies, early so soon as we understand them, we look at the schedule, we see how they're going to appear in the project on the timeline and start developing strategies can minimize those as much as possible or
exploit them.
Then we want to monitor manage and update the risk register as often as possible usually projects I've been associated with we usually have a monthly meeting on them and we review them, we, you know, each risk as an owner that is responsible for tracking
and managing that risk event. And then lastly, and most importantly, risk management is a team effort it takes the subject matter experts the designers the construction people, it's everybody involved in the project that contributes to the risk management
effort.
And so with that, I will turn into that gene back to you.
All right, thank you so very much. I think we have time for a couple of the questions who have come in from the live audience, I'm just going to start reading through them, and then we'll go through some closing reminders for those of you who cannot stay
though please open up the chat I did push out some final links in the meeting chat if you have to hop off so important URLs are in there now, but for those who can stay with us give me a few more moments and let's get through some of these questions.
So one of the first questions, we had a lot of people interested in the software packages, you were referencing and showing illustrations from some people wanted to confirm if Monte Carlo was the name of the software or is it the name of the method, or
both. So could you just confirm what software you were using. Okay, the software that I showed examples of here we're from a company called palisade, and the name of the software is called a risk, they have a number of software that they publish with
at risk is the one that I use for all of my risk crystal ball is developed by Oracle and crystal ball obviously is the name of that and they're both probably the most used and accepted programs on the market, both very very good quality.
Okay. I actually use that back in college.
Okay, um, one of the attendees wondered how do you set your minimum and maximum cost range in your risk register is that exclusively based on professional judgment.
Generally, yes.
Again, this is where we, if you have the ability or have historical information, you can sort of get an idea of how those have landed historically. If not, it's subjective it's based on subject matter expert input, the estimator input UI it's.
You can also look at it as many of us are familiar with the estimate classification they will define the range of accuracy, whether it's a class five class three or class one type of estimate, you know, the closer you move the class one the tide of that
is, I use that as a starting point for my range development. So if it says, minus 15 plus 20. That's the range I'm going to use up at that most likely by this is target.
So, so Jean and I think I'll make the comment that at Jacobs we have a lot of subject matter experts, and for specific technologies, so we really do rely on debt staff to help us less you know the most likely the least likely and calculate costs so it's
usually a team effort between the tech team and the estimating team to develop those those ranges. But, yeah, it has to do a lot with experience and the people who've performed the work.
Okay.
Um, we also just had a quick question. Someone wanted to know is power the kilowatts cost, becoming a more dominant or complex, time of day or seasonal risk in your experience.
It has, and what was very interesting in looking at the power costs for.
In this case it was Southern Cal Edison, they do not publish the Publish rates for residential clients, but they do not publish it for industrial clients.
And the only way we were able to find the $4, a kilowatt hour, was we we found there.
The SCEPUC some middles, and how they develop their rates.
So that's how we found out that the peak rate was going to be $4, a kilowatt hour. Okay. And, yeah, it's a huge spike, and it's in the small print of everyone's contract that if it hits 91 degrees, you're going to get hit with that peak rate for that
day and it's, you know, it's easy to do the math once you figure that out.
Okay, question number one just briefly.
Both of those programs.
You know crystal ball and at risk are actually programs that embed into your Excel program so once you load the program, they become a part of your Excel program.
Okay. Well, that's how I remember it working.
Somebody else brought up another product, another software package called model risk, which has a Monte Carlo package that is useful and apparently free just wondered if you've ever use something called model risk.
No, I have not. Okay. Okay.
All right. Well, let's take this one last question here and we'll go through our final reminders before we close things out. So, some of the attendees or just curious about your success rate in using these methods.
So they really want to know, you know, how successful Have you been in the last five years. Do you have profitability percentages based on the fees using these types of risk methods.
Well, I can comment on that. I do a lot of deep dive reviews for our design build teams and, and they typically will spend the time and money going through and doing a Monte Carlo analysis on rest.
And like time we get to the point where we're submitting a final design and price.
The average has been around three 3.1% risk value on the total contract amount.
And one thing that I always do is, is we refer to the type of technology being used, and the value of the contract you know whether or not it's a $200 million project or a $2 billion project.
And the good news for us, is it appears that we've been spending. Maybe 1.7 to 1.8% of that risk value and executing the work.
In most of those cases we are sharing that with our clients.
But, yeah, right right now we've been doing very well with how we manage risk.
Excellent.
All right, well we are just a minute past our scheduled end time so I think at this point I'm going to go ahead and wrap up our q amp a segment. And I'm going to switch gears here to call up a different presentation deck and put that up on the screen,
but I do want to thank both of our presenters for joining us and answering those questions and sharing their expertise, I really do appreciate that and the audience has been sending in a couple of Thank you messages as well.
So, I want to go through a couple of quick final reminders and I don't want to steal the thunder from Josh who I know is on the line still and So Josh, I know you you lead in by talking about it.
Do you want to remind the audience about this ongoing partnership and then I'll come in and close this out.
I'm happy to.
So, I want to remind you guys about some of the upcoming webinars that we have the BI monthly webinars. So we have one more in the afternoon today at 330.
And we also have one in September coming up as well.
We have the EPA SME construction management training that we do jointly and, again, feel free to visit the website@www.dchws.org. For more info about that.
Yep. Here's the info for the upcoming webinars that we have.
And I'll take it from there, Josh.
Thank you so very much. So again, I encourage those of you who joined us. I appreciate each and every one of you for taking time out of your day both our presenters organizers and attendees.
I'll remind everybody that you can visit us at the queen of information network or clue n.org website to learn more about other resources and online events related to hazardous waste a clean up.
We also send out a free monthly newsletter tech direct we're will announce webinars like today's session so I encourage you to sign up for tech direct if you don't receive it, I think it's one of my most favorite emails, it shows up in my inbox each month.
So you can visit us at clue in and sign up for tech direct there.
I will also remind everyone that presentation materials are already posted on the seminar homepage. So it's shown here on the slide I also put it in the meeting chat for those who are still with us you can open up the chat and see public messages that
I've sent. So there's a complete copy of the materials, there's also information about our speakers if you want to follow up with any of them we have their bios up there as well.
And for those of you who can and will or are interested in a certificate thanks for joining us today. I encourage you to fill out that online feedback form and let us know what you thought of our delivery.
So there's a link to the seminar feedback for my also place that earlier in the meeting chat, you can find the tab on the seminar homepage to that form.
For those of you who are looking for, see user PDH is, we do not issue those hours, but I can give you a participation certificate, if you fill out the form for feedback and check a box at the very bottom certifying that you were here for the whole session,
as long as you fill out the feedback form and check that box you'll then immediately have access to a participation certificate that you can save and for your own records, oftentimes our attendees are able to take that, along with a copy of the slides
and other resources that we post online and use that to document to get credit with their own institution. If you happen to share the line with multiple people each person can fill out that feedback form to get their own certificate, even if they didn't
register individually. And if you happen to be one of the lucky ones replaying the recorded version of today's session. I'll direct your attention up in the upper right, you can't see it now but for those of you on the recorded version you will be able
to see a feedback form button in the upper right where you can click to fill out the feedback form based off your replay of the archives to get the very same certificate.
So with that, I will issue out one last reminder that our session was recorded and I'll send an email as soon as that archive is available for on demand playback.
I'll remind everybody you can get to all of our fabulous resources for our session on the seminar homepage. And I want to thank all of our speakers and organizers for joining us and again all of the attendees who hopped on and participated in today's