Official US Government Icon

Official websites use .gov
A .gov website belongs to an official government organization in the United States.

Secure Site Icon

Secure .gov websites use HTTPS
A lock ( ) or https:// means you’ve safely connected to the .gov website. Share sensitive information only on official, secure websites.

Smart City Challenge Information Session 9 Transcript

Evaluation and the Smart City Challenge

[Please stand by for  realtime captions.] >> [Please stand by for  realtime  captions.]  >> Ladies and gentlemen thank you  for standing by. The conference  will begin  momentarily.  >> Ladies and gentlemen thank you  for standing by today's program  will be starting momentarily. Please  continue to hold and thank you for  your  patience.  >> Ladies and gentlemen, thank you  for  standing by. Welcome to today's  evaluation and smart  city challenge. At this time  all participants are in a listen  only mode. If you should require  assistance during today's call please  press star and  then zero. I will not turn things  over to Brian Cronin. 

Good afternoon everyone, thank  you and welcome to the The Smart  City Challenge   evaluation webinar.  And I think our last webinar  and then we heard you loud and clear  and we will let you go write your  proposal we felt like and you felt  like the opening kickoff meeting  a few weeks back that  we should die a little bit more  into the evaluation component. Today  I am going to start  this out and we will kind of go  through the Notice of Funding Opportunity  to make sure it's clear as to what  we are expecting you to tell us  as you submit your proposal of  that were going to talk about our  expectations related to performance  measurement  and evaluation and talk a little  bit more and answer your questions  and then Kevin will  provide information on other technical  resources and information that out  there that should you need more  information, you're obviously welcome  to  call at but we wanted to highlight  those resources and there is some  next  steps information and we will talk  about that. Thank  you and as  a reminder, we start looking at  evaluation, we are talking about  the outcome for the smart  city challenge. As he  made clear in the notice is really  all about improving safety, mobility,  and opportunity and arresting  climate change. We are looking to  capture measures and determine the  outcome from the The Smart City  Challenge  on these different  challenge  area. 

So in the goals on page 10 and  11 we talk about really  starting at this is exactly your  challenge so while we put a framework around  evaluation talk about different  performance measures it is ultimately  grounded in the challenges you face.  So Vulcan Inc.  understand the impact  from the technology,  the applications, that will help  address  these challenges. Looking at evolutionary  steps to integrating technology  so were looking at how  to demonstrate and evaluate the  impact of the technologies also  understand the technical policy  and institutional  arrangement needed to make  this change. We are want  to look at reproducibility so we  really want there is an aspect of  the evaluation to try to understand  how this will work not only in your  city of other cities across the  country and tie into our  ability to  do technology transfer and outreach  without thinking to go through the  development process. And then ultimately  we want to work with the  federal partners and look at how  we accelerate the deployment of  clean transportation  and connected and automated vehicle  technologies. 

In the notice in  section A we talk about performance  measurement plan  and about U.S.  DOT  tracking surveys that will  be  going on during the time of this  notice. That we will talk about  data management and support filters  article roles  in how you do anything of management  plan how to support independent  evaluation, how you write data  to support that evaluation and providing  some data  on it research data exchange so  others can analyze and look at the  research data to help understand  how does technology is impacting  their cities.  And what you're going  Dto need  to do in section talks about  the application process and the  technical approach around doing  those things. The performance measurement  plan looks  at identifying performance measures  and quantitative  performance targets. What we need  to discuss this type of data needed  to support performance measurement  in your demonstration  and evaluation. We want to look  at hypothesis and methodologies  for collecting data before hand,  data during  the lifecycle of the  demonstration and as well as cost  data and information on the timeline  and application availability deployed  in your demonstration. And how the  performance recognition will be  released as  open data and data for other researchers  and technology we  move forward. 

In the notice we also talk about  setting different performance measures  so for instance some items  we identified as possible was that  would make sense in a smart city  application around traffic related  fatalities and injuries, transition  to clean multimodal this objective,  cost savings, and you can  read these but it's critical  to identify your performance  measures how you want to do that  in the  smart city. How you do your  data management and support for  independent  evaluation. It going to be we are  providing independent evaluation  undergirding the data that you generate.  They are going to look at  the hypothesis that you recommend  and work  with you as you understand which  the greater going to collect and  how they're going to be able to  assess if you have those goals so  they will be  looking at on the data that qualitative  evaluation methodologies to look  at before and after, cost-benefit,  user acceptance or satisfaction,  document  lessons learned, and your purchase  in overcoming challenges as he went  through implementation, costs and  impact, and really when it comes  down to these you achieve  your vision?  Was the demonstration ultimately  achieving the vision in the smart  city  challenge. 

Continuing more on the data management  and evaluation report, unique evaluation  support plan how you are going to  work with  you and --  independent evaluators. To get clarity  as we get into the demonstration  of roles and responsibilities and  is really about understanding what  the hell you're going to collect,  and the metadata to describe  that  data as to what the data is in the  purpose and the how and what  it's for. How you're going to monitor  the performance measures  and target, availability talk about  how the site and the staff are  able to help the independent evaluator  in terms of their role in providing  that data and information  to them so they can understand your  data and applications from that.  If we decide after a and interviews  are needed, to  supplement evaluation then helping  to do that. Your role  is to support the independent evaluation,  access to the site, in fact staff  to conduct experiments and  injuries. 

Considering on your  systems must be capable of generating  the data needed to calculate measures  that you well-defined so we need  to be looking into how well  those systems are performing and  what's needed to generate the data  you need for  your evaluation and measurement.  You have to look at monitor the  quality of the data if you go mom  and mom were you thought we were  collecting data and then you shouldn't  have any copies have processes in  place to be able to really understand  your getting the data needed at  the time of its and accurate and  quality  for supporting. And then transmission  of that data to support  the evaluator. Data collected for  the evaluation will be owned by  the U.S.  DOT  and required to support independent  evaluation must be  shared. 

We have this research data exchange  and that's only been using to share  research data with the community  at large. We had various  connected vehicle data samples,  we had data from the safety  pilot, we had data from the connected  vehicle pilot, is not necessarily  your whole data set from the smart  city but indicated  that at least representative enough  that can be used and we would  like to house some of the research  data exchange you will is articulate  to us as we  go through and part of the data  management what information is going  to be available and shared so we  can make them available to the  research community.  That for the want to analyze how  a smart city and technologies and  applications can be used to support  their  city. 

In  section D your proposal should really  provide measurable goals and objectives  describe your approach for monetary  impact on the demonstration  across major different goal areas  and then identify a set of target  for measures that relates to the  impact of your proposed demonstration.  With that foundation I'm going to  hand it over to Marcia Gulati for  more details about independent  evaluation. 

Good morning and good afternoon  to everyone who is joining us on  the phone today. Thank you for walking  us through the elements  of the tran08's everybody is  clear on the requirements that are  expected related to evaluation and  at the end of the  short webinar if you have any questions,  please make sure to ask us for clarification  so we know that you know that  always requirements are understood  the way they are listed. For the  next few minutes you will  should focus and I will be talking  to you about the more practical  considerations as you develop  your performance measurement plan  and the other documents that  Brian  described.  >> What you are about to do  is truly the big  and bold  and exciting. And it is indeed a once-in-a-lifetime  opportunity to be a living laboratory  for the nation with respect to smart  and connected cities. They not only  demonstrate the innovative use of  technologies and policies and partnerships  but you realize you your role in  ensuring the nation and probably  the world because it will be international  interest, the amazing things that  you have accomplished with  investment that is being  made here. So you have truly some  unprecedented opportunities to baseline  where you are and then show  the improvements in your city with  the use of these new technologies  and approaches  and partnerships. You have even  more exciting opportunities to develop  compelling performance measures  and innovative ways of testing and  evaluating them. Your efforts will  serve  as a map for other cities and also  an example for the people in United  States to provide them  very quantifiable and dependable  benefits related to this  you chessmen that we are  making. 

Given this, there is or  could be  our example of our thinking about  logical way to approach  some of the questions you  should probably be asking yourself  as you develop the elements  of the NOFO   related to evaluation. These are  some fairly big  strategic lessons.  Several questions that may drive  many others is in the  end what do you and your partners  really want to get from  this investment? Not only investment  of federal money by the  investment of resources in your  jurisdiction and your time and your  people and  your brainpower. What do you want  to show the world and tell the world  immediate about what you have accomplished  with the The Smart City Challenge.  And what you want to have learned  from this  grand adventure. For each jurisdiction  the answer to these patients will  definitely be different  but ultimately these are the questions  I think that you  need to start  thinking about. And then this starts  to drill down deeper from these  larger questions actually, the  performance measures and hypotheses  that mean the most to you. Once  you are clear on those,  it's kind of like reverse engineering,  you need to start thinking about  what data you will need to test  those hypotheses and those  performance measures. What data  we need from who and how much? How  long is it going to take to get  it? What kind of agreement you have  to have a place in order to have  those data? And I make a point of  this because it's not just transportation  data, that you will want to be dealing  with, it's data from all sectors  of the city because the smart city  challenge isn't just a  transportation project. This always  to be  very clear that in the near term  is also important in terms of baselining.  You will  need to give baselining some serious  thought because it needs to  be very clear on what data you need  to do that because you don't really  get a second chance to do that baselining  and at the end of your project one  of the most people questions or  things that we tend to ask ourselves  is we really should have collected  data  on X because that means  a lot for stakeholders  and ourselves. Think in advance  about your baselining is  important. 

I get some guidelines in terms  of approach. Set your performance  measures that quantitative performance  targets with your stakeholders at  the  beginning of the demo and articulate  that in the performance  measurement plan. You are going  to want to  think about the most effective way  of partnering with independent evaluators  and articulate that in  your proposal. We like  to see you  establishing hypotheses and here  is an example. Hypotheses  are typically opening statements  if I do  ask them why  will happen. If transit signal priority is deployed  on all city buses, then 80% of all  buses will arrive at their destination  on time. The last one is  establishing baseline conditions  so you can establish a before  and after  jurisdiction. 

As mentioned before a  good evaluation allows  you to highlight the amazing things  you have accomplished and show off  potential of investment for that  technology.  Another point of values that you  gain visibility into the effectiveness  and operation of all aspects of  your city that you currently don't  have on a very granular way anyhow  an opportunity to go cross  sector partnerships that are in  some ways based on the kind of data  exchange you set at  the beginning. And keeping in mind  that this is not just a  transport project. It's critical  to start thinking of what kind of  data do we need to start thinking  about  between the transport and the healthcare  sector, or  public safety and integrate  those into the performance measures  that you will develop particularly  important in addressing the outcome  and the NOFO  related to letters opportunity   which is not  specifically directed to  just transportation. It's about  economic development and society  and jobs and the  economy so you  need to begin to be able to  answer questions for city officials  other decision-makers and your citizens  about the performance of this project  and not only at the  end. 

To that end, the  evaluation itself cancer multiple  goals. To  replicate -- not  only identified what is going right  what's been done well, but to  also identified what is not going  well and  why. Hopefully most things you put  out there in the environment  will go right but some  might  not. Honestly, much of  this challenges so on the cutting  edge of so many things that you  will be surprised that everything  you thought would go right actually  did but it's important  you document even  those instances where things didn't  go as well so other people  can learn how to mitigate those  problems or avoid  them. Because  you will be doing something that  no one has tried before. You will  invent new ways to solve new problems  in knowing what you try,  will reduce the time, that frustration  ultimately for you and your colleagues  and also for those of  you who are  representing cities that may want  to pursue this in the  future. 

We also ask you if  it helpful  as a navigation tool imagine you  were the second city trying this  The Smart City Challenge  . What would you want to learn from  the first in order so you can be  successful? Treat that city the  way you want to be treated. Try  to get the  details of what we expect to happen,  and if it  doesn't, happen what does that  mean? 

Some more technical guidance  on  best practices. It's important to  choose performance measures that  are of value to your stakeholders  is important to talk to them about  what's important to them. It seems  to be a no-brainer  but sometimes these discussions  are left off the table.  You need to establish clear traceability  for goals to measures and  from system input  to outcome, this will help guide  your data collection and also your  hypothesis development. You need  to develop a data  management prospect is clean and  efficient to integrate multisource  data into a common format and database.  This is only  help you and the need to have  a plan for automated to verify your  data quality and perform periodically  to ensure that whatever equipment  that is being used to generate the  data you need  is accurate and in a healthy  state. 

This is just a summary of what  we have been talking about. There  are three major elements here the  first  one the The Smart City  Challenge  the bullets underneath  are some of the things that you  will be expected to provide us  information on  and then  feed those and work with the independent  evaluators  who undertake if not all at least  the vast majority of  the bullets listed under independent  evaluator. And also to be able to  figure out what is going to  be placed in  the RDE  so other jurisdictions can take  advantage of the hard work that  you have  done. 

Just so you know that all this  work is done in the  context  with JPO in a larger program  that essentially starts to track  all over the United States  the performance of  various ITS systems the  effectiveness of them, the  installation and eventual replacement  of different  generations of ITS  technology  and it supports also  making strategic effectiveness  assessment about what gets deployed,  the weight of employment, dealing  with the development across the  spectrum of ITS   so just know that the results from  the smart city challenge evaluation  will be part of the larger scheme  of things and that goes back  to the notion of being a living  laboratory that can help the Department  of Transportation actually get a  granular view into how this next  generation of ITS   takes off and is  deployed. 

We thought we were  going to hold for questions until  the end? There is a  couple. 

I will read the  first one. Do we need to develop an evaluation  support plan as part of the proposal,  or do you just want to see evidence  that we will be able to create a  strong  plan? 

We've had this question for the  other called we don't develop a  plan in  the proposal to  provide evidence that you have the  capabilities to do  so. 


You don't have to create the  evaluation support  plan as  the proposal, you need to show evidence  that what to do and then we  are hiring the independent evaluator  but you need to have support  to see -- feed your data  help create your evaluation plan  and support of  that. 

The  bottom line U.S. DOT will contract  separately  with an independent  evaluator. 

I don't see any other questions  in the  chatbox. Kevin, do you want  to? 

Just add the questions and we  will come back to  them.  >> Technical resources  now we are for weeks and  30 minutes from when the proposal  is to do so you have seen much of  this information before, this is  just a reminder that there are resources  out there and if there's something  missing here, the  other things you are addressing  in your  proposal, that

     -- you can ask up, we can  research it. This  list of eight information sessions  today's webinar will be number  nine, that will be up in a day or  two. This'll be available also.  Again most of this sessions were  from December and January  not only the presentations are  up on the side  of the transcript as well as you  can replay the  whole webinar obviously most don't  have time to deal with that at this  point, but it's  available. 

Next I know that were never questions  related to  system engineering.  Federal highway begins  a site and  decide training the consortium for  ICS training  and education. The federal highway  does have  some good not only training  but material on how to do a system  engineering plan so this site is  a good resource  and again working with the visual  audio team that you're going to  have to point to will be able  to prepare and of the system engineering  work later. I would hope at  this point everyone is familiar  with this.

     And again on each presentation we  have  the URL, you will have to cut and paste  them. 

The next few slides are  from the  ITS  JPO  sites  so this is just to cover  lots of materials, more than you  need for addressing your  technical proposal. We just provided  overview here and this is again,  most of you have seen this. So what  we want to highlight here are connected  vehicle courses  -- we just  had a little  technical issue. This gives  an overview again and I think we  have presented as material and our  previous  workshop.  >> If there is one site that all of  a lot  of use for any of the smart  city is this is what  we have for connected vehicle pilots  because in part  this is in some respects  similar to [Indiscernible] some  of the connected  vehicle aspect probably not to the  same skilled than you do so much  more. There are ways of dealing  with the  performance measurement,  dealing with  security credential  managing system, and then we have  presentations, reports,  guided material, all in this site.  This is a very rich  resource and will continue to be  of  use to all the cities going forward  that you're looking to do this type  of work or as part of future projects. 

Similarly  we have much more  detail on ITS   national architecture that the next  layer down talking about connected  vehicles is really another addition  to another layer on the ITS  architecture.  For the most part  a smart city in  highly connected so much of what's  applicable in the  connected vehicle reference interpretation  architecture will need to be part  of  your project for your technical  proposal, to  the extent it's a matter of how  much in-depth you're going into  this area, this is certainly  a resource that should be able to  address anything you need at this  point. Again we can provide  the experts in  this area and provide  responses through [Indiscernible]  if there's any questions that come  in the next few weeks. The  last item supported from the  ITS site  since this  is a  key aspect of what we are looking  and smart city is  accessible transportation. There  is a multiagency project that  we host that involves  folks from DOT  and outside calls the accessible  transportation  research -- accessible transportation  technology research. A lot of  material  on characteristics, technology solutions,  not just pure  transportation but ways for  dealing with transportation, units  of transportation, dealing with  for a range of  disability categories. And  that is the  last resource  related question. The one topic  we want to hit next  before questions is general next  steps because  [Indiscernible name] is  here. 

Hey guys, I hope everybody is  doing well and just wanted to talk  a little bit about the upcoming  city benefits before I do that I  want  to highlight what candidates talking  about in terms of technical assistance  resources. There might be people  in your city there somebody  that in your city that's just  really focused on connected vehicles  or somebody might be outside of  your  team, this is an opportunity for  you to connect folks across your  network to  these various resources and communities  of practice. So this is  less about getting your information  for your proposal and more about  helping you get list  over all on terms of the  smartest of your city. If not  limited to just the people writing  your proposals I hope you will think  about being the resource of resources  for all those folks around your  city and your partners. A couple  of things on next steps for  city business. I don't have anything  on the slide but just  some clarification. The secretary  is coming, I think you all have  your scheduled dates  and times, it's a two-hour visit,  the first  30 minutes will be that the mayor  and the secretary. We are open to  the mayor having  another -- somebody asked the mayor to bring  the governor, yes, we want to keep  this a very  small meeting we would encourage  you not to open it up to your private  partners, keep  it governmental body can be just  a one-on-one meeting the mayor and  the secretary but we do need to  see who is in the meeting. Let us  know who's going to be there beyond  the mayor. Two is  the Roundtable, we will give you  an invitation we have to see  the list. I know some of you have  unique circumstances we talked about  about how we will configure  that Roundtable but we still need  to have a list  from you for our counsel review.  This is just a standard operating  procedure we use that the  secretary the. We ask that you  submit that I think I sent a spreadsheet  for you to put that into it if  it's too hard  to find, I will take it in an email,  and is to be name, title, organization,  and then once our counsel review  that, we will give you  the invite to share and manage all  that and it will get a final RSVP  list from you ahead  of that. If anybody wants to talk  off-line about your individual circumstances,  I'm happy to do that. The third  piece will be a press -- or  of a gaggle  pull aside, it's going  to be open press but the press won't  be allowed to ask any questions  in that environment.  So we will have a time set aside  for press afterwards for the mayor  and the secretary for  any press questions but not a formal  press conference style. In terms  of driving press to  your event, either done or Andy  will be in touch with  you folks in the  mayor's office.  Questions, comments, and love you  guys are all sending me your --  I don't have too many list so we  would be good to have that. So we  can get those  vetted. 


Thoughts about any  of this? The city  visits, evaluations, -- I have one  more thing after we talk all this  through and then remind me to come  back. 

Folks, the phone lines  are open. If you would like to speak  you may have to take yourself  off mute? 

Let's just check  in. 

Anyone from Pittsburgh  on?  >> Rob is here as well as the  rest of my team. 

Are you guys done with  jet black? 

No. I'm still jet black  right now. That was very excellent,  thank you for hosting that. I think  we are on  schedule here. We have to run our  list of participants by  the mayor and then we will get it  to you as soon as  possible. 

Any questions on the evaluation? 


Just as a  reminder folks, just tell me in  your chatbox if you're going to  send me your stuff. Kansas City,  how are you guys  doing? 

San  Francisco? 

We are doing good. Where  just working through the multiple  [Indiscernible] we have heard from  you guys that may eight was actually  confirmed so we can start preparing  a  list. 

When I go back to my desk, I  will send you a new  email. 


You guys doing  okay? 

I can't  hear you.  [Laughter]. 


What happened to everyone?  I know you guys are on. 

I'm here, it's Catherine, we  are doing good. 

Can you guys send me your  list? 

We will have some more eyes  2.-- 

We will send it to you. 

With that  Portland? 

Kansas City? Have  we heard  from Austin? Who am I missing? 


Randy? Who is  on? 

I know you guys are  out  there. 

Katie, are you  still  on? 

 Okay, send  me your list  to out. Real  quick on June  7, 8 we are doing oral presentation  on June 7 and eight, I think you  got the amendments, you want to  highlight was in the  amendment? 

In our amendment one we just  tried to give a framework of the  plans for orals, the two components  being technical teams with the 45  minute presentation on the topics  prescribed  in the NOFO and the second component  is the mayor's pitch  . 

45 minutes of presentation and  time with the Q&A with the technical  review panel. That presentation  is just your team in  the room with the technical  review panel, not everyone listening  to everyone else's oral  presentations so it will be a 45  minute presentation, you have to  take all of it, if you only want  to present for 50 minutes, that's  fine. And then  open Q&A so they can get a  clarification or more of a  conversation. For 30 minutes  after the  45 minutes. We anticipate having  for presentations on the 7th  and three on  the 8th we get a schedule out. We  will  set up a camp for you guys so you  have your individual rooms and you  can Out in your future headquarters  and then on  the 8th there will be a public event  where mayors will make a  pitch that also include  the video and  the  3D representation. Autodesk  or other. C gas can start thinking  about that as well. I know you're  still writing so you can't put it  all together but as you thinking  about schedule, the time on the  8thafternoon of  the  is TBD  and we would like all the players  to be here but if the mayor cannot  be here,  he's designee, and the location  is also still on  that TBD, it will be  here  in Washington but you are still  exploring where it will take place.  We will try to live stream those  pitches and open  them up to a live audience and open  press as well. The  45 minutes and 30  minute presentation is just you,  your team by herself any  technical panel and the public event  on the  8th. 

The oral presentation part one  just as a reminder you can have  up to six people presenting, you  have others in the room a maximum  of six people presenting. For  part two the mayor's page, we  have indicated the mayor and up  to two other personnel as  well. 

That a  shorter pitch. The mayor's pitch  is a 10 minute maximum to  include a two minute  video. 

Any questions  on that? The amendment went out on The Smart  City Challenge  so if you have not  seen it, send us a note and we will  resend it. 

It was issued on the 18th and  I sent it directly to  all finalists. Location wise  to keep in mind the technical presentations  are anticipated to be in the media  center here at U.S. DOT.  

Parts to mayor's pitch  is TBD but in the  DC  area. 

When are you anticipating to  give up a schedule of the city's? 

We are working  on it, I can't answer that question  because I don't know  the answer. We will  try to get it to you as soon as  we  can. 


Other  questions? I got  excited? -- Are you  guys  excited 


There was one  more question. What are the relationships between  the elements and  this  challenge?  >> The ITS architecture   is a that interoperability for our  information and  data across ultimately all transportation  systems. For the smart city were  moving into some data elements that  go beyond transportation so they  may not fit exactly into the ITS  architecture  that you  will need to have a plan  for that. All the elements are addressed  by the  architecture but the transportation  is you have a framework for all  your information will be interoperable  and how you will be able to share  data. Anybody else from DOT [Indiscernible  - too far from  mic]. 

There's also the elements that  part of this  effort is also to be  able to feed updates and enhancements  to the architecture, because there  are new element to the architecture  and  the CVRA that the smart city challenge  will address in the course of business  so will be great to get that input  in terms of data flows. 

Ultimately  were developing modifying standards  and work with industry to change  status all the time that this will  lead to an  automated vehicle , there are very few interoperability  standards. 

One of the things they had anticipated  when this idea putting glimmer  and I  is that to really be able to begin  to thoroughly explore the data exchanges  between the transportation sector  and other sectors in the  city's economy. To  and from. The ability to get some  visibility into does bidirectional  data flows between transport  and other is going to be invaluable  going forward in terms of creating  space and architecture for all that.  The back  help? -- Does  that help? Phone lines  are  open. 

She said it  does. 

Other questions,  comments,  clarifications? 


Somebody  is typing. We can hear  you typing. You can  just  talk. 

Thank you for your participation  today and certainly continue to  submit any questions you  have to The Smart City  Challenge 

On the technical committees you  expect the mayor to be in that technical  committee or -- 

Is totally up  to you. Right, the  mayor and the  technical committee is up  to them. We would like for them  to do  the pitch. We hope that they would  come from the public pitch and they  are welcome, that's really  your call whether you want to have  them in the technical pitch. 

Of the technical pitch is really  do not thimbles so you're looking  for  -- 

Thank you. 

Other questions? You  guys know where to find out,  don't  hesitate to send questions, I  will look for  your lists, we will get the final  schedule out to everyone so they  can have it again for travel and  we will talk to guys soon. Thank  you. 

The dots complete our conference  for today. Thank you  for  your participation. You may now  disconnect. 

 [Event concluded]