Skip to end of metadata
Go to start of metadata

Decisions:

N/A

Action Items:

N/A

Notes:

  • Additional comments made in zoom chat during the meeting. Topics include training, ECSS, visualizing these data (other topics as well).  See chat below notes section.
  • Add to the objectives on first/second slide - maintain or improve the quality of KPIs in addition to reducing the quantity. 
  • John T and Bob C (NSF) to review the IPR appendix for metrics as well to determine what should be elevated, maintained, or removed.
  • Issues with how the KPIs are displayed in summary form. Need to rethink the visualizations/tables.
  • New/sustained URMs metrics/KPI should not be as repetitive as it is in the current IPR - Linda. Lizanne - this would only reported and described at the highest level. There may be other metrics at the L3 level that get described in the body of the report. (Follow up - once Linda saw the L2 metrics happy with the changes and not concerned with repetition anymore)
  • Dave/Karla - looks like existing communities are not well represented. For example only one sustained metric is under "deepen use" white the other 2 are under extend use. 
  • Ensure "sustained" means the same across the different L2 areas and metrics.
  • John T. - lose sense of impact by only measuring counts (i.e. number of new, number of sustained etc.). Lizanne - many qualitative impact measures exist within the broader portfolio but were not elevated/maintained. Can revisit these. 
  • Hours of live training delivered by XSEDE staff - change to hours received (by attendees). Lots of nuances discussed in the zoom chat. Challenges include not knowing how long people have stayed during the training. 
  • "XCI - total number of capabilities deployed for use in production." John would like this to change to "total number of capabilities in production."
  • (From chat) Kandace - I want to comment that on the last slide where raising awareness says it will be a composite of media measures, we need to consider that efforts to raise awareness extend beyond ER and media efforts. For this to be accurate we should consider all efforts across the project that raise awareness.

  • John C. - would like to see ECSS show up in "provide excellent user support." Was not visible at the project level. L2 KPIs cover deep use to existing communities only.
  • John T. - hard for our community to adopt what looks like an insufficient/incomplete measure when not all areas are represented for a particular goal/subgoal. What we need, however, are indicators. This is difficult because we want to make an exact, correct, complete picture but a KPI is none of those things. Lizanne - there is a rich array of other metrics with existing longitudinal data that can be used to expand/enrich these. This is not to be a full blown picture but a way to take a temperature to see if the project is healthy. 
  • John T. - would it be odd to ask the review panel to rate us on the KPIs? Lizanne - have seen this before but needs to be very structured and they need to be provided with data. Dave L. - need to frame it around a review of the process and KPI. Don't make it a rating of a rating. Lizanne - consider doing this with the XAB instead since they know the project really well. 
  • (from chat) Dave H - Just a quibble: For Ops, why the “user satisfaction with XOC-closed” tickets isn’t linked to “Provide Excellent User Support” subgoal. Scott - 
  • It should be. I think that is a typo. We have it as supporting Provide Excellent User Support on on our end
  • Project is required to present metrics at the mid-year review.

 

Zoom Chat text:

10:18:55 From Marlon Pierce : Shouldn’t “Extend use (new communities)” also be an ECSS area? NIP
10:19:15 From Karla Gendler : Yes. I think there may be some confusion. We’ll have to wait until the next few slides.
10:19:47 From Lorna Rivera : ya. the number 5 in that column is for the project as a whole. we’re trying not to repeat them at the L2 level but that’s definitely up for discussion
10:23:44 From John Cazes : wouldn’t ecss be in sustain the ecosystem? we don’t just help new users
10:25:39 From Lorna Rivera : ya the table is confusing. You’ll notice in these slides now that some of the project KPIs are owned by L2 areas though that wasn’t so clear on the last table. hopefully this helps. we definitely welcome feedback on the measures and also on how to display the data
10:25:40 From Susan Mehringer : is hours of live training delivered proposed by “hours presented” or “hours attended”?
10:26:17 From Jay Alameda : I think presented, but I'm not sure. This one is not easy to deal with
10:26:45 From Lorna Rivera : we’ve been thinking about it as presented but John commented yesterday on attended. should discuss more
10:27:35 From Jay Alameda : presented would be hard, attended, we are not at all set up to deal with that
10:27:45 From Lorna Rivera : good feedback
10:28:05 From Karla Gendler : I think we talked about this in the CEE discussion and both of those measures were stated that it would be hard to gather
10:28:33 From Jay Alameda : that is consistent with my view from the ESTEO world :)
10:30:33 From John Towns : The question from me is: what would it take to be able to collect the necessary data?
10:33:38 From Susan Mehringer : Collecting hours of training delivered would be easy (I think). Collecting hours of training attended would be very difficult; I’m having a tough time getting remote event moderators to even collect a sign-in sheet… I think to collect actual attendee hours, we’d need to collect sign-in & sign-out.
10:35:23 From Victor : Is there an XSEDE mobile website or phone app, could add the check in, check out to that. Maytal might have some ideas
10:36:11 From Jay Alameda : I concur with Susan - and Victor, we don't really have a training platform that folks would have to log into (and out of) at the beginning and/or end of training. the training calendar is not this, of course.
10:36:53 From Susan Mehringer : It can be difficult to get people to self-report
10:37:23 From Jay Alameda : difficult to impossible, I think.
10:37:36 From Victor : If we make it easy you have a much better chance to get people to do it. There is always a few people who dont want to. Can't fix that, but can make it easier
10:38:35 From Victor : I think perhaps the portal has mobile capabilities. Should explore like Jay said a "training platform" that might make this easy.
10:38:36 From Jay Alameda : I've done some experiments in ECSS, making self-reporting easy (really easy) and it doesn't matter. something like 1-2% report. I have to count by hand every time.
10:39:10 From Jay Alameda : need a carrot, need folks to need something on the platform that they can't live without. that makes it easier.
10:39:17 From Lorna Rivera : right
10:40:17 From Victor : If the instructor view of the mobile website could see the registrations, then the class could be held up until check ins are at least some percentage complete
10:46:23 From Kandace Turner : Since we're short on time and I can't comment, I want to comment that on the last slide where raising awareness says it will be a composite of media measures, we need to consider that efforts to raise awareness extend beyond ER and media efforts. For this to be accurate we should consider all efforts across the project that raise awareness.
10:46:44 From Lorna Rivera : thanks Kandace! i’ll add to notes
10:49:12 From Kandace Turner : Thanks
10:49:26 From Lorna Rivera : np ;)
10:50:31 From Ester Soriano : What is CEE?
10:50:37 From Lorna Rivera : used to be TEOS
10:50:45 From Lorna Rivera : community engagement and enrichment
10:50:51 From Ester Soriano : okay, thanks!
10:50:55 From Lorna Rivera : ;)
10:51:02 From Jay Alameda : there will be a quiz :)
10:51:19 From Ester Soriano : Hahaha - I won’t participate. I am the newbie here
10:51:20 From Karla Gendler : It a combination of UE (user engagement) and TEOS from XSEDE1
10:51:33 From Lorna Rivera : thanks Karla!
10:59:34 From David Hart : Just a quibble: For Ops, why the “user satisfaction with XOC-closed” tickets isn’t linked to “Provide Excellent User Support” subgoal.
11:02:15 From Lorna Rivera : good point
11:02:18 From Lorna Rivera : i’ll add to notes
11:03:18 From Lonnie Davis Crosby : It should be. I think that is a typo. We have it as supporting Provide Excellent User Support on on our end - Scott
11:04:57 From Lorna Rivera : thanks Scott!
11:05:23 From Lonnie Davis Crosby : ;)

 


 

 

 

 

  • No labels