Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

TimeItemWhoNotes
 Welcome 
  • Brief XAB introductions
 Previous summaryKarin Remington
  • ACTION XAB members review summary and let Karin know any objections
  • Audience includes other XAB members not in attendance or for historical documentation as well as an official report NSF might see about the role of the XAB to XSEDE
 Impact Metrics: Project-Wide KPIs and MetricsJohn Towns
  • Taking a top-down view of the XSEDE KPIs and metrics, similar to the ROI analysis led by Craig Stewart
  • NSF reviews have been pushing science impact metrics despite their difficulty in measuring
  • What are other mechanisms by which we could measure XSEDE's impact?
  • Cliff - reviewed document quantifying XSEDE's work. Context: why XSEDE award was made, it is not curiosity-driven. XSEDE is an agent carrying out NSF's mission. Document gets very self-focused rather than helping NSF carry out its mission. XSEDE doesn't need to advance science, NSF does. XSEDE can facilitate advancing science. XSEDE doesn't fund the research but the infrastructure. Would be nice to see narrative about important break-throughs, relative ranking of a subjective view of how NSF is advancing its mission
  • John - NSF has encouraged the use of anecdotal stories, which is not a numerical metric. Collectively the project has worked very hard to strike a balance between the numbers and the stories including the annual highlights book. Most of the metrics are bubbling up and have an operational aspect. The metrics are not getting at the heart of the matter: the science we are impacting and the role XSEDE is playing. Panel isn't dissatisfied with the metrics we have. It is a lack of broader indicators of how the role XSEDE has has an impact.
  • Cliff - strategic plan and fundamental mission of NSF as an agent
  • Tom - scientist will give success stories based upon research and not the broader impacts on workforce development. Perhaps we need to ask more pointed questions of the researchers about
    • Metrics in different areas, stories in different areas, too
  • Rama - demand for usage in the cloud but a lack of skilled staff able to leverage. Is that something XSEDE should take on or be included in the charter? Adopt the open-source software and how to exploit usage. Huge demand for skills right now and XSEDE might be able to help.
  • John - XSEDE has a role to play, structural and cultural challenges in the community from which XSEDE staff come from. Many are resistant, have been moving in that direction with many of the newer resources. Story is good, but what NSF wants to know is the impact that has on the domains and research–which is not specific to the resources. Illustrate at a high level successes, but we need to articulate the role XSEDE has played in that success.
  • Rama - help scientists rapidly adopt to get ahead of the curve
  • Cliff - it is unfair for NSF to ask you to produce metrics that are different than it has as an agency
  • John - always ask NSF for their metrics, and they have nothing
  • Cliff - integrative ability of XSEDE to bring so much activity together in a synergistic way. Whether it be partners or how different partners work together to enhance the environment of how researchers carry out their research.
  • John - yes, how do we articulate that at a higher level? Are there other top-down metrics we are missing?
  • Cliff - how many elements a year or other metrics were integrated to accomplish that? It wasn't just providing cycles. Knowing how scientists do their work and their outcomes it would be go to show the integrative nature by showing how many areas enabled that research
  • Karin - being able to point to collaborative teams that came together to make the work happen. It does appear that NSF is passing the buck, but we can still provide information to potentially improve their game as well, as an opportunity for shared best practices
    • DOE panel for measuring process. This is difficult to do. Each lab has found a way to measure it that is meaningful: collaborations that are launched, patents, etc.
  • Rama - metrics defined by the scientists who are using XSEDE in a way that quantifies their usage. Having the time, expertise, and the value they see XSEDE brings
  • Karin - agree, what data systems track allocations
  • John - extensive data: request, submission, usage of which resources. Oriented toward operations. We don't know what the impact is on that particular domain or across multiple domains. Do interview PIs to provide rating impact level, not just efficiencies but the scientific impact. Interesting data. We are being asked for a more holistic view of how XSEDE is impacting science as a whole. XSEDE is being seen as a pioneer; they understand XSEDE is having a strong impact. Would like this to be a collaborative activity between XSEDE and NSF.
  • Phil - approach could be to build a case for a measure by noting: the world is competing for people's attention; scientists act on their own best interest, and they are doing their work. If XSEDE holds their attention, we are holding their attention. Count minutes people spend at conferences, logged in, using resources, etc.
  • Cliff - ask users what the important measures they think are important about XSEDE ACTION Emre take to UAC
  • Cliff - overwhelming how many metrics, could help to prioritize if a picture becomes clear
  • Karin - could questions be asked at the end of their allocation process?
  • John - we'll leave that to staff to consider implementation
  • Ralph - we ask users who are coming back for a 2nd allocation what they did the first time. Trouble is we interact with them quite a bit before they have the publication; recent work is not reflected in that metric. Our experience with sending post-allocation surveys has a terrible return. They time to ask them is when they are asking for something more.
  • Craig - NSF asks XSEDE to be an exemplar of a distributed organization. We are in the lead in terms of understanding our ROI as a cyberinfrastructure organization. We can provide a leadership position to get at the root of the impact metric
  • Ralph - we have been collecting data: when ECSS interviews PIs to estimate how much time it would get you to be where you are without ECSS support. 3 months of ECSS time: 15 months of the entire project.
  • Karin - close out process
  • Cliff - ask researchers to share their report
  • Karin - NSF could require additional input on XSEDE

Presentation Materials

Action items

...