Child pages
  • XSEDE Project-wide KPIs & Metrics

Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

KPI

Program Year

Target

RP1

RP2

RP3

RP4

Total

Owner(s)

Average ECSS impact rating

RY5

      

ECSS (§4)

RY4

      

RY3

      

RY2

4 out of 5 / qtr4.114.03   

RY1

4 out of 5 / qtr

*4.564.61

3.29

4.14

 

Definition/Description: After an ECSS project is marked as completed (i.e. has a workplan, work has progressed, and a final report has been filed), the L2 Directors interview the PIs, preferably via phone, and ask them to rate the impact of the ECSS support on their research on a scale of 1 to 5.

Collection methodology: PIs are contacted by the L2 Directors post filing of a final report by ECSS staff members. The L2 Director asks the PI to rate the impact of the ECSS support on their research on a scale of one to five and this number is recorded in a spreadsheet which is shared with the L2 Area Project Manager (PM). The PM then transfers this number to Sciforma. Each individual impact rating that is reported for the reporting period is then added together and divided by the total number of interviews conducted for the average.

Average satisfaction with ECSS supportRY5      ECSS (§4)
RY4      
RY3      
RY24.5 out of 5 / qtr4.654.56   
RY14.5 out of 5 / qtr*4.864.72

4.64

4.54

 

Definition/Description: After an ECSS project is marked as completed (i.e. has a workplan, work has progressed, and a final report has been filed), the L2 Directors interview the PIs, preferably via phone, and ask them to rate their satisfaction with the ECSS support they have received on a scale of 1 to 5.

Collection methodology: PIs are contacted by the L2 Directors post filing of a final report by ECSS staff members. The L2 Director asks the PI to rate their satisfaction with the support received on a scale of one to five and this number is recorded in a spreadsheet which is shared with the L2 Area Project Manager (PM). The PM then transfers this number to Sciforma. Each individual satisfaction rating that is reported for the reporting period is then added together and divided by the total number of interviews conducted for the average.

Number of completed ECSS projects

RY5

      

ECSS (§4)

RY4

      

RY3

      

RY2

50 / yr169   

RY1

50 / yr

*1013

25

48

 

Definition/Description: The total number of completed ECSS projects consists of projects that were completed through work in ESRT, ESCC, and ESSGW. A completed project is defined as a project that has progressed through the complete support pipeline.  This pipeline includes steps such as assignment of a consultant, production of a work plan, execution of the work plan and reporting of progress through quarterly reports, and the filing of a final project report.

Collection methodology: The number of completed projects is tracked in Sciforma, XSEDE’s project management software. A report has been created in Sciforma that queries all the ECSS projects, filters for allocations that have ended in that quarter and also for projects that have a final report.

...

KPI

Program Year

Target

RP1

RP2

RP3

RP4

Total

Owner(s)

Number of new users from underrepresented

communities and non-traditional disciplines of XSEDE resources and services

RY 5

      

ECSS – NIP (§4.3)


CEE - Broadening Participation (§3.4)

RY 4

      

RY 3

      

RY 2

1,100 / yr352274   

RY 1

> 200 / yr

*297227

398

922

 

Definition/Description: This KPI tracks progress in extending participation in XSEDE by first-time users from new communities.   

Collection methodology: This is the sum of the number of new users from underrepresented communities measured by Community Engagement and Enrichment - WBS 2.1 and the number of new users from non-traditional disciplines measured by Novel and Innovative Projects - WBS 2.2.3 

Number of sustained users from underrepresented

communities and non-traditional disciplines of XSEDE resources and services

RY 5

      

ECSS – NIP (§4.3)


CEE - Broadening Participation (§3.4)

RY 4

      

RY 3

      

RY 2

3,500 / yr1,1901,117   

RY 1

2,600 / yr1

*773755

1,251

2,779

 

Definition/Description: This KPI tracks progress in extending participation in XSEDE by persistent engagement of users from new communities. 

Collection methodology: This is the sum of the number of persistent users from underrepresented communities measured by Community Engagement and Enrichment - WBS 2.1 and the number of persistent users from non-traditional disciplines measured by Novel and Innovative Projects - WBS 2.2.3. 

...

KPI

Program Year

Target

RP1

RP2

RP3

RP4

Total

Owner(s)

Number of attendees in synchronous and

asynchronous training

RY 5

      

CEE - Workforce Development (§3.2)

RY 4

      

RY 3

      

RY 2

5,200 / yr1,305890   

RY 1

6,000 / yr1

*1,3041,804

1,264

3,652

 

Definition/Description: We will report number of attendees in two categories:

  • The total number of attendees at synchronous events (in-person and webinar), where an attendee is any XSEDE User Portal registrant who attends the event for any length of time.  For an in-person workshop, attendees are those who physically attend, and for webinars, attendees are those who sign in with their XSEDE User Portal username; and

  • The number of classes taken by all attendees, where an attendee is anyone who loaded page(s) in an online training module.  Regardless of activity level, an attendee will be counted as taking a course no more than once in a given quarter.  Only attendees who signed in with an XUP username are count.

Collection methodology:

    • For synchronous events, information about attendees (XUP username, date, and course) who register and participate using their XSEDE User Portal username are recorded and uploaded to a central metrics database via the course coordinator.   Walk-ins are not counted.

    • For asynchronous events, attendee information (XUP username, date, and course) is pulled from CI-Tutor and Cornell Virtual Workshop databases and uploaded to the XSEDE User Portal database by the training lead and is uploaded to a central metrics database. Visiting pages in one asynchronous module multiple times during a quarter count as one participant. If this person visits the course again the next quarter, it is counted as a new class that quarter, as it’s the equivalent of a person attending the same course/event a second time.

Average impact assessment of training for attendees registered through the XSEDE User Portal

RY 5

      

CEE - Workforce Development (§3.2)

RY 4

      

RY 3

      

RY 2

4 out of 5 / qtr4.294.34   

RY 1

4 out of 5 / qtr

*4.544.39

4.28

4.36

 

Definition/Description: This is the average of all attendees’ self-rating of each event in which they attend, on a scale of 1 to 5.

Collection methodology: The data is collected from post event surveys and recorded in the database. Specific items included in this index are:

    • Q1. The training session fulfilled my expectations.

    • Q2. The trainer stimulated my interest.

    • Q6. The training session was well-organized.

    • Q8. I was able to easily access this training session.

    • Q10. Overall I would rate my experience as successful.

...

KPI

Program Year

Target

RP1

RP2

RP3

RP4

Total

Owner(s)

Number of pageviews to the XSEDE website

RY 5

      

Community Engagement and Enrichment - UII (§3.5)

RY 4

      

RY 3

      

RY 2

80,000 / qtr63,998NA1   

RY 1

80,000 / qtr

*49,40965,157

68,227

183,212

 

Definition/Description: Per Google Analytics, “a pageview is defined as a view of a page on your site that is being tracked by the Analytics tracking code. If a user clicks reload after reaching the page, this is counted as an additional pageview. If a user navigates to a different page and then returns to the original page, a second pageview is recorded as well.”

Collection methodology: This is the number reported in the Google Analytics tracking software for the given time period.

Number of pageviews to the XSEDE User Portal

RY 5

      

Community Engagement and Enrichment - UII (§3.5)

RY 4

      

RY 3

      

RY 2

250,000 / qtr256,482282,722   

RY 1

100,000 / qtr

*183,408219,644

265,151

670,080

 

Definition/Description: Per Google Analytics, “a pageview is defined as a view of a page on your site that is being tracked by the Analytics tracking code. If a user clicks reload after reaching the page, this is counted as an additional pageview. If a user navigates to a different page and then returns to the original page, a second pageview is recorded as well.”

Collection methodology: This is the number reported in the Google Analytics tracking software for the given time period.

Number of Social Media impressions

RY 5

      

Program Office - ER (§8.2)

RY 4

      

RY 3

      

RY 2

300,00 / yr69,60755,506   

RY 1

190,000 / yr

*52,200128,675

81,332

262,207

 

Definition/Description: Number of people who have seen XSEDE interactions on Facebook + Twitter. Both Facebook and Twitter count “impressions,” which is essentially how many people saw a certain post. A user of FB or Twitter no longer has to directly “like” or “retweet” a post to get this number - “impressions” would refer to people who see the post based on their friends or followers sharing our information. It is a better way of collecting social media awareness than just “followers” or “shares.”

Collection methodology: Facebook and Twitter both have internal methods of tracking metrics - we simply go to those sites and find the “impressions” numbers we need at the end of each reporting cycle. Anyone with access to the FB and Twitter pages can easily gather this information.

Number of media hits

RY 5

      

Program Office - ER (§8.2)

RY 4

      

RY 3

      

RY 2

169 / yr4230   

RY 1

140 / yr

*3230

18

80

 

Definition/Description: This is the number of XSEDE-related stories we find in the media, many times through Google alerts for “XSEDE,” that mentions XSEDE by name. Often, we manually search “XSEDE,” as well, in case the daily alert email is missed.  

Collection methodology: NCSA tracks media hits for both NCSA and XSEDE, so that is a baseline collection of numbers. The ER team can then do additional manual searching - as an example, oftentimes the hits of HPCwire are not collected on the NCSA page, so an additional manual count is needed.

...

KPI

Program Year

Target

RP1

RP2

RP3

RP4

Total

Owner(s)

Number of new capabilities made

available for production deployment

RY 5

      

XCI (§5)

RY 4

      

RY 3

      

RY 2

7 / yr12   

RY 1

7 / yr

*00

6

6

 

Definition/Description: New use cases that have been fully enabled using new or enhanced components and made available by RACD completed CDP activities.  

Collection methodology: Count is done manually and maintained in JIRA.

Average satisfaction rating of

XCI services

RY 5

      

XCI (§5)

RY 4

      

RY 3

      

RY 2

4.7 out of 5 / yr4.254.5   

RY 1

4 out of 5 / yr

*4.84.81

4.5

4.5

 

Definition/Description: Customer satisfaction rating of XCI staff provided services (does not include software services which are covered in RACD metrics)

Collection methodology: Customer satisfaction survey(s).

...

KPI

Program Year

Target

RP1

RP2

RP3

RP4

Total

Owner(s)

Average rating of staff regarding how

well-prepared they feel to perform their jobs

RY 5

      

Program Office - Strategic Planning, Policy & Evaluation (§8.5)

RY 4

      

RY 3

      

RY 2

4 out of 5 / yr-3.4   

RY 1

4 out of 5 / yr

*--

-

-

 

Definition/Description: An annual Staff Climate Study is a survey administered and analyzed by an external evaluation team as part of an effort to understand whether XSEDE staff feel adequately prepared to conduct their XSEDE work. The survey study includes items regarding staff training.

Collection methodology: Respondents rate closed-ended Likert scale items regarding training on scale of 1 (strongly disagree) to 5 (strongly agree). The average of all staff responses for a given year is calculated to determine this metric. The specific item used to calculate this metric is Q2m: “I have access to adequate training to conduct my XSEDE-related work.”

...

KPI

Program Year

Target

RP1

RP2

RP3

RP4

Total

Owner(s)

Percentage of recommendations addressed

by relevant project areas

RY 5

      

Program Office - Strategic Planning, Policy & Evaluation (§8.5)

RY 4

      

RY 3

      

RY 2

90% / qtr2315   

RY 1

90% / qtr

*NA100

57

67

 

Definition/Description: The SP&E team will calculate the metric based off the data from two measurements; total key recommendations made and total key recommendations addressed. Total key recommendations addressed will be divided by the total key recommendations made at the point in time of data capture (at the end of the IPR reporting period).

Collection methodology: The number of key recommendations made according to the annual XSEDE Climate Study findings + the total number of recommendations recorded on the XSEDE Project-Wide Improvements & Recommendations Google Sheet, such as XAB, NSF, and SP&E recommendations. XSEDE staff, not the SP&E team, are accountable for entering information to track improvements and recommendations.

NOTE: An improvement made is not necessarily a recommendation addressed; but a recommendation addressed is an improvement made and therefore should also be tracked in the XSEDE Project-Wide Improvements & Recommendations Google Sheet once fully implemented; it is OK to have duplication between Improvements fully implemented and Recommendations addressed.

...

Area Metric

Program Year

Target

RP1

RP2

RP3

RP4

Total

 Goal Supported

Number of new users of XSEDE

resources and services

RY 5

      

Deepen/Extend – Extend use to new communities

RY 4

      

RY 3

      

RY 2

2,000 / qtr2,3052,813   

RY 1

> 1,000 / qtr

*1,97311,8491

 2,359

6,181 

 

Definition/Description: This is the number of portal accounts created within the selected time period

Collection methodology: This is the number of portal accounts created with Liferay. This is queried in the Liferay database by looking in the Users table and seeing how many accounts have a timestamp in the created field between the selected dates.

Number of sustained users of XSEDE resources and services

RY 5

      

Deepen/Extend – Deepen use to existing communities

RY 4

      

RY 3

      

RY 2

3,000 / qtr3,9623,754   

RY 1

> 5,000 / qtr

*4,7554,4461

4,924 

6,186 

 

Definition/Description: This is the number of active users logged into the XSEDE User Portal within the selected time period.

Collection methodology: There is a usage log parsing script that collects the tomcat logs and parses out the relevant data during the selected time period.

Number of new users from underrepresented

communities using XSEDE resources and services

(KPI - composite)


RY 5

      

Deepen/Extend – Extend use to new communities

RY 4

      

RY 3

      

RY 2

150/ qtr251175   

RY 1

> 100 / qtr

*150135

240 

525

 

Definition/Description:First-time awardees of XSEDE compute resources or an “allocation” (i.e. Startup, Research, Education, etc.) from underrepresented communities including women and racial/ethnic domestic minorities in HPC as well as anyone from a Minority Serving Institution (MSI) as defined by the Carnegie Classification of Institutions of Higher Education are tracked.

Collection methodology: XSEDE portal users are able to specify their race/ethnicity, gender, and institution in their user profile and are encouraged to complete this data when registering for training events through the XSEDE portal. Date of allocation awards is also recorded in the XSEDE database (XDCDB). A query is then run to generate the number of individuals meeting the aforementioned criteria within a given quarter. Care is taken to not over represent this metric by ensuring an individual who meets multiple criteria is only accounted for one time.

Number of sustained users from underrepresented

communities using XSEDE resources and services

(KPI - composite)

RY 5

      

Deepen/Extend – Deepen use to existing communities

RY 4

      

RY 3

      

RY 2

1,500 / yr490408   

RY 1

> 1,000 / yr

*322238

535

1,095

 

Definition/Description: Awardees of XSEDE compute resources or an “allocation” (i.e. Startup, Research, Education, etc.) from underrepresented communities including women and racial/ethnic domestic minorities in HPC as well as anyone from a Minority Serving Institution (MSI) as defined by the Carnegie Classification of Institutions of Higher Education are tracked that access their “allocation” during the time period.

Collection methodology: XSEDE portal users are able to specify their race/ethnicity, gender, and institution in their user profile and are encouraged to complete this data when registering for training events through the XSEDE portal. Date of allocation awards is also recorded in the XSEDE database (XDCDB). A query is then run to generate the number of individuals meeting the aforementioned criteria within a given quarter. Care is taken to not over represent this metric by ensuring an individual who meets multiple criteria is only accounted for one time.

Number of attendees in synchronous and

asynchronous training

RY 5

      

Deepen/Extend – Prepare the current and next generation

RY 4

      

RY 3

      

RY 2

5,200 / yr1,305890   

RY 1

> 6,000 / yr2

*1,3041,084

 1,264

3,652 

 

Definition/Description:

  • We will report number of attendees in two categories:

    • The total number of attendees at synchronous events (in-person and webinar), where an attendee is any XSEDE User Portal registrant who attends the event for any length of time.  For an in-person workshop, attendees are those who physically attend, and for webinars, attendees are those who sign in with their XSEDE User Portal username; and

    • The number of classes taken by all attendees, where an attendee is anyone who loaded page(s) in an online training module.  Regardless of activity level, an attendee will be counted as taking a course no more than once in a given quarter.  Only attendees who signed in with an XUP username are counted

Collection methodology:

    • For synchronous events, information about attendees (XUP username, date, and course) who register and participate using their XSEDE User Portal username are recorded and uploaded to a central metrics database via the course coordinator.   Walk-ins are not counted.

    • For asynchronous events, attendee information (XUP username, date, and course) is pulled from CI-Tutor and Cornell Virtual Workshop databases and uploaded to the XSEDE User Portal database by the training lead and is uploaded to a central metrics database. Visiting pages in one asynchronous module multiple times during a quarter count as one participant. If this person visits the course again the next quarter, it is counted as a new class that quarter, as it’s the equivalent of a person attending the same course/event a second time.


Average impact assessment of training for attendees registered through XSEDE User Portal

RY 5

      

Deepen/Extend – Prepare the current and next generation

RY 4

      

RY 3

      

RY 2

4 out of 5 / qtr4.294.34   

RY 1

4 out of 5 / qtr

*4.544.39

4.28 

4.36

 

Definition/Description: This is the average of all attendees’ self-rating of each event in which they attend, on a scale of 1 to 5.

Collection methodology: 

  • The data is collected from post event surveys and recorded in the database. Specific items included in this index are:

    • Q1. The training session fulfilled my expectations.

    • Q2. The trainer stimulated my interest.

    • Q6. The training session was well-organized.

    • Q8. I was able to easily access this training session.

    • Q10. Overall I would rate my experience as successful.


Number of pageviews to the XSEDE website

RY 5

      

Deepen/Extend – Raise awareness of the value of advanced digital research services

RY 4

      

RY 3

      

RY 2

80,000 / qtr63,998NA2   

RY 1

80,000 / qtr

*49,40965,157

68,227 

183,212 

 

Definition/Description: Per Google Analytics, “a pageview is defined as a view of a page on your site that is being tracked by the Analytics tracking code. If a user clicks reload after reaching the page, this is counted as an additional pageview. If a user navigates to a different page and then returns to the original page, a second pageview is recorded as well.”

Collection methodology: This is the number reported in the Google Analytics tracking software for the given time period.

Number of pageviews to the XSEDE User Portal

RY 5

      

Deepen/Extend – Raise awareness of the value of advanced digital research services

RY 4

      

RY 3

      

RY 2

250,000 / qtr256,482282,722   

RY 1

100,000 / qtr

*183,408219,664

261,115

670,080 

 

Definition/Description:Per Google Analytics, “a pageview is defined as a view of a page on your site that is being tracked by the Analytics tracking code. If a user clicks reload after reaching the page, this is counted as an additional pageview. If a user navigates to a different page and then returns to the original page, a second pageview is recorded as well.”

Collection methodology: This is the number reported in the Google Analytics tracking software for the given time period.

...

Area Metric

Program Year

Target

RP1

RP2

RP3

RP4

Total

 Goal Supported

Number of unique attendees, synchronous training

RY 5

      

Deepen/Extend – Prepare the current and next generation

RY 4

      

RY 3

      

RY 2

1,200 / yr325438   

RY 1

1,600 / yr

*112371

551 

981

 

Definition/Description: Number of unique attendees at synchronous events (in-person and webinar), where an attendee is any XSEDE User Portal registrant who attends the event for any length of time.  For an in-person workshop, attendees are those who physically attend, and for webinars, attendees are those who sign in with their XSEDE User Portal username.

Collection methodology: Information about attendee (XUP username, date, and course) who register and participate using their XSEDE User Portal username are recorded by the session moderator and uploaded to a central metrics database via the course coordinator. For the annual total, the query is rerun to look at the year as a whole and not just the quarter.  Walk-ins are not counted.

Number of total attendees, synchronous training

(One person can take several classes)

RY 5

      

Deepen/Extend – Prepare the current and next generation

RY 4

      

RY 3

      

RY 2

1,400 / yr921484   

RY 1

2,000 / yr

*188382

 614

1,184

 

Definition/Description: The total number of attendees at synchronous events (in-person and webinar), where an attendee is any XSEDE User Portal registrant who attends the event for any length of time.  For an in-person workshop, attendees are those who physically attend, and for webinars, attendees are those who sign in with their XSEDE User Portal username. 

Collection methodology: Information about attendees (XUP username, date, and course) who register and participate using their XSEDE User Portal username are recorded and uploaded to a central metrics database via the course coordinator.   Walk-ins are not counted.

Number of unique attendees, asynchronous training

RY 5

      

Deepen/Extend – Prepare the current and next generation

RY 4

      

RY 3

      

RY 2

1,200 / yr148170   

RY 1

2,000 / yr

*215213

211 

520

 

Definition/Description: Number of unique attendees using asynchronous materials (CI-Tutor and Cornell Virtual Workshop), where an attendee is any XSEDE User Portal  user logged in who visited at least one page in an online course.

Collection methodology: Attendee information (XUP username, date, and course) is pulled from CI-Tutor and Cornell Virtual Workshop databases and uploaded to the XSEDE User Portal database by the training lead  and is uploaded to a central metrics database, for the time period listed.

Number of total attendees, asynchronous training

(One person can take several classes)


RY 5

      

Deepen/Extend – Prepare the current and next generation

RY 4

      

RY 3

      

RY 2

4,000 / yr384406   

RY 1

4,000 / yr

*1,116702

650 

2,468

 

Definition/Description: The number of classes taken by all attendees, where an attendee is anyone who loaded page(s) in an online training module.  Regardless of activity level, an attendee will be counted as taking a course no more than once in a given quarter.  Only attendees who signed in with an XUP username are counted. 

Collection methodology: Attendee information (XUP username, date, and course) is pulled from CI-Tutor and Cornell Virtual Workshop databases and uploaded to the XSEDE User Portal database by the training lead and is uploaded to a central metrics database. Visiting pages in one asynchronous module multiple times during a quarter count as one participant. If this person visits the course again the next quarter, it is counted as a new class that quarter, as it’s the equivalent of a person attending the same course/event a second time.

Average impact assessment of training for attendees

registered through XSEDE User Portal

RY 5

      

Deepen/Extend – Prepare the current and next generation

RY 4

      

RY 3

      

RY 2

4 out of 5 /qtr4.294.34   

RY 1

4 out of 5 /qtr

*4.544.39

4.28 

4.36

 

Definition/Description: This is the average of all attendees’ self-rating of each event in which they attend, on a scale of 1 to 5.

Collection methodology: This data is collected from post event surveys and recorded in the database. Specific items included in this index are:

    • Q1. The training session fulfilled my expectations.

    • Q2. The trainer stimulated my interest.

    • Q6. The training session was well-organized.

    • Q8. I was able to easily access this training session.

    • Q10. Overall I would rate my experience as successful.

Number of formal degree, minor, and certificate

programs added to the curricula

RY 5

      

Deepen/Extend – Prepare the current and next generation

RY 4

      

RY 3

      

RY 2

3 / yr00   

RY 1

3 / yr

*10

2

 

Definition/Description: The number of degrees, minors, and certificate programs addressing computational and data-enabled science and engineering added to course curricula in higher education institutions.  This will span all disciplines. 

Collection methodology: This is self-reported by these institutions as a result of XSEDE polling them on an annual basis. Pre and post NCSI workshop surveys are sent for each event. A follow up survey approximately 6 months post participation is also conducted. The data is stored in a surveymonkey account. The education lead pulls these numbers from the interim reports that the evaluation team provides based on the survey data.

Number of training materials contributed to public repository

RY 5

      

Deepen/Extend – Prepare the current and next generation

RY 4

      

RY 3

      

RY 2

50/yr010   

RY 1

40 / yr

*1025

 10

45

 

Definition/Description: This is a count of training and education materials that are posted to the HPC University repository (http://www.hpcuniversity.org).

Collection methodology: This data is maintained by the HPC University web site and is extracted quarterly.

Number of materials downloaded from the repository

RY 5

      

Deepen/Extend – Prepare the current and next generation

RY 4

      

RY 3

      

RY 2

62,000 / yr16,54521,340   

RY 1

56,000 / yr

*11,11425,233

25,279 

61,626

 

Definition/Description: This is a count of training and education materials downloaded from the HPC University repository (http://www.hpcuniversity.org).

Collection methodology: This data is maintained by the HPC University web site and is extracted quarterly.

Number of computational science modules added to courses in higher education institutions

RY 5

      

Deepen/Extend – Prepare the current and next generation

RY 4

      

RY 3

      

RY 2

40 / yr18-   

RY 1

40 / yr

*---NA1

 

Definition/Description: The number of courses addressing computational and data-enabled science and engineering modules added to courses in higher education institutions.  This spans all disciplines.

Collection methodology: This is self-reported by these institutions as a result of us polling them on an annual basis.  We poll the faculty and the institutions with whom we are working and have contact over the years.

Number of students benefitting from XSEDE resources and services

RY 5

      

Deepen/Extend – Prepare the current and next generation

RY 4

      

RY 3

      

RY 2

950 / qtr1,7221,478   

RY 1

50 / qtr

*99721,148

2,679

4,824

 

Definition/Description: This counts the total number of students supported by XSEDE to work on XSEDE projects and to attend the annual XSEDE Conference. 

Collection methodology: This data is provided by the organizations receiving student financial support from XSEDE on a quarterly basis.

Percentage of under-represented students benefitting

from XSEDE resources and services

RY 5

      

Deepen/Extend – Prepare the current and next generation

RY 4

      

RY 3

      

RY 2

50% / qtr2827   

RY 1

50% / qtr

*34233

19 

28

 

Definition/Description: Percentage of underrepresented students out of total students supported by XSEDE to work on XSEDE projects and to attend the annual XSEDE/PEARC Conference.  

Collection methodology: This data is provided by the organizations receiving student financial support from XSEDE on a quarterly basis

...

Area Metric

Program Year

Target

RP1

RP2

RP3

RP4

Total

 Goal Supported

Percentage of active and new PIs

contacted quarterly

RY 5

      

Sustain – Provide excellent user support

RY 4

      

RY 3

      

RY 2

100% / qtr

100

(2,040)

100

(2.026)

   

RY 1

100% / qtr

*

100

(1,533)

100

(1,586)

100
(2,062)2

**

 

Definition/Description: The percentage of PIs with an active allocation that are sent an email to identify user issues that need to be addressed (user requirements) and pass the issues on to the appropriate XSEDE group for resolution.

Collection methodology: Members of the User Engagement team send emails, acknowledge user responses, create XSEDE tickets on the user’s behalf, track the issues to resolution, and close the loop with the user. Currently a spreadsheet is maintained that includes the number of PIs contacted and the date the emails were sent. The goal is to use Jira to keep track of this metric.

Percentage of user requirements resolved1

RY 5

      

Sustain – Provide excellent user support

RY 4

      

RY 3

      

RY 2

100% / yr78 (36/44)102 (47/46)   

RY 1

100% / yr1

*

50

(16/32)

89

(40/45)

75
(40/53)

74
(96/130)

 

Definition/Description: The percentage of user requirements that are resolved.

Collection methodology: Currently a spreadsheet is maintained that includes the status of each ticket and we report on the number of tickets that have been resolved in the quarter. The goal is to use Jira to track and report this metric.

...

Area Metric

Program Year

Target

RP1

RP2

RP3

RP4

Total

 Goal Supported

Number of new under-represented individuals

using XSEDE resources and services

(KPI)

RY 5

      

Deepen/Extend – Extend use to new communities

RY 4

      

RY 3

      

RY 2

150 / qtr251175   

RY 1

> 100 / yr

*150135

240

525

 

Definition/Description: First-time awardees of XSEDE compute resources or an “allocation” (i.e. Startup, Research, Education, etc.) from underrepresented communities including women and racial/ethnic domestic minorities in HPC as well as anyone from a Minority Serving Institution (MSI) as defined by the Carnegie Classification of Institutions of Higher Education are tracked.

Collection methodology: XSEDE portal users are able to specify their race/ethnicity, gender, and institution in their user profile and are encouraged to complete this data when registering for training events through the XSEDE portal. Date of allocation awards is also recorded in the XSEDE database (XDCDB). A query is then run to generate the number of individuals meeting the aforementioned criteria within a given quarter. Care is taken to not over represent this metric by ensuring an individual who meets multiple criteria is only accounted for one time.

Number of sustained under-represented individuals

using XSEDE resources and services

(KPI)

RY 5

      

Deepen/Extend – Deepen use to existing communities

RY 4

      

RY 3

      

RY 2

1,500 / yr490408   

RY 1

> 1,000 / yr

*322238

535

1,095

 

Definition/Description: Awardees of XSEDE compute resources or an “allocation” (i.e. Startup, Research, Education, etc.) from underrepresented communities including women and racial/ethnic domestic minorities in HPC as well as anyone from a Minority Serving Institution (MSI) as defined by the Carnegie Classification of Institutions of Higher Education are tracked that access their “allocation” during the time period.

Collection methodology: XSEDE portal users are able to specify their race/ethnicity, gender, and institution in their user profile and are encouraged to complete this data when registering for training events through the XSEDE portal. Date of allocation awards is also recorded in the XSEDE database (XDCDB). A query is then run to generate the number of individuals meeting the aforementioned criteria within a given quarter. Care is taken to not over represent this metric by ensuring an individual who meets multiple criteria is only accounted for one time.

Longitudinal Assessment of Inclusion in XSEDE via the Staff Climate Survey

RY 5

      

Advance – Enhance the array of technical expertise and support services

RY 4

      

RY 3

      

RY 2

5% improvement / yr-4.25   

RY 1

5% improvement / yr

*---

 

Definition/Description: A measure of XSEDE staff perception and experience relating to inclusion in the project.

Collection methodology: A mean index score is generated from annual Staff Climate Study responses to items within the "Inclusion" dimension of the study. This score is then compared to the previous year to determine percent change on an annual basis.

Longitudinal Assessment of Equity in XSEDE via the Staff Climate Survey



       
RY 5      Advance – Enhance the array of technical expertise and support services



RY 4      
RY 3      
RY 25% improvement / yr -

-5

   
RY 15% improvement / yr* ----
 

Definition/Description: A measure of XSEDE staff perception and experience relating to diversity in the project.

Collection methodology: A mean index score is generated from annual Staff Climate Study responses to items within the "diversity" dimension of the study. This score is then compared to the previous year to determine percent change on an annual basis.

...

Area Metric

Program Year

Target

RP1

RP2

RP3

RP4

Total

 Goal Supported

Number of new users of XSEDE

resources and services

RY 5

      

Deepen/Extend – Extend use to new communities

RY 4

      

RY 3

      

RY 2

2,000 / qtr2,3052,813   

RY 1

>1,000 / qtr

*1,88121,7702

2,359

6,181

 

Definition/Description: This is the number of portal accounts created within the selected time period.

Collection methodology: The Liferay Database is queried for the number of users in the Users table that was created during the selected time periods. Sample query: select screenName, createDate from User_ where createDate between '2016-09-01 00:00:00' AND '2016-10-31 00:00:00';

Number of sustained users of XSEDE

resources and services

RY 5

      

Deepen/Extend – Deepen use to existing communities

RY 4

      

RY 3

      

RY 2

3,000 / qtr3,9623,754   

RY 1

>5,000 / qtr

*

4,7552

4,4462

4,924

6,168

 

Definition/Description: This is the number of users logged into the XSEDE User Portal within the selected time period that were existing XUP users before the selected reporting period. Users who have created their account during that time period are not counted.

Collection methodology: The UII team provides the list of username’s logged in during this period to the evaluation team, they filter out the accounts that were created during this time to get the total number of sustained users.

Number of unique pageviews to the XSEDE website

RY 5

      

Deepen/Extend – Raise awareness of the value of advanced digital services

RY 4

      

RY 3

      

RY 2

80,000 / qtr63,998NA3   

RY 1

80,000 / qtr

*49,40965,157

68,227

183,212

 

Definition/Description: Per Google Analytics, “a pageview is defined as a view of a page on your site that is being tracked by the Analytics tracking code. If a user clicks reload after reaching the page, this is counted as an additional pageview. If a user navigates to a different page and then returns to the original page, a second pageview is recorded as well.”

Collection methodology: This is the number reported in the Google Analytics tracking software for the given time period.

Number of pageviews to the

XSEDE User Portal

RY 5

      

Deepen/Extend – Raise awareness of the value of advanced digital services

RY 4

      

RY 3

      

RY 2

250,000 / qtr256,482282,722   

RY 1

100,000 / qtr

*183,408219,664

265,115

670,080

 

Definition/Description: Per Google Analytics, “a pageview is defined as a view of a page on your site that is being tracked by the Analytics tracking code. If a user clicks reload after reaching the page, this is counted as an additional pageview. If a user navigates to a different page and then returns to the original page, a second pageview is recorded as well.”

Collection methodology: This is the number reported in the Google Analytics tracking software for the given time period.

User satisfaction with the website

RY 5

      

Sustain – Provide excellent user support

RY 4

      

RY 3

      

RY 2

4 out of 5 / yr4.17-   

RY 1

4 out of 5 / yr

*----

 

Definition/Description: The user satisfaction rating of the web site as determined by the Annual User Survey on a scale of 1-5.

Collection methodology: An annual user survey is emailed to all users of XSEDE resources and users are asked to rate their satisfaction with the website based on a scale of 1-5. The statement is “Please rate your overall satisfaction with XSEDE on a scale of 1 to 5, with 1 being “very dissatisfied” and 5 being “very satisfied.” If you have no basis for rating your satisfaction, please select "Not applicable.”

User satisfaction with the XSEDE User Portal

RY 5

      

Sustain – Provide excellent user support

RY 4

      

RY 3

      

RY 2

4 out of 5 / yr4.18-   

RY 1

4 out of 5 / yr

*----

 

Definition/Description: The user satisfaction rating of the portal from the Annual User Survey on a scale of 1-5.

Collection methodology: An annual user survey is emailed to all users of XSEDE resources and users are asked to rate their satisfaction with the website based on a scale of 1-5. The statement is “Please rate your overall satisfaction with XSEDE on a scale of 1 to 5, with 1 being “very dissatisfied” and 5 being “very satisfied.” If you have no basis for rating your satisfaction, please select "Not applicable.”

User satisfaction with user documentation

RY 5

      

Sustain – Provide excellent user support

RY 4

      

RY 3

      

RY 2

4.25 out of 5 / yr4.07-   

RY 1

4 out of 5 / yr

*-4.221-4.22

 

Definition/Description: The user satisfaction rating of user documentation from the Annual User Survey on a scale of 1-5.

Collection methodology: An annual user survey is emailed to all users of XSEDE resources and users are asked to rate their satisfaction with the user documentation based on a scale of 1-5.

...

Area Metric

Program Year

Target

RP1

RP2

RP3

RP4

Total

 Goal Supported

Number of institutions with a Champion

RY 5

      

Deepen/Extend – Deepen use to existing communities

RY 4

      

RY 3

      

RY 2

240218238   

RY 1

225 / yr

*224231

234

234

 

Definition/Description: The total number of Institutions that have a signed MOU with the Champion program. 

Collection methodology:  These are collected from a spreadsheet kept by the Campus Engagement leadership and on the public website listing Champion institutions.

Number of unique contributions to the Champion email list (campuschampions@xsede.org)

RY 5

      

Deepen/Extend – Deepen use to existing communities

RY 4

      

RY 3

      

RY 2

100 / yr129112   

RY 1

50 / yr

*90112

125

190

 

Definition/Description: The total number of distinct champions that sent a message to the champion email list.

Collection methodology: Archives are available at https://mhonarc.xsede.org/campuschampions/  as mbox files that can be parsed and counted.

Number of activities that (i) expand the emerging CI workforce and/or (ii) improve the extant CI workforce, participated in by members of the Campus Engagement team

RY 5

      

Deepen/Extend – Deepen use to existing communities

RY 4

      

RY 3

      

RY 2

40 / yr2830   

RY 1

20 / yr

*1011

10

30

 

Definition/Description: CI workforce development activities engaged in by any member of the Campus Engagement team, including but not limited to the Campus Engagement Level 3 Co-Managers, the Campus Champions Coordinator, Campus Engagement staff and leadership team, and the Campus Champions. This includes but isn't limited to all  education, outreach, training, mentoring, team-building, fostering of relevant communities, professional development, recruitment, proposal submissions, workshops, dissemination (for example, papers, posters, presentations etc, at conferences, in journals, and in other venues), which are intended to, or have the effect of, expanding and/or improving the CI workforce, across all segments (academic, industry, government, non-profit/non-governmental).

Collection methodology: Written reports (documents, e-mails, etc) provided by any and all members of the Campus Engagement team, as enumerated above, including but not limited to any internal survey or other data collection results.

...

Area Metric

Program Year

Target

RP1

RP2

RP3

RP4

Total

 Goal Supported

Average ECSS impact rating

(KPI)

RY 5

      

Deepen/Extend – Deepening use to existing communities

RY 4

      

RY 3

      

RY 2

4 out of 5 / yr4.114.03   

RY 1

4 out of 5 / qtr

*4.564.61

3.29 

4.14

 

Definition/Description: After an ECSS project is marked as completed (i.e. has a workplan, work has progressed, and a final report has been filed), the L2 Directors interview the PIs, preferably via phone, and ask them to rate the impact of the ECSS support on their research on a scale of 1 to 5.

Collection methodology: PIs are contacted by the L2 Directors post filing of a final report by ECSS staff members. The L2 Director asks the PI to rate the impact of the ECSS support on their research on a scale of one to five and this number is recorded in a spreadsheet, which is provided to the L2 Project Manager (PM). The PM then transfers this number to Sciforma. Each individual impact rating that is reported for the reporting period is then added together and divided by the total number of interviews conducted for the average.

Average satisfaction with ECSS support

(KPI)

RY 5

      

Deepen/Extend – Deepen use to existing communities

RY 4

      

RY 3

      

RY 2

4.5 out of 5 / yr4.654.56   

RY 1

4.5 out of 5 / qtr

*4.864.72

4.64 

4.54

 

Definition/Description: After an ECSS project is marked as completed (i.e. has a workplan, work has progressed, and a final report has been filed), the L2 Directors interview the PIs, preferably via phone, and ask them to rate their satisfaction with the ECSS support they have received on a scale of 1 to 5.

Collection methodology: PIs are contacted by the L2 Directors post filing of a final report by ECSS staff members. The L2 Director asks the PI to rate their satisfaction with the support received on a scale of one to five and this number is recorded in a spreadsheet, which is provided to the L2 Project Manager (PM). The PM then transfers this number to Sciforma. Each individual satisfaction rating that is reported for the reporting period is then added together and divided by the total number of interviews conducted for the average.

Number of completed ECSS projects

(ESRT  +  ESCC  +  ESSGW)

(KPI)

RY5      Deepen/Extend – Deepening use to existing communities
RY4      
RY3      
RY250 / yr169   

RY1

50 / yr

*1013

25 

48

 

Definition/Description: The total number of completed ECSS projects consists of projects that were completed through work in ESRT, ESCC, and ESSGW. A completed project is defined as a project that has progressed through the complete support pipeline.  This pipeline includes steps such as assignment of a consultant, production of a work plan, execution of the work plan and reporting of progress through quarterly reports, and the filing of a final project report.

Collection methodology: The number of completed projects is tracked in Sciforma, XSEDE’s project management software. A report has been created in Sciforma that queries all the ECSS projects, filters for allocations that have ended in that quarter and also for projects that have a final report.

 

Number of new users from non-traditional

disciplines of XSEDE resources and services

(KPI - composite)

RY 5

      

Deepen/Extend – Extend use to new communities

       

RY 4

      

RY 3

      

RY 2

500 / yr10199   

RY 1

400 / yr

*14792

158 

397

 

Definition/Description: This metric tracks the progress of extending XSEDE allocations to first-time users from fields of science (FOS) that have not been significant consumers of HPC resources and services.  

Collection methodology: A set of 60 FOS have been identified in the XD Central Database (XDCDB), each of whose usage over the past 10 years is below 0.5% of the total normalized usage. A scripted query to XDCDB counts the total number of users on the grants newly activated from these FOS.

Number of sustained users from non-traditional

disciplines of XSEDE resources and services

(KPI - composite)

RY 5

      

Deepen/Extend – Deepen use to existing communities

RY 4

      

RY 3

      

RY 2

500 / qtr700709   

RY 1

400 / qtr1

*451517

716 

1,684

 

Definition/Description: This metric tracks the progress of users from fields of science (FOS) that have not been significant consumers of HPC resources and services, who are benefiting from their XSEDE allocations in a sustained manner.  

Collection methodology: A set of 60 FOS have been identified in the XD Central Database (XDCDB), each of whose usage over the past 10 years is below 0.5% of the total normalized usage. A scripted query to XDCDB counts the total number of users on the grants from these FOS that have used at least 10% of their allocated usage in the past year.

Average estimated months saved due to ECSS support2RY 5      Deepen/Extend – Deepen use to existing communities
RY 4      
RY 3      
RY 212 mo / project13.6112.23   
RY 1******

 

Definition/Description: This metric is an average amount of months a PI estimates to have saved due to ECSS support and is ascertained during the PI interview conducted by ECSS L2 management. The metric will be reported by adding the number of months estimated to have been saved due to ECSS support divided by the total number of PI interviews conducted during the reporting period. 

Collection methodology: During this interview, the PI is asked to estimate the time in months that it would have taken to get to the same stage that the project achieved if the project did not have ECSS Support. The number that would be reported as a metric is the average estimated months saved due to ECSS Support.

...

Area Metric

Program Year

Target

RP1

RP2

RP3

RP4

Total

 Goal Supported

Average ESRT impact rating

(KPI)

RY 5

      

Deepen/Extend – Deepen use to existing communities

RY 4

      

RY 3

      

RY 2

4 out of 5 / yr3.74.63   

RY 1

4 out of 5 / qtr

*54.67

5

4.67

 

Definition/Description: After an ESRT project is marked as completed (i.e. has a workplan, work has progressed, and a final report has been filed), the L2 Directors interview the PIs, preferably via phone, and ask them to rate the impact of the ESRT support on their research on a scale of 1 to 5. 

Collection methodology: PIs are contacted by the L2 Directors post filing of a final report by ESRT staff members. The L2 Director asks the PI to rate the impact of the ESRT support on their research on a scale of one to five and this number is recorded in a spreadsheet, which is shared with the L2 Area Project Manager (PM). The PM then transfers this number to Sciforma. Each individual satisfaction rating that is reported for the reporting period is then added together and divided by the total number of interviews conducted for the average. 

Average satisfaction with ESRT support

(KPI)

RY 5

 

      

Deepen/Extend – Deepen use to existing communities

RY 4

      

RY 3

      

RY 2

4.5 out of 5 / yr4.64.75   

RY 1

4.5 out of 5 / qtr

*54.33

5

4.86

 

Definition/Description: After an ESRT project is marked as completed (i.e. has a workplan, work has progressed, and a final report has been filed), the L2 Directors interview the PIs, preferably via phone, and ask them to rate their satisfaction with the ECSS support they have received on a scale of 1 to 5. 

Collection methodology: PIs are contacted by the L2 Directors post filing of a final report by ESRT staff members. The L2 Director asks the PI to rate their satisfaction with the support received on a scale of one to five and this number is recorded in a spreadsheet which is shared with the L2 Area Project Manager (PM). The PM then transfers this number to Sciforma. Each individual satisfaction rating that is reported for the reporting period is then added together and divided by the total number of interviews conducted for the average. 

Number of completed ESRT projects

(KPI)

RY 5      Deepen/Extend – Deepen use to existing communities
RY 4       
 RY 3      
 RY 230 / yr102   
 RY 1

30 / yr

*36

15

24

 

Definition/Description: A completed project is defined as a project that has progressed through the complete support pipeline.  This pipeline includes steps such as assignment of a consultant, production of a work plan, execution of the work plan and reporting of progress through quarterly reports and the filing of a final project report.

Collection methodology: The number of completed projects is tracked in Sciforma, XSEDE’s project management software. A report has been created in Sciforma that queries all the ECSS projects, filters for allocations that have ended in that quarter and also for projects that have a final report. This report can be filtered by the 3 project areas in ECSS: ESRT, ESCC, or ESSGW.

Average estimated months saved due to ESRT support1RY 5      Deepen/Extend – Deepen use to existing communities
RY 4      
RY 3      
RY 26 mo / project18.316   
RY 1******
 

Definition/Description: This metric is an average amount of months a PI estimates to have saved due to ESRT support and is ascertained during the PI interview conducted by ECSS L2 management. The metric will be reported by adding the number of months estimated to have been saved due to ESRT support divided by the total number of PI interviews conducted during the reporting period. 

Collection methodology: During this interview, the PI is asked to estimate the time in months that it would have taken to get to the same stage that the project achieved if the project did not have ESRT Support. The number that would be reported as a metric is the average estimated months saved due to ESRT Support.

...

Area Metric

Program Year

Target

RP1

RP2

RP3

RP4

Total

Goal Supported

Number of new users from non-traditional

disciplines using XSEDE resources and services

(KPI)

RY 5

      

Deepen/Extend – Extend use to new communities

RY 4

      

RY 3

      

RY 2

500 / yr10199   

RY 1

400 / yr

*14792

158 

397

 

Definition/Description: This metric tracks the progress of extending XSEDE allocations to first-time users from fields of science (FOS) that have not been significant consumers of HPC resources and services.  

Collection methodology: A set of 60 FOS have been identified in the XD Central Database (XDCDB), each of whose usage over the past 10 years is below 0.5% of the total normalized usage. A scripted query to XDCDB counts the total number of users on the grants newly activated from these FOS.

Number of sustained users from non-traditional

disciplines using XSEDE resources and services

(KPI- composite)

RY 5

      

Deepen/Extend – Deepen use to existing communities

RY 4

      

RY 3

      

RY 2

500 / qtr700709   

RY 1

400 / qtr1

*451517

 716

1,684

 

Definition/Description: This metric tracks the progress of users from fields of science (FOS) that have not been significant consumers of HPC resources and services, who are benefiting from their XSEDE allocations in a sustained manner.  

Collection methodology: A set of 60 FOS have been identified in the XD Central Database (XDCDB), each of whose usage over the past 10 years is below 0.5% of the total normalized usage. A scripted query to XDCDB counts the total number of users on the grants from these FOS that have used at least 10% of their allocated usage in the past year.

Number of new XSEDE projects from target

communities generated by NIP

RY 5

      

Deepen/Extend – Extend use to new communities

RY 4

      

RY 3

      

RY 2

30 / yr1110   

RY 1

20 / yr

*165

 17

38

 

Definition/Description: This metric tracks the success of the personal efforts of NIP staff members to generate new XSEDE projects from fields of science (FOS) that have not been significant consumers of HPC resources and services.

Collection methodology: First-time grants generated by the  personal efforts of NIP staff members are flagged in XDCDB. A scripted query reports the number of these grants.

Number of successful XSEDE projects from

target communities mentored by NIP

RY 5

      

Deepen/Extend – Extend use to new communities

RY 4

      

RY 3

      

RY 2

25 / qtr2435   

RY 1

20 / qtr1

*2325

34 

82

 

Definition/Description: This metric tracks the success of the personal efforts of NIP staff members to mentor sustained projects from fields of science (FOS) that have not been significant consumers of HPC resources and services.

Collection methodology: Grants generated by the  personal efforts of NIP staff members are flagged in XDCDB. A scripted query reports the number of these grants that have used at least 10% of their allocated usage in the past year..

...

Area Metric

Program Year

Target

RP1

RP2

RP3

RP4

Total

Goal Supported

Average ESCC impact rating

(KPI)

RY 5

      

Deepen/Extend – Deepen use to existing communities

RY 4

      

RY 3

      

PY27

4 out of 5 / yr51   

RY 1

4 out of 5 / qtr

*4.675

0

3.00

 

Definition/Description: After an ESCC project is marked as completed (i.e. has a workplan, work has progressed, and a final report has been filed), the L2 Directors interview the PIs, preferably via phone, and ask them to rate the impact of the ESCC support on their research on a scale of 1 to 5.

Collection methodology: PIs are contacted by the L2 Directors post filing of a final report by ESCC staff members. The L2 Director asks the PI to rate the impact of the ESCC support on their research on a scale of one to five and this number is recorded in a spreadsheet which is shared with the L2 Area Project Manager (PM). The PM then transfers this number to Sciforma. Each individual satisfaction rating that is reported for the reporting period is then added together and divided by the total number of interviews conducted for the average.

Average satisfaction with ESCC support

(KPI)

RY 5

      

Deepen/Extend – Deepen use to existing communities

RY 4

      

RY 3

      

RY 2

4.5 out of 5 / yr52.75   

RY 1

4.5 out of 5 / qtr

*45

4

4.57

 

Definition/Description: After an ESCC project is marked as completed (i.e. has a workplan, work has progressed, and a final report has been filed), the L2 Directors interview the PIs, preferably via phone, and ask them to rate their satisfaction with the ESCC support they have received on a scale of 1 to 5.

Collection methodology: PIs are contacted by the L2 Directors post filing of a final report by ESCC staff members. The L2 Director asks the PI to rate their satisfaction with the support received on a scale of one to five and this number is recorded in a spreadsheet which is shared with the L2 Area Project Manager (PM). The PM then transfers this number to Sciforma. Each individual satisfaction rating that is reported for the reporting period is then added together and divided by the total number of interviews conducted for the average.

Number of completed ESCC projects

(KPI)

RY 5      Deepen/Extend – Deepening use to existing communities
RY 4      
RY 3      
RY 210 / yr44   
RY 1

10 / yr

*34

3

10

 

Definition/Description: A completed project is defined as a project that has progressed through the complete support pipeline.  This pipeline includes steps such as assignment of a consultant, production of a work plan, execution of the work plan and reporting of progress through quarterly reports, and the filing of a final project report.

Collection methodology: The number of completed projects is tracked in Sciforma, XSEDE’s project management software. A report has been created in Sciforma that queries all the ECSS projects, filters for allocations that have ended in that quarter and also for projects that have a final report. This report can be filtered by the 3 project areas in ECSS: ESRT, ESCC, or ESSGW.

Average estimated months saved due to ESCC support1RY 5      Deepen/Extend – Deepening use to existing communities
RY 4      
RY 3      
RY 212 mo / project11.53   
RY 1******
 

Definition/Description: This metric is an average amount of months a PI estimates to have saved due to ESCC support and is ascertained during the PI interview conducted by ECSS L2 management. The metric will be reported by adding the number of months estimated to have been saved due to ESCC support divided by the total number of PI interviews conducted during the reporting period. 

Collection methodology: During this interview, the PI is asked to estimate the time in months that it would have taken to get to the same stage that the project achieved if the project did not have ESCC Support. The number that would be reported as a metric is the average estimated months saved due to ESCC Support.

...

Area Metric

Program Year

Target

RP1

RP2

RP3

RP4

Total

Goal Supported

Average ESSGW impact rating

(KPI)

RY 5

      

Deepen/Extend – Deepen use to existing communities

RY 4

      

RY 3

      

RY 2

4 out of 5 / yr4.254.00   

RY 1

4 out of 5 / qtr

*44.7

4.5 

4.54

 

Definition/Description: After an ESSGW project is marked as completed (i.e. has a workplan, work has progressed, and a final report has been filed), the L2 Directors interview the PIs, preferably via phone, and ask them to rate the impact of the ESSGW support on their research on a scale of 1 to 5.

Collection methodology: PIs are contacted by the L2 Directors post filing of a final report by ESSGW staff members. The L2 Director asks the PI to rate the impact of the ESSGW support on their research on a scale of one to five and this number is recorded in a spreadsheet which is shared with the L2 Area Project Manager (PM). The PM then transfers this number to Sciforma. Each individual satisfaction rating that is reported for the reporting period is then added together and divided by the total number of interviews conducted for the average.

Average satisfaction with ESSGW support

(KPI)

RY 5

      

Deepen/Extend – Deepen use to existing communities

RY 4

      

RY 3

      

RY 2

4.5 out of 5 / yr4.54.80   

RY 1

4.5 out of 5 / qtr

*54.7

4.88 

4.75

 

Definition/Description: After an ESSGW project is marked as completed (i.e. has a workplan, work has progressed, and a final report has been filed), the L2 Directors interview the PIs, preferably via phone, and ask them to rate their satisfaction with the ESSGW support they have received on a scale of 1 to 5.

Collection methodology: PIs are contacted by the L2 Directors post filing of a final report by ESSGW staff members. The L2 Director asks the PI to rate their satisfaction with the support received on a scale of one to five and this number is recorded in a spreadsheet which is shared with the L2 Area Project Manager (PM). The PM then transfers this number to Sciforma. Each individual satisfaction rating that is reported for the reporting period is then added together and divided by the total number of interviews conducted for the average.

Number of completed ESSGW projects

(KPI)

RY 5      Deepen/Extend – Deepen use to existing communities
RY 4      
RY 3      
RY 210 / yr23   
RY 1

10 / yr

*43

14

 

Definition/Description: A completed project is defined as a project that has progressed through the complete support pipeline.  This pipeline includes steps such as assignment of a consultant, production of a work plan, execution of the work plan and reporting of progress through quarterly reports, and the filing of a final project report.

Collection methodology: The number of completed projects is tracked in Sciforma, XSEDE’s project management software. A report has been created in Sciforma that queries all the ECSS projects, filters for allocations that have ended in that quarter and also for projects that have a final report. This report can be filtered by the 3 project areas in ECSS: ESRT, ESCC, or ESSGW.

Average estimated months saved due to ESSGW support1

RY 5      Deepen/Extend – Deepen use to existing communities
RY 4      
RY 3      
RY 212 mo / project4.0011.40   
RY 1******
 

Definition/Description: This metric is an average amount of months a PI estimates to have saved due to ESSGW support and is ascertained during the PI interview conducted by ECSS L2 management. The metric will be reported by adding the number of months estimated to have been saved due to ESSGW support divided by the total number of PI interviews conducted during the reporting period. 

Collection methodology: During this interview, the PI is asked to estimate the time in months that it would have taken to get to the same stage that the project achieved if the project did not have ESSGW Support. The number that would be reported as a metric is the average estimated months saved due to ESSGW Support.

...