Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

KPI

Program Year

Target

RP1

RP2

RP3

RP4

Total

Owner(s)

Number of sustained users of XSEDE resources and services via the portal2





RY64,500 / qtr5,1484,722



 

 

CEE (§3)



RY5

 4,500 / qtr 4,6444,489 4,494  5,280 7,014

RY4

 3,500 / qtr 4,137 4,728 4,070 4,6156,578 

RY3

3,500 / qtr4,1964,0893,099 4,864 6,851

RY2

3,000 / qtr

3,962

3,754

2,488

3,020

4,527

RY1

> 5,000 / qtr

*

4,755

4,446

4,924

6,186

 

Definition/Description: Number of sustained users within the current reporting period. A "sustained user" is any individual who created a portal account prior to the current reporting period and logged in during the reporting period.


Collection methodology: There is a usage log parsing script that collects the tomcat logs and parses out the relevant data during the selected time period.

Number of sustained underrepresented individuals using XSEDE resources and services via the portal2RY61,750 / yr1,009869





CEE (§3)



RY5 1,750 / yr831  805763 1,013 1,266 
RY4 1,750 / yr 625 809 564 705 1,014
RY31,500 / yr5295091343 636 1,818
RY21,500 / yr

490

408

296

402

1,640

RY1> 1,000 / yr*

322

238

535

1,095

 

Definition/Description: Number of sustained underrepresented users. A "sustained user" is any individual who created a portal account prior to the current reporting period and logged in during the reporting period. Only individuals from underrepresented communities including women and racial/ethnic domestic minorities in HPC are counted. Users identifying with multiple underrepresented communities are only counted once. 

Collection methodology: There is a usage log parsing script that collects the tomcat logs and parses out the relevant data during the selected time period. Self reported demographic XUP data is then matched to these data to select underrepresented users including women and racial/ethnic minorities from the list. Users identifying with multiple underrepresented communities are only counted once.

Percentage of sustained allocation users from non-traditional disciplines of XSEDE resources and services



RY633% / yr24.424.7





ECSS (§4)



RY5

 33% / yr23.2  23.824.9 26.0 33.3 

RY4

 22% / yr 21.1 22.7 23.0 23.530.4 

RY3

20% / qtr22.120.725.8 22.7 22.8

RY2

21.5

20.4

21.1

21.0

21.0

RY1

*

*

18.2

19.9

18.6

18.9

 

Definition/Description: Number of sustained allocation users from non-traditional disciplines in HPC divided by the total number of sustained allocation users within the reporting period.

Collection methodology: A set of 63 fields of science (FOS) have been identified in the XD Central Database (XDCDB), each of whose usage over the past 10 years is below 0.5% of the total normalized usage. A scripted query to XDCDB counts “sustained allocation users” as the total number of users on grants from these FOS that are active during the reporting period and have used at least 10% of their allocated usage on at least one service provider (SP). This is divided by the total number of users on all the grants that that are active during the reporting period and have used at least 10% of their allocated usage on at least one SP. 

...

KPI

Program Year

Target

RP1

RP2

RP3

RP4

Total

Owner(s)

Number of new users of XSEDE resources and services via the portal




RY 62,500/qtr1,6032,377



 

 

 

CEE (§3)



RY 5

 2,500/qtr2,157 2,612 1,972 2,211 9,050 

RY 4

 3,000 / qtr 2,415 3,209 2,365 3,089 11,078

RY 3

3,000 / qtr

1,9051

2,76312,52712,757  9,952

RY 2

2,000 / qtr

2,305

2,813

2,346

2,917

10,381

RY 1

>1,000 / qtr

*

1,973

1,849

2,359

6,181

 

Definition/Description: Number of new users of XSEDE resources and services as determined by the number of portal accounts created within the reporting period.

Collection methodology: The Liferay Database is queried for the number of users in the Users table that was created during the selected time periods. Sample query: select screenName, createDate from User_ where createDate between '2016-09-01 00:00:00' AND '2016-10-31 00:00:00';

Number of new underrepresented individuals using XSEDE resources and services via the portal



RY 6250 / qtr181163



 


CEE (§3)



RY 5

 250 / qtr301 159 188 219  998

RY 4

 175 / qtr 380 332 214 400 1,326

RY 3

200 / qtr

134

155129 238 656

RY 2

150 / qtr

251

175

222

234

950

RY 1

100 / qtr

*

150

135

240

525

 

Definition/Description: Number of new underrepresented users. A "new user" is any individual who created a portal account within the reporting period. Only individuals from underrepresented communities including women and racial/ethnic domestic minorities in HPC are counted. Users identifying with multiple underrepresented communities are only counted once. 

Collection methodology: XSEDE portal users are able to specify their race/ethnicity, gender, and institution in their user profile and are encouraged to complete this data when registering for training events through the XSEDE portal. Date of allocation awards is also recorded in the XSEDE database (XDCDB). A query is then run to generate the number of individuals meeting the aforementioned criteria within a given quarter. Care is taken to not over represent this metric by ensuring an individual who meets multiple criteria is only accounted for one time.

Percentage of new allocation users from non-traditional disciplines of XSEDE resources and servicesRY 635% / yr30.930.0



 

 

ECSS (§4)



RY 5 35% / yr30.1  35.039.7 35.4 40.0 
RY 4 35% / yr 26.4 37.1 33.4 38.434.8 
RY 330% / yr 33.126.038.732.5  32.8
RY 2

 NA

24.8

26.0

26.6

33.9

27.8

RY 1

 NA

*

21.8

24.9

31.0

25.7

 

Definition/Description: Number of new allocation users from non-traditional disciplines in HPC divided by the total number of new allocation users within the reporting period. 

Collection methodology: A set of 63 fields of science (FOS) have been identified in the XD Central Database (XDCDB), each of whose usage over the past 10 years is below 0.5% of the total normalized usage. A scripted query to XDCDB counts “new allocation users” as the total number of users on the grants from these FOS that have been newly awarded during the reporting period. This is divided by the total number of users on all the grants that have been awarded during the reporting period.

...

KPI

Program Year

Target

RP1

RP2

RP3

RP4

Total

Owner(s)

Number of participant hours of live training delivered by XSEDE





RY 645,000 / yr16,31512,531



 

 

CEE (§3)



RY 5

 40,000 / yr19,751  16,1115,689  15,51557,066 

RY 4

 40,000 / yr 15,461 16,135 6,380 12,667 50,643

RY 3

40,000 / yr

14,140

8,2747,25912,352  42,025

RY 2

 NA1

12,787

8,876

6,004

14,753

42,421

RY 1

 NA1

5,994

3,770

5,180

9,199

24,143

 

Definition/Description: Number of total participant hours spent in XSEDE live, in-person, or webcast events during the current reporting period.

Collection methodology: Measured using registrations via the XSEDE User Portal. Does not include training events that do not require registration such as Cornell Virtual Workshop, CI-Tutor, Youtube videos, etc. 

...

KPI

Program Year

Target

RP1

RP2

RP3

RP4

Total

Owner(s)

Grand (aggregate) mean of XSEDE User Survey awareness items regarding XSEDE resources and services


RY 63.7 of 5 / yr3.6-



 

 

Program Office (§8)



RY 5

 3.7 of 5 / yr3.8  -3.8 

RY 4

 3.7 of 5 / yr 3.8 - - - 3.8

RY 3

3.5 of 5 / yr

3.7

-- - 3.7

RY 2

 *

3.6

-

-

-

3.6

RY 1

*

-

-

-

NA

 

Definition/Description: Aggregate mean rating of user awareness of XSEDE resources and services as measured by the annual User Survey on a scale of 1 (not at all aware) - 5 (extremely aware). The core items included in the calculation for this purpose are: Mission, Computational Resources, Data Storage Services, Viz Services, Science Gateways, XUP, Data Transfer Services, Website, Training Opportunities, Online Technical Documentation; Community Engagement and Enrichment Opportunities (f.k.a., Education and Outreach Opportunities); Help Desk Services, and ECSS.

Collection methodology: Aggregate rating of awareness scale items measuring current XSEDE resources and services listed on the annual XSEDE User Survey.

Number of social media impressions over time1 





RY 6424,000 / yr91,77085,506





Program Office (§8)



RY 5

  430,000/ yr

86,046 

 101,394106,263 109,800 403,503 

RY 4

 426,198 / yr 87,48275,16484,185 110,904 357,735

RY 3

359,714 / yr

112,806

63,26993,80385,287355,165

RY 2

NA

69,607

55,506

59,490

128,180

312,783

RY 1

NA

*

52,200

128,675

88,332

269,207

 

Definition/Description: Each reporting period the number of people who have seen XSEDE interactions on Facebook + Twitter is reported. "Impressions" are generated by Facebook and Twitter to determine how many individuals have seen XSEDE interactions or posts on said platforms. This includes "likes," "retweets," "followers," “shares,” as well as people who see the post based on their friends or followers sharing XSEDE information. At the end of each reporting year, impressions across the four reporting periods for that year are summed, and a percentage increase over the total number from the previous reporting year is calculated as follows: Number of social media impressions within the current reporting period year (x) minus the number of social media impressions during the previous reporting year (y). This number is divided by the number of social media impressions in the previous reporting year (y) to determine percent increase (((x-y)/y)*100)).

Collection methodology: The External Relations team pulls the information from Facebook and Twitters' internal tracking methods at the end of each report period. The percent increase is then computed based on the previous reporting year data.

...

KPI

Program Year

Target

RP1

RP2

RP3

RP4

Total

Owner(s)

Total number of capabilities in production1





RY 6123 by end of RY6121125



 

 

XCI (§5)




RY 5

 110 by end of RY5102 104 109  118 118

RY 4

 100 by end of RY4 88 8891 102 102

RY 3

81 by end of RY3

76

851

87

87 

87 

RY 2

NA

72

74

74

75

75

RY 1

NA

*

63

63

69

69

 

Definition/Description: Total number of capabilities in production across XSEDE SPs, central operations, partner service providers (Globus, etc.), campuses, science gateways, etc.

Collection methodology: A list of capabilities in production through XSEDE is kept in the Community Software Repository which catalogs Use Cases and Capability Delivery Plans.

...

KPI

Program Year

Target

RP1

RP2

RP3

RP4

Total

Owner(s)

Grand (aggregate) mean of XSEDE User Survey satisfaction items regarding XSEDE technical support services1



RY 63.5 of 5 / yr4.4-



 

 

Program Office (§8)



RY 5

 3.5 of 5 / yr4.4  -4.4 

RY 4

 3.5 of 5 / yr 3.9 -- - 3.9

RY 3

3.5 of 5 / yr

3.6

-- - 3.6

RY 2

NA

3.4

-

-

-

3.4

RY 1

NA

*

*

*

*

*

 

Definition/Description: Aggregate rating of user satisfaction with XSEDE technical expertise and support services via the annual User Survey on a scale of 1 (very dissatisfied) - 5 (very satisfied).

Collection methodology: Aggregate rating of all current "XSEDE Software and Service Components" measured using the "frequency of performance meeting expectations" scale on the annual XSEDE User Survey.

...

KPI

Program Year

Target

RP1

RP2

RP3

RP4

Total

Owner(s)

Mean composite availability of core services (%)





RY 699.9% / qtr99.999.9


 

 

Operations (§6)


RY 5

 99.9% / qtr

99.9  99.999.9 99.9 99.9 

RY 4

 99.9% / qtr 99.9 99.9 99.999.9 99.9

RY 3

99.9% / qtr

99.9

99.999.999.9  99.9

RY 2

99.9% / qtr99.899.999.999.999.9

RY 1

99.0% / qtr

*

99.9

99.9

99.9

99.9

 

Definition/Description: Mean composite availability of core services is the geometric mean of % core enterprise services availability and % POPS/XRAS availability. Availability percentage of each component is measured separately, then aggregated and averaged using a geometric mean to determine the composite availability.

Mean composite of: "Average availability of critical enterprise services (%) [geometric mean]" (Systems Operational Support 2.4.5) and "Availability of XRAS systems" (AA&AM 2.5.3).

Collection Methodology:
Average availability of critical enterprise services (%) [geometric mean]: The data is stored locally to the Nagios service at IU. Nagios is an application that constantly logs the availability of core central services. These logs are queried to determine if/when a service becomes unavailable to users.

Availability of XRAS systems:
There is a table acct.downtime in the XDCDB, which is manually completed when there is an outage in the XDCDB, a network outage disrupting availability of the XRAS services, or an outage of the XSEDE User Portal. Currently, outages are detected via a variety of e-mail notifications, and the data is manually entered into the XDCDB.

...

KPI

Program Year

Target

RP1

RP2

RP3

RP4

Total

Owner(s)

Mean time to ticket resolution (hours)





RY 6< 16 hrs/ qtr14.617.6


 

 

Operations (§6)


RY 5

  < 16 hrs/ qtr15.3  23.218.8 12.2 17.4 

RY 4

 < 16 hrs/ qtr 14.0 14.621.0 14.3 16.0

RY 3

< 16 hrs/ qtr1

12.3

18.523.217.5 17.9

RY 2

< 24 hrs/ qtr2620.122.815.021.0

RY 1

< 24 hrs/ qtr

*24.028.2

23.1

25.1

 

Definition/Description: Mean time to ticket resolution within XSEDE WBS ticket queues and the XOC queue. This metric does not apply to Service Provider ticket queues.

Collection methodology: The data is stored locally via the Request Tracker service, which is the XSEDE ticket software and which is located at TACC. Reports are generated against the RT database to determine MTTR and other metrics.

Mean rating of user satisfaction with allocations process and support services1




RY 6 4 of 5 / yr4.1314.2


 

 

RAS (§7)



RY 5

  4 of 5 / yr

4.4  4.34.3 4.3 4.3 

RY 4

 4 of 5 / yr 4.2 4.34.4 4.2 4.3

RY 3

4 of 5 / yr

4.1

4.24.1 4.4 4.2

RY 2

4 of 5 / yr

4.1

4.0

4.1

3.9

4.0

RY 1

4 of 5 / yr

*

4.0

4.0

4.0

4.0

 

Definition/Description: Aggregate mean rating of user satisfaction with allocations process and XRAS services on a scale of 1 (not at all satisfied) - 5 (extremely satisfied).

Collection methodology: Average of the L2 KPI satisfaction ratings for the allocations process and for the XRAS system.

Percentage of research requests successful (not rejected)RY 685.0% / qtr85.476.0




RAS (§7)



RY 5 85.0% / qtr80.0 84.0  84.0 91.0 84.8
RY 4 85.0% / qtr 81.0 80.0 78.0 87.0 82.0
RY 3

85.0% / qtr

65.0

70.072.075.0  70.5
RY 2

85.0% / qtr

70.0

69.0

72.0

68.0

69.8

RY 1

85.0% / qtr

*

76.0

75.0

74.0

75.0

 

Definition/Description: Number of Research allocation requests receiving some non-zero award from an XRAC meeting divided by the number of Research allocation requests submitted to the same XRAC meeting.

Collection methodology: Request data comes from the Quarterly XRAC meetings and is stored in XRAS as well as some meeting spreadsheets.

...

KPI

Program Year

Target

RP1

RP2

RP3

RP4

Total

Owner(s)

Mean rating of importance of XSEDE resources and services to researcher productivity





RY 64.4 of 5 / yr4.2-


 

 

Program Office (§8)



RY 5

  4.4 of 5 / yr 4.2 - 4.2

RY 4

 4.4 of 5 / yr 4.2 -- - 4.2

RY 3

4.2 of 5 / yr

4.4

- - - 4.4

RY 2

 

4.42

-

-

-

4.4

RY 1

 

4.32

-

-

-

4.3

 

Definition/Description: Mean rating of importance of XSEDE resources and services to researcher productivity as measured by the annual User Survey on a scale of 1 (not important at all) - 5 (essential).

Collection methodology: A mean rating is calculated from the responses to the annual XSEDE User Survey item, "How important are XSEDE resources to your work?"

Percentage of users who indicate the use of XSEDE-managed and/or XSEDE-associated resources in the creation of their work product1RY 6 80%/yr88-




Program Office (§8)

RY 5 80%/yr83  - 83
RY 4 80%/yr 79 -- - 79
RY 379%/yr79- - - 79
RY 2******
RY 1******
 

Definition/Description: Percentage of users who self-report the use of XSEDE resources in their work product when responding to the annual User Survey.

Collection methodology: Annual User Survey response. The Annual User Survey is a representative sample of XSEDE Users who have interacted with XSEDE via the portal or an allocation within 2 years of the current dissemination date. When particular groups are targeted, it is to increase the response rate from that group in order to ensure adequate representation that mimics the general population. The evaluation team is confident that the responses from this group can be generalized.

...

KPI

Program Year

Target

RP1

RP2

RP3

RP4

Total

Owner(s)

Percentage of Project Improvement Fund funded projects resulting in innovations in the XSEDE organization

RY 670% / yr--




Program Office (§8)



RY 5  70% / yr - 66.766.7 
RY 4 70% / yr - -- 66.7 66.7
RY 3

60% / yr

-

--71.4 71.4 
RY 2 *

*

*

*

*

*

RY 1 *

*

*

*

*

*

 

Definition/Description:  Number of completed PIF (Project Improvement Fund) funded projects resulting in innovations as determined by SMT review divided by the total number of completed PIF funded projects to date. 

Collection methodology: The results of each Project Improvement Fund approved submissions to date is evaluated, at the end of the project year, to assess the level of innovation. This is a manual effort completed by the SP&E team. 

 

Mean rating of innovation within the organization by XSEDE staff





RY 64 of 5 / yr-4.1


 

 

Program Office (§8)



RY 5

  4 of 5 / yr - 4.2 - -4.2 

RY 4

 4 of 5 / yr - 4.0- - 4.0

RY 3

3.5 of 5 / yr

-

4.0- -4.0 

RY 2

*

*

*

*

*

RY 1

 *

*

*

*

*

*

 

Definition/Description: Mean rating of innovation within the organization by XSEDE employees as measured by the annual XSEDE Staff Climate Study on a scale of 1 (not at all innovative) - 5 (highly innovative).

Collection methodology: New item added to 2018 XSEDE Staff Climate Study where staff will rate the level of innovation within the organization of a scale of 1 (low/negative) to 5 (high/positive). 

...

KPI

Report Year

Target

RP1

RP2

RP3

RP4

Total

 Goal Supported

Number of students benefiting from XSEDE resources and services through training, XSEDE projects, or conference attendance


RY 62,000 / qtr2,2982.027


 

 

Deepen/Extend – Extend use to new communities



RY 5

 2,000 / qtr 2,277 2,0431,944 2,509 5,761 

RY 4

 1,500 / qtr 2,210 2,634 1,649 2,252 6,196

RY 3

1,250 / qtr

1,613 

1,6661,145 2,1042,323

RY 2

950 / qtr

1,722

1,478

1,170

1,522

1,802

RY 1

50 / qtr

*

997

815

2,679

3,122

 

Definition/Description: Number of students who participated in XSEDE training, were supported by XSEDE to work on XSEDE projects, received funding to attend the annual PEARC Conference (formerly XSEDExy), or SC conference as indicated by registrations in the XSEDE portal during the current reporting period.

Collection methodology: Measured using registrations via the XSEDE User Portal, participants in CEE student programs (i.e. EMPOWER), and PEARC student program attendees.

Number of under-represented students benefiting from XSEDE resources and services through training, XSEDE projects, or conference attendance





RY 6650 / qtr775666


 

 

Deepen/Extend – Deepen use to existing communities



RY 5

 650 / qtr 741 632 616 792 1,970

RY 4

 500 / qtr 674 7941 492 753 2,072

RY 3

625 / qtr

449

438307436 1,630 

RY 2

475 / qtr

488

399

347

423

1,104

RY 1

50 / qtr

*

34 

33

19

28

 

Definition/Description: Number of underrepresented students who participated in XSEDE training, were supported by XSEDE to work on XSEDE projects, received funding to attend the annual PEARC Conference (formerly XSEDExy), or SC conferences as indicated by registrations in the XSEDE portal during the current reporting period. Only students from underrepresented communities including women and racial/ethnic domestic minorities in HPC are counted in the numerator. Students identifying with multiple underrepresented communities are only counted once.

Collection methodology: Measured using registrations via the XSEDE User Portal, participants in CEE student programs (i.e. EMPOWER), and PEARC student program attendees. Only students from underrepresented communities including women and racial/ethnic domestic minorities in HPC are counted in the numerator. Students identifying with multiple underrepresented communities are only counted once.

Grand (aggregate) mean rating of Post Training Event Survey items related to training impact for attendees registered through the portal


RY 64.4 of 5 / qtr4.54.5


 

 

Deepen/Extend – Extend use to new communities



RY 5

  4.4 of 5 / qtr 4.6 4.54.4 4.6 4.5 

RY 4

 4.4 of 5 / qtr 4.5 4.34.3 4.6 4.4

RY 3

4.4 of 5 / qtr

4.5 

4.44.5 4.3 4.4

RY 2

4 of 5 / qtr

4.3

4.3

4.6

4.5

4.4

RY 1

4 of 5 / qtr

*

4.5

4.4 

4.3

4.4

 

Definition/Description: Aggregate mean rating of training impact for attendees registered via the XSEDE User Portal for XSEDE delivered training sessions as measured by the Post Training Event Survey on a scale of 1 (low impact) to 5 (high impact).

Collection methodology: The data is collected from post event surveys and recorded in the database. Specific items included in this index are:

Q1. The training session fulfilled my expectations.
Q2. The trainer stimulated my interest.
Q6. The training session was well-organized.
Q8. I was able to easily access this training session.
Q10. Overall I would rate my experience as successful.

Number of institutions with a Champion





RY 6345344349


 

 

Deepen/Extend – Deepen use to existing communities



RY 5

 340327  332333 337337 

RY 4

 300304 305 315 325 325

RY 3

250

259 

266277 284 284

RY 2

240

218

238

239

246 

246

RY 1

225

*

224

231

234

234

 

Definition/Description: Total number of institutions that have a signed letter of collaboration with the Champions program.

Collection methodology: All champions are tracked using a master spreadsheet managed by champion program managers and coordinator. All champions in the spreadsheet with a current corresponding letter of collaboration with XSEDE are counted for this metric.

Percentage of user requirements addressed within 30 days





RY 698% / qtr100 (11/11)

100 (12/12)




 

Deepen/Extend – Prepare the current and next generation



RY 5

  98% / qtr100 (16/16) 

 100

(20/20)

100 (28/28) 100 (26/26)  100 (90/90)

RY 4

 98% / qtr

 100
(34/34)

 96
(23/24)

 100 (26/26)

 97
(36/37)

 98
(119/121)

RY 3

100% / qtr

89 (40/45) 

90 (47/52)100 (36/36)

100 (27/27)

 94 (150/160)

RY 2

100% / qtr

78 (36/44)

102 (47/46)

86 (32/37)

93 (41/44)

91 (156/157)

RY 1

100% / qtr

*

50 (16/32)

89 (40/45)

75 (40/53)

74 (96/130)

 

Definition/Description: The number of user requirements addressed within 30 days during the current reporting year divided by the total number of requirements identified by the XSEDE User Engagement team through regular email contact with all XSEDE PIs. Total number of requirements identified does not include those identified within the last 30 days of the reporting year.

Collection methodology: All requirements addressed within 30 days tracked in JIRA within the current reporting year divided by the total number of requirements in JIRA within the same year that are over 30 days old. For purposes of this metric, the time period includes the last 30 days of the previous reporting year but does not include the last 30 days of the current reporting year. Requirements are considered addressed when the PI is presented with a solution via email, a ticket is submitted via the XSEDE ticket system, or a use case is submitted for further consideration.

...

KPI

Report Year

Target

RP1

RP2

RP3

RP4

Total

 Goal Supported

Number of completed ECSS projects





RY 650 / yr2417


 

Deepen/Extend – Deepening use to existing communities



RY 5

 45 / yr17  1018 10 55 

RY 4

 45 / yr 15 11 16 10 52

RY 3

45 / yr

17 

1012 8 47

RY 2

50 / yr

16

9

10

12

47

RY 1

50 / yr

*

10

13

25

48

 

Definition/Description: The total number of completed ECSS projects in ESRT, ESCC, and ESSGW within the current reporting period. A completed project is defined as having progressed through the full support pipeline, namely assigning a consultant, producing a work plan, executing the work plan, reporting progress through quarterly reports, and filing of a final project report.

Collection methodology: The number of completed projects is tracked in JIRA, XSEDE’s project management software. A report has been created in JIRA that queries all the ECSS projects, filters for allocations that have ended in that quarter and also for projects that have a final report.

Grand (aggregate) mean rating of ECSS impact by PIs measured by ECSS Project Exit Survey items


RY 6 4 of 5 / yr4.04.9


 

 

Deepen/Extend – Deepen use to existing communities



RY 5

  4 of 5 / yr4.1  4.2 4.54.8 4.3 

RY 4

 4 of 5 / yr 3.0 4.7 4.3 4.7 4.5

RY 3

4 of 5 / yr

 NA

3.94.4 4.3 4.2

RY 2

4 of 5 / yr

4.1 

4.0

3.8 

 4.3

4.0

RY 1

4 of 5 / qtr

*

4.6 

4.6

3.3

4.1

 

Definition/Description: Aggregate mean rating of ECSS impact by PIs collected after an ECSS project is complete (i.e. has a workplan, work has progressed, and a final report has been filed) measured by the ECSS Project Exit Survey on a scale of 1 (low impact) - 5 (high impact).

Collection methodology:

Grand (aggregate) mean rating of PI satisfaction with ECSS support measured by ECSS Project Exit Survey items

RY64.5 of 5 / yr4.35.0




Deepen/Extend – Deepening use to existing communities



RY5  4.5 of 5 / yr4.8  4.93.8 5.0 4.8 
RY4 4.5 of 5 / yr 3.0 5.0 5.0 4.7 4.8
RY34.5 of 5 / yrNA4.34.8 4.54.6 
RY24.5 of 5 / yr4.74.64.2

4.8

4.5

RY1

4.5 of 5 / qtr

*4.94.7

4.6

4.5

 

Definition/Description: Aggregate mean rating of PI satisfaction with ECSS support collected after an ECSS project is complete (i.e. has a work plan, work has progressed, and a final report has been filed) measured by the ECSS Project Exit Survey on a scale of 1 (not at all satisfied) - 5 extremely satisfied).

Collection methodology: Measured by the average of selected question(s) of the PI's satisfaction survey.


...

KPI

Report Year

Target

RP1

RP2

RP3

RP4

Total

Goal Supported

Grand (aggregate) mean rating of XSEDE User Survey user satisfaction items regarding XCI software and technical services, capabilities, and resources4

RY 64 of 5 / yr4.1-


 

 

Advance – Create an open and evolving e-infrastructure



RY 5

 4 of 5 / yr4.4  - 4.4

RY 4

* * *** *

RY 3

*****

RY 2

****

*

RY 1

*

***

*

*

 

Definition/Description: Aggregate rating of USER satisfaction with XSEDE software and technical services, capabilities, and resources via the annual User Survey on a scale of 1–5, where “1” is “performance never meets expectations” and “5” is “performance always meets expectations.”

Collection methodology: Aggregate ratings of USER-facing XCI software and technical services, capabilities and resources as measured using the "frequency of performance meeting expectations" scale on the annual XSEDE User Survey, to include the following items: Authenticating to the XSEDE User Portal and other web services using campus InCommon credentials (CILogon); Using XSEDE credentials to log into non-XSEDE Web services (e.g., GENI and OrCID.); Globus Transfer Service; Globus Sharing Service; Globus Connect for moving data between personal systems and XSEDE resources; Job start prediction service (Karnak); Login to XSEDE affiliated web sites without entering a password every time; Cluster integration software (XCBC); Scientific software for campus clusters (XNIT); Cluster integration tutorials and documentation; and Cluster integration site visits.

Grand (aggregate) mean rating of XSEDE User Survey Service Provider satisfaction items regarding XCI software and technical services, capabilitiies, and resources



RY 64 of 5 / yr4.4-


Advance – Create an open and evolving e-infrastructure
RY 54 of 5 / yr4.5---4.5
RY 4******
RY 3******
RY 2******
RY 1******

Definition/Description: Aggregate rating of SP satisfaction with XSEDE software and technical services, capabilities, and resources via the annual User Survey on a scale of 1–5, where “1” is “performance never meets expectations” and “5” is “performance always meets expectations.”

Collection methodology: Aggregate ratings of SP-facing XCI software and technical services, capabilities and resources as measured using the "frequency of performance meeting expectations" scale on the annual XSEDE User Survey, to include the following items: IPF software used to publish resource information to XSEDE; XSEDE accepted CA certificates; XSEDE Resource Operational Status Dashboards; XSEDE Resource Integration Steps Dashboard; Resource Description Repository (RDR); Install software from XSEDE RPM repositories on software.xsede.org; Opportunities to discuss software needs, delivery activities, deployment, and usage; Opportunities to participate in software technical reviews; Software Integration Support; Software Usage Instrumentation Support; Use case requirements analysis processes; and Use case development processes; and Use case prioritization processes. In order to get an adequate number of responses from Service Providers, that population will be over-sampled. If that does not result in an adequate response rate, these questions will be pulled and sent to the SPs separately to improve the number of responses.

Number of non-XSEDE partnerships with XCI2

 

RY 6

14 / yr1414
  

 

 

Advance – Create an open and evolving e-infrastructure

RY 518 / yr14113161521

RY 4

 12 / yr 9 1011 21 21

RY 3

8 / yr131211 1220

RY 2

----

 -

8

RY 1

-

*--

-

8

 

Definition/Description: Number of XCI partnerships supported/facilitated between CRI and SPs, national CI organizations, and campus CI providers.

Collection methodology: Non-XSEDE partnerships are tracked through the ticket system and through JIRA.

Mean time to issue resolution (days)RY 6< 10 days/ qtr2.81.7


 

 

Sustain — Provide excellent user support 


RY 5 < 10 days/ qtr 4.2 6.0 10.1 2.45.3 
RY 4 < 10 days/ qtr 5.4 4.58.6 35.8 13.7
RY 3< 14 days/ qtr22.37.32.9 2.98.8 
RY 2< 30 days4.07.08.0

5.0

6.1

RY 1

< 45 days

*7.04.016.09.0
 

Definition/Description: Average number of days from defect or support request submitted to solution communicated.

Collection methodology: The data is stored locally via the Request Tracker service, which is the XSEDE ticket software. Reports are generated against the RT database to determine MTTR and other metrics.

...

KPI

Report Year

Target

RP1

RP2

RP3

RP4

Total

Goal Supported

Mean rating of user satisfaction with tickets closed by the XOC





RY 64.5 of 5 / qtr4.74.7


 

 

Sustain – Provide excellent user support.


RY 5

  4.5 of 5 / qtr4.7  4.74.7 4.7 4.7 

RY 4

 4.5 of 5 / qtr 4.8 4.74.74.7 4.7

RY 3

4.5 of 5 / qtr4.54.94.6 4.7 4.7

RY 2

4.5 of 5 / qtr4.54.84.8

4.4

4.6

RY 1

4 of 5 / qtr

*4.84.2

5.0

4.7

 

Definition/Description: Mean rating of satisfaction by users who submitted help tickets that were closed by the XOC during the specified reporting period as measured by the XSEDE Operations Point-of-Service Satisfaction Survey on a scale of 1 (not at all satisfied) - 5 (extremely satisfied).

Collection methodology: Within two hours of the resolution and closure of a ticket, the user receives an e-mail containing a link to a Survey Monkey survey. The survey consists of five questions using a 5 point Likert scale and the results are stored in Survey Monkey.

Hours of downtime with direct user impacts from XSEDE security incidents

 




RY 60 hrs/ qtr00


 

 

Sustain – Provide reliable, and secure infrastructure



RY 5

  0 hrs/ qtr 0

RY 4

 0 hrs/ qtr 0 022 22

RY 3

0 hrs/ qtr000 0 0

RY 2

0 hrs/ qtr000

0

0

RY 1

<24 hrs/ qtr

*0146

0

146

 

Definition/Description: Hours of downtime is calculated as the Time to Return to Repair (TTR) across all applicable services and incidents during the quarter in order to measure resource unavailability as a result of an XSEDE-wide security event (involving a Tier 1 service or spanning more than one SP).

Collection methodology: The Cybersecurity co-leads determine that an XSEDE-wide security event is responsible for an outage. The Sysops lead will determine the TTR based on monitoring and other information in logs and tickets, and the TTR value will be the data point reported.

...

KPI

Report Year

Target

RP1

RP2

RP3

RP4

Total

Goal Supported

Mean rating of user satisfaction with allocations process





RY 64 of 5 / qtr4.14.2


 

 

Sustain – Provide excellent user support



RY 5

 4 of 5 / qtr4.4  4.34.4  4.3 4.3

RY 4

 4 of 5 / qtr 4.2 4.34.4 4.3 4.3

RY 3

4 of 5 / qtr4.14.14.2 4.4 4.2

RY 2

4 of 5 / qtr4.14.04.1

3.9

4.0

RY 1

4 of 5 / qtr

*4.04.0

4.0

4.0

 

Definition/Description: Aggregate mean rating of user satisfaction across 10 facets of allocation policies and procedures related to the submission, review, and administration of allocation requests on a scale of 1 (not at all satisfied) - 5 (extremely satisfied).

Collection methodology: Quarterly Post-Allocations Submission Survey of persons who have submitted allocation requests in the prior quarter.

Mean rating of user satisfaction with XRAS





RY 64 of 5 / qtr4.14.2


 

 

Sustain – Provide reliable, efficient, and secure infrastructure



RY 5

 4 of 5 / qtr 4.34.3 4.3 4.2 4.3 

RY 4

 4 of 5 / qtr 4.3 4.3 4.4 4.2 4.3

RY 3

4 of 5 / qtr4.04.24.1 4.34.2 

RY 2

4 of 5 / qtr4.04.04.1

4.0

4.0

RY 1

4 of 5 / qtr

*4.04.0

4.0

4.0

 

Definition/Description: Aggregate mean rating of user satisfaction with XRAS (XSEDE Resource Allocation System) across 8 facets of the user-facing submission process on a scale of 1 (not at all satisfied) - 5 (extremely satisfied).

Collection methodology: Quarterly Post-Allocations Submission Survey of persons who have submitted allocation requests in the prior quarter

...

KPI

Report Year

Target

RP1

RP2

RP3

RP4

Total

Goal Supported

Variance between relevant report submission and due date (days) 





RY 6000


 

 

Sustain — Operate an effective and productive virtual organization 



RY 5

 0 0 0 0 0

RY 4

 0 0 00 -14 -1

RY 3

0000 0 0

RY 2

0000

0

0

RY 1

0

*NA0

0

0

 

Definition/Description: Variance in number of calendar days between report submission and due date for the previous reporting period. In accordance with NSF quarterly and annual reporting requirements, deadlines are determined together by the XSEDE PI and NSF Project Officer and then communicated to the XSEDE team to establish the reporting schedule used to calculate this metric.

Collection methodology: The XSEDE PI tracks when the relevant document was submitted to FastLane and records the number of days between the due date and this date in a spreadsheet.

Percentage of sub-award invoices processed within target duration





RY 690% / qtr100100


 

 

 

Sustain — Operate an effective and productive virtual organization 



RY 5

 90% / qtr 92.6 90.25 91.3 96.2 92.6

RY 4

 95% / qtr 95.0 100.079.5 90.9 91.3

RY 3

95% / qtr82.477.894.492.9  86.9

RY 2

95% / qtr100.090.988.0

67.4

74.6

RY 1

90% / qtr

*NA1100.0

NA2

100

 

Definition/Description: Number of invoices processed within the target duration divided by the total number of invoices processed. Duration is determined by the number of calendar days from acceptance of submitted invoice to invoice paid. The target duration is less than 45 calendar days. 

Collection methodology: Data is logged and stored in the Business Operations master spreadsheet located on the Business Operations Google Drive folder.

Percentage of recommendations addressed by relevant project areas within 90 days

 





RY 6 90% / yrNA100


 

 

Sustain – Operate an effective and productive virtual organization



RY 5

  90% / yr100  90100 NA3  97

RY 4

 90% / yr 47.0 48.00.1 0 24

RY 3

90% / yr46.074.0062.0 61

RY 2

90% / qtr23.015.037.0

 49.0

31

RY 1

90% / qtr

*NA3100.0

57.0

67

 

Definition/Description: Percentage of recommendations made to XSEDE that were addressed (i.e. change action taken or decision for no change) by Level 2 program area managers. The percentage is calculated using recommendations from the XSEDE Advisory Board, NSF Review Panels, Service Provider Forum, User Advisory Committee, and the XSEDE Staff Climate Study conducted by the external evaluation team. 

Collection methodology: The number of key recommendations made according to the total number of recommendations recorded on the XSEDE Project-Wide Improvements & Recommendations Google Sheet, such as XAB, NSF, annual XSEDE Climate Study findings, and other SP&E recommendations. XSEDE staff, not the SP&E team, are accountable for entering information to track improvements and recommendations.

NOTE: An improvement made is not necessarily a recommendation addressed, but any change made directly related to a recommendation is an improvement made and therefore should also be tracked in the XSEDE Project-Wide Improvements & Recommendations Google Sheet once fully implemented; it is OK to have duplication between Improvements fully implemented and Recommendations addressed. 


Grand (aggregate) mean of Staff Climate Study satisfaction items regarding content and accessibility of the XSEDE Staff Wiki 

RY 63.9 of 5 / yr -4.0


Sustain — Operate an effective and productive virtual organization 





RY 5 3.9 of 5 / yr 3.9  - 3.9
RY 43.9 of 5 / yr  - 3.8- - 3.8
RY 33.5 of 5 / yr -3.9- - 3.9
RY 2
*****
RY 1
*****
 

Definition/Description: Aggregate mean rating of staff satisfaction with the available content and ability to access information relevant to their work via the XSEDE Staff Wiki as measured annually by the XSEDE Staff Climate Study on a scale of 1 (not at all satisfied) - 5 (extremely satisfied), items 1A and 1C.

Collection methodology: A mean index score is generated from annual Staff Climate Study responses to all items within the "XSEDE Staff Wiki and Website" dimension of the study to determine aggregate mean.

Number of staff publications

RY 640 / yr122


 

 

Sustain — Operate an innovative virtual organization 





RY 5 50 / yr 1611 16 45 
RY 4  32 / yr 11 447 18 80
RY 320 / yr 19167 648 
RY 220 / yr 260

1

9
RY 170 / yr *50

 13

18

 

Definition/Description: Number of technical reports, presentations, and peer-reviewed publications authored or co-authored by XSEDE staff to date within the current reporting year.

Collection methodology: When an XSEDE staff member has a publication, he or she enters the publication data into their profile on the XUP. XSEDE documents added to IDEALS by XSEDE staff are also included. This data is then summed for the appropriate quarter and reported.

Grand (aggregate) mean of Staff Climate Study awareness items regarding inclusion 





RY 64.1 / yr-4.3


 

 

Sustain – Operate an innovative virtual organization



RY 5

 4.1 / yr 4.3 - 4.3

RY 4

 4.1 / yr - 4.2- - 4.2

RY 3

4.1 / yr-4.1 - - 4.1

RY 2

 **4.3*

*

4.3

RY 1

 *

***

*

*

 

Definition/Description: Aggregate mean rating of staff perception and experience relating to inclusion in the project as measured annually by the XSEDE Staff Climate Study on a scale of 1 (low inclusion) - 5 (high inclusion).

Collection methodology: A mean index score is generated from annual Staff Climate Study responses to items within the "Inclusion" dimension of the study. This score is then compared to the previous year to determine percent change on an annual basis.

Grand (aggregate) mean of Staff Climate Study awareness items regarding equity





RY 64.0 / yr-4.2


 

 

Sustain – Operate an innovative virtual organization


RY 5

 4.0 / yr 4.4 -4.4 

RY 4

 4.0 / yr - 4.2- - 4.2

RY 3

4.0 / yr-4.1- - 4.1

RY 2

*****

RY 1

 *

***

*

*

 

Definition/Description: Aggregate mean rating of staff perception and experience relating to diversity and equity in the project as measured annually by the XSEDE Staff Climate Study on a scale of 1 (low equity) - 5 (high equity).

Collection methodology: A mean index score is generated from annual Staff Climate Study responses to items within the "diversity" dimension of the study. This score is then compared to the previous year to determine percent change on an annual basis.

Number of XSEDE-related media hitsRY 6 600 / yr13194


 

 

Deepen/ Extend — Raise awareness of the value of advanced digital services 

RY 5  325 / yr624 467 326 121  1,538
RY 4  165 / yr 66 92101 237 496
RY 3 169 / yr232954 60 166
RY 2 169 / yr42304429145
RY 1 147 / yr*32301880
 

Definition/Description: Number of stories in the media within the reporting period which mention “XSEDE” and/or “Extreme Science and Engineering Discovery Environment” by name as found using Cision. 

Collection methodology: These media hits (i.e. non-XSEDE hosted/sponsored articles, blog posts, etc .) are found using Cision. This number should not include social media hits, as those are tracked separately. It should also not include content published by wire services and news release distribution services, nor should it include technical journal publications.

...