Page History

CEE (§3)

RY5

RY4

RY3

RY2

3,962

3,754

2,488

3,020

4,527

RY1

> 5,000 / qtr

4,755

4,446

4,924

6,186

Definition/Description: Number of sustained users within the current reporting period. A "sustained user" is any individual who created a portal account prior to the current reporting period and logged in during the reporting period.

Collection methodology: There is a usage log parsing script that collects the tomcat logs and parses out the relevant data during the selected time period.

CEE (§3)

490

408

296

402

1,640

322

238

535

1,095

Definition/Description: Number of sustained underrepresented users. A "sustained user" is any individual who created a portal account prior to the current reporting period and logged in during the reporting period. Only individuals from underrepresented communities including women and racial/ethnic domestic minorities in HPC are counted. Users identifying with multiple underrepresented communities are only counted once. 

Collection methodology: There is a usage log parsing script that collects the tomcat logs and parses out the relevant data during the selected time period. Self reported demographic XUP data is then matched to these data to select underrepresented users including women and racial/ethnic minorities from the list. Users identifying with multiple underrepresented communities are only counted once.

Percentage of sustained allocation users from non-traditional disciplines of XSEDE resources and services

ECSS (§4)

RY5

RY4

RY3

RY2

21.5

20.4

21.1

21.0

21.0

RY1

*

18.2

19.9

18.6

18.9

Definition/Description: Number of sustained allocation users from non-traditional disciplines in HPC divided by the total number of sustained allocation users within the reporting period.

Collection methodology: A set of 63 fields of science (FOS) have been identified in the XD Central Database (XDCDB), each of whose usage over the past 10 years is below 0.5% of the total normalized usage. A scripted query to XDCDB counts “sustained allocation users” as the total number of users on grants from these FOS that are active during the reporting period and have used at least 10% of their allocated usage on at least one service provider (SP). This is divided by the total number of users on all the grants that that are active during the reporting period and have used at least 10% of their allocated usage on at least one SP. 

Number of new users of XSEDE resources and services via the portal

CEE (§3)

RY 5

RY 4

RY 3

3,000 / qtr

1,905¹

RY 2

2,000 / qtr

2,305

2,813

2,346

2,917

10,381

RY 1

>1,000 / qtr

*

1,973

1,849

2,359

6,181

Definition/Description: Number of new users of XSEDE resources and services as determined by the number of portal accounts created within the reporting period.

Collection methodology: The Liferay Database is queried for the number of users in the Users table that was created during the selected time periods. Sample query: select screenName, createDate from User_ where createDate between '2016-09-01 00:00:00' AND '2016-10-31 00:00:00';

Number of new underrepresented individuals using XSEDE resources and services via the portal

CEE (§3)

RY 5

RY 4

RY 3

200 / qtr

134

RY 2

150 / qtr

251

175

222

234

950

RY 1

100 / qtr

*

150

135

240

525

Definition/Description: Number of new underrepresented users. A "new user" is any individual who created a portal account within the reporting period. Only individuals from underrepresented communities including women and racial/ethnic domestic minorities in HPC are counted. Users identifying with multiple underrepresented communities are only counted once. 

Collection methodology: XSEDE portal users are able to specify their race/ethnicity, gender, and institution in their user profile and are encouraged to complete this data when registering for training events through the XSEDE portal. Date of allocation awards is also recorded in the XSEDE database (XDCDB). A query is then run to generate the number of individuals meeting the aforementioned criteria within a given quarter. Care is taken to not over represent this metric by ensuring an individual who meets multiple criteria is only accounted for one time.

ECSS (§4)

 NA

24.8

26.0

26.6

33.9

27.8

 NA

*

21.8

24.9

31.0

25.7

Definition/Description: Number of new allocation users from non-traditional disciplines in HPC divided by the total number of new allocation users within the reporting period. 

Collection methodology: A set of 63 fields of science (FOS) have been identified in the XD Central Database (XDCDB), each of whose usage over the past 10 years is below 0.5% of the total normalized usage. A scripted query to XDCDB counts “new allocation users” as the total number of users on the grants from these FOS that have been newly awarded during the reporting period. This is divided by the total number of users on all the grants that have been awarded during the reporting period.

Number of participant hours of live training delivered by XSEDE

CEE (§3)

RY 5

RY 4

RY 3

40,000 / yr

14,140

RY 2

12,787

8,876

6,004

14,753

42,421

RY 1

 NA¹

5,994

3,770

5,180

9,199

24,143

Definition/Description: Number of total participant hours spent in XSEDE live, in-person, or webcast events during the current reporting period.

Collection methodology: Measured using registrations via the XSEDE User Portal. Does not include training events that do not require registration such as Cornell Virtual Workshop, CI-Tutor, Youtube videos, etc. 

Grand (aggregate) mean of XSEDE User Survey awareness items regarding XSEDE resources and services

Program Office (§8)

RY 5

RY 4

RY 3

3.5 of 5 / yr

3.7

RY 2

 *

3.6

-

-

-

3.6

RY 1

* 

*

-

-

-

NA

Definition/Description: Aggregate mean rating of user awareness of XSEDE resources and services as measured by the annual User Survey on a scale of 1 (not at all aware) - 5 (extremely aware). The core items included in the calculation for this purpose are: Mission, Computational Resources, Data Storage Services, Viz Services, Science Gateways, XUP, Data Transfer Services, Website, Training Opportunities, Online Technical Documentation; Community Engagement and Enrichment Opportunities (f.k.a., Education and Outreach Opportunities); Help Desk Services, and ECSS.

Collection methodology: Aggregate rating of awareness scale items measuring current XSEDE resources and services listed on the annual XSEDE User Survey.

Number of social media impressions over time¹ 

Program Office (§8)

RY 5

86,046 

RY 4

RY 3

359,714 / yr

112,806

RY 2

NA

69,607

55,506

59,490

128,180

312,783

RY 1

NA

*

52,200

128,675

88,332

269,207

Definition/Description: Each reporting period the number of people who have seen XSEDE interactions on Facebook + Twitter is reported. "Impressions" are generated by Facebook and Twitter to determine how many individuals have seen XSEDE interactions or posts on said platforms. This includes "likes," "retweets," "followers," “shares,” as well as people who see the post based on their friends or followers sharing XSEDE information. At the end of each reporting year, impressions across the four reporting periods for that year are summed, and a percentage increase over the total number from the previous reporting year is calculated as follows: Number of social media impressions within the current reporting period year (x) minus the number of social media impressions during the previous reporting year (y). This number is divided by the number of social media impressions in the previous reporting year (y) to determine percent increase (((x-y)/y)*100)).

Collection methodology: The External Relations team pulls the information from Facebook and Twitters' internal tracking methods at the end of each report period. The percent increase is then computed based on the previous reporting year data.

Total number of capabilities in production¹

XCI (§5)

RY 5

RY 4

RY 3

81 by end of RY3

76

85¹

87

87 

87 

RY 2

NA

72

74

74

75

75

RY 1

NA

*

63

63

69

69

Definition/Description: Total number of capabilities in production across XSEDE SPs, central operations, partner service providers (Globus, etc.), campuses, science gateways, etc.

Collection methodology: A list of capabilities in production through XSEDE is kept in the Community Software Repository which catalogs Use Cases and Capability Delivery Plans.

Grand (aggregate) mean of XSEDE User Survey satisfaction items regarding XSEDE technical support services¹

Program Office (§8)

RY 5

RY 4

RY 3

3.5 of 5 / yr

3.6

RY 2

NA

3.4

-

-

-

3.4

RY 1

NA

*

*

*

*

*

Definition/Description: Aggregate rating of user satisfaction with XSEDE technical expertise and support services via the annual User Survey on a scale of 1 (very dissatisfied) - 5 (very satisfied).

Collection methodology: Aggregate rating of all current "XSEDE Software and Service Components" measured using the "frequency of performance meeting expectations" scale on the annual XSEDE User Survey.

Mean composite availability of core services (%)

Operations (§6)

RY 5

 99.9% / qtr

RY 4

RY 3

99.9% / qtr

99.9

RY 2

RY 1

99.0% / qtr

*

99.9

99.9

99.9

99.9

Definition/Description: Mean composite availability of core services is the geometric mean of % core enterprise services availability and % POPS/XRAS availability. Availability percentage of each component is measured separately, then aggregated and averaged using a geometric mean to determine the composite availability.

Mean composite of: "Average availability of critical enterprise services (%) [geometric mean]" (Systems Operational Support 2.4.5) and "Availability of XRAS systems" (AA&AM 2.5.3).

Collection Methodology: Average availability of critical enterprise services (%) [geometric mean]: The data is stored locally to the Nagios service at IU. Nagios is an application that constantly logs the availability of core central services. These logs are queried to determine if/when a service becomes unavailable to users.

Availability of XRAS systems: There is a table acct.downtime in the XDCDB, which is manually completed when there is an outage in the XDCDB, a network outage disrupting availability of the XRAS services, or an outage of the XSEDE User Portal. Currently, outages are detected via a variety of e-mail notifications, and the data is manually entered into the XDCDB.

Mean time to ticket resolution (hours)

Operations (§6)

RY 5

RY 4

RY 3

< 16 hrs/ qtr¹

12.3

RY 2

RY 1

< 24 hrs/ qtr

23.1

25.1

Definition/Description: Mean time to ticket resolution within XSEDE WBS ticket queues and the XOC queue. This metric does not apply to Service Provider ticket queues.

Collection methodology: The data is stored locally via the Request Tracker service, which is the XSEDE ticket software and which is located at TACC. Reports are generated against the RT database to determine MTTR and other metrics.

Mean rating of user satisfaction with allocations process and support services¹

RAS (§7)

RY 5

  4 of 5 / yr

RY 4

RY 3

4 of 5 / yr

4.1

RY 2

4 of 5 / yr

4.1

4.0

4.1

3.9

4.0

RY 1

4 of 5 / yr

*

4.0

4.0

4.0

4.0

Definition/Description: Aggregate mean rating of user satisfaction with allocations process and XRAS services on a scale of 1 (not at all satisfied) - 5 (extremely satisfied).

Collection methodology: Average of the L2 KPI satisfaction ratings for the allocations process and for the XRAS system.

RAS (§7)

85.0% / qtr

65.0

85.0% / qtr

70.0

69.0

72.0

68.0

69.8

85.0% / qtr

*

76.0

75.0

74.0

75.0

Definition/Description: Number of Research allocation requests receiving some non-zero award from an XRAC meeting divided by the number of Research allocation requests submitted to the same XRAC meeting.

Collection methodology: Request data comes from the Quarterly XRAC meetings and is stored in XRAS as well as some meeting spreadsheets.

Mean rating of importance of XSEDE resources and services to researcher productivity

Program Office (§8)

RY 5

RY 4

RY 3

4.2 of 5 / yr

4.4

RY 2

4.4²

-

-

-

4.4

RY 1

4.3²

-

-

-

4.3

Definition/Description: Mean rating of importance of XSEDE resources and services to researcher productivity as measured by the annual User Survey on a scale of 1 (not important at all) - 5 (essential).

Collection methodology: A mean rating is calculated from the responses to the annual XSEDE User Survey item, "How important are XSEDE resources to your work?"

Program Office (§8)

Definition/Description: Percentage of users who self-report the use of XSEDE resources in their work product when responding to the annual User Survey.

Collection methodology: Annual User Survey response. The Annual User Survey is a representative sample of XSEDE Users who have interacted with XSEDE via the portal or an allocation within 2 years of the current dissemination date. When particular groups are targeted, it is to increase the response rate from that group in order to ensure adequate representation that mimics the general population. The evaluation team is confident that the responses from this group can be generalized.

Percentage of Project Improvement Fund funded projects resulting in innovations in the XSEDE organization

Program Office (§8)

60% / yr

-

*

*

*

*

*

*

*

*

*

*

Definition/Description:  Number of completed PIF (Project Improvement Fund) funded projects resulting in innovations as determined by SMT review divided by the total number of completed PIF funded projects to date. 

Collection methodology: The results of each Project Improvement Fund approved submissions to date is evaluated, at the end of the project year, to assess the level of innovation. This is a manual effort completed by the SP&E team. 

Mean rating of innovation within the organization by XSEDE staff

Program Office (§8)

RY 5

RY 4

RY 3

3.5 of 5 / yr

-

RY 2

*

*

*

*

*

RY 1

 *

*

*

*

*

*

Definition/Description: Mean rating of innovation within the organization by XSEDE employees as measured by the annual XSEDE Staff Climate Study on a scale of 1 (not at all innovative) - 5 (highly innovative).

Collection methodology: New item added to 2018 XSEDE Staff Climate Study where staff will rate the level of innovation within the organization of a scale of 1 (low/negative) to 5 (high/positive). 

Number of students benefiting from XSEDE resources and services through training, XSEDE projects, or conference attendance

Deepen/Extend – Extend use to new communities

RY 5

RY 4

RY 3

1,250 / qtr

1,613 

RY 2

950 / qtr

1,722

1,478

1,170

1,522

1,802

RY 1

50 / qtr

*

997

815

2,679

3,122

Definition/Description: Number of students who participated in XSEDE training, were supported by XSEDE to work on XSEDE projects, received funding to attend the annual PEARC Conference (formerly XSEDExy), or SC conference as indicated by registrations in the XSEDE portal during the current reporting period.

Collection methodology: Measured using registrations via the XSEDE User Portal, participants in CEE student programs (i.e. EMPOWER), and PEARC student program attendees.

Number of under-represented students benefiting from XSEDE resources and services through training, XSEDE projects, or conference attendance

Deepen/Extend – Deepen use to existing communities

RY 5

RY 4

RY 3

625 / qtr

449

RY 2

475 / qtr

488

399

347

423

1,104

RY 1

50 / qtr

*

34 

33

19

28

Definition/Description: Number of underrepresented students who participated in XSEDE training, were supported by XSEDE to work on XSEDE projects, received funding to attend the annual PEARC Conference (formerly XSEDExy), or SC conferences as indicated by registrations in the XSEDE portal during the current reporting period. Only students from underrepresented communities including women and racial/ethnic domestic minorities in HPC are counted in the numerator. Students identifying with multiple underrepresented communities are only counted once.

Collection methodology: Measured using registrations via the XSEDE User Portal, participants in CEE student programs (i.e. EMPOWER), and PEARC student program attendees. Only students from underrepresented communities including women and racial/ethnic domestic minorities in HPC are counted in the numerator. Students identifying with multiple underrepresented communities are only counted once.

Grand (aggregate) mean rating of Post Training Event Survey items related to training impact for attendees registered through the portal

Deepen/Extend – Extend use to new communities

RY 5

RY 4

RY 3

4.4 of 5 / qtr

4.5 

RY 2

4 of 5 / qtr

4.3

4.3

4.6

4.5

4.4

RY 1

4 of 5 / qtr

*

4.5

4.4 

4.3

4.4

Definition/Description: Aggregate mean rating of training impact for attendees registered via the XSEDE User Portal for XSEDE delivered training sessions as measured by the Post Training Event Survey on a scale of 1 (low impact) to 5 (high impact).

Collection methodology: The data is collected from post event surveys and recorded in the database. Specific items included in this index are:

Q1. The training session fulfilled my expectations.
Q2. The trainer stimulated my interest.
Q6. The training session was well-organized.
Q8. I was able to easily access this training session.
Q10. Overall I would rate my experience as successful.

Number of institutions with a Champion

Deepen/Extend – Deepen use to existing communities

RY 5

RY 4

RY 3

250

259 

RY 2

240

218

238

239

246 

246

RY 1

225

*

224

231

234

234

Definition/Description: Total number of institutions that have a signed letter of collaboration with the Champions program.

Collection methodology: All champions are tracked using a master spreadsheet managed by champion program managers and coordinator. All champions in the spreadsheet with a current corresponding letter of collaboration with XSEDE are counted for this metric.

Percentage of user requirements addressed within 30 days

100 (12/12)

Deepen/Extend – Prepare the current and next generation

RY 5

 100

(20/20)

RY 4

 100
(34/34)

 96
(23/24)

 97
(36/37)

RY 3

100% / qtr

89 (40/45) 

100 (27/27)

RY 2

100% / qtr

78 (36/44)

102 (47/46)

86 (32/37)

93 (41/44)

91 (156/157)

RY 1

100% / qtr

*

50 (16/32)

89 (40/45)

75 (40/53)

74 (96/130)

Definition/Description: The number of user requirements addressed within 30 days during the current reporting year divided by the total number of requirements identified by the XSEDE User Engagement team through regular email contact with all XSEDE PIs. Total number of requirements identified does not include those identified within the last 30 days of the reporting year.

Collection methodology: All requirements addressed within 30 days tracked in JIRA within the current reporting year divided by the total number of requirements in JIRA within the same year that are over 30 days old. For purposes of this metric, the time period includes the last 30 days of the previous reporting year but does not include the last 30 days of the current reporting year. Requirements are considered addressed when the PI is presented with a solution via email, a ticket is submitted via the XSEDE ticket system, or a use case is submitted for further consideration.

Number of completed ECSS projects

Deepen/Extend – Deepening use to existing communities

RY 5

RY 4

RY 3

45 / yr

17 

RY 2

50 / yr

16

9

10

12

47

RY 1

50 / yr

*

10

13

25

48

Definition/Description: The total number of completed ECSS projects in ESRT, ESCC, and ESSGW within the current reporting period. A completed project is defined as having progressed through the full support pipeline, namely assigning a consultant, producing a work plan, executing the work plan, reporting progress through quarterly reports, and filing of a final project report.

Collection methodology: The number of completed projects is tracked in JIRA, XSEDE’s project management software. A report has been created in JIRA that queries all the ECSS projects, filters for allocations that have ended in that quarter and also for projects that have a final report.

Grand (aggregate) mean rating of ECSS impact by PIs measured by ECSS Project Exit Survey items

Deepen/Extend – Deepen use to existing communities

RY 5

RY 4

RY 3

4 of 5 / yr

 NA

RY 2

4 of 5 / yr

4.1 

4.0

3.8 

 4.3

4.0

RY 1

4 of 5 / qtr

*

4.6 

4.6

3.3

4.1

Definition/Description: Aggregate mean rating of ECSS impact by PIs collected after an ECSS project is complete (i.e. has a workplan, work has progressed, and a final report has been filed) measured by the ECSS Project Exit Survey on a scale of 1 (low impact) - 5 (high impact).

Collection methodology:

Grand (aggregate) mean rating of PI satisfaction with ECSS support measured by ECSS Project Exit Survey items

Deepen/Extend – Deepening use to existing communities

4.8

4.5

RY1

4.5 of 5 / qtr

4.6

4.5

Definition/Description: Aggregate mean rating of PI satisfaction with ECSS support collected after an ECSS project is complete (i.e. has a work plan, work has progressed, and a final report has been filed) measured by the ECSS Project Exit Survey on a scale of 1 (not at all satisfied) - 5 extremely satisfied).

Collection methodology: Measured by the average of selected question(s) of the PI's satisfaction survey.

Grand (aggregate) mean rating of XSEDE User Survey user satisfaction items regarding XCI software and technical services, capabilities, and resources⁴

Advance – Create an open and evolving e-infrastructure

RY 5

RY 4

RY 3

RY 2

*

* 

RY 1

*

*

*

Definition/Description: Aggregate rating of USER satisfaction with XSEDE software and technical services, capabilities, and resources via the annual User Survey on a scale of 1–5, where “1” is “performance never meets expectations” and “5” is “performance always meets expectations.”

Collection methodology: Aggregate ratings of USER-facing XCI software and technical services, capabilities and resources as measured using the "frequency of performance meeting expectations" scale on the annual XSEDE User Survey, to include the following items: Authenticating to the XSEDE User Portal and other web services using campus InCommon credentials (CILogon); Using XSEDE credentials to log into non-XSEDE Web services (e.g., GENI and OrCID.); Globus Transfer Service; Globus Sharing Service; Globus Connect for moving data between personal systems and XSEDE resources; Job start prediction service (Karnak); Login to XSEDE affiliated web sites without entering a password every time; Cluster integration software (XCBC); Scientific software for campus clusters (XNIT); Cluster integration tutorials and documentation; and Cluster integration site visits.

Grand (aggregate) mean rating of XSEDE User Survey Service Provider satisfaction items regarding XCI software and technical services, capabilitiies, and resources

Definition/Description: Aggregate rating of SP satisfaction with XSEDE software and technical services, capabilities, and resources via the annual User Survey on a scale of 1–5, where “1” is “performance never meets expectations” and “5” is “performance always meets expectations.”

Collection methodology: Aggregate ratings of SP-facing XCI software and technical services, capabilities and resources as measured using the "frequency of performance meeting expectations" scale on the annual XSEDE User Survey, to include the following items: IPF software used to publish resource information to XSEDE; XSEDE accepted CA certificates; XSEDE Resource Operational Status Dashboards; XSEDE Resource Integration Steps Dashboard; Resource Description Repository (RDR); Install software from XSEDE RPM repositories on software.xsede.org; Opportunities to discuss software needs, delivery activities, deployment, and usage; Opportunities to participate in software technical reviews; Software Integration Support; Software Usage Instrumentation Support; Use case requirements analysis processes; and Use case development processes; and Use case prioritization processes. In order to get an adequate number of responses from Service Providers, that population will be over-sampled. If that does not result in an adequate response rate, these questions will be pulled and sent to the SPs separately to improve the number of responses.

Number of non-XSEDE partnerships with XCI²

RY 6

Advance – Create an open and evolving e-infrastructure

RY 4

RY 3

RY 2

 -

8

RY 1

-

-

8

Definition/Description: Number of XCI partnerships supported/facilitated between CRI and SPs, national CI organizations, and campus CI providers.

Collection methodology: Non-XSEDE partnerships are tracked through the ticket system and through JIRA.

Sustain — Provide excellent user support 

5.0

6.1

< 45 days

Definition/Description: Average number of days from defect or support request submitted to solution communicated.

Collection methodology: The data is stored locally via the Request Tracker service, which is the XSEDE ticket software. Reports are generated against the RT database to determine MTTR and other metrics.

Mean rating of user satisfaction with tickets closed by the XOC

Sustain – Provide excellent user support.

RY 5

RY 4

RY 3

RY 2

4.4

RY 1

4 of 5 / qtr

5.0

4.7

Definition/Description: Mean rating of satisfaction by users who submitted help tickets that were closed by the XOC during the specified reporting period as measured by the XSEDE Operations Point-of-Service Satisfaction Survey on a scale of 1 (not at all satisfied) - 5 (extremely satisfied).

Collection methodology: Within two hours of the resolution and closure of a ticket, the user receives an e-mail containing a link to a Survey Monkey survey. The survey consists of five questions using a 5 point Likert scale and the results are stored in Survey Monkey.

Hours of downtime with direct user impacts from XSEDE security incidents

Sustain – Provide reliable, and secure infrastructure

RY 5

RY 4

RY 3

RY 2

0

0

RY 1

<24 hrs/ qtr

0

146

Definition/Description: Hours of downtime is calculated as the Time to Return to Repair (TTR) across all applicable services and incidents during the quarter in order to measure resource unavailability as a result of an XSEDE-wide security event (involving a Tier 1 service or spanning more than one SP).

Collection methodology: The Cybersecurity co-leads determine that an XSEDE-wide security event is responsible for an outage. The Sysops lead will determine the TTR based on monitoring and other information in logs and tickets, and the TTR value will be the data point reported.

Mean rating of user satisfaction with allocations process

Sustain – Provide excellent user support

RY 5

RY 4

RY 3

RY 2

3.9

4.0

RY 1

4 of 5 / qtr

4.0

4.0

Definition/Description: Aggregate mean rating of user satisfaction across 10 facets of allocation policies and procedures related to the submission, review, and administration of allocation requests on a scale of 1 (not at all satisfied) - 5 (extremely satisfied).

Collection methodology: Quarterly Post-Allocations Submission Survey of persons who have submitted allocation requests in the prior quarter.

Mean rating of user satisfaction with XRAS

Sustain – Provide reliable, efficient, and secure infrastructure

RY 5

RY 4

RY 3

RY 2

4.0

4.0

RY 1

4 of 5 / qtr

4.0

4.0

Definition/Description: Aggregate mean rating of user satisfaction with XRAS (XSEDE Resource Allocation System) across 8 facets of the user-facing submission process on a scale of 1 (not at all satisfied) - 5 (extremely satisfied).

Collection methodology: Quarterly Post-Allocations Submission Survey of persons who have submitted allocation requests in the prior quarter

Variance between relevant report submission and due date (days) 

Sustain — Operate an effective and productive virtual organization 

RY 5

RY 4

RY 3

RY 2

0

0

RY 1

0

0

0

Definition/Description: Variance in number of calendar days between report submission and due date for the previous reporting period. In accordance with NSF quarterly and annual reporting requirements, deadlines are determined together by the XSEDE PI and NSF Project Officer and then communicated to the XSEDE team to establish the reporting schedule used to calculate this metric.

Collection methodology: The XSEDE PI tracks when the relevant document was submitted to FastLane and records the number of days between the due date and this date in a spreadsheet.

Percentage of sub-award invoices processed within target duration

Sustain — Operate an effective and productive virtual organization 

RY 5

RY 4

RY 3

RY 2

67.4

74.6

RY 1

90% / qtr

NA²

100

Definition/Description: Number of invoices processed within the target duration divided by the total number of invoices processed. Duration is determined by the number of calendar days from acceptance of submitted invoice to invoice paid. The target duration is less than 45 calendar days. 

Collection methodology: Data is logged and stored in the Business Operations master spreadsheet located on the Business Operations Google Drive folder.

Percentage of recommendations addressed by relevant project areas within 90 days

Sustain – Operate an effective and productive virtual organization

RY 5

RY 4

RY 3

RY 2

 49.0

31

RY 1

90% / qtr

57.0

67

Definition/Description: Percentage of recommendations made to XSEDE that were addressed (i.e. change action taken or decision for no change) by Level 2 program area managers. The percentage is calculated using recommendations from the XSEDE Advisory Board, NSF Review Panels, Service Provider Forum, User Advisory Committee, and the XSEDE Staff Climate Study conducted by the external evaluation team. 

Collection methodology: The number of key recommendations made according to the total number of recommendations recorded on the XSEDE Project-Wide Improvements & Recommendations Google Sheet, such as XAB, NSF, annual XSEDE Climate Study findings, and other SP&E recommendations. XSEDE staff, not the SP&E team, are accountable for entering information to track improvements and recommendations.

NOTE: An improvement made is not necessarily a recommendation addressed, but any change made directly related to a recommendation is an improvement made and therefore should also be tracked in the XSEDE Project-Wide Improvements & Recommendations Google Sheet once fully implemented; it is OK to have duplication between Improvements fully implemented and Recommendations addressed. 

Grand (aggregate) mean of Staff Climate Study satisfaction items regarding content and accessibility of the XSEDE Staff Wiki 

Definition/Description: Aggregate mean rating of staff satisfaction with the available content and ability to access information relevant to their work via the XSEDE Staff Wiki as measured annually by the XSEDE Staff Climate Study on a scale of 1 (not at all satisfied) - 5 (extremely satisfied), items 1A and 1C.

Collection methodology: A mean index score is generated from annual Staff Climate Study responses to all items within the "XSEDE Staff Wiki and Website" dimension of the study to determine aggregate mean.

Number of staff publications

Sustain — Operate an innovative virtual organization 

1

 13

18

Definition/Description: Number of technical reports, presentations, and peer-reviewed publications authored or co-authored by XSEDE staff to date within the current reporting year.

Collection methodology: When an XSEDE staff member has a publication, he or she enters the publication data into their profile on the XUP. XSEDE documents added to IDEALS by XSEDE staff are also included. This data is then summed for the appropriate quarter and reported.

Grand (aggregate) mean of Staff Climate Study awareness items regarding inclusion 

Sustain – Operate an innovative virtual organization

RY 5

RY 4

RY 3

RY 2

*

4.3

RY 1

 *

*

*

Definition/Description: Aggregate mean rating of staff perception and experience relating to inclusion in the project as measured annually by the XSEDE Staff Climate Study on a scale of 1 (low inclusion) - 5 (high inclusion).

Collection methodology: A mean index score is generated from annual Staff Climate Study responses to items within the "Inclusion" dimension of the study. This score is then compared to the previous year to determine percent change on an annual basis.

Grand (aggregate) mean of Staff Climate Study awareness items regarding equity

Sustain – Operate an innovative virtual organization

RY 5

RY 4

RY 3

RY 2

RY 1

 *

*

*

Definition/Description: Aggregate mean rating of staff perception and experience relating to diversity and equity in the project as measured annually by the XSEDE Staff Climate Study on a scale of 1 (low equity) - 5 (high equity).

Collection methodology: A mean index score is generated from annual Staff Climate Study responses to items within the "diversity" dimension of the study. This score is then compared to the previous year to determine percent change on an annual basis.

Deepen/ Extend — Raise awareness of the value of advanced digital services 

Definition/Description: Number of stories in the media within the reporting period which mention “XSEDE” and/or “Extreme Science and Engineering Discovery Environment” by name as found using Cision. 

Collection methodology: These media hits (i.e. non-XSEDE hosted/sponsored articles, blog posts, etc .) are found using Cision. This number should not include social media hits, as those are tracked separately. It should also not include content published by wire services and news release distribution services, nor should it include technical journal publications.