Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

Table 2-4: KPIs for the sub-goal of raise awareness of the value of advanced digital research services

KPI

Program Year

Target

RP1

RP2

RP3

RP4

Total

Owner(s)

Grand (aggregate) mean of XSEDE User Survey awareness items regarding XSEDE resources and services

(1-5 point Likert scale)


RY 63.7 of 5 / yr





 

 

Program Office (§8)



RY 5

 3.7 of 5 / yr3.8  -3.8 

RY 4

 3.7 of 5 / yr 3.8 - - - 3.8

RY 3

3.5 of 5 / yr

3.7

-- - 3.7

RY 2

 *

3.6

-

-

-

3.6

RY 1

*

-

-

-

NA

 

Definition/Description: Aggregate mean rating of user awareness of XSEDE resources and services as measured by the annual User Survey on a scale of 1 (not at all aware) - 5 (extremely aware). The core items included in the calculation for this purpose are: Mission, Computational Resources, Data Storage Services, Viz Services, Science Gateways, XUP, Data Transfer Services, Website, Training Opportunities, Online Technical Documentation; Community Engagement and Enrichment Opportunities (f.k.a., Education and Outreach Opportunities); Help Desk Services, and ECSS.

Collection methodology: Aggregate rating of awareness scale items measuring current XSEDE resources and services listed on the annual XSEDE User Survey.

Number of social media impressions over time1 





RY 6424,000 / yr







Program Office (§8)



RY 5

  430,000/ yr

86,046 

 101,394106,263 109,800 403,503 

RY 4

 426,198 / yr 87,48275,16484,185 110,904 357,735

RY 3

359,714 / yr

112,806

63,26993,80385,287355,165

RY 2

NA

69,607

55,506

59,490

128,180

312,783

RY 1

NA

*

52,200

128,675

88,332

269,207

 

Definition/Description: Each reporting period the number of people who have seen XSEDE interactions on Facebook + Twitter is reported. "Impressions" are generated by Facebook and Twitter to determine how many individuals have seen XSEDE interactions or posts on said platforms. This includes "likes," "retweets," "followers," “shares,” as well as people who see the post based on their friends or followers sharing XSEDE information. At the end of each reporting year, impressions across the four reporting periods for that year are summed, and a percentage increase over the total number from the previous reporting year is calculated as follows: Number of social media impressions within the current reporting period year (x) minus the number of social media impressions during the previous reporting year (y). This number is divided by the number of social media impressions in the previous reporting year (y) to determine percent increase (((x-y)/y)*100)).

Collection methodology: The External Relations team pulls the information from Facebook and Twitters' internal tracking methods at the end of each report period. The percent increase is then computed based on the previous reporting year data.

 

-  Data reportedly annually.

This was a new KPI in RY3. Data provided for RY1 and RY2 was reported retroactively. Beginning in RY5, this KPI is calculated as the number of social media impressions with the annual target calculated based on a 20% increase over the previous year.

2.2       Advance the Ecosystem

...

KPI

Program Year

Target

RP1

RP2

RP3

RP4

Total

Owner(s)

Grand (aggregate) mean of XSEDE User Survey satisfaction items regarding XSEDE technical support services1 (1-5 point Likert scale)



RY 63.8 of 5 / yr





 

 

Program Office (§8)



RY 5

 3.5 of 5 / yr4.4  -4.4 

RY 4

 3.5 of 5 / yr 3.9 -- - 3.9

RY 3

3.5 of 5 / yr

3.6

-- - 3.6

RY 2

NA

3.4

-

-

-

3.4

RY 1

NA

*

*

*

*

*

 

Definition/Description: Aggregate rating of user satisfaction with XSEDE technical expertise and support services via the annual User Survey on a scale of 1 (very dissatisfied) - 5 (very satisfied).

Collection methodology: Aggregate rating of all current "XSEDE Software and Service Components" measured using the "frequency of performance meeting expectations" scale on the annual XSEDE User Survey.

...

KPI

Program Year

Target

RP1

RP2

RP3

RP4

Total

Owner(s)

Mean time to ticket resolution (hours)





RY 6< 16 hrs/ qtr




 

 

Operations (§6)


RY 5

  < 16 hrs/ qtr15.3  23.218.8 12.2 17.4 

RY 4

 < 16 hrs/ qtr 14.0 14.621.0 14.3 16.0

RY 3

< 16 hrs/ qtr1

12.3

18.523.217.5 17.9

RY 2

< 24 hrs/ qtr2620.122.815.021.0

RY 1

< 24 hrs/ qtr

*24.028.2

23.1

25.1

 

Definition/Description: Mean time to ticket resolution within XSEDE WBS ticket queues and the XOC queue. This metric does not apply to Service Provider ticket queues.

Collection methodology: The data is stored locally via the Request Tracker service, which is the XSEDE ticket software and which is located at TACC. Reports are generated against the RT database to determine MTTR and other metrics.

Mean rating of user satisfaction with allocations process and support services1

(1-5 point Likert scale)




RY 6 4 of 5 / yr




 

 

RAS (§7)



RY 5

  4 of 5 / yr

4.4  4.34.3 4.3 4.3 

RY 4

 4 of 5 / yr 4.2 4.34.4 4.2 4.3

RY 3

4 of 5 / yr

4.1

4.24.1 4.4 4.2

RY 2

4 of 5 / yr

4.1

4.0

4.1

3.9

4.0

RY 1

4 of 5 / yr

*

4.0

4.0

4.0

4.0

 

Definition/Description: Aggregate mean rating of user satisfaction with allocations process and XRAS services on a scale of 1 (not at all satisfied) - 5 (extremely satisfied).

Collection methodology: Average of the L2 KPI satisfaction ratings for the allocations process and for the XRAS system.

Percentage of research requests successful (not rejected)RY 685.0% / qtr






RAS (§7)



RY 5 85.0% / qtr80.0 84.0  84.0 91.0 84.8
RY 4 85.0% / qtr 81.0 80.0 78.0 87.0 82.0
RY 3

85.0% / qtr

65.0

70.072.075.0  70.5
RY 2

85.0% / qtr

70.0

69.0

72.0

68.0

69.8

RY 1

85.0% / qtr

*

76.0

75.0

74.0

75.0

 

Definition/Description: Number of Research allocation requests receiving some non-zero award from an XRAC meeting divided by the number of Research allocation requests submitted to the same XRAC meeting.

Collection methodology: Request data comes from the Quarterly XRAC meetings and is stored in XRAS as well as some meeting spreadsheets.

...

KPI

Program Year

Target

RP1

RP2

RP3

RP4

Total

Owner(s)

Percentage of Project Improvement Fund funded projects resulting in innovations in the XSEDE organization

RY 670% / yr






Program Office (§8)



RY 5  70% / yr - 66.766.7 
RY 4 70% / yr - -- 66.7 66.7
RY 3

60% / yr

-

--71.4 71.4 
RY 2 *

*

*

*

*

*

RY 1 *

*

*

*

*

*

 

Definition/Description:  Number of completed PIF (Project Improvement Fund) funded projects resulting in innovations as determined by SMT review divided by the total number of completed PIF funded projects to date. 

Collection methodology: The results of each Project Improvement Fund approved submissions to date is evaluated, at the end of the project year, to assess the level of innovation. This is a manual effort completed by the SP&E team. 

 

Mean rating of innovation within the organization by XSEDE staff (1-5 point Likert scale)





RY 64 of 5 / yr




 

 

Program Office (§8)



RY 5

  4 of 5 / yr - 4.2 - -4.2 

RY 4

 4 of 5 / yr - 4.0- - 4.0

RY 3

3.5 of 5 / yr

-

4.0- -4.0 

RY 2

*

*

*

*

*

RY 1

 *

*

*

*

*

*

 

Definition/Description: Mean rating of innovation within the organization by XSEDE employees as measured by the annual XSEDE Staff Climate Study on a scale of 1 (not at all innovative) - 5 (highly innovative).

Collection methodology: New item added to 2018 XSEDE Staff Climate Study where staff will rate the level of innovation within the organization of a scale of 1 (low/negative) to 5 (high/positive). 

...

KPI

Report Year

Target

RP1

RP2

RP3

RP4

Total

 Goal Supported

Number of completed ECSS projects





RY 645 / yr




 

Deepen/Extend – Deepening use to existing communities



RY 5

 45 / yr17  1018 10 55 

RY 4

 45 / yr 15 11 16 10 52

RY 3

45 / yr

17 

1012 8 47

RY 2

50 / yr

16

9

10

12

47

RY 1

50 / yr

*

10

13

25

48

 

Definition/Description: The total number of completed ECSS projects in ESRT, ESCC, and ESSGW within the current reporting period. A completed project is defined as having progressed through the full support pipeline, namely assigning a consultant, producing a work plan, executing the work plan, reporting progress through quarterly reports, and filing of a final project report.

Collection methodology: The number of completed projects is tracked in JIRA, XSEDE’s project management software. A report has been created in JIRA that queries all the ECSS projects, filters for allocations that have ended in that quarter and also for projects that have a final report.

Grand (aggregate) mean rating of ECSS impact by PIs measured by ECSS Project Exit Survey items (1-5 point Likert scale)


RY 6 4 of 5 / yr




 

 

Deepen/Extend – Deepen use to existing communities



RY 5

  4 of 5 / yr4.1  4.2 4.54.8 4.3 

RY 4

 4 of 5 / yr 3.0 4.7 4.3 4.7 4.5

RY 3

4 of 5 / yr

 NA

3.94.4 4.3 4.2

RY 2

4 of 5 / yr

4.1 

4.0

3.8 

 4.3

4.0

RY 1

4 of 5 / qtr

*

4.6 

4.6

3.3

4.1

 

Definition/Description: Aggregate mean rating of ECSS impact by PIs collected after an ECSS project is complete (i.e. has a workplan, work has progressed, and a final report has been filed) measured by the ECSS Project Exit Survey on a scale of 1 (low impact) - 5 (high impact).

Collection methodology:

Grand (aggregate) mean rating of PI satisfaction with ECSS support measured by ECSS Project Exit Survey items (1-5 point Likert scale)

RY64.5 of 5 / yr






Deepen/Extend – Deepening use to existing communities



RY5  4.5 of 5 / yr4.8  4.93.8 5.0 4.8 
RY4 4.5 of 5 / yr 3.0 5.0 5.0 4.7 4.8
RY34.5 of 5 / yrNA4.34.8 4.54.6 
RY24.5 of 5 / yr4.74.64.2

4.8

4.5

RY1

4.5 of 5 / qtr

*4.94.7

4.6

4.5

 

Definition/Description: Aggregate mean rating of PI satisfaction with ECSS support collected after an ECSS project is complete (i.e. has a work plan, work has progressed, and a final report has been filed) measured by the ECSS Project Exit Survey on a scale of 1 (not at all satisfied) - 5 extremely satisfied).

Collection methodology: Measured by the average of selected question(s) of the PI's satisfaction survey.


...

KPI

Report Year

Target

RP1

RP2

RP3

RP4

Total

Goal Supported

Grand (aggregate) mean rating of XSEDE User Survey user satisfaction items regarding XCI software and technical services, capabilities, and resources4

 

RY 64 of 5 / yr




 

 

Advance – Create an open and evolving e-infrastructure



RY 5

 4 of 5 / yr4.4  - 4.4

RY 4

* * *** *

RY 3

*****

RY 2

****

*

RY 1

*

***

*

*

 

Definition/Description: Aggregate rating of USER satisfaction with XSEDE software and technical services, capabilities, and resources via the annual User Survey on a scale of 1–5, where “1” is “performance never meets expectations” and “5” is “performance always meets expectations.”

Collection methodology: Aggregate ratings of USER-facing XCI software and technical services, capabilities and resources as measured using the "frequency of performance meeting expectations" scale on the annual XSEDE User Survey, to include the following items: Authenticating to the XSEDE User Portal and other web services using campus InCommon credentials (CILogon); Using XSEDE credentials to log into non-XSEDE Web services (e.g., GENI and OrCID.); Globus Transfer Service; Globus Sharing Service; Globus Connect for moving data between personal systems and XSEDE resources; Job start prediction service (Karnak); Login to XSEDE affiliated web sites without entering a password every time; Cluster integration software (XCBC); Scientific software for campus clusters (XNIT); Cluster integration tutorials and documentation; and Cluster integration site visits.

Grand (aggregate) mean rating of XSEDE User Survey Service Provider satisfaction items regarding XCI software and technical services, capabilitiies, and resources

(1-5 point Likert scale)



RY 64 of 5 / yr




Advance – Create an open and evolving e-infrastructure
RY 54 of 5 / yr4.5---4.5
RY 4******
RY 3******
RY 2******
RY 1******

Definition/Description: Aggregate rating of SP satisfaction with XSEDE software and technical services, capabilities, and resources via the annual User Survey on a scale of 1–5, where “1” is “performance never meets expectations” and “5” is “performance always meets expectations.”

Collection methodology: Aggregate ratings of SP-facing XCI software and technical services, capabilities and resources as measured using the "frequency of performance meeting expectations" scale on the annual XSEDE User Survey, to include the following items: IPF software used to publish resource information to XSEDE; XSEDE accepted CA certificates; XSEDE Resource Operational Status Dashboards; XSEDE Resource Integration Steps Dashboard; Resource Description Repository (RDR); Install software from XSEDE RPM repositories on software.xsede.org; Opportunities to discuss software needs, delivery activities, deployment, and usage; Opportunities to participate in software technical reviews; Software Integration Support; Software Usage Instrumentation Support; Use case requirements analysis processes; and Use case development processes; and Use case prioritization processes. In order to get an adequate number of responses from Service Providers, that population will be over-sampled. If that does not result in an adequate response rate, these questions will be pulled and sent to the SPs separately to improve the number of responses.

Number of non-XSEDE partnerships with XCI2

 

RY 6

14 / yr


  

 

 

Advance – Create an open and evolving e-infrastructure

RY 518 / yr14113161521

RY 4

 12 / yr 9 1011 21 21

RY 3

8 / yr131211 1220

RY 2

----

 -

8

RY 1

-

*--

-

8

 

Definition/Description: Number of XCI partnerships supported/facilitated between CRI and SPs, national CI organizations, and campus CI providers.

Collection methodology: Non-XSEDE partnerships are tracked through the ticket system and through JIRA.

Mean time to issue resolution (days)RY 6< 10 days/ qtr




 

 

Sustain — Provide excellent user support 


RY 5 < 10 days/ qtr 4.2 6.0 10.1 2.45.3 
RY 4 < 10 days/ qtr 5.4 4.58.6 35.8 13.7
RY 3< 14 days/ qtr22.37.32.9 2.98.8 
RY 2< 30 days4.07.08.0

5.0

6.1

RY 1

< 45 days

*7.04.016.09.0
 

Definition/Description: Average number of days from defect or support request submitted to solution communicated.

Collection methodology: The data is stored locally via the Request Tracker service, which is the XSEDE ticket software. Reports are generated against the RT database to determine MTTR and other metrics.

...