Skip to end of metadata
Go to start of metadata

Changes resulting from a project-wide KPI review were implemented beginning with RY3, RP1 (May - July 2018). As a result, significant changes were made to XSEDE reporting. This page now reflects those changes. For results of KPIs and Metrics prior to May 2018, see RY1-RY2: XSEDE Project-wide KPIs & Metrics (ARCHIVED)

More Information: What is a KPI - KPI Guidelines - Annual KPI Reviews 

1    Executive Summary

Computing across all fields of scholarship is becoming ubiquitous. Digital technologies underpin, accelerate, and enable new, even transformational, research in all domains. Researchers continue to integrate an increasingly diverse set of distributed resources and instruments directly into their research and educational pursuits. Access to an array of integrated and well-supported high-end digital services is critical for the advancement of knowledge. XSEDE (the Extreme Science and Engineering Discovery Environment) is a socio-technical platform that integrates and coordinates advanced digital services within the national ecosystem to support contemporary science. This ecosystem involves a highly distributed, yet integrated and coordinated, assemblage of software, supercomputers, visualization systems, storage systems, networks, portals and gateways, collections of data, instruments, and personnel with specific expertise. XSEDE supports the need for an advanced digital services ecosystem distributed beyond the scope of a single institution and provides a long-term platform to empower modern science and engineering research and education. As a significant contributor to this ecosystem, driven by the needs of the open research community, XSEDE substantially enhances the productivity of a growing community of scholars, researchers, and engineers. XSEDE federates with other high-end facilities and campus-based resources, serving as the foundation for a national e-science infrastructure with tremendous potential for enabling new advancements in research and education. Our vision is a world of digitally-enabled scholars, researchers, and engineers participating in multidisciplinary collaborations while seamlessly accessing computing resources and sharing data to tackle society’s grand challenges.

Researchers use advanced digital resources and services every day to expand their understanding of our world. More pointedly, research now requires more than just supercomputers and XSEDE represents a step toward a more comprehensive and cohesive set of advanced digital services through our mission: to substantially enhance the productivity of a growing community of scholars, researchers, and engineers through access to advanced digital services that support open research; and to coordinate and add significant value to the leading cyberinfrastructure resources funded by the NSF and other agencies.  XSEDE has developed its strategic goals in a manner consistent with NSF’s strategic plan, Building the Future: Investing in Discovery and Innovation - NSF Strategic Plan for Fiscal Years (FY) 2018 - 2022,1

NSF’s strategies stated broadly in the Cyberinfrastructure Framework for 21st Century Science and Engineering,2 vision document, and, the more specifically relevant, Advanced Computing Infrastructure: Vision and Strategic Plan3 document.

 

1.1       Strategic Goals

To support our mission and to guide the project’s activities toward the realization of our vision, three strategic goals are defined: 

Deepen and Extend Use: XSEDE will deepen the use—make more effective use—of the advanced digital services ecosystem by existing scholars, researchers, and engineers, and extend the use to new communities. We will contribute to preparation—workforce development—of the current and next generation of scholars, researchers, and engineers in the use of advanced digital services via training, education, and outreach; and we will raise the general awareness of the value of advanced digital services.

Advance the Ecosystem: Exploiting its internal efforts and drawing on those of others, XSEDE will advance the broader ecosystem of advanced digital services by creating an open and evolving e-infrastructure, and by enhancing the array of technical expertise and support services offered.

Sustain the Ecosystem: XSEDE will sustain the advanced digital services ecosystem by ensuring and maintaining a reliable, efficient, and secure infrastructure, and providing excellent user support services. XSEDE will further operate an effective, productive, and innovative virtual organization.

The strategic goals of XSEDE cover a considerable scope. To ensure we are delivering on our mission and to assess progress toward our vision, we have identified key metrics to measure our progress toward meeting each sub-goal. These Key Performance Indicators (KPIs) are a high-level encapsulation of our project metrics that measure how well we are meeting each sub-goal. Planning is driven by our vision, mission, goals, and these metrics—which are in turn rooted in the needs and requirements of the communities we serve.

The key concept is not that the KPIs themselves must have a direct causal effect on eventual outcomes, or measure eventual outcomes or long-term impacts, but rather that the KPIs are chosen so that actions and decisions which move the metrics in the desired direction also move the organization in the direction of the desired outcomes and goals.

As discussed in our preceding report, the XSEDE Reporting Year 2 Annual Report and Project Year 8 Program Plan[1], a significant review of our KPIs and metrics was conducted in the latter half of calendar year 2017. This resulted in considerable changes in metrics as we refined them. Table 1?1 reflects the revised summary of KPIs against which we will be reporting in this and subsequent reports.

The table below shows the project’s three strategic goals and associated sub-goals along with the KPIs used to measure progress toward achieving those goals.

Table 1-1: Summary of key performance indicators (KPIs) for XSEDE.

Strategic Goals

Sub-goals

KPIs

Deepen and Extend Use

Deepen use (existing communities)

  • Number of sustained users of XSEDE resources and services via the portal
  • Number of sustained underrepresented individuals using XSEDE resources and services via the portal
  • Percentage of sustained allocation users from non-traditional disciplines of XSEDE resources and service

Extend use (new communities)

  • Number of new users of XSEDE resources and services via the portal
  • Number of new underrepresented individuals using XSEDE resources and services via the portal
  • Percentage of new allocation users from non-traditional disciplines of XSEDE resources and services

Prepare the current and next generation

  • Number of participant hours of live training delivered by XSEDE

Raise awareness of the value of advanced digital services

  • Aggregate mean rating of user awareness of XSEDE resources and services
  • Percent increase in social media impressions over time

Advance the Ecosystem

Create an open and evolving e-infrastructure

  • Total number of capabilities in production

Enhance the array of technical expertise and support services

  • Aggregate mean rating of user satisfaction with XSEDE technical expertise and support services

Sustain the Ecosystem

Provide reliable, efficient, and secure infrastructure

  • Mean composite availability of core services (%)

Provide excellent user support

  • Mean time to ticket resolution (hours)
  • Aggregate mean rating of user satisfaction with allocations process and support services
  • Percentage of research requests successful (not rejected)

Operate an effective and productive virtual organization

  • Mean rating of importance of XSEDE resources and services to researcher productivity
  • Percentage of users who indicate the use of XSEDE-managed and/or XSEDE-associated resources in the creation of their work product

Operate an innovative virtual organization

  • Percentage of Project Improvement Fund proposals resulting in innovations in the XSEDE organization
  • Mean rating of innovation within the organization by XSEDE staff

[Table of Contents]

2    Discussion of Strategic Goals and Key Performance Indicators

The strategic goals of XSEDE cover a considerable scope. Additionally, the specific activities within our scope are often very detailed; therefore, to ensure that this significant and detailed scope will ultimately deliver our mission and realize our vision, we decompose the three strategic goals into components or sub-goals to be considered individually.

In determining the best measures of progress toward each of the sub-goals, KPIs that correlate to impact on the scientific community are used. These often pair measurements of outcome with an assessment of quality or impact to provide both a sense of scope and significance of the supporting activities.

 

2.1       Deepen and Extend Use

XSEDE will 1) deepen the use—make more effective use—of the advanced digital services ecosystem by existing scholars, researchers, and engineers and 2) extend the use to new communities. We will 3) contribute to preparation—workforce development—of scholars, researchers, and engineers in the use of advanced digital technologies via training, education, and outreach; and we will 4) raise the general awareness of the value of advanced digital research services.

2.1.1 Deepen use to Existing Communities

XSEDE engages in a range of activities that serve to deepen use including identifying new technologies and new service providers, evolving the e-infrastructure, and enhancing the research prowess of current and future researchers. However, the ongoing use of resources and services available via XSEDE is the key indicator of this deepening use. As a result, the project has chosen three KPIs (Table 2-1) that together measure the ongoing engagement with the community with an emphasis on exposing the diversity of those consuming these services: 1) Number of sustained users of XSEDE resources and services via the portal, 2) Number of sustained underrepresented individuals using XSEDE resources and services via the portal, and 3) Percentage of sustained allocation users from non-traditional disciplines of XSEDE resources and services. 

Table 2-1: KPIs for the sub-goal of deepen use (existing communities). 

KPI

Program Year

Target

RP1

RP2

RP3

RP4

Total

Owner(s)

Number of sustained users of XSEDE resources and services via the portal

RY5

      

 

 

CEE (§3)

RY4

      

RY3

3,500 / qtr4,1964,089   

RY2

3,000 / qtr

3,962

3,754

2,488

3,020

4,527

RY1

> 5,000 / qtr

*

4,755

4,446

4,924

6,186

 

Definition/Description: Number of sustained users within the current reporting period. A "sustained user" is any individual who created a portal account prior to the current reporting period and logged in during the reporting period.


Collection methodology: There is a usage log parsing script that collects the tomcat logs and parses out the relevant data during the selected time period.

Number of sustained underrepresented individuals using XSEDE resources and services via the portalRY5      



CEE (§3)

RY4      
RY31,500 / yr529506   
RY21,500 / yr

490

408

296

402

1,640

RY1> 1,000 / yr*

322

238

535

1,095

 

Definition/Description: Number of sustained underrepresented users. A "sustained user" is any individual who created a portal account prior to the current reporting period and logged in during the reporting period. Only individuals from underrepresented communities including women and racial/ethnic domestic minorities in HPC are counted in the numerator. Users identifying with multiple underrepresented communities are only counted once.

Collection methodology: This is the sum of the number of persistent users from underrepresented communities measured by Community Engagement and Enrichment - WBS 2.1 and the number of persistent users from non-traditional disciplines measured by Novel and Innovative Projects - WBS 2.2.3.

Percentage of sustained allocation users from non-traditional disciplines of XSEDE resources and services

RY5

      



ECSS (§4)

RY4

      

RY3

20 /
qtr
22.120.7   

RY2

 

21.5

20.4

21.1

21.0

21.0

RY1

 

*

18.2

19.9

18.6

18.9

 

Definition/Description: The total number of completed ECSS projects consists of projects that were completed through work in ESRT, ESCC, and ESSGW. A completed project is defined as a project that has progressed through the complete support pipeline.  This pipeline includes steps such as assignment of a consultant, production of a work plan, execution of the work plan and reporting of progress through quarterly reports, and the filing of a final project report.

Collection methodology: The number of completed projects is tracked in Sciforma, XSEDE’s project management software. A report has been created in Sciforma that queries all the ECSS projects, filters for allocations that have ended in that quarter and also for projects that have a final report.

 * NOTE: RY1 is unique and only spans September 2016-April 2017.  RY1 RP1 doesn’t exist due to the shortened RY. RY1, RP2 only includes two months (September-October 2016). 


2.1.2 Extend Use to New Communities

New communities are defined as fields of science, industry, and underrepresented communities that represent less than one percent of XSEDE Resource Allocation Committee (XRAC) allocations. The Novel & Innovative Projects (NIP) team and the Broadening Participation team both work to bring advanced digital services to new communities. XSEDE measures both the number of new users and the number of new users on research projects from underrepresented communities and non-traditional disciplines of XSEDE resources and services as the indicators of progress  (Table 2-2).

Table 2-2: KPIs for the sub-goal of extend use (new communities).

KPI

Program Year

Target

RP1

RP2

RP3

RP4

Total

Owner(s)

Number of new users of XSEDE resources and services via the portal

RY 5

      

 

 

 

CEE (§3)

RY 4

      

RY 3

3,000/
qtr

1,095

2,743   

RY 2

2,000/
qtr

2,305

2,813

2,346

2,917

10,381

RY 1

>1,000/
qtr

*

1,973

1,849

2,359

6,181

 

Definition/Description: Number of new users of XSEDE resources and services as determined by the number of portal accounts created within the reporting period.

Collection methodology: The Liferay Database is queried for the number of users in the Users table that was created during the selected time periods. Sample query: select screenName, createDate from User_ where createDate between '2016-09-01 00:00:00' AND '2016-10-31 00:00:00'

Number of new underrepresented individuals using XSEDE resources and services via the portal

RY 5

      

 


CEE (§3)

RY 4

      

RY 3

200/qtr

134

155   

RY 2

150/qtr

251

175

222

234

950

RY 1

100/qtr

*

150

135

240

525

 

Definition/Description: Number of new underrepresented users. A "new user" is any individual who created a portal account within the reporting period. Only individuals from underrepresented communities including women and racial/ethnic domestic minorities in HPC are counted in the numerator. Users identifying with multiple underrepresented communities are only counted once.

Collection methodology: XSEDE portal users are able to specify their race/ethnicity, gender, and institution in their user profile and are encouraged to complete this data when registering for training events through the XSEDE portal. Date of allocation awards is also recorded in the XSEDE database (XDCDB). A query is then run to generate the number of individuals meeting the aforementioned criteria within a given quarter. Care is taken to not over represent this metric by ensuring an individual who meets multiple criteria is only accounted for one time.

Percentage of new allocation users from non-traditional disciplines of XSEDE resources and servicesRY 5      

 

 

ECSS (§4)

RY 4      
RY 330 / yr 33.126.0   
RY 2

 

24.8

26.0

26.6

33.9

27.8

RY 1

 

*

21.8

24.9

31.0

25.7

 

Definition/Description: Number of new allocation users from non-traditional disciplines in HPC divided by the total number of new allocation users within the reporting period. A set of 63 fields of science (FOS) have been identified in the XD Central Database (XDCDB), each of whose usage over the past 10 years is below 0.5% of the total normalized usage. A scripted query to XDCDB counts “new allocation users” as the total number of users on the grants from these FOS that have been newly awarded during the reporting period. This is divided by the total number of users on all the grants that have been awarded during the reporting period.

Collection methodology:This is the sum of the number of new users from underrepresented communities measured by Community Engagement and Enrichment - WBS 2.1 and the number of new users from non-traditional disciplines measured by Novel and Innovative Projects - WBS 2.2.3.

 * NOTE: RY1 is unique and only spans September 2016-April 2017.  RY1 RP1 doesn’t exist due to the shortened RY. RY1, RP2 only includes two months (September-October 2016). 

 

2.1.3 Prepare the Current and Next Generation

Part of XSEDE’s mission is to provide a broad community of existing and future researchers with access and training to use advanced digital services via the sub-goal of preparing the current and next generation of computationally-savvy researchers. While many activities support this sub-goal, such as the various Champion (§3.6), Student Engagement (§3.4), and Education (§3.2) programs, the training offered through Community Engagement & Enrichment (CEE) impacts the most people directly. Therefore, the key indicator (Table 2-3) of performance toward this goal, which is reflective of industry standards, is the number of participant hours of live training delivered by XSEDE.

Table 2-3: KPIs for the sub-goal of prepare the current and next generation

KPI

Program Year

Target

RP1

RP2

RP3

RP4

Total

Owner(s)

Number of participant hours of live training delivered by XSEDE

RY 5

      

 

 

CEE (§3)

RY 4

      

RY 3

40,000 /
yr

14,140

8,274   

RY 2

 

NA

NA

NA

NA

NA

RY 1

 

NA

NA

NA

NA

NA

 

Definition/Description: Number of total participant hours spent in XSEDE live, in-person, or webcast events during the current reporting period.

Collection methodology: Measured using registrations via the XSEDE User Portal. Does not include training events that do not require registration such as Cornell Virtual Workshop, CI-Tutor, Youtube videos, etc.


2.1.4 Raise Awareness of the value of advanced digital services

While many activities led by teams throughout the XSEDE organization, such as our Workforce Development (§3.2), User Engagement (§3.3), Broadening Participation (§3.4), and Campus Engagement (§3.6) contribute to our ability to raise the general awareness of the value of advanced digital research services, we have chosen to focus on measures in two areas (Table 2-4): user input and social media. Desirable trends in these key outcomes can be correlated to success for this sub-goal.

Table 2-4: KPIs for the sub-goal of raise awareness of the value of advanced digital research services

KPI

Program Year

Target

RP1

RP2

RP3

RP4

Total

Owner(s)

Aggregate mean rating of user awareness of XSEDE resources and services

RY 5

      

 

 

Program Office (§8)

RY 4

      

RY 3

3.5 of 5 / yr

3.69

-   

RY 2

 

3.59

-

-

-

3.59

RY 1

 

*

-

-

-

NA

 

Definition/Description: Per Google Analytics, “a pageview is defined as a view of a page on your site that is being tracked by the Analytics tracking code. If a user clicks reload after reaching the page, this is counted as an additional pageview. If a user navigates to a different page and then returns to the original page, a second pageview is recorded as well.”

Collection methodology: Aggregate rating of awareness scale items measuring current XSEDE resources and services listed on the annual XSEDE User Survey.

Percent increase in social media impressions over time

RY 5

      



Program Office (§8)

RY 4

      

RY 3

15 / yr

62

14   

RY 2

NA

-14

-20

-2

93

9

RY 1

NA

*

NA

147

-37

NA

 

Definition/Description: Number of social media impressions to date within the current reporting year (x) minus the number of social media impressions during the same period in the previous reporting year (y). This number is divided by the number of social media impressions during the same period in the previous reporting year (y) to determine percent increase (((x-y)/y)*100)). "Impressions" are generated by Facebook and Twitter to determine how many individuals have seen XSEDE interactions or posts on said platforms. This includes "likes," "retweets," "followers," “shares,” as well as people who see the post based on their friends or followers sharing XSEDE information.

Collection methodology: The information is pulled from Facebook and Twitters' internal tracking methods at the end of each report period. The percent increase is then computed based on the previous reporting period data.

 * NOTE: RY1 is unique and only spans September 2016-April 2017.  RY1 RP1 doesn’t exist due to the shortened RY. RY1, RP2 only includes two months (September-October 2016). 

-  Data reportedly annually.

2.2       Advance the Ecosystem

Exploiting its internal efforts and drawing on those of others, XSEDE will advance the broader ecosystem of advanced digital services by 1) creating an open and evolving e-infrastructure, and by 2) enhancing the array of technical expertise and support services offered.

2.2.1 Create an Open and Evolving e-Infrastructure

There are a variety of factors that affect the evolution of the e-infrastructure. These range from external factors, such as the number of XSEDE Federation members and the variety of services they provide, to internal factors, like Operations (§6) of critical infrastructure and services and the evaluation and integration of new capabilities. While we actively seek new Federation members and Service Providers, as well as partnerships with national and international cyberinfrastructure projects, we view our role as connectors of these elements to have the most impact. Thus, XSEDE focuses on the number of new capabilities in production as an indicator of performance with respect to this sub-goal (Table 2-5).

Table 2-5: KPIs for the sub-goal of create an open and evolving e-infrastructure

KPI

Program Year

Target

RP1

RP2

RP3

RP4

Total

Owner(s)

Total number of capabilities in production

RY 5

      

 

 

XCI (§5)

RY 4

      

RY 3

81

76

86

 

 

 

RY 2

*

72

74

74

75

75

RY 1

*

*

63

63

69

69

 

Definition/Description: Total number of capabilities in production across XSEDE SPs, central operations, partner service providers (Globus, etc.), campuses, science gateways, etc.

Collection methodology: A list of capabilities in production through XSEDE is kept in the Community Software Repository which catalogs Use Cases and Capability Delivery Plans.

 * NOTE: RY1 is unique and only spans September 2016-April 2017.  RY1 RP1 doesn’t exist due to the shortened RY. RY1, RP2 only includes two months (September-October 2016). 

 

2.2.2 Enhance the Array of Technical Expertise and Support Services

To enhance the technical expertise of our staff to offer an evolving set of support services, we will continue many activities including workshops, symposia, and training events hosted by Extended Collaborative Support Services (ECSS) and Service Providers (§4.6). The KPI for this is feedback provided from the XSEDE user-base through the annual user survey (Table 2-6).

Table 2-6: KPIs for the sub-goal of enhance the array of technical expertise and support services.

KPI

Program Year

Target

RP1

RP2

RP3

RP4

Total

Owner(s)

Aggregate mean rating of user satisfaction with XSEDE technical expertise and support services

RY 5

      

 

 

Program Office (§8)

RY 4

      

RY 3

3.5 of 5/yr

3.64

3.69   

RY 2

N/A

3.44

-

-

-

3.44

RY 1

N/A

*

*

*

*

*

 

Definition/Description: An annual Staff Climate Study is a survey administered and analyzed by an external evaluation team as part of an effort to understand whether XSEDE staff feel adequately prepared to conduct their XSEDE work. The survey study includes items regarding staff training.

Collection methodology: Respondents rate closed-ended Likert scale items regarding training on scale of 1 (strongly disagree) to 5 (strongly agree). The average of all staff responses for a given year is calculated to determine this metric. The specific item used to calculate this metric is Q2m: “I have access to adequate training to conduct my XSEDE-related work.”

 * NOTE: RY1 is unique and only spans September 2016-April 2017.  RY1 RP1 doesn’t exist due to the shortened RY. RY1, RP2 only includes two months (September-October 2016). 

- Data reported annually. 

 

2.3       Sustain the Ecosystem

XSEDE will sustain the advanced digital services ecosystem by 1) ensuring and maintaining a reliable, efficient, and secure infrastructure, and 2) providing excellent user support services. Furthermore, XSEDE will operate an 3) effective, 4) productive, and 5) innovative virtual organization.

2.3.1 Provide Reliable, Efficient, and Secure Infrastructure

Many activities support the provisioning and support of reliable, efficient and secure infrastructure—such as User Interfaces & Online Information (§3.5), Security (§6.2), Data Transfer Services (§6.3), Systems Operations and Support (§6.5), support for Allocations (§7.2), and Allocations, Accounting & Account Management (§7.3)—but perhaps the truest measure of an infrastructure’s reliability is its robustness as reflected by sustained availability. Thus, the KPI for this sub-goal is the mean composite availability of core services, shown as a percentage (Table 2-7), measured as a geometric mean. This is a composite measure of the availability of critical enterprise services and the XRAS allocations request management service.

Table 2-7: KPIs for the sub-goal of provide reliable, efficient, and secure infrastructure

KPI

Program Year

Target

RP1

RP2

RP3

RP4

Total

Owner(s)

Mean composite availability of core services

(geometric mean of critical services and XRAS)

RY 5

 

     

 

 

Operations (§6)

RY 4

      

RY 3

99.9 / qtr

99.9

99.9   

RY 2

99.9 / qtr99.899.999.999.999.9

RY 1

99.0 / qtr

*

99.9

99.9

99.9

99.9

 

Definition/Description: The percent average composite availability of core services is the  geometric mean of % core enterprise services availability and % POPS/XRAS availability. Because the availability percentage of each of these components is measured separately, they are aggregated and then averaged using a geometric mean to determine the composite availability.

Collection methodology: See the individual collection methodologies for each component in the responsible WBS area below: Core enterprise services (Systems Operational Support 2.4.5) and POPS/XRAS (AA&AM 2.5.3).

 * NOTE: RY1 is unique and only spans September 2016-April 2017.  RY1 RP1 doesn’t exist due to the shortened RY. RY1, RP2 only includes two months (September-October 2016). 

2.3.2 Provide Excellent User Support 

Although nearly every group in the organization has some support function, we have chosen to focus on metrics with respect to two primary support interfaces to the community: the XSEDE Operations Center (XOC) and the Resource Allocation Services (RAS) team. The XOC is the frontline centralized support group that either resolves or escalates tickets to the appropriate resolution center depending on the request. RAS is responsible for the allocations process and the allocation request system. These two support interfaces are the focus for gauging the progress towards achieving the sub-goal of providing excellent user support, specifically: 1) the mean time to resolution on support tickets that are resolved by the XOC or routed to, and resolved by, other XSEDE areas, 2) the aggregate mean rating of user satisfaction with allocations process and support services measured via a quarterly survey of users who have interacted with the allocations request system and the allocations process more generally; and 3) the percentage of research requests successful (not rejected) determined following the quarterly allocations session (Table 2-8).

Table 2-8: KPIs for the sub-goal of provide excellent user support

KPI

Program Year

Target

RP1

RP2

RP3

RP4

Total

Owner(s)

Mean time to ticket resolution (hours)

RY 5

      

 

 

Operations (§6)

RY 4

      

RY 3

< 24 / qtr

12.3

18.5   

RY 2

< 24 / qtr2620.122.815.021.0

RY 1

< 24 / qtr

*24.028.2

23.1

25.1

 

Definition/Description: XSEDE-funded areas are measured to ensure that, on average, tickets are resolved in a timely manner. This includes all XSEDE WBS ticket queues as well as the XSEDE Operations Center (XOC) queue. This metric does NOT apply to Service Provider ticket queues. End-users expect for their issues to be responded to within a reasonable amount of time, so the target is a resolution time under 24 hours.

Collection methodology: The data is stored locally via the Request Tracker service, which is the XSEDE ticket software and which is located at TACC. Reports are generated against the RT database to determine MTTR and other metrics.

Aggregate mean rating of user satisfaction with allocations process and support services

RY 5

 

     

 

 

RAS (§7)

RY 4

      

RY 3

4 of 5 / yr

4.06

4.16   

RY 2

4 of 5 / yr

4.08

4.00

4.14

3.86

4.02

RY 1

4 of 5 / yr

*

3.98

4.03

4.03

4.01

 

Definition/Description: Aggregate mean rating of user satisfaction with allocations process and other support services as measured by the Post Allocations Submission Survey administered prior to award notification on a scale of 1 (not at all satisfied) - 5 (extremely satisfied).

Collection methodology: Because this is a composite KPI that consists of an AM from XSEDE Allocations Process & Policies (WBS 2.5.2) and an AM from Allocations, Accounting & Account Management (WBS 2.5.3), see those sections below for how the data is collected each AM.

Percentage of research requests successful (not rejected)RY 5      



RAS (§7)

RY 4      
RY 3

85.0 / qtr

65.0

70.0   
RY 2

85.0 / qtr

70.0

69.0

72.0

68.0

69.75

RY 1

85.0 / qtr

*

76.0

75.0

74.0

75.0

 

Definition/Description: Number of Research allocation requests receiving some non-zero award from an XRAC meeting divided by the number of Research allocation requests submitted to the same XRAC meeting.

Collection methodology: Request data comes from the Quarterly XRAC meetings and is stored in XRAS as well as some meeting spreadsheets.

 * NOTE: RY1 is unique and only spans September 2016-April 2017.  RY1 RP1 doesn’t exist due to the shortened RY. RY1, RP2 only includes two months (September-October 2016). 


2.3.3 Effective and Productive Virtual Organization

During the first five years of XSEDE, in conjunction with developing a methodology for driving and assessing performance excellence, XSEDE adopted the Baldrige Criteria 4 and has assessed and applied applicable criteria from all seven criteria by that methodology. These include annual reviews of the vision, mission, strategic goals, project-wide processes and standards (KPIs); user and staff surveys (§3.3, §8.5); stakeholder communications (§8.1); advisory boards (§8.1); community engagement (§3); workforce enhancement (§3.2); and the analysis of organizational data that leads to organizational learning, strategic improvement, and innovation. With this foundation, it is now appropriate to look to the XSEDE users to give us an indication of our effectiveness by rating the importance of the resources and services provided by XSEDE (Table 2-9).

Table 2-9: KPIs for the sub-goal of operate an effective and productive virtual organization.

KPI

Program Year

Target

RP1

RP2

RP3

RP4

Total

Owner(s)

Mean rating of importance of XSEDE resources

and services to researcher productivity

RY 5

      

 

 

Program Office (§8)

RY 4

      

RY 3

4.2 of 5/yr

4.43

-   

RY 2

 

4.41

-

-

-

4.41

RY 1

 

4.32

-

-

-

4.32

 

Definition/Description: Mean rating of importance of XSEDE resources and services to researcher productivity as measured by the annual User Survey on a scale of 1 (not important at all) - 5 (essential).

Collection methodology: A mean rating is calculated from the responses to the annual XSEDE User Survey item, "How important are XSEDE resources to your work?"

Percentage of users who indicate the use of XSEDE-managed and/or XSEDE-associated resources in the creation of their work product1RY 5      



Program Office (§8)

RY 4      
RY 379-79   
RY 2******
RY 1******
 

Definition/Description:

Collection methodology:


2.3.4 Innovative Virtual Organization

Measuring innovation for an organization like XSEDE (or for organizations in general) is difficult and represents an area of open research. After much thought and discussion both internally and with external stakeholders and advisors, we have identified two indicators that correlate to innovation within the project: 1) percentage of Project Improvement Fund proposals resulting in innovations in the XSEDE organization and 2) mean rating of innovation within the organization by XSEDE staff (Table 2-10). The first indicator is a measurement of our ability to fund smaller innovative improvements within the project; the second measures how staff rate the level of innovation within the project. These KPIs will continue to be the subject of an open conversation within the organization and with stakeholders and advisors as XSEDE assesses these measurements and how to best quantify innovation.

Table 2-10: KPIs for the sub-goal of operate an innovative organization

KPI

Program Year

Target

RP1

RP2

RP3

RP4

Total

Owner(s)

Percentage of Project Improvement Fund

proposals resulting in innovations in the XSEDE organization

RY 5      



Program Office (§8)

RY 4      
RY 3

60 / yr

-

-   
RY 2 

*

*

*

*

*

RY 1 

*

*

*

*

*

 

Definition/Description:  Number of completed PIF (Project Improvement Fund) projects resulting in innovations as determined by SMT review divided by the total number of completed PIF projects to date.

Collection methodology: The results of each Project Improvement Fund approved submissioned for a project year is evalauted, at the end of the project year, to assess the level of innovation. This is a manual effort completed by the SP&E team.

 

Mean rating of innovation within the

organization by XSEDE staff

RY 5

      

 

 

Program Office (§8)

RY 4

      

RY 3

3.5 of 5/
yr

-

3.95   

RY 2

 

*

*

*

*

*

RY 1

 

*

*

*

*

*

 

Definition/Description: Mean rating of innovation within the organization by XSEDE employees as measured by the annual XSEDE Staff Climate Study on a scale of 1 (not at all innovative) - 5 (highly innovative).

Collection methodology: New item added to 2018 XSEDE Staff Climate Study where staff will rate the level of innovation within the organization of a scale of 1 (low/negative) to 5 (high/positive).


[Table of Contents]

3       Community Engagement & Enrichment - CEE (WBS 2.1) 

At the core of Community Engagement & Enrichment (CEE) is the researcher, broadly defined to include anyone who uses or may potentially use the array of resources and services offered by XSEDE. The CEE team is dedicated to actively engaging a broad and diverse cross-section of the open science community, bringing together those interested in using, integrating with, enabling, and enhancing the national cyberinfrastructure. Vital to the CEE mission is the persistent relationship with existing and future users, including allocated users, training participants, XSEDE collaborators, and campus personnel. CEE will unify public offerings to provide a more consistent, clear, and concise message about XSEDE resources and services, and bring together those aspects of XSEDE that have as their mission teaching, informing, and engaging those interested in advanced cyberinfrastructure.

The five components of CEE are Workforce Development (§3.2), which includes Training, Education and Student Preparation, User Engagement (§3.3), Broadening Participation (§3.4), User Interfaces & Online Information (§3.5), and Campus Engagement (§3.6). These five teams ensure routine collection and reporting of XSEDE’s actions to address user requirements. They provide a consistent suite of web-based information and documentation and engage with a broad range of campus personnel to ensure that XSEDE’s resources and services complement those offered by campuses. Additionally, CEE teams expand workforce development efforts to enable many more researchers, faculty, staff, and students to make effective use of local, regional, and national advanced digital resources. CEE expands efforts to broaden the diversity of the community utilizing advanced digital resources. The CEE team tightly coordinates with the rest of XSEDE, particularly Extended Collaborative Support Service (ECSS)  (ECSS) (§4), Resource Allocation Services (§7), XSEDE Cyberinfrastructure Integration (§5), and External Relations (§8.2).

CEE is focused on personal interactions, ensuring that existing users, potential users, and the general public have sufficient access to materials and have a positive and effective experience with XSEDE public offerings and frontline user support. As such, the CEE Key Performance Indicators are designed to broadly assess this performance. CEE focuses on metrics that quantify how many users in aggregate are benefiting from XSEDE resources and services. Additionally, CEE focuses on how well the user base is sustained over time and how well training offerings evolve with changing user community needs.

Table 3-1: KPIs for Community Engagement & Enrichment (CEE).

KPI

Report Year

Target

RP1

RP2

RP3

RP4

Total

 Goal Supported

Number of students benefiting from XSEDE resources and services through training,  XSEDE projects, or conference attendance

RY 5

      

 

 

Deepen/Extend – Extend use to new communities

RY 4

      

RY 3

1,250 / qtr

1,613 

1,666   

RY 2

950 / qtr

1,722

1,478

1,170

1,522

3,720

RY 1

50 / qtr

*

997

815

2,679

4,824

 

Definition/Description: Number of students who participated in XSEDE training, were supported by XSEDE to work on XSEDE projects, received funding to attend the annual PEARC Conference (formerly XSEDExy), or SC conference as indicated by registrations in the XSEDE portal during the current reporting period.

Collection methodology: Measured using registrations via the XSEDE User Portal, participants in CEE student programs (i.e. EMPOWER), and PEARC student program attendees.

Percentage of under-represented students benefiting from XSEDE resources and services through training, XSEDE projects, or conference attendance

RY 5

      

 

 

Deepen/Extend – Deepen use to existing communities

RY 4

      

RY 3

50 / qtr

 28

26   

RY 2

50 / qtr

28

27

30

28

30

RY 1

50 / qtr

*

34 

33

19

28

 

Definition/Description: Number of underrepresented students out of total students who participated in XSEDE training, were supported by XSEDE to work on XSEDE projects, received funding to attend the annual PEARC Conference (formerly XSEDExy), or SC conferences as indicated by registrations in the XSEDE portal during the current reporting period. Only students from underrepresented communities including women and racial/ethnic domestic minorities in HPC are counted in the numerator. Students identifying with multiple underrepresented communities are only counted once.

Collection methodology: Measured using registrations via the XSEDE User Portal, participants in CEE student programs (i.e. EMPOWER), and PEARC student program attendees. Only students from underrepresented communities including women and racial/ethnic domestic minorities in HPC are counted in the numerator. Students identifying with multiple underrepresented communities are only counted once.

Aggregate mean rating of training impact for attendees registered through the portal


RY 5

      

 

 

Deepen/Extend – Extend use to new communities

RY 4

      

RY 3

4.4 of 5 /qtr

4.50 

4.42   

RY 2

4 of 5 /qtr

4.29

4.34

4.57

4.50

4.42

RY 1

4 of 5 /qtr

*

4.54

4.39 

4.28

4.36

 

Definition/Description: Aggregate mean rating of training impact for attendees registered via the XSEDE User Portal for XSEDE delivered training sessions as measured by the Post Training Event Survey on a scale of 1 (low impact) to 5 (high impact).

Collection methodology: The data is collected from post event surveys and recorded in the database. Specific items included in this index are:

  • Q1. The training session fulfilled my expectations.
  • Q2. The trainer stimulated my interest.
  • Q6. The training session was well-organized.
  • Q8. I was able to easily access this training session.
  • Q10. Overall I would rate my experience as successful.

Number of institutions with a Champion

RY 5

      

 

 

Deepen/Extend – Deepen use to existing communities

RY 4

      

RY 3

250

259 

266   

RY 2

240

218

238

239

246 

246

RY 1

225

*

224

231

234

234

 

Definition/Description: Total number of institutions that have a signed MOU with the Champions program

Collection methodology: The number of completed projects is tracked in Sciforma, XSEDE’s project management software. A report has been created in Sciforma that queries all the ECSS projects, filters for allocations that have ended in that quarter and also for projects that have a final report.

Percentage of user requirements addressed

RY 5

      

 

Deepen/Extend – Prepare the current and next generation

RY 4

      

RY 3

100 / qtr

89 (40/45) 

90 (47/52)   

RY 2

100 / qtr

78 (36/44)

102 (47/46)

86 (32/37)

93 (41/44)

91 (156/157)

RY 1

100 / qtr

*

50 (16/32)

89 (40/45)

75 (40/53)

74 (96/130)

 

Definition/Description: The number of user requirements addressed within the current reporting year divided by the total number of requirements identified by the XSEDE User Engagement team through regular email contact with all XSEDE PIs within the same reporting period. Requirements are considered addressed when the PI is presented with a solution via email, a ticket is submitted via the XSEDE ticket system, or a use case is submitted for further consideration.

Collection methodology: Currently a spreadsheet is maintained that includes the status of each ticket and we report on the number of tickets that have been resolved in the quarter. The goal is to use Jira to track and report this metric.

 * NOTE: RY1 is unique and only spans September 2016-April 2017.  RY1 RP1 doesn’t exist due to the shortened RY. RY1, RP2 only includes two months (September-October 2016). 


3.1 CEE Director's Office (WBS 2.1.1)

The CEE Director’s Office has been established to provide the necessary oversight to ensure the greatest efficiency and effectiveness of the CEE area. This oversight includes providing direction to the L3 management team, coordination of, and participation in, CEE planning activities and reports through the area’s Project Manager, and monitoring compliance with budgets, and retarget effort if necessary. The Director’s Office also attends and supports the preparation of project level reviews and activities.

The CEE Director’s Office will continue to manage and set direction for CEE activities and responsibilities. They will contribute to and attend bi-weekly Senior Management Team calls, contribute to the project level plan, schedule, and budget, contribute to XSEDE quarterly, annual, and other reports as required by the NSF and attend XSEDE quarterly and annual meetings. Lastly, the Director’s Office will advise the XSEDE PI on many issues, especially those relevant to this WBS area.

3.2 Workforce Development (WBS 2.1.2) 

The Workforce Development mission is to provide a continuum of learning resources and services designed to address the needs and requirements of researchers, educators, developers, integrators, and students utilizing advanced digital resources. This includes providing professional development for XSEDE team members.

Workforce Development fulfills its mission through an integrated suite of training, education, and student preparation activities to address formal and informal learning about advanced digital resources. CEE–Workforce Development provides business and industry with access to XSEDE’s workforce development efforts including training services and student internships that have historically proven beneficial to industry.

Workforce Development is comprised of three areas: Training, Education, and Student Preparation. The Training team develops and delivers training programs to enhance the skills of the national open science community and ensure productive use of XSEDE’s cyberinfrastructure. The Education team works closely with Training and Student Preparation to support faculty in all fields of study with their incorporation of advanced digital technology capabilities within the undergraduate and graduate curriculum. The Student Preparation program actively recruits students to use the aforementioned training and education offerings to enable the use of XSEDE resources by undergraduate and graduate students to motivate and prepare them to pursue advanced studies and careers to advance discovery and scholarly studies.

3.3 User Engagement (WBS 2.1.3) 

The mission of the User Engagement (UE) team is to capture community needs, requirements, and recommendations for improvements to XSEDE’s resources and services, and report to the national community how their feedback is being addressed. XSEDE places an emphasis on maintaining consistent user contact, traceability in tracking user issues, and closing the feedback loop.

3.4 Broadening Participation (WBS 2.1.4) 

Broadening Participation’s mission is to engage underrepresented minority researchers from domains that are not traditional users of HPC and from Minority Serving Institutions. This target audience ranges from potential users with no computational experience to computationally savvy researchers, educators, Champions, and administrators who will promote change at their institutions for increased use of advanced digital services for research and teaching.

Broadening Participation will continue the most effective recruitment activities— conference exhibiting, campus visits, and regional workshops—while increasing national impact through new partnerships and the utilization of lower cost awareness strategies to continue the growth in new users from underrepresented communities. The Diversity Forum and the Minority Research Community listservs and community calls focus on user persistence in their use of XSEDE services and their deepening engagement through participation in committees such as the User Advisory Committee (UAC) and XRAC, and participation in Champions, Campus Bridging, and other programs. Persistent institutional engagement is enabled by curriculum reform and larger numbers of researchers adopting the use of advanced digital resources as a standard research method.

3.5 User Interfaces & Online Information (WBS 2.1.5) 

User Interfaces & Online Information (UII) is committed to enabling the discovery, understanding, and effective utilization of XSEDE’s powerful capabilities and services. Through UII’s ongoing effort to improve and engage a variety of audiences via the XSEDE website and user portal, UII has an immediate impact on a variety of stakeholders including the general public, potential and current users, educators, services providers, campus affiliates, and funding agencies. These stakeholders will gain valuable information about XSEDE through an information rich website, the XSEDE User Portal, and a uniform set of user documentation.

3.6 Campus Engagement (WBS 2.1.6) 

The Campus Engagement program promotes and facilitates the effective participation of a diverse national community of campuses in the application of advanced digital resources and services to accelerate discovery, enhance education, and foster scholarly achievement.

Campus Engagement, via the Campus Champions, works directly with institutions across the U.S. both to facilitate computing and data-intensive research and education, nationally and with collaborators worldwide, and to expand the scale, scope, ambition and impact of these endeavors. This is done by increasing scalable, sustainable institutional uptake of advanced digital services from providers at all levels (workgroup, institutional, regional, national, and international), fostering a broader, deeper, more agile, more sustainable and more diverse nationwide cyberinfrastructure ecosystem across all levels, and cultivating inter-institutional interchange of resources, expertise and support. Campus Engagement also aims to assist with the establishment and expansion of consortia (e.g., intra-state, regional, domain-specific) that collaborate to better serve the needs of their advanced computing stakeholders.

 

[Table of Contents]

4       Extended Collaborative Support Service - ECSS (WBS 2.2) 

The Extended Collaborative Support Service (ECSS) improves the productivity of the XSEDE user community through meaningful collaborations and well-planned training activities. The objective is to optimize applications, improve work and data flows, increase effective use of the XSEDE digital infrastructure, and broadly expand the XSEDE user base by engaging members of underrepresented communities and domain areas. The ECSS program provides professionals who can be part of a collaborative team—dedicated staff who develop deep, collaborative relationships with XSEDE users—helping them make best use of XSEDE resources to advance their work. These professionals possess combined expertise in many fields of computational science and engineering. They have a deep knowledge of underlying computer systems and of the design and implementation principles for optimally mapping scientific problems, codes, and middleware to these resources. ECSS includes experts in not just the traditional use of advanced computing systems but also in data-intensive work, workflow engineering, and the enhancement of scientific gateways.

ECSS projects fall into five categories: Extended Support for Research Teams (ESRT) (§4.2), Novel and Innovative Projects (NIP) (§4.3), Extended Support for Community Codes (ESCC) (§4.4), Extended Support for Science Gateways (ESSGW) (§4.5), and Extended Support for Training, Education and Outreach (ESTEO) (§4.6). Project-based ECSS support is requested by researchers via the XSEDE peer-review allocation process, or, in some cases, suggested by reviewers as something that would benefit the researchers. If reviewers recommend support and if staff resources are available, projects progress through three activities. First, the project is assigned to an ECSS expert. Second, the project is quantified with the formation of a work plan through collaboration with the research group. The work plan includes concrete quarterly goals and staffing commitments from both the PI team and ECSS. Third, when the project is completed, the ECSS expert produces a final report with input from the research group. A successful project is the completion of all three phases. Each state of the progression is measured to provide an assessment of progress. Submission of work plans within 45 days of initial contact, 90% of projects with work plans completed, and 85% of completed projects with final reports within three months are additional criteria for success. The ECSS managers review work plans and also track progress via Interim Project Reports.

Table 4-1: KPIs for Extended Collaborative Support Service (ECSS).

KPI

Report Year

Target

RP1

RP2

RP3

RP4

Total

 Goal Supported

Number of completed ECSS projects

RY 5

      

 

Deepen/Extend – Deepening use to existing communities

RY 4

      

RY 3

45 / yr

17 

10   

RY 2

50 / yr

16

9

10

12

47

RY 1

50 / yr

*

10

13

25

48

 

Definition/Description: The total number of completed ECSS projects in ESRT, ESCC, and ESSGW within the current reporting period. A completed project is defined as having progressed through the full support pipeline, namely assigning a consultant, producing a work plan, executing the work plan, reporting progress through quarterly reports, and filing of a final project report.

Collection methodology: The number of completed projects is tracked in Sciforma, XSEDE’s project management software. A report has been created in Sciforma that queries all the ECSS projects, filters for allocations that have ended in that quarter and also for projects that have a final report.

Aggregate mean rating of ECSS impact by PIs

RY 5

      

 

 

Deepen/Extend – Deepen use to existing communities

RY 4

      

RY 3

4 of 5 / yr

 NA

3.93   

RY 2

4 of 5 / yr

4.11 

4.03

3.82 

 4.31

4.03

RY 1

4 of 5 / qtr

*

4.56 

4.61

3.29

4.14

 

Definition/Description: After an ECSS project is marked as completed (i.e. has a workplan, work has progressed, and a final report has been filed), the L2 Directors interview the PIs, preferably via phone, and ask them to rate their satisfaction with the ECSS support they have received on a scale of 1 to 5.

Collection methodology: PIs are contacted by the L2 Directors post filing of a final report by ECSS staff members. The L2 Director asks the PI to rate their satisfaction with the support received on a scale of one to five and this number is recorded in a spreadsheet, which is provided to the L2 Project Manager (PM). The PM then transfers this number to Sciforma. Each individual satisfaction rating that is reported for the reporting period is then added together and divided by the total number of interviews conducted for the average.

Aggregate mean rating of PI satisfaction with ECSS support

RY5      



Deepen/Extend – Deepening use to existing communities

RY4      
RY34.5 of 5 / yrNA4.31   
RY24.5 of 5 / yr4.654.564.17

4.75

4.47

RY1

4.5 of 5 / qtr

*4.864.72

4.64

4.54

 

Definition/Description: Aggregate mean rating of PI satisfaction with ECSS support collected after an ECSS project is complete (i.e. has a work plan, work has progressed, and a final report has been filed) measured by the ECSS Project Exit Survey on a scale of 1 (not at all satisfied) - 5 extremely satisfied).

Collection methodology: Measured by the average of selected question(s) of the PI's satisfaction survey.

 

 * NOTE: RY1 is unique and only spans September 2016-April 2017.  RY1 RP1 doesn’t exist due to the shortened RY. RY1, RP2 only includes two months (September-October 2016). 

NA:  There are no Impact and Satisfaction ratings as ECSS recently changed the way these assessments are recorded. Future ratings obtained from the PI interviews will be validated by an anonymous survey conducted by the XSEDE’s Strategic Planning, Policy & Evaluation (SP&E) team. 

4.1 ECSS Director's Office (WBS 2.2.1)

The ECSS Director’s Office has been established to provide the necessary oversight to ensure the greatest efficiency and effectiveness of the ECSS area. This oversight includes providing direction to the L3 management team, coordination of and participation in ECSS planning activities and reports through the area’s Project Manager, and monitoring compliance with budgets, and retargeting effort, if necessary. The Director’s Office also attends and supports the preparation of project level reviews and activities. The ECSS Director’s Office will continue to manage and set direction for ECSS activities and responsibilities. They will contribute to and attend bi-weekly Senior Management Team calls, contribute to the project level plan, schedule, and budget, contribute to XSEDE quarterly, annual, and other reports as required by the NSF, and attend XSEDE quarterly and annual meetings. The Director’s Office will advise the XSEDE PI on many issues, especially those relevant to this WBS area. The office consists of two Level 2 Co-Directors, Philip Blood, who manages ESRT and NIP activities, and Bob Sinkovits, who manages ESCC, ESSGW, and ESTEO activities. The office also has three project managers (Marques Bland, Sonia Nayak, and Leslie Morsek). As noted before, Nancy Wilkins-Diehr and Ralph Roskies have both stepped down from their roles in XSEDE. Wilkins-Diehr supported transition of leadership responsibilities through June 2018.

Blood and Sinkovits carry out the post-project interviews with all project PIs who have received ECSS support, both to get their assessment of how the project went, and to hear and act on any concerns they may express. Sinkovits also organizes the monthly symposium series, serves as one of the contributors to staff training, and runs the Campus Champions Fellows program (§3.6). Blood convenes User Advisory Committee meetings and supports the User Advisory Committee Chair in his or her role.

The project managers aid in the management of the day-to-day activities of ECSS, which includes the management of project requests (XRAC and startups), active projects, project assignments, and staffing. They continuously refine the ECSS project lifecycle, further defining processes to aid in the management of over 100 active projects. They also administer JIRA for the management and tracking of projects, both for the managers and directors of ECSS and for ECSS staff.

4.2 Extended Support for Research Teams (WBS 2.2.2) 

Extended Support for Research Teams (ESRT) accelerates scientific discovery by collaborating with researchers, engineers, and scholars to optimize their application codes, improve their work and data flows, and increase the effectiveness of their use of XSEDE digital infrastructure.

ESRT projects are initiated as a result of support requests or recommendations obtained during the allocation process. Most projects focus on research codes associated with specific research teams, as community codes fall under ESCC (§4.4) but are not exclusively restricted to this classification. The primary mandate of ESRT is the support of individual research teams within the context of their research goals.

4.3 Novel & Innovative Projects (WBS 2.2.3) 

Novel and Innovative Projects (NIP) accelerates research, scholarship, and education provided by new communities that can strongly benefit from the use of XSEDE’s ecosystem of advanced digital services. Working closely with the XSEDE Outreach team, the NIP team identifies a subset of scientists, scholars, and educators from new communities, i.e., from disciplines or demographics that have not yet made significant use of advanced computing infrastructure, who are now committed to projects that appear to require XSEDE services, and are in a good position to use them efficiently. NIP staff then provide personal mentoring to these projects, helping them to obtain XSEDE allocations and use them successfully.

XSEDE projects generated by, and mentored by, the personal efforts of the NIP experts should stimulate additional practitioners in their field to become interested in XSEDE. Strategies used include building and promotion of science gateways serving communities of end-users and the enhancement of the Domain Champions program by which successful practitioners spread the word about the benefits of XSEDE to their colleagues.

4.4 Extended Support for Community Codes (WBS 2.2.4) 

Extended Collaborative Support for Community Codes (ESCC) extends the use of XSEDE resources by collaborating with researchers and community code developers to deploy, harden, and optimize software systems necessary for research communities to create new knowledge.

ESCC supports users via requested projects and XSEDE-initiated projects. ESCC projects may be created in two different ways. Most ESCC projects are initiated as a result of requests for assistance during the allocation process. These projects are similar in nature to ESRT projects but involve community codes rather than codes developed for and by individual research groups. ESCC projects may also be initiated by staff to support a community’s needs.

4.5 Extended Support for Science Gateways (WBS 2.2.5) 

Extended Support for Science Gateways (ESSGW) broadens science impact and accelerates scientific discovery by collaborating in the development and enhancement of science-centric gateway interfaces and by fostering a science gateway community ecosystem.

4.6 ECSS Communities – Extended Support for Education, Outreach, & Training (WBS 2.2.6) 

Extended Collaborative Support for Training, Education & Outreach (ESTEO) prepares the current and next generation of researchers, engineers, and scholars in the use of advanced digital technologies by providing the technical support for Training, Education, and Outreach planned activities.

Typical events include train-the-trainers events, on-site classes requested by Campus Champions, regional workshops, conferences, and summer schools (including the International HPC Summer School). Staff also create and review online documentation and training modules. This on-demand training is increasingly popular with the user community when both time and travel budgets are limited.

[Table of Contents]

5        XSEDE Cyberinfrastructure Integration (XCI, WBS 2.3) 

The mission of XSEDE Cyberinfrastructure Integration (XCI) is to facilitate interaction, sharing, interoperability, and compatibility of all relevant software and related services across the national CI community, building and improving upon the foundational efforts of XSEDE.

XCI envisions a national cyberinfrastructure that is consistent and straightforward to understand, and practical for use by researchers and students. Service to XSEDE Service Providers (SPs) is a particularly important aspect of XCI’s activities. Service Providers should find it as simple as possible to provide resources to the national research community through shared expertise and, when appropriate, software and tools. We strive to make it possible for researchers and students to effortlessly use computational and data analysis resources ranging from those allocated by XSEDE to campus-based CI facilities, an individual’s workstation, and commercial cloud providers, and to interact with these resources via CI software services such as science gateways and Globus Online. Through XCI, XSEDE serves an aligning function within the nation, not by rigorously defining a particular architecture, but rather by assembling a technical infrastructure that facilitates interaction and interoperability across the national CI ecosystem. In turn, this infrastructure is adopted by campus, regional, and national CI providers because it makes their task of delivering services easier and the delivered services better. The suite of interoperable and compatible software tools that XSEDE will make available to the CI community will be based on those already in use, and services will be added that address emerging needs including data and computational services.

The XSEDE User Requirements Evaluation and Prioritization Working Group (UREP) prioritizes users’ needs for XCI. The UREP provides the primary direction and prioritization for the Requirements Analysis and Capability Delivery team (RACD). Given that, as a source of prioritization, the group of NSF-funded Level 1 SPs is generally customer number one for XCI. These are the nationally-accessible cyberinfrastructure resources created and operated by Service Providers that are required to be interoperable with XSEDE services in general, and to be allocated via the XRAC process. The Level 1 SPs, then, are the cyberinfrastructure Service Providers most invested in having important requirements from XSEDE and important requirements for XSEDE (and XCI) to fulfill. Level 2 and 3 SPs represent the next two most motivated and engaged groups of cyberinfrastructure providers in the US. The SP Forum is another source for user needs and priorities; the members of the SP Forum constitute national leaders and exemplars for the US national cyberinfrastructure community as a whole. By engaging with Level 1, 2, and 3 SPs and the SP Forum in particular, we believe that we can get the most detailed statements of needs and priorities. New tools implemented under the leadership of XCI are most likely to be widely and quickly adopted by the national community of CI providers if they are first adopted by members of the SP Forum. Because the XSEDE Cyberinfrastructure Resource Integration team (CRI) deals primarily with Level 1, 2, and 3 SPs, along with campus cyberinfrastructure administrators and support experts, the SP Forum and Campus Champions are XCI’s primary sources of direction regarding prioritization of efforts.

The Community Software Repository (CSR) is a new service and tool catalog for the national research community to facilitate connecting resources, software, and services into the broader cyberinfrastructure ecosystem. 

Table 5-1: KPIs for XSEDE Cyberinfrastructure Integration (XCI).

KPI

Report Year

Target

RP1

RP2

RP3

RP4

Total

Goal Supported

Aggregate mean rating of satisfaction with XCI services

 

RY 5

      

 

 

Advance – Create an open and evolving e-infrastructure

RY 4

      

RY 3

4.5 of 5 / yr4.124.50   

RY 2

4 out of 5 / yr4.254.504.40

4.65

4.45 

RY 1

4 out of 5 / yr

*4.804.80

4.50 

4.50

 

Definition/Description: Aggregate mean rating of customer satisfaction with XCI-provided services as measured by the annual User Survey, the CSR rating tool, and direct customer interaction with users on a scale of 1 (not at all satisfied) - 5 (extremely satisfied).

Collection methodology: User interviews and micro-surveys done on an as need basis after services have been delivered.

Number of non-XSEDE partnerships with XCI

 

RY 5

      

 

 

Advance – Create an open and evolving e-infrastructure

RY 4

      

RY 3

8 / yr124   

RY 2

----

 -

8

RY 1

-

*--

-

8

 

Definition/Description: Number of XCI partnerships supported/facilitated between CRI and SPs, national CI organizations, and campus CI providers.

Collection methodology: Non-XSEDE partnerships are tracked through the ticket system and through JIRA.

Mean time to issue resolution (days)RY 5      

 

 

Sustain — Provide excellent user support 

RY 4      
RY 314 / qtr22.37.3   
RY 2< 30 days4.07.08.0

5.0

6.1

RY 1

< 45 days

*7.04.016.09.0
 

Definition/Description: Average number of days from defect or support request submitted to solution communicated.

Collection methodology: The data is stored locally via the Request Tracker service, which is the XSEDE ticket software. Reports are generated against the RT database to determine MTTR and other metrics.

 * NOTE: RY1 is unique and only spans September 2016-April 2017.  RY1 RP1 doesn’t exist due to the shortened RY. RY1, RP2 only includes two months (September-October 2016). 


5.1 XCI Director's Office (WBS 2.3.1)

The XCI Director’s Office has been established to provide necessary oversight to ensure the greatest efficiency and effectiveness of the XCI area. This oversight includes providing direction to the L3 management team, coordination of and participation in XCI planning activities and reports through the area’s project manager, and monitoring compliance with budgets, retargeting effort, if necessary. The Director’s Office also attends and supports the preparation of project level reviews and activities.

The XCI Director’s Office will continue to manage and set direction for XCI activities and responsibilities. They will contribute to and attend bi-weekly Senior Management Team calls; contribute to the project level plan, schedule, and budget; contribute to XSEDE IPRs, annual reports and other reports as required by the NSF; and attend XSEDE quarterly and annual meetings. Lastly, the Director’s Office will advise the XSEDE PI on many issues, especially those relevant to this WBS area.

5.2 Requirements Analysis and Capability Delivery (RACD, WBS 2.3.2) 

The Requirements Analysis & Capability Delivery (RACD) team facilitates the integration, maintenance, and support of cyberinfrastructure capabilities addressing user technical requirements. The process begins by preparing Capability Delivery Plans (CDPs) that describe the technical gaps in XSEDE’s prioritized Use Cases. To fill the gaps, RACD evaluates and/or tests existing software solutions, engages with software providers, and facilitates software and service integration. To ensure software and service adoption and ROI, RACD will involve users, Service Providers, and operators in an integration process that uses engineering best practices and instruments components to measure usage. Once components are integrated, RACD will facilitate software maintenance and enhancements in response to evolving user needs and an evolving infrastructure environment.

5.3 XSEDE Cyberinfrastructure Resource Integration (WBS 2.3.3) 

The mission of the Cyberinfrastructure Resource Integration (CRI) team is to work with SPs, CI providers, and campuses to maximize the aggregate utility of national cyberinfrastructure. CRI will facilitate the incorporation of XSEDE software at SPs and encourage SPs to publish their information in the RDR. CRI’s activities are reflected in the uptake of CRI-integrated toolkits, such as the XSEDE Campus Bridging Cluster toolkit and XSEDE National Integration Toolkit, but also Globus Transfer clients and other toolkits as developed.

 

[Table of Contents]

6        XSEDE Operations (WBS 2.4) 

The mission of XSEDE Operations is to install, connect, maintain, secure, and evolve an integrated cyberinfrastructure that incorporates a wide range of digital capabilities to support national scientific, engineering, and scholarly research efforts.

In addition to the Operations Director’s Office (§6.1), Operations staff is subdivided into four teams based on the work breakdown structure: Cybersecurity (§6.2), Data Transfer Services (DTS, §6.3), XSEDE Operations Center (XOC, §6.4), and Systems Operational Support (SysOps, §6.5). The Operations management team meets weekly and individual Operations groups meet approximately bi-weekly with all meeting minutes posted to the XSEDE wiki. 

Table 6-1: KPIs for XSEDE Operations.

KPI

Report Year

Target

RP1

RP2

RP3

RP4

Total

Goal Supported

Mean rating of user satisfaction with tickets closed by the XOC

RY 5

      

 

 

Sustain – Provide reliable, and secure infrastructure

RY 4

      

RY 3

4.5 of 5 / qtr4.54.9   

RY 2

4.5 of 5 / qtr4.54.84.8

4.4

4.6

RY 1

4 of 5 / qtr

*4.84.2

5.0

4.7

 

Definition/Description: Mean rating of satisfaction by users who submitted help tickets that were closed by the XOC during the specified reporting period as measured by the XSEDE Operations Point-of-Service Satisfaction Survey on a scale of 1 (not at all satisfied) - 5 (extremely satisfied).

Collection methodology: Within two hours of the resolution and closure of a ticket, the user receives an e-mail containing a link to a Survey Monkey survey. The survey consists of five questions using a 5 point Likert scale and the results are stored in Survey Monkey.

Hours of downtime with direct user impacts from XSEDE security incidents

 

RY 5

      

 

 

Sustain – Provide reliable, and secure infrastructure

RY 4

      

RY 3

0 / qtr00   

RY 2

0 / qtr000

0

0

RY 1

<24 / qtr

*0146

0

146

 

Definition/Description: This metric will measure resource unavailability for users as a result of an XSEDE-wide security event (involving a Tier 1 service or spanning more than one SP). The metric will be calculated as the Time to Return to Repair (TTR) summed across all applicable services and incidents during the quarter.

Collection methodology: The Cybersecurity co-leads determine that an XSEDE-wide security event is responsible for an outage. The Sysops lead will determine the TTR based on monitoring and other information in logs and tickets, and the TTR value will be the data point reported.

 * NOTE: RY1 is unique and only spans September 2016-April 2017.  RY1 RP1 doesn’t exist due to the shortened RY. RY1, RP2 only includes two months (September-October 2016). 

6.1 Operations Director's Office (WBS 2.4.1)

The Operations Director’s Office has been established to provide the necessary oversight to ensure the greatest efficiency and effectiveness of the Operations area. This oversight includes providing direction to the L3 management team, coordination of and participation in Operations planning activities and reports through the area’s Project Manager, and monitoring compliance with budgets, retargeting effort if necessary. The Director’s Office also attends and supports the preparation of project level reviews and activities.

The Operations Director’s Office will continue to manage and set direction for Operations activities and responsibilities. The Office will contribute to and attend bi-weekly Senior Management Team calls, contribute to the project-level plan, schedule, and budget, contribute to XSEDE IPR, annual, and other reports as required by the NSF, and attend XSEDE quarterly and annual meetings. Lastly, the Director’s Office will advise the XSEDE PI on many issues, especially those relevant to this WBS area.

6.2 Cybersecurity (WBS 2.4.2) 

The Cybersecurity Security (SecOps) group protects the confidentiality, integrity and availability of XSEDE resources and services. Users expect XSEDE resources to be reliable and secure, thus the security team’s goal is to minimize any interruption of services related to a security event.

6.3 Data Transfer Services (WBS 2.4.3) 

The Data Transfer Services (DTS) group facilitates data movement and management for the community by maintaining and continuously evolving XSEDE data services and resources.

6.4 XSEDE Operations Center (WBS 2.4.4) 

The XSEDE Operations Center (XOC) staff serve as user advocates, providing timely and accurate assistance to the XSEDE community, while simultaneously monitoring and troubleshooting user-facing systems and services.

6.5 Systems Operational Support (WBS 2.4.5) 

Systems Operational Support (SysOps) provides enterprise-level support and system administration for all XSEDE central services.

 

[Table of Contents]

7     Resource Allocation Service (WBS 2.5) 

The Resource Allocation Service (RAS) is building on XSEDE’s current allocation processes and evolving to meet the challenges presented by new types of resources to be allocated via XSEDE, new computing and data modalities to support increasingly diverse research needs, and large-scale demands from the user community for limited XSEDE-allocated resources. RAS is pursuing these objectives through three activities: managing the XSEDE allocations process in coordination with the XD Service Providers, enhancing and maintaining the RAS infrastructure and services, and anticipating changing community needs.

Table 7-1: KPIs for Resource Allocation Service (RAS).

KPI

Report Year

Target

RP1

RP2

RP3

RP4

Total

Goal Supported

Average mean rating of user satisfaction with allocations process

RY 5

      

 

 

Sustain – Provide excellent user support

RY 4

      

RY 3

4 of 5 / qtr4.084.13   

RY 2

4 of 5 / qtr4.143.994.13

3.86

4.03

RY 1

4 of 5 / qtr

*3.984.03

4.03

4.01

 

Definition/Description: Aggregate mean rating of user satisfaction with allocation policies and procedures for the submission, review, and administration of allocation requests as measured by the Post Allocations Submission Survey on a scale of 1 (not at all satisfied) - 5 (extremely satisfied).

Collection methodology: Quarterly survey of persons who have submitted allocation requests in the prior quarter. Aggregate mean rating of satisfaction with the allocations process (e.g. time to prepare documents, documentation, support available, reviewer comments, etc.) on a scale of 1 (not at all satisfied) - 5 (extremely satisfied).

Aggregate mean rating of user satisfaction with XRAS

RY 5

      

 

 

Sustain – Provide reliable, efficient, and secure infrastructure

RY 4

      

RY 3

4 of 5 / qtr4.034.18   

RY 2

4 of 5 / qtr4.034.004.14

3.91

4.02

RY 1

4 of 5 / qtr

*3.984.03

4.03

4.01

 

Definition/Description: Aggregate mean rating of user satisfaction with XRAS (XSEDE Resource Allocation System), the primary tool for submission, review, and administration of allocation requests as measured by the Post Allocation Submission Survey on a scale of 1 (not at all satisfied) - 5 (extremely satisfied).

Collection methodology: Quarterly survey of persons who have submitted allocation requests in the prior quarter. Aggregate mean rating of satisfaction with the XRAS system (e.g. ease of use for submission, responsiveness, ease of finding information about outcomes, etc.), on a scale of 1 (not at all satisfied) - 5 (extremely satisfied).

 * NOTE: RY1 is unique and only spans September 2016-April 2017.  RY1 RP1 doesn’t exist due to the shortened RY. RY1, RP2 only includes two months (September-October 2016). 

7.1 RAS Director's Office (WBS 2.5.1)

The RAS Director’s Office has been established to provide the necessary oversight to ensure the greatest efficiency and effectiveness of the RAS area. This oversight includes providing direction to the L3 management team, coordination of and participation in RAS planning activities and reports through the area’s Project Manager, and monitoring compliance with budgets, and retarget effort if necessary. The Director’s Office also attends and supports the preparation of project-level reviews and activities. The RAS Director's Office also contributes to an analytics effort to support NSF, Service Providers, and XSEDE in understanding and projecting the stewardship of, demand for, and impact of CI resources and services.

The RAS Director’s Office will continue to manage and set direction for RAS activities and responsibilities. They will contribute to and attend bi-weekly Senior Management Team calls, contribute to the project-level plan, schedule, and budget, contribute to XSEDE IPR, annual, and other reports as required by the NSF, and attend XSEDE quarterly and annual meetings. Lastly, the Director’s Office will advise the XSEDE PI on many issues, especially those relevant to this WBS area.

7.2 XSEDE Allocations Process & Policies (WBS 2.5.2) 

Allocations enable the national open science community to easily gain access to XSEDE’s advanced digital resources, allowing them to achieve their research and education goals.

7.3 Allocations, Accounting & Account Management CI (WBS 2.5.3) 

The Allocations, Accounting and Account Management CI (A3M) group maintains and improves the interfaces, databases, and data transfer mechanisms for XSEDE-wide resource allocations, accounting of resource usage, and user account management.

 

[Table of Contents]

8     Program Office (WBS 2.6) 

The purpose of the Program Office is to ensure critical project level functions are in place and operating effectively and efficiently. The oversight provided via the Project Office is necessary to provide consistent guidance and leadership to the L3 managers across the project. A common and consistent approach to managing projects and risks is provided by the Project Management, Reporting and Risk Management (PM&R) team, while Business Operations manages all financial functions and sub-awards. The crucial aspect of communications to all stakeholders is the focus of the External Relations team. Finally, Strategy, Planning, Policy, Evaluation & Organizational Improvement (SP&E) focuses attention in precisely those areas to ensure the best possible structure continues to exist within XSEDE to allow the support of all significant project activities and enable efficient and effective performance of all project responsibilities.

Table 8-1: KPIs for the Program Office.

KPI

Report Year

Target

RP1

RP2

RP3

RP4

Total

Goal Supported

Variance between relevant report submission and due date (days) 

RY 5

      

 

 

Sustain — Operate an effective and productive virtual organization 

RY 4

      

RY 3

000   

RY 2

0000

0

0

RY 1

0

*NA0

0

0

 

Definition/Description: Variance in number of calendar days between report submission and due date for the previous reporting period. In accordance with NSF quarterly and annual reporting requirements, deadlines are determined together by the XSEDE PI and NSF Project Officer and then communicated to the XSEDE team to establish the reporting schedule used to calculate this metric.

Collection methodology: The XSEDE PI tracks when the relevant document was submitted to FastLane and records the number of days between the due date and this date in a spreadsheet.

Percentage of sub-award invoices processed within target duration

RY 5

      

 

 

 

Sustain — Operate an effective and productive virtual organization 

RY 4

      

RY 3

95 /
qtr
82.477.8   

RY 2

95 /
qtr
10090.988

67.4

74.6

RY 1

90 /
qtr

*NA100

-

100

 

Definition/Description: Number of invoices processed within the target duration divided by the total number of invoices processed. Duration is determined by the number of calendar days from acceptance of submitted invoice to invoice paid.

Collection methodology: Data is logged and stored in the Business Operations master spreadsheet located on the Business Operations Google Drive folder.

Percentage of recommendations addressed by relevant project areas within 60 days

 

RY 5

      

 

 

Sustain – Operate an effective and productive virtual organization

RY 4

      

RY 3

90 / yr4674   

RY 2

90 / qtr231537

 49

31

RY 1

90 / qtr

*NA100

57 

67

 

Definition/Description: Percentage of recommendations made to XSEDE that were addressed by Level 2 program area managers. Percentage is calculated using recommendations from the XSEDE Advisory Board, NSF Review Panels, Service Provider Forum, User Advisory Committee, and the XSEDE Staff Climate Study conducted by the external evaluation team.

Collection methodology: The number of key recommendations made according to the annual XSEDE Climate Study findings + the total number of recommendations recorded on the XSEDE Project-Wide Improvements & Recommendations Google Sheet, such as XAB, NSF, and SP&E recommendations. XSEDE staff, not the SP&E team, are accountable for entering information to track improvements and recommendations.


NOTE: An improvement made is not necessarily a recommendation addressed; but a recommendation addressed is an improvement made and therefore should also be tracked in the XSEDE Project-Wide Improvements & Recommendations Google Sheet once fully implemented; it is OK to have duplication between Improvements fully implemented and Recommendations addressed.

Aggregate mean rating of satisfaction with content and accessibility of the XSEDE Staff Wiki

RY 5      Sustain — Operate an innovative virtual organization 




RY 4      
RY 33.5 of 5 / yr-3.91   
RY 2 NANANA

NA

NA

RY 1 *NANA

 NA

NA
 

Definition/Description: Aggregate mean rating of staff satisfaction with the available content and ability to access information relevant to their work via the XSEDE Staff Wiki as measured annually by the XSEDE Staff Climate Study on a scale of 1 (not at all satisfied) - 5 (extremely satisfied), items 1A and 1C.

Collection methodology: A mean index score is generated from annual Staff Climate Study responses to all items within the "XSEDE Staff Wiki and Website" dimension of the study to determine aggregate mean.

Number of staff publications

RY 5      

 

 

Sustain — Operate an innovative virtual organization 




RY 4      
RY 320 / yr1916   
RY 220 / yr260

1

9
RY 170 / yr*50

 13

18

 

Definition/Description: Number of technical reports, presentations, and peer-reviewed publications authored or co-authored by XSEDE staff to date within the current reporting year.

Collection methodology: When an XSEDE staff member has a publication, he or she enters the publication data into their profile on the XUP. XSEDE documents added to IDEALS by XSEDE staff are also included. This data is then summed for the appropriate quarter and reported.

Aggregate mean rating of Inclusion in XSEDE

RY 5

      

 

 

Sustain – Operate an innovative virtual organization

RY 4

      

RY 3

4.1 / yr-4.14   

RY 2

 *4.25*

*

4.25

RY 1

 

***

*

*

 

Definition/Description: Aggregate mean rating of staff perception and experience relating to inclusion in the project as measured annually by the XSEDE Staff Climate Study on a scale of 1 (low inclusion) - 5 (high inclusion).

Collection methodology: A mean index score is generated from annual Staff Climate Study responses to items within the "Inclusion" dimension of the study. This score is then compared to the previous year to determine percent change on an annual basis.

Aggregate mean rating of Equity in XSEDE

RY 5

      

 

 

Sustain – Operate an innovative virtual organization

RY 4

      

RY 3

4.0 / yr-    

RY 2

 *****

RY 1

 

***

*

*

 

Definition/Description: Aggregate mean rating of staff perception and experience relating to diversity and equity in the project as measured annually by the XSEDE Staff Climate Study on a scale of 1 (low equity) - 5 (high equity).

Collection methodology: A mean index score is generated from annual Staff Climate Study responses to items within the "diversity" dimension of the study. This score is then compared to the previous year to determine percent change on an annual basis.

Number of XSEDE-related social media impressionsRY 5       

 

 

Deepen/ Extend — Raise awareness of the value of advanced digital services 

RY 4       
RY 3320,000 /yr112,80663,269   
RY 2 300,00 /yr69,60755,50659,490128,180312,783
RY 1 190,000 /yr*52,200128,67588,332262,207
 

Definition/Description: Number of people who have seen XSEDE interactions on Facebook + Twitter. "Impressions" are generated by Facebook and Twitter to determine how many individuals have seen XSEDE interactions or posts on said platforms. This includes "likes," "retweets," "followers," “shares,” as well as people who see the post based on their friends or followers sharing XSEDE information.

Collection methodology: Facebook and Twitter both have internal methods of tracking metrics - we simply go to those sites and find the “impressions” numbers we need at the end of each reporting cycle. Anyone with access to the FB and Twitter pages can easily gather this information.

Number of XSEDE-related media hitsRY 5       

 

 

Deepen/ Extend — Raise awareness of the value of advanced digital services 

RY 4       
RY 3 169 / yr2329   
RY 2 169 / yr42304429145
RY 1 147 / yr*32301880
 

Definition/Description: Number of XSEDE-related stories found in the media through Google alerts of manual searches for “XSEDE,” that mention XSEDE by name within the current reporting period.

Collection methodology: NCSA tracks media hits for both NCSA and XSEDE, so that is a baseline collection of numbers. The ER team can then do additional manual searching - as an example, oftentimes the hits of HPCwire are not collected on the NCSA page, so an additional manual count is needed.

 * NOTE: RY1 is unique and only spans September 2016-April 2017.  RY1 RP1 doesn’t exist due to the shortened RY. RY1, RP2 only includes two months (September-October 2016). 

- Data reported annually.

8.1 Project Office (WBS 2.6.1) 

The Project Office has been established to provide the necessary oversight to ensure the greatest efficiency and effectiveness of the Program Office area and to establish responsibility for assuring advisory activities of the project occur. This oversight includes providing direction to the L3 management team and coordination of and participation in Program Office planning activities and reports through the area’s Project Manager. The Project Office also attends and supports the preparation of project-level reviews and activities. Importantly, the Project Office is responsible for ensuring that the XSEDE Advisory Board, the User Advisory Committee, and the SP Forum are functioning. The Project Office is responsible for coordination of project-level meetings such as the bi-weekly Senior Management Team (SMT) teleconference calls and the project quarterly meetings. Lastly, the Project Office will advise the XSEDE PI on many issues, especially those relevant to this WBS area.

8.2  External Relations (WBS 2.6.2) 

External Relations’ (ER) mission is to communicate the value and importance of XSEDE to all stakeholders (including the internal audience) through creative and strategic communications.

8.3 Project Management, Reporting, & Risk Management (WBS 2.6.3) 

The Project Management, Reporting & Risk Management (PM&R) team enables an effective virtual organization through the application of project management principles, provides visibility to project progress, successes, and challenges, brings new ideas and management practices into the project, and disseminates lessons learned in XSEDE to other virtual organizations. Communication is critical to success in this highly distributed virtual organization.

8.4 Business Operations (WBS 2.6.4) 

The Business Operations (BO) group, working closely with staff at the University of Illinois’ Grants and Contracts Office (GCO) and National Center for Supercomputing Applications’ (NCSA) Business Office, manages budgetary issues and sub-awards, and ensures timely processing of sub-award amendments and invoices.

8.5 Strategic Planning, Policy & Evaluation (WBS 2.6.5) 

XSEDE dedicates effort to project-wide strategic planning, policy development, evaluation and assessment, and organizational improvement in support of sustaining an effective and productive virtual organization.

XSEDE has engaged an independent Evaluation Team designed to provide XSEDE with information to guide program improvement and assess the impact of XSEDE services. Evaluations are based on five primary data sources: (1) an Annual User Survey that is part of the XSEDE annual report and program plan; (2) an Enhanced Longitudinal Study, encompassing additional target groups (e.g., faculty, institutions, disciplines, etc.) and additional measures (e.g., publications, citations, research funding, promotion and tenure, etc.); (3) an Annual XSEDE Staff Climate Study; (4) XSEDE KPIs, Area Metrics, and Organizational Improvement efforts, including ensuring that procedures are in place to assess these data; and (5) Specialized Studies as contracted by Level 2 directors and the Program Office.

 

[Table of Contents]

Footnotes
Ref Notes
1 https://www.nsf.gov/pubs/2018/nsf18045/nsf18045.pdf 
2 http://www.nsf.gov/cise/aci/cif21/CIF21Vision2012current.pdf
3 http://www.nsf.gov/publications/pub_summ.jsp?ods_key=nsf12051
4 http://www.nist.gov/baldrige/

 

 



  • No labels