[In progress] Collection methodology for XCI satisfaction KPI.
Concern about variance in values for this when the annual XSEDE User Survey is included vs. other reporting periods when it’s not. Questions: Remove user survey? Consistent data sources for each reporting period every year?
Aggregate mean rating of satisfaction with XCI services
Description: Average mean rating of customer satisfaction with XCI-provided services as measured by the annual User Survey, the CSR rating tool, and direct customer interaction with users on a scale of 1 (not at all satisfied) - 5 (extremely satisfied).
Collection methodology: User interviews and micro-surveys done on an as need basis after services have been delivered.
At the October 3, 2019 XCI meeting, external evaluation team members (Julie and Lorna) joined the L3 XCI leads to discuss this metric further.
The team agrees to continue with plans from the September 2019 Quarterly meeting to reduce the KPI data source to the annual User Survey only. The XCI team will continue to collect data (i.e. surveys, etc) for internal purposes and may report these metrics in future IPR appendices.
The XCI team is ok with having an annual measure and not a quarterly measure for IPR reporting.
Julie provided a copy of the draft 2019 User Survey instrument to XCI staff for review.
Team discussed the possibility of an SP microsurvey but decided a special module in the Annual User Survey was ideal/sufficient and would streamline the data collection and reporting process as the XCI IPR metrics would be collected and reported as other project-wide metrics that are generated from the Annual User Survey.
XCI participants will be provided by XCI team to Julie to ensure evaluation team includes them in distribution list for the annual User Survey.
10/3/2019 Next steps:
XCI L3s will review the User Survey and make edits. The team will reach out to Julie and Lorna as needed during this process.
Julie and Lorna will join the Oct 17 XCI meeting to review any changes from L3s.
[Resolved] Performance of Project KPI: Aggregate mean rating of user satisfaction with XSEDE technical support services
Current target: 3.5/5: Should this target be raised?
RY3: 3.6; RY4: 3.9
Since this is measured via the annual user survey, it is actually assessing performance/satisfaction of the previous year. Thus, a change in the RY5 target is really a change in the expectation of satisfaction for RY4. If this will be changed for the RY5 user survey, it should be decided and communicated to the project very soon.
The decision was made to not increase this substantially as the increase from RY3 to RY4 has not been seen historically. The external evaluation team recommends increasing this to 3.75 at the most until RY5 data is reported.
Revised Survey Question
Includes questions from Service Provider micro-survey:
(USER) Logged in using an SSH command line
Login to SP resources thru the single sign-on hub (login.xsede.org) using XSEDE credentials Direct login to SP resources using SP specific credentials
Direct login to SP resources with XSEDE credential using MyProxy and GSI OpenSSH
Login to XSEDE using multiple factors (only rate this capability if you know you are using it)
XSEDE allocation lookup command line tool (xdusage)
(USER) Accessed resources using campus federated credentials or XSEDE credentials
Authenticating to the XSEDE User Portal and other web services using campus InCommon credentials (CILogon)
Using XSEDE credentials to log into non-XSEDE Web services (e.g., GENI and OrCID.)
(USER) Utilized XSEDE-provided Globus data management services
Globus Transfer Service
Globus Sharing Service
Globus Connect for moving data between personal systems and XSEDE resources
(USER) Ran batch jobs
The job start prediction service (Karnak)
(NEW) (USER) Access XSEDE affiliated web sites using the "XSEDE Login" button (web single sign-on)
Login to XSEDE affiliated web sites without entering a password every time
(NEW) (USER) Would you be interested if XCI's Resource Integration team released/curated a repository of containers (a)
(NEW) (USER) Access the Research Software Portal (software.xsede.org) to discover or share software
Search all available software and services in the XSEDE community
Advertise software my team makes available to the XSEDE community
Describe a new user story or use case
Discover already supported use cases
(NEW) (USER) If we may contact you to follow up on your comments and/or to gauge your interest in participating in a use-case study, please share your contact information. (a)
(GATEWAY) Support a gateways integrated with XSEDE
The gateways submit attributes service (API)
Globus Auth for user login to gateway
(NEW) (SERVICE PROVIDER) Integrate a Service Provider resource into XSEDE
IPF software used to publish resource information to XSEDE
Discuss software needs, delivery activities, deployment, and usage
Participate in software technical reviews
(NEW) (SERVICE PROVIDER) XCI software related technical services (a)
Software Integration Support
Software Usage Instrumentation Support
Use case requirements analysis processes
Use case development processes
Use case prioritization processes
(NEW) (SERVICE PROVIDER) What level Service Provider (SP) do you represent? (a)
(NEW) (SERVICE PROVIDER) How many users does your SP support? Please provide your best estimate (a)
(NEW) (SERVICE PROVIDER) How long has your institution collaborated with the XSEDE project (or its predecessor, TeraGrid)? (a)
(NEW) (SERVICE PROVIDER) Please share with us any comments, concerns or suggestions about the services, software, and/or associated components offered by the XSEDE Cyberinfrastructure Integration group. (a)
(a) both awareness and satisfaction ratings from XCI Software and Services Micro-Survey Spring 2018.
Aggregate mean rating of user satisfaction with XSEDE technical support services
Aggregate rating of user satisfaction with XSEDE technical expertise and support services via the annual User Survey on a scale of 1 (very dissatisfied) - 5 (very satisfied).
Aggregate rating of all current "XSEDE Software and Service Components" measured using the "frequency of performance meeting expectations" scale on the annual XSEDE User Survey.