Review Rubric

To help the XRAC reviewers ensure that requests address the relevant criteria and other mandatory request components, the attached Review Rubric is provided prior to each XRAC meeting.

Review Criteria

Source: Allocations Policies, section 7.2

All resource requests are reviewed against three criteria, which apply across all types of resources, with the level of detail of the review rising with the size of the requested resources:

    1. Appropriateness of Methodology: For compute requests, the choice of applications, methods, algorithms and techniques to be employed to accomplish the stated scientific objectives should be reasonably described and motivated. For storage requests, the data usage, access methods, algorithms and techniques to be employed to accomplish the stated research objectives should be reasonably described and motivated. For shared collections, requestors should describe the public or community access methods to be provided.

    2. Appropriateness of Research Plan: The steps in the research plan should explain how the research objectives would be achieved. For compute resources, the proposed computations should encompass simulation parameters (step size, time scale, ensemble parameters, etc.) that are needed to obtain accurate and meaningful results, as well as the human resources that can be devoted to the task. The amount of resources requested should be derived from the methodology and the research plan. If there are serious concerns about the research plan, reviewers will document those concerns in their reviews, and the PI may choose to address those concerns in an Appeal.

    3. Efficient Use of Resources: The resources selected should be used as efficiently as is reasonably possible and in accordance with the recommended use guidelines of those resources. For computational resources, performance and parallel scaling data should be provided along with a discussion of optimization and/or parallelization work to be done to improve the applications. If the reviewers conclude that the request is more appropriate on XSEDE resources other than those requested, they may recommend an allocation on those other resources instead.

In considering these criteria, scientific productivity is the end goal. If adapting to less familiar but, in the view of the panel, better architectures or algorithms requires a significant learning curve for the proposer, with a concomitant interruption of scientific productivity, the panel may suggest the alternatives, but nevertheless grant the requested resources with the proposed architecture and methods. In exceptional cases, where the reviewers conclude "beyond a reasonable doubt" that the proposed methods are so inefficient that they amount to a waste of public resources, they should not approve the request until their concerns are addressed by the proposer. Reviewers are encouraged to recommend extended collaborative support, even for those proposers who have not requested it, as part of their recommendations for addressing shortcomings. For storage resources, information on required performance and expected access patterns should be provided for all data and collections to be stored and used along with a discussion of work done or planned to improve the efficiency of the data use.