15 February, 2017 - version 2.2

Document History

Relevant Sections

Version

Date

Changes

Author

Entire Document

1.0

Apr 5, 2013

First public release

XRAS Team

Entire document

1.1

Oct 15, 2013

Replace “POPS 2.0” with “XRAS”

D. Hart

Section 4 and 5

2.1

Jan 19, 2017

Update Section 4

Alina Banerjee

Entire document

2.2

Feb 15, 2017

Formatting, Table of contents, XDMoD description, Appendix, addition of screenshots

Alina Banerjee

Table of Contents

Introduction ................................................................

The Purpose of this Document ..............................................

Document Management and Configuration Control ............................

XRAS architecture ..........................................................

Submission GUI ..............................................................

Review GUI ..................................................................

Administrative (Admin) GUI .................................................

Reporting service (XDMoD) ..................................................

XRAS Submit API .............................................................

XRAS-review component ( integrated with the Review GUI ) ..................

XRAS-admin component ( integrated with the Admin GUI ) ....................

AMIE interface to accounting service ....................................

REST interface to RDR ....................................................

Interface to XDMoD .......................................................

Database - XDCDB ............................................................

Important note on the XDCDB ...............................................

Rules Engine ................................................................

Deployment Notes ............................................................

Deployed by XRAS SaaS customers ...........................................

Deployed as XSEDE centralized services ...................................

References .................................................................

Appendix: XRAS software-as-a-service (SaaS) ..........................

Overview of and rationale for the XRAS architecture .....................

Architectural alternatives .................................................

SaaS advantages .............................................................

 

List of Abbreviations

Abbreviation

Name

XSEDE

Extreme Science and Engineering Discovery Environment

XRAS

XSEDE Resource Allocations Services

XUP

XRAS User Portal

XDMoD

XD Metrics on Demand

IIS

Integrated Information Services

RDR

Resource Description Repository

Introduction

The eXtreme Science and Engineering Discovery Environment (XSEDE) provides researchers, educators, and students across the United States and beyond with access to powerful computers, data, tools, and other resources to conduct research and improve the planet. A key element of XSEDE infrastructure is the centralized process by which Allocations on participating resources are requested, reviewed, and awarded.

The Purpose of this Document

This document describes the Level 3 decomposition of XRAS’s architecture. As such, this document relies on terminology, definitions, and general architectural abstractions described in the Level 1 and 2 decompositions of the overall XSEDE architecture [ 1 ].

Document Management and Configuration Control

The version documented here is 2.2 of the XRAS Level 3 Decomposition.

XRAS architecture

The XRAS use cases led the XRAS team to define the high-level architecture summarized below. Supporting services shown here reflect the current XSEDE services, but XRAS will be designed to allow XRAS customer sites to designate alternate services.

Submission GUI

Description : The Submission GUI provides users with the ability to submit, modify, and view the status of allocation requests. For XSEDE, the Submission GUI will be deployed within the XUP, but other XRAS customers can implement the GUI in their own sites as long as they comply with the XRAS Submit API. Sites can customize a Submission GUI integrated within their own site portals or web sites and provide additional functionality above and beyond that required by XRAS. The Submission GUI determines how submission data entry is presented to the user.

Target user : Researchers, students, and staff who need to submit an allocation Request.

Screenshot :

Uses: The Submit service, XSEDE (or other customer-designated) identity and authentication services.

Deployment view : Provided via XUP, accessible via Web browsers.

Review GUI

Description : The Review GUI provides the interface through which persons designated as Reviewers for a given Allocations Process see their review assignments, view Submission documents, and submit their reviews and recommendations. The Review GUI will be provided by XRAS (note: Unlike the Submit GUI, the Review GUI directly interacts with both the XRAS database, the Rules Engine and the relevant Allocations Processes such as the Accounting Service).

Target user : A user who has been designated a reviewer for a given Allocation Process.

Screenshot :

Uses : The XRAS-review service, XSEDE (or other customer-designated) identity and authentication services

Deployment view : Provided by a single, global XRAS server, accessible via Web browser.

Administrative (Admin) GUI

Description: The Administrative (Admin) GUI will be used primarily by the Allocations Administrator for each XRAS Client Organization. Certain service provider (SP) administrators may have limited interactions with this GUI to customize allocations-specific Resource preferences. The Admin GUI supports the management of review Panels, allocation Opportunities, Meeting preparation, Allocation awards, and limited reporting functions.

Target user: A user who has been designated the Allocations Administrator for one of the Allocations Processes hosted by the XRAS service.

Screenshot:

 

Uses: The XRAS-admin service, XSEDE (or other customer-designated) identity and authentication services;

Deployment view : Provided by a single, global XRAS server, accessible via Web browser.

Reporting service (XDMoD)

Description: Applicable XRAS data within the XSEDE context will be available via the XD Metrics on Demand (XDMoD) interface [ 4 ]. Non-XSEDE customers will have the option of leveraging XDMoD capabilities.

Target user: Allocations stakeholders, such as service providers, XSEDE (or other program) management, funding agency officials, and the public.

Example screenshot:

Uses: The XDMoD service (described elsewhere).

Deployment view : Provided by a single, global XDMoD server, accessible via Web browser.

XRAS Submit API

Description : This API provides the Allocations Process knowledge for the Submission GUI. When contacted by the Submission GUI, the XRAS Submit API first provides valid Request options and pending Request information for the Submitter. Based on a selection relayed by the Submission GUI, XRAS provides the required components (or saved partial submission) for the selected allocation Request. Upon being provided a completed Submission, the Submit API provides acknowledgment to the Submission GUI.

Provides :

            REST-based JSON interface for allocation Submission GUI.

Uses :

            Identity and authentication interfaces

            XRAS database

Deployment view : One global instance using a schema in the XDCDB for the XRAS database.

XRAS-review component ( integrated with the Review GUI )

Description : This component provides the allocations process knowledge for the review of allocation Requests. When a designated Reviewer authenticates to the Review GUI, the XRAS-review component within the Review GUI provides the Reviewer with options for viewing their review assignments associated with a given allocation Opportunity, downloading documents and viewing information about selected review assignments, and submitting their reviews and recommendations.

Provides :

            The Review GUI, customized for a specific Allocations Process.

Uses :

            Identity and authentication interfaces

            XRAS database

Deployment view : Separate deployment instances for XRAS Review GUI exist for each Allocations process (E.g. NCAR)

XRAS-admin component ( integrated with the Admin GUI )

Description : This component provides the allocations process knowledge and supporting tools for the administration of the review process, the award of allocation requests, and related management tasks. The XRAS Admin GUI interacts with the designated accounting, resource discovery, and reporting services to support the Allocations Process. Within the XSEDE context, these services are provided by the XDCDB, IIS, RDR and XDMoD.

Provides :

            The Admin GUI, customized for a specific Allocations Process.

Uses :

            Identity and authentication interfaces

            AMIE interface to the designated accounting service

            REST interface to the resource discovery service

            TBD interface to the designated reporting service

            XRAS database

Deployment view : Separate deployment instances exist for each Allocations process.

 

AMIE interface to accounting service

            AMIE is the system and protocol for exchanging accounting information among XSEDE systems [ 5 ].

              When XDCDB gets projects from XRAS via the XSEDE Accounting Service, it sends these projects to XSEDE resources using AMIE and when NCAR gets projects from XRAS via its own Accounting Service, it has its own mechanisms for setting up projects and accounts at NCAR.

REST interface to RDR

XSEDE currently operates RESTful interfaces to an Integrated Information Service (IIS) and Resource Description Repository (RDR) as its resource discovery service(s). RDR is the primary data source XRAS uses for resource data. Any changes made using the RDR user interface at RDR XSEDE are also immediately reflected in the XRAS resource tables. RDR updates both its own tables and the XRAS resource tables. In the RDR interface there is an “Allocations” section for each resource where the data used by XRAS can be viewed and modified.

Interface to XDMoD

XDMoD has a replication of the XDCDB running at their location.  This provides relevant information to the XDMoD reporting process. Client Organizations leveraging XRAS will be have the option of integrating their leveraging XSEDE’s XDMoD service, or exporting Allocation data for manual reporting.

Database - XDCDB

While the XRAS service layer embeds a significant amount of allocations process knowledge and business logic, the resource layer consists of a global XRAS service instance, to be deployed on a server or VM within the XSEDE infrastructure, and a central database instance. The XRAS service will leverage the existing XDCDB PostgreSQL server, while ensuring application-level independence of the XRAS database schema (i.e., no inter-schema joins into the other XDCDB schemas such as accounting “acct” or XUP “portal”).

The resource layer components of the other XSEDE supporting services not discussed here.

Important note on the XDCDB

Within the context of XRAS, it is important to clarify what is meant by “XDCDB.” Most references to the XD Central Database (XDCDB) encompass the XDCDB’s key functionality as the core of the XSEDE-wide accounting system. However, the “acct” schema is one of only 24 schemas within the XDCDB, and thus “XDCDB” can just as validly refer to XSEDE’s central database server and resource here at NCSA and fail-over at PSC.

As a database resource, XDCDB also provides resource layer support for the XUP, the RDR, and the AMIE protocol, among others. Furthermore, as a result of past development decisions, a number of logical databases within the XDCDB, implemented as a schema or collection schemas (including, but not limited to, the accounting and portal schemas) provide the de facto authoritative “Identity Service” for XSEDE.

Rules Engine

Description : The rules engine is an API with several purposes:

  1. Determine what actions a user can take in a Submit GUI

            Can the user submit a New or Renewal for an opportunity?

            Can the user submit additional actions on an existing request?

  1. Validate a user's action before allowing submission in Submit GUI

            Is valid data provided for all required fields?

            Is the PI valid for this request?

  1. Generate request numbers

            New requests from the Submit GUI

            When changing request number in the Admin GUI

  1. Send notifications

            Action Finalized - notification sent when action submitted

            Action Approved - notification sent when a submitted action is approved

            Action Rejected - notification sent when a submitted action is rejected

            Review submitted - notification sent when a review is submitted

            Reviewer Assigned - notification sent when a reviewer is assigned to review a proposal

            Reviewer Notifiy - notification sent to the admins when all the reviews for a specific request have been submitted. In that case the rules sends a notification so admins can start processing it (approve/reject)

The rules controlling all of these end points are in the rule_book_* tables of xras schema

Provides : REST-based JSON interface for XRAS Submit-API and Admin UI.

Uses :

  • Identity, accounting and authentication interfaces 
  • XRAS database

Deployment view : One global instance using a schema in the XDCDB for the XRAS database.

Deployment Notes

This section consolidates and summarizes the brief deployment notes provided throughout this document.

Deployed by XRAS SaaS customers

The following required and optional components will be developed and deployed by XRAS SaaS customers. The relevant XSEDE components are included here.

            Optional: Submission GUI(s). For XSEDE, the Submission GUI will be deployed within the XUP.

            Required: Identity and authentication services. For XSEDE, this includes various data sources within the XDCDB as well as the Kerberos service.

            Optional: Accounting service/AMIE. XSEDE has a well-defined XSEDE-wide accounting system. Non-XSEDE customers may elect for manual data export.

            Optional: Reporting service. For XSEDE, this is XDMoD. Non-XSEDE customers may elect for manual data export and reporting options.

Deployed as XSEDE centralized services

The following components will be deployed by XSEDE as centralized services, to be leverage by all XRAS SaaS customers.

            Review and Administrative GUIs

            XRAS-submit, XRAS-review, and XRAS-admin services

            XRAS database (within the XDCDB server)

            Resource discovery service—currently RDR/IIS within XSEDE

References

  1. Bachmann, F., Foster, I., Grimshaw, A., Lifka, D., Riedel, M. and Tuecke, S. XSEDE Architecture Level 1 and 2 Decomposition, XSEDE, 2012.
  2. Schneider, B., e-mail message to authors, February 5, 2013.
  3. HPCWorld Consortium. Handbook of HPC e-Science Infrastructure Allocation Reviewing, Selection and Management, 2011, http://www.hpcworld.eu/files/intranet/handbook.pdf.
  4. Thomas R. Furlani, Matthew D. Jones, Steven M. Gallo, Andrew E. Bruno, Charng-Da Lu, Amin Ghadersohi, Ryan J. Gentner, Abani K. Patra, Robert L. DeLeon, Gregor von Laszewski, Lizhe Wang, and Ann Zimmerman. 2011. Performance metrics and auditing framework for high performance computer systems. In Proceedings of the 2011 TeraGrid Conference: Extreme Digital Discovery (TG '11). ACM, New York, NY, USA. http://doi.acm.org/10.1145/2016741.2016759
  5. AMIE in the TeraGrid, last updated April 22, 2010, available from http://software.teragrid.org/amie/doc/AMIE-TeraGrid.pdf.

Appendix: XRAS software-as-a-service (SaaS)

Prior to August 2017, the XSEDE allocations process used the POPS system, which provided the tools and interfaces for the submission, review, and administration of allocations. In designing, documenting, and implementing the new XRAS software architecture, we have sought to follow best practices from the XSEDE systems engineering process.

Overview of and rationale for the XRAS architecture

The XRAS effort was launched to address three high-level objectives:

  1. To refine and revise a legacy system composed of more than 10 years’ worth of incremental improvements. The Allocations Process evolved over time and early implementation decisions are not always well-suited to support modern processes needed in XSEDE.
  2. To better integrate and position XRAS within the current XSEDE infrastructure. The POPS system was built before the TeraGrid program existed and, thus, higher-level architectural decisions that made sense at the time need to be revisited and aligned within the XSEDE architecture.
  3. To address an expanded set of needs by XRAS stakeholders. These needs range from low-level user interface improvements to much higher-level management needs.

The first two objectives have been addressed through XRAS design and implementation processes, and many decisions in this architecture document have been made with these objectives in mind. Within the third objective, however, lies the origin of the most significant architectural decision for XRAS and the driver for the whole of Level 3 architecture.

Architectural alternatives

Notably, a key requirement from NSF [ 2 ] was for an allocations system that would serve not only the needs of XSEDE, but also the Allocations Process needs of other NSF cyberinfrastructure investments, in such a way as to help streamline the effort required to support allocations across a broader set of resources.

This requirement from NSF dictated that XRAS could not merely be “just like now, but better.” Rather, the end product needed to be applicable beyond the current XSEDE resource environment. The XRAS team considered three possible deliverables:

            a “shrink-wrapped” software application that could be downloaded, installed, and customized by any non-XSEDE provider;

            A “Grants.gov”-like, unified allocation submission service that could accept allocation requests for both XSEDE and non-XSEDE resources, making potentially multiple review processes transparent to the submitter; or

            An allocations software-as-a-service (SaaS) that would allow both XSEDE and non-XSEDE allocations processes to be instantiated and managed separately, while leveraging a common review-process infrastructure.

The XRAS team chose to pursue the SaaS model because the first two options had major limitations. First, the “shrink-wrapped” software application would ultimately require a considerable degree of customization and administrative overhead by non-XSEDE providers because of the system’s reliance on a number of supporting services, including identity, discovery, authentication, accounting, and reporting. Aside from these significant obstacles to adoption, the “shrink-wrapped” model is not as well aligned with XSEDE’s cyberinfrastructure mission and goals.

The Grants.gov approach was not chosen primarily due to issues at several policy levels. This model would have required collective action and consensus not only at the provider level but also at the funding agency level (potentially intra- and inter-agency) to be widely adopted. The XRAS team felt that the political decisions were beyond their charter and schedule. Furthermore, the Grants.gov model could be conceived as an extension to the SaaS model so XRAS would be properly positioned, had stakeholder decisions later indicated that a Grants.gov-like approach was warranted.

SaaS advantages

The XRAS SaaS model has several advantages. It fits well into the overall XSEDE architecture and the Allocations Process is a good fit for the SaaS approach. While there may be low-level differences between Allocations Processes and policies, all Allocation Processes have steps in common, and this higher-level or abstract Allocations Process has been documented [ 3 ]. Because of these commonalities, the SaaS approach requires only a modest extension of a well-designed XSEDE-only model that clearly specifies interfaces between different components (services, API) of the XRAS architecture. The primary new functionality is to instantiate a new Allocations Process, as well as to provide administrative interfaces for customizing lower-level details — for example, customizing the data entry and documents required for Submissions.

Figure 1 outlines the steps and sub-steps that are common to most Allocation Processes. The color-coding represents the primary stakeholders or actors responsible. Yellow activities (1.1, 1.2, and 2.1) are the responsibility of funding agencies or program leadership and are pre-requisites to establishing an Allocations Process, while orange activities (6.1, 6.2, 6.3, and 6.5) are post-decision operational and execution activities. The remaining steps, in shades of green, fall within the scope of XRAS and have been addressed in the XRAS use cases defined for the system.

Figure 1. Review process elements for a “standard” allocations process [3].

The following summary of the steps in Figure 1 shows how the XRAS use cases align with the Allocation Process standard.

            Step 1. After the funding agency and program management define allocation policies, the Allocation Administrator (also called coordinator or manager) of a prospective Client Organization visits XRAS to establish an Allocation Process (1.4) and designate members of a Review Panel or Panels (1.3). Also in this step, the Resources to be allocated are discovered and integrated into the Allocation Process.

            Step 2. Outside of XRAS, the Allocation Administrator publicizes the Opportunity to submit allocation Requests. Next, interested users prepare and use the submission interface created by (or provided to) the Client Organization to provide the necessary materials (2.2). The Allocation Administrator uses the XRAS administrative interface to review Submissions for compliance with policies (2.3) and prepares the Requests for review or (in certain cases) bypasses review and makes Allocation awards.

            Step 3. The Allocation Administrator uses XRAS to assign Panel members to review Submissions (3.1), and the Reviewers use the XRAS review interface to view materials and enter reviews (3.2). In the “right to reply” step (3.3), Submitters have an opportunity to rebut Reviewer comments before award decisions are made. Outside of XRAS, the Allocation Administrator incorporates Reviewer recommendations and rebuttals and applies policy rules and stakeholder input to determine the final Allocation awards (3.4).

            Step 4. The Allocation Administrator then uses the XRAS admin interface to enter Allocation awards, subsequently notifying the awardees (4.1) and relevant resource providers (4.2).

            Step 5. XRAS will support an Appeal process (5.1, 5.2, and 5.3) that provides Submitters a mechanism via the submission interface to challenge unfavorable decisions.

            Step 6. After awards are made, Submitters use their Allocations to conduct the proposed work (6.1) and the work is tracked by the appropriate accounting system (6.2). Submission of a final report (6.3) will be supported by XRAS. The “call-back” procedure (6.4) refers to potentially reducing allocation awards after a period of non- or under-performance and making them available for re-allocation, typically a resource provider operational decision.