Contents

Introduction

Connections

Performance

Methods

Examples

Resources

Glossary

Getting Started Guide
Methods and Issues 
of Distributed Computing

1. Middleware Solutions
1. CORBA
2. Legion
3. Globus
2. Challenges
1. Security
2. Accessing Remote Resources
3. Resource Discovery
4. Comunication/Data Access
5. Fault Detection/Tolerance
3. Middleware Summary

The massive growth of the Internet has given rise to a wide variety of protocols, libraries, and applications allowing various resources to talk to one another. Developing distributed applications has become increasingly difficult when faced with the growing number of technologies from which to choose. This situation is further complicated by the growing number of technical issues that need to be addressed when attempting to develop an application.

This chapter attempts to provide basic information about some of the technical issues involved in distributed computing, and how they are addressed by some of the current paradigms. This discussion is limited to TCP/IP, CORBA, Legion, and Globus. Although TCP/IP does not provide a framework to address the various challenges raised in developing applications on the Grid, it is discussed briefly because it is used by most, if not all, of the other distributed computing frameworks. Whether you incorporate Legion, Globus,  CORBA, or some other middlware solution into your application, you are indirectly using TCP/IP at some point to communicate data. Understanding a little about how TCP/IP works can help you see how other distributed computing frameworks perform communication and understand some of the additional overhead resulting in decreased network performance.

1. The TCP/IP Protocols

The development of TCP/IP created the notion of layered protocols. IP (Internet Protocol), the most basic protocol of TCP/IP, provides the minimum requirements to route a message and move it from one machine to another over a network interface. For example, the ping command queries the network availability of remote hosts using IP (or raw sockets). Sockets are the end points of a connection and are created with and used by function calls. The Connections Section provides more information on network connections. Application developers typically don't program at the IP (Internet Protocol) level because IP is too low-level and does not meet general reliability requirements that most distributed applications have.

The User Datagram Protocol (UDP) provides a bit more functionality than using raw sockets and is a very basic transport protocol built on top of IP. Because UDP does not guarantee that messages will reach their final destinations, UDP is used for certain applications like streaming audio and video where packet loss recovery is not as important. The loss of a packet or two in a whole series of packets comprising a video image does not greatly change the end user's experience.

Like UDP, the Transmission Control Protocol (TCP) is also layered on top of IP but provides more reliabilty by ensuring that packets that are dropped get retransmitted and reach their destination in the order they were sent. TCP addresses the following transport issues:

• handshaking mechanisms to establish a connection between two machines
• capabilities for flow control (how much data can be sent at a time)
• congestion control (what to do when packets are dropped)
• polling for messages (how long to wait for an incoming packet)
• retransmission of lost or corrupted data

1. initialize the socket
2. bind the socket to a chosen port greater than 1024
3. listen for incoming connections
4. accept (or not) any incoming requests

1. initialize a socket
2. connect to the server

Unfortunately, sockets programming is still too low-level for developers not intimately familiar with some of the more esoteric and finer details of TCP/IP and network programming. For this reason, developers typically rely on distributed computing frameworks, or middleware, discussed below to more effectively enable application development. See W. Richard Stevens, Unix Network Programming, Vol. 1 for more information.

2. Middleware Solutions

• more robust communications libraries that take advantage of TCP/IP and provide the ability for clients and servers to communicate via remote method invocations or remote procedure calls. This allow clients and servers to specify remote tasks besides just sending data, which is an advantage.
• advanced architectures for addressing other technical challenges that face the design and development of a distributed application. The most common issues that arise are security, resource management, resource discovery, and fault tolerance in a heterogenous environment.

Keep in mind as you read that CORBA's Interface Definition Language makes it possible for both Legion and Globus to interoperate with it. Developers can therefore take advantage of the various services Legion and Globus offer while still working within the CORBA model.

2.1 CORBA

CORBA defines an Interface Definition Language (IDL) and the Application Programming Interfaces (API) that enable client/server interaction within a particular Object Request Broker (ORB). The ORB is the middleware that establishes the client-server relationships between objects. Client and server applications (objects) are specified using IDL and then compiled into the stub (or skeleton) interfaces (the actual code to describe the object's methods). The languages supported by the IDL compiler depend on the ORB implementation. CORBA clients and servers can be developed to dynamically (at run-time) invoke remote methods using the Dynamic Invocation Interface for the client and the Dynamic Skeleton Interface (DSI) for the server implementation.

2.2 Legion

Legion sits on top of a user's operating system, negotiating between that computer's resources and whichever resources or applications are required. Legion handles resources scheduling and security issues. A user-controlled naming system called context space is used, so that users can easily monitor and use objects. A Legion resource maintains Core Objects, such as the host and vault objects used to represent processors and persistent storage respectively. As part of Legion, many high-level Unix-like tools exist for working with Legion objects. Using Legion's Interface Description Language (IDL), implementation code can be written in any of the supported languages: Java, C, C++, Fortran, and the CORBA IDL.

2.3 Globus

Systems with Globus deployed are, like Legion, capable of secure authentication and job scheduling. However, the design approach taken by the Globus developers is entirely unlike Legion or CORBA. Rather than define an object-oriented architecture, Globus provides libraries, written in C, for addressing individual areas like security, remote job submission, file staging, and resource discovery. In the Globus framework, applications can take advantage of only the libraries they need without adopting an entire architecture to create a distributed application. Globus also offers a number of higher-level tools to allow users to stage files, execute programs remotely, and inquire about other resources.

3. Challenges

3.1 Security

Your decision to adopt a particular distributed computing framework may be highly dependent on the security requirements of the site resources you plan on using.

CORBA.   The Object Management Group defined four security specifications:

1. The CORBA Security Service Specification, which provides mechanisms to address delegation (passing credentials around), authentication, and access control using a variety of mechanisms and policies including X.509 certificates (authentication of entities involved in a distributed directory service), SSL (Secure Sockets Layer, a standard security protocol), and Kerberos. The security service is dependent on a vendor's implementation of the Object Request Broker (ORB).
2. The Secure Interoperability/secIOP Specification provides mechanisms for secure interoperability between different vendor's CORBA security services.
3. The CORBA ORB-SSL Integration Specification discusses the use of SSL within the Internet Inter-Orb Protocol-level (IIOP) of the ORB. (See Communication below for information on IIOP.)
4. The CORBA/Firewall Specification describes how a firewall can process IIOP, provide access control to client and server objects and protecting server objects.

• authentication is supported via Kerberos v5, and PKI-based (public key infrastructure) security using X.509 certificates is under development.
• an access control module known as MayI permits developers to define their own method for gaining access to a Legion object.
• allowing for three message-layer security modes based on the RSAREF 2.0 libraries: private (encrypted communication), protected (fast digested communication with unforgeable secrets), and no security.

• standard X.509 certificates that are commonly used in secure Internet transactions to provide a user with a global identity.
• authentication using the Generic Security Services API under Eric Young's implementation of SSL.
• single sign-on, which means a user only has to login once to use all Grid compute resources to which the user has authorization.
• Mapping of a global identity, a Grid user identification, to your local resource login is by means of a grid-mapfile that associates the identity with the resource login.

3.2 Accessing Remote Resources

CORBA.   Although CORBA provides no direct support for the execution and scheduling of programs on remote computers, this feature could be designed using remote method invocation. Remote method invocation (RMI) allows one CORBA object, or program, to call functions from another CORBA object on another computer. For example, an existing code or existing executable could be "wrapped" into a CORBA server object that could then be invoked by the client using RMI. A drawback in the high-performance computing world is that CORBA has no support for specific scheduling systems such as LSF, PBS, or NQE. It is up to the developer to design server-side code that can create and submit a batch script to the scheduling system if necessary.

Legion.   Legion allows you to run your existing code on Legion resources by following three steps:

1. compile your code with the Legion libraries and header files
2. register the code using something like legion_register_program (or legion_mpi_register for an MPI code)
3. run the program using something like legion_run (or legion_mpi_run for an MPI code)

A Collection is a Legion object (what actually runs in the computer) used to collect and maintain information about currently registered host and vault objects. Tools are provided for system administrators of hosts (computational resources) and vaults (storage resources) to set a scheduling policy and add resources to a Collection.

Globus.   When using Globus, globus-job-run is a useful command that is based on the Globus Resource and Allocation Manager (GRAM). As a part of GRAM, a gatekeeper runs on systems deploying Globus. A gatekeeper is responsible for deciding if you have been granted permission to use the resource. It does this by comparing your certificate name with a list of authorized users in a gridmap file. Once authorized, the gatekeeper starts up a job-manager that is responsible for executing and monitoring your job. The gatekeeper can run jobs in interactive mode, or it can submit the job to be run by the local scheduling system on a given queue. Common schedulers are PBS (Portable Batch System), LSF (Load Sharing Facility), LoadLeveler, or NQE (Network Queuing Environment). The Resource Specification Language (RSL) is another Globus component, used by GRAM to specify executables, input, output, and other information for running jobs. RSL strings are attribute value pairs like memory=X or directory=/u/ncsa/novotny.

3.3 Resource Discovery

CORBA.   CORBA provides various services that are useful for locating CORBA server objects. The Corba Naming Service provides a white-pages directory that can query CORBA objects by name. A CORBA server must register its name and object reference with the naming service, where you can query by name as you do with the white pages of your phonebook. The CORBA Trader Service is useful for querying objects by property, much the way you look services up in the yellow pages.

Legion.   Legion offers a common context space to display Legion objects. Higher-level tools exist for querying hosts (compute resource objects) and vaults (storage server objects) and other objects. However, Legion expects you to add the objects to your context space before it can be queried.

Globus.   In Globus, the Grid Information Service (GIS) helps Grid users locate resources and resource attributes from a dynamic database. Typically, when a resource registers itself under Globus, information such as memory, number of processors, and other capabilities are communicated to an LDAP (Lightweight Directory Access Protocol) server. LDAP has become a standard protocol for maintaining a catalog in a directory tree structure. Information about the Globus implementation of the LDAP server is available on the Globus website. Also provided is information about computer attributes like memory, operating system, cache, and local scheduling system.

3.4 Communication / Data Access

CORBA.   In CORBA, communication between client and server is done by the object request broker (ORB) allowing remote method invocations. The underlying protocol between ORBs is the General Inter-Orb Protocol (GIOP). GIOP specifies the message layouts and types. The common data representation (CDR) details the primitive data types, the composition of constructed types, and the format and layout of the various pseudo-objects. The CDR takes care of the data conversion between client and server machines. The GIOP has certain transport assumptions. For instance, it is based on a connection-oriented and reliable byte stream, which fits into the model of TCP/IP. The Internet Inter-Orb Protocol (IIOP) is a mapping of GIOP into a TCP/IP session. It is a commonly deployed standard between ORBs running on the Internet.

Legion.   Under Legion, communication between objects is achieved via non-blocking remote method invocations. Method invocations are subject to a scheduling and security policy. Legion uses a three-level naming system (see Figure 4).

Legion objects are maintained in a single, shared context space and are referred to by their context names. Context names are mapped to LOIDs (Legion Object Identifiers) that contain a public key from RSA Security Inc. Because LOIDs are location-independent, they are mapped to a LOA (Legion Object Address) for communication. A LOA is a physical address that contains the required information (IP address and port number) to permit other objects to communicate with the object. MPI processes and PVM tasks are represented as Legion objects, enabling heterogenous parallel computing.

Globus.   Similar to TCP/IP, Globus also provides layered communication protocols above TCP/IP. The simplest, globus_io, is a communications library that sits atop TCP/IP and provides a higher level sockets library capable of asynchronous, multithreaded communication. This communication is possible without you knowing the ugly details about asynchronous I/O and POSIX threads. The globus_io library not only allows you to set socket attributes, but uses GSI, the Grid Security Infrastructure to create secure communications channels and authorize connections. The globus_io library is used by GRAM for client/gatekeeper handshaking and by the Globus Access to Secondary Storage (GASS) component. GASS extends the globus_io library by including support for the HTTP/1.1 protocol and GSI, so that data can be transferred securely via https.

Nexus is a core communications library in Globus that provides a multimethod communications protocol capable of supporting TCP, UDP, reliable and unreliable multicast, shared memory and Posix threads. Nexus is intended to be used more by compiler writers and low-level Grid developers. One of the most exciting uses of Nexus is the MPICH-G library. MPICH-G allows parallel programs that have been written using MPI to run effectively over the wide-area network across heterogenous machines simply by compiling and linking your MPI code with the MPICH-G headers and libraries.

3.5 Fault Detection / Tolerance

CORBA.   In CORBA, fault tolerance must be devised by the application developer. One approach is object/data replication. In order to minimize machine failure, a server object is replicated across multiple servers. Databases, too, may be mirrored (replicated) at additional sites. A client request may be handled by a server that contains a pointer to a replicated server and database. If the replicated server dies, the query can be re-routed to another server, transparently to the client. As long as one replicated server is still up and running, the client will eventually get the result.

Legion.   Legion provides a checkpointing library for SPMD-style (Single Program Multiple Data) applications. In SPMD applications, each task is typically responsible for a region of data and exchanging data between neighboring tasks. Library routines can be used to create a "checkpoint" file that periodically saves the necessary data that is required when restarting the program in case of a failure. Special options used when running a program via legion_mpi_run (see Connections) can activate checkpointing and specify how long after a failure the application should be restarted (if at all).

Globus.   The Globus Heart Beat Monitor (HBM) provides a library intended to allow applications to register with a local monitor and provide on-going status to a data collector located on another machine. The data collector can then be queried for status of the running application. As of this writing, the Heart Beat Monitor (HBM) is not quite ready for prime-time, but may evolve into a Globus event model in the future.
 

4. Middleware Summary

Contents

Introduction

Connections

Performance

Methods

Examples

Resources

Glossary