Scalable Management and Data Mining using
Astrolabe
Robbert van Renesse, Kenneth Birman, Dan Dumitriu, Werner Vogels
✁
Department of Computer Science
Cornell University, Ithaca, NY 14853
rvr,ken,dumitriu,vogels@cs.cornell.edu
✂
Abstract. Astrolabe is a new kind of peer-to-peer system implementing a hierarchical distributed database abstraction. Although deigned for scalable management and data mining, the system can also support wide-area multicast and
offers powerful aggregation mechanisms that permit applications to build customized virtual databases by extracting and summarizing data located throughout
a large network. In contrast to other peer-to-peer systems, the Astrolabe hierarchy
is purely an abstraction constructed by running our protocol on the participating
hosts – there are no servers, and the system doesn’t superimpose a specialized
routing infrastructure or employ a DHT. This paper focuses on wide-area implementation challenges.
1 Introduction
To a growing degree, applications are expected to be self-configuring and self-managing,
and as the range of permissible configurations grows, this is becoming an enormously
complex undertaking. Indeed, the management subsystem for a distributed system is
often more complex than the application itself. Yet the technology options for building
management mechanisms have lagged. Current solutions, such as cluster management
systems, directory services, and event notification services, either do not scale adequately or are designed for relatively static settings.
In this paper, we describe a new information management service called Astrolabe. Astrolabe monitors the dynamically changing state of a collection of distributed
resources, reporting summaries of this information to its users. Like DNS, Astrolabe
organizes the resources into a hierarchy of domains, and associates attributes with each
domain. Unlike DNS, no servers are associated with domains, the attributes may be
highly dynamic, and updates propagate quickly; typically, in tens of seconds.
Astrolabe continuously computes summaries of the data in the system using on-thefly aggregation. The aggregation mechanism is controlled by SQL queries, and can be
understood as a type of data mining capability. For example, Astrolabe aggregation can
be used to monitor the status of a set of servers scattered within the network, to locate a
desired resource on the basis of its attribute values, or to compute a summary description
✄
This research was funded in part by DARPA/AFRL-IFGA grant F30602-99-1-0532, in part by
a grant under NASA’s REE program administered by JPL, in part by NSF-CISE grant 9703470,
and in part by the AFRL/Cornell Information Assurance Institute.
2
of loads on critical network components. As this information changes, Astrolabe will
automatically and rapidly recompute the associated aggregates and report the changes
to applications that have registered their interest.
The Astrolabe system looks to a user much like a database, although it is a virtual
database that does not reside on a centralized server. This database presentation extends
to several aspects. Most importantly, each domain can be viewed as a relational table
containing the attributes of its child domains, which in turn can be queried using SQL.
Also, using database integration mechanisms like ODBC and JDBC standard database
programming tools can access and manipulate the data available through Astrolabe.
The design of Astrolabe reflects four principles:
1. Scalability through hierarchy: Astrolabe achieves scalability through its domain
hierarchy. Given bounds on the size and amount of information in a domain, the
computational, storage and communication costs of Astrolabe are also bounded.
2. Flexibility through mobile code: A restricted form of mobile code, in the form of
SQL aggregation queries, allows users to customize Astrolabe on the fly.
3. Robustness through a randomized peer-to-peer protocol: Systems based on centralized servers are vulnerable to failures, attacks, and mismanagement. Astrolabe
agents run on each host, communicating through an epidemic protocol that is highly
tolerant of failures, easy to deploy, and efficient.
4. Security through certificates: Astrolabe uses digital signatures to identify and reject
potentially corrupted data and to control access to potentially costly operations.
This paper is organized as follows. Section 2 describes the Astrolabe system itself,
while its use is illustrated in Section 3. To avoid overlap with work published elsewhere,
we focus on wide-area communication challenges here. Accordingly, the basic lowlevel communication and addressing mechanisms used by Astrolabe are the subject of
Section 4. Section 5 describes Astrolabe’s self-configuration strategy. How Astrolabe
operates in the presence of firewalls is the topic of Section 6. In Section 7 we describe
various related work in the peer-to-peer area. Finally, Section 8 concludes.
2 Astrolabe
The goal of Astrolabe is to maintain a dynamically updated data structure reflecting
the status and other information contributed by hosts in a potentially large system. For
reasons of scalability, the hosts are organized into a domain hierarchy, in which each
participating machine is a leaf domains (see Figure 1). The leafs may be visualized
as tuples in a database: they correspond to a single host and have a set of attributes,
which can be base values (integers, floating point numbers, etc) or an XML object. In
contrast to these leaf attributes, which are directly updated by hosts, the attributes of an
internal domain (those corresponding to “higher levels” in the hierarchy) are generated
by aggregating (summarizing) attributes of its child domains.
The implementation of Astrolabe is entirely peer-to-peer: the system has no servers,
nor are any of its agents configured to be responsible for any particular domain. Instead,
each host runs an agent process that communicates with other agents through an epidemic protocol or gossip [DGH 87]. The data structures and protocols are designed
such that the service scales well:
☎
3
Fig. 1. An example of a three-level Astrolabe tree. The top-level root domain has three child
domains. Each domain, including the leaf domains (the hosts), has an attribute list. Each host
runs a Astrolabe agent.
– The memory used by each host grows logarithmically with the membership size;
– The size of gossip messages grows logarithmically with the size of the membership;
– If configured well (more on this in Section 5), the gossip load on network links
grows logarithmically with the size of the membership, and is independent of the
update rate;
– The latency grows logarithmically with the size of the membership. Latency is defined as the time it takes to take a snapshot of the entire membership and aggregate
all its attributes;
– Astrolabe is tolerant of severe message loss and host failures, and deals with network partitioning and recovery.
In practice, even if the gossip load is low (Astrolabe agents are typically configured
to gossip gossip only once every few seconds), updates propagate very quickly, and is
typically within a minute even for very large deployments [vRB02].
In the description of the implementation of Astrolabe below, we omit all details
except those that are necessary in order to understand the function of Astrolabe, and the
issues that relate to peer-to-peer communication in the Internet. A much more detailed
description and evaluation of Astrolabe appears in [vRB02].
As there is exactly one Astrolabe agent for each leaf domain, we name Astrolabe
agents by their corresponding domain names. Each Astrolabe agent maintains, for each
domain that it is a member of, a relational table called the domain table. For example,
the agent “/nl/amsterdam/vu” has domain tables for “/”, “/nl”, and “/nl/amsterdam” (see
Figure 2). A domain table of a domain contains a row for each of its child domains, and
a column for each attribute name. One of the rows in the table is the agent’s own row,
which corresponds to that child domain that the agent is a member of as well. Using
a SQL aggregation function, a domain table may be aggregated by computing some
form of summary of the contents to form a single row. The rows from the children of
a domain are concatenated to form that domain’s table in the agent, and this repeats to
the root of the Astrolabe hierarchy.
Since multiple agents may be in the same domain, the corresponding domain table
is replicated on all these agents. For example, both the agents “/nl/amsterdam/vu” and
“/nl/utrecht/uu” maintain the “/” and “/nl” tables. A replicated table is kept approxi-
4
/
/nl
own
fr
nl
uk
/nl/amsterdam
own
amsterdam
maastricht
rotterdam
utrecht
own
uva
vu
✆
Fig. 2. A simplified representation of the data structure maintained by the agent corresponding to
/nl/amsterdam/vu.
mately consistent using an epidemic protocol. Each agent in a domain calculates, using
an aggregation function, a small set of representative agents for its domain. Typically,
Astrolabe is configured to use up to three representative agents for each domain. The
representative agents of the child domains of a parent domain run the epidemic protocol
for the parent’s domain table. On a regular basis, say once a second, each agent X that
is a representative for some child domain chooses another child domain at random, and
then a representative agent Y within the chosen child domain, also at random. X sends
the parent’s table to Y. Y merges this table with its own table, and sends the result back
to X, so that X and Y now agree on the contents of their tables.
The rule by which such tables are merged is central to the behavior of the system as
a whole. The basic idea is as follows. Y adopts into the merged table rows from X for
child domains that were not in Y’s original table, as well as rows for child domains that
are more current than in Y’s original table. To determine currency, agents timestamp
rows each time they are updated by writing (in case of leaf domains) or by generation
(in case of internal domains). Unfortunately, this requires all clocks to be synchronized
which is, at least in today’s Internet, far from being the case.
To solve this problem, each row is tagged with the generator: the domain name of
the agent that wrote or generated the row (in addition to the timestamp). Agents also
maintain, for each row in each table, the set of generators from which they received
updates for that row, along with the timestamp on the last received update. The merge
operation is now executed as follows. When receiving a row in a gossip message, the
agent adopts it, as is, if it is created by a previously unknown generator. If it is a known
generator, the row is adopted if and only if the row’s timestamp is more recent than the
last received timestamp from that generator. This way, only timestamps from the same
agent are compared with one another, and thus no clock synchronization is necessary.
When no update has been received from a particular generator for some time period
, that generator is considered faulty, and eventually forgotten (after time
for reasons that go beyond the scope of this paper, but which are described in [vRMH98]).
should be chosen so that the probability of any old gossips from this generator still going around is very low [vRB02, vRMH98]. Since gossips disseminate in time
,
this typically is not very long, and can be determined by techniques of epidemic analysis or simulation. If there are no more known generators for a domain, the domain itself
is removed from the agent’s domain table.
✝✆
✆
✞✠✟☛✡✌☞✎✍✑✏✓✒
5
3 Using Astrolabe
Applications invoke Astrolabe interfaces through calls to a library (see Table 1). The
library allows applications to peruse all the information in the Astrolabe tree, setting up
new connections as necessary. The creation and termination of connections is transparent to application processes, so the programmer can think of Astrolabe as a ubiquitous
service, somewhat analogous to the DNS.
Table 1. Application Programmer Interface.
Method
find contacts(time, scope)
set contacts(addresses)
get attributes(domain, event queue)
get children(domain, event queue)
set attribute(domain, attribute, value)
Description
search for Astrolabe agents in the given time and scope
specify addresses of initial agents to connect to
report updates to attributes of domain
report updates to domain membership
update the given attribute
In addition to its native interface, the library has an SQL interface that allows applications to view each node in the domain tree as a relational database table, with a row
for each child domain and a column for each attribute. The programmer can then simply
invoke SQL operators to retrieve data from the tables. Using selection, join, and union
operations, the programmer can create new views of the Astrolabe data that are independent of the physical hierarchy of the Astrolabe tree. An ODBC driver is available
for this SQL interface, so that many existing database tools can use Astrolabe directly,
and many databases can import data from Astrolabe.
New aggregation functions can be installed dynamically, and their dissemination
piggybacks on the gossip protocol. This way an Astrolabe hierarchy can be customized
for the applications that use it. The code of these functions is embedded in so-called aggregation function certificates (AFCs), which are signed certificates installed as domain
attributes.
We also use AFCs for purposes other than aggregation. An Information Request
AFC specifies what information the application wants to retrieve at each participating
host, and how to aggregate this information in the domain hierarchy. (both are specified
using SQL queries). A Configuration AFC specifies run-time parameters that applications may use for dynamic on-line configuration.
Example: Peer-to-peer Multicast
In [vRB02] we present a number of possible uses for Astrolabe, such as for locating
”nearby” resources by using ODBC to query the local domain, monitoring the dynamically evolving state of a subset of hosts in a large network using an aggregation function,
or tracking down desired resources in very large settings. To avoid repeating that material here, we now present a different example of how Mariner might be used. Many
6
distributed games and other applications require a form of multicast that scales well, is
fairly reliable, and does not put a TCP-unfriendly load on the Internet. In the face of
slow participants, the multicast protocol’s flow control mechanism should not force the
entire system to grind to a halt. This section describes such a multicast facility. It uses
Astrolabe to track the set of multicast recipients, but then but sets up a separate tree of
TCP connections for actually transporting multicast messages.
Each multicast group has a name, say “game”. A participant expresses its interest in
receiving messages for this group by installing its TCP/IP address (or addresses) in the
attribute “game” of its leaf domain’s MIB. This attribute is aggregated using the query
SELECT FIRST(3, game) AS game
That is, each domain selects three of its participants’ TCP/IP addresses. (FIRST is
an often-used Astrolabe extension to SQL.)
Participants exchange messages of the form (domain, data). A participant that wants
to initiate a multicast lists the child domains of the root domain, and, for each child that
has a non-empty “game” attribute, sends the message (child-domain, data) to a selected
participant for that child domain (more on this selection later). Each time a participant
receives a message (domain, data), it finds the child domains of the given domain that
have non-empty “game” attributes and recursively continues the dissemination process.
The TCP connections that are created are cached. This effectively constructs a tree
of TCP connections that spans the set of participants. This tree is automatically updated
as Astrolabe reports domain membership updates.
To make sure that the dissemination latency does not suffer from slow participants
in the tree, some measures must be taken. First, a participant could post (in Astrolabe)
the rate of messages that it is able to process. The aggregation query can then be updated
as follows to select only the highest performing participants for “internal routers.”
SELECT FIRST(3, game) AS game ORDER BY rate
Senders can also monitor their outgoing TCP pipes. If one fills up, they may want
to try another participant for the corresponding domain. It is even possible to use more
than one participant to construct a “fat tree” for dissemination, but then care should
be taken to reconstruct the order of messages. These mechanisms together effectively
route messages around slow parts of the Internet, much like Resilient Overlay Networks [ABKM01] accomplishes for point-to-point traffic. Notice that this makes our
multicast ”TCP-friendly”, in the sense that if a router becomes overloaded and starts
dropping messages, the multicast protocol will reduce the load imposed on that router.
This property is rarely seen in Internet multicast protocols.
Our solution can also be used to implement the Publish/Subscribe paradigm. In
these systems [OPSS93], receivers subscribe to topics of interest, and publishers post
messages to topics. The multicast protocol described above can easily implement Publish/Subscribe, as well as a generalized concept that we call selective multicast or selective Publish/Subscribe.
The idea is to tag messages with a SQL condition, chosen by the publishers. For
example, a publisher that wants to send an update to all hosts that have a version of some
object that is less than 3.1, this condition could be “MIN(version) < 3.1”. The
7
participants in the multicast protocol above use this condition to decide to which other
participants should receive the message. In the example used above, when receiving a
message (domain, data, condition), a participant executes the following SQL query to
find out which participants to forward the message to:
SELECT game
FROM domain
WHERE condition
In this idea, the publisher specifies the set of receivers. Basic Publish/Subscribe can
then be expressed as the publisher specifying that a message should be delivered to all
subscribers to a particular topic.
The simplest way to accomplish this type of routing is to create a new attribute
by the name of the topic. Subscribers set this attribute to 1, and the attribute is aggre”. However, such an
gated by taking the sum. The condition is then “attribute
approach would scale poorly if a system has large number of possible topics, since it
requires one bit each.
A solution that scales much better is to use a Bloom filter [Blo70].1 This solution
uses a single attribute that contains a fixed-size bit map. The attribute is aggregated using bitwise OR. Topic names are hashed to a bit in this bit map. The condition tagged to
the message is “BITSET(HASH(topic))”. In the case of hash collisions, a message
may reach some non-subscribing destinations, but would be filtered and ignored at the
last hop.
✔✖✕
4 Low-level Communication
The preceding example showed how a multicast protocol could be layered over Astrolabe, running on TCP channels. However, for its own communication, Astrolabe uses
UDP/IP, HTTP (on top of TCP/IP or SSL), or both. To support HTTP, Astrolabe agents
act both as HTTP servers and clients. To enable communication through firewalls, Astrolabe makes use of Application Level Gateways, but for scalability concerns this is
only done as a last resort (see Section 6). This section discusses some of the challenges that arose in running Astrolabe in wide-area settings. Before we discuss the
actual communication in more detail, we will first describe the concept of realms, and
how addressing is done in Astrolabe.
A realm is a set of hosts and a communication protocol. For example, the tuple
(“Cornell Computer Science Department”, UDP) forms a realm, as does (“Core Internet”, HTTP). (“Core Internet” is the set of those hosts on the main Internet that do not
reside behind firewalls.) Each realm has a unique identifier of the form name:protocol,
for example “cornellcs:udp” and “internet:http”. The hosts in a realm form an equivalence class, in that they can all be accessed using the realm’s protocol in the same way.
A host can be in more than one realm, and can have more than one address in the same
1
✗
This idea is also used in various other distributed systems, including the directory service of
the Ninja system [GWVB 01].
8
realm. No two hosts in the same realm can have the same address, but the same address
may be used in different realms.
UDP addresses are of the form “IP-address:port” (e.g., “10.0.0.4:6422”) or “DNSname:port” (e.g., “rome.cs.cornell.edu:6422”). HTTP addresses are of the form “agentname@TCP-address”, where “agent-name” is the Astrolabe domain name of the agent,
and “TCP-address,” as in UDP addresses, consists of a port and either an IP address or
a DNS name. For example, “/usa/ny/ithaca/cornell/cs/rome@10.0.0.4:2246”.
We define an extended address to be the triple (realm identifier, address, preference).
For example, (“cornellcs:udp”, 10.0.0.4:6422, 5). A host has a set of these addresses,
and can indicate its preference for certain addresses using the preference field. We call
the set of extended addresses of a host the contact for that host. The contact is dynamic as addresses may appear and disappear over time as the administrator of the host
connects to, or disconnects from, ISPs and VPNs.
Unlike agent’s contacts, agent’s names are constant. In order to simplify configuration, we observed that realms often coincide with Astrolabe domains, and thus named
realms using their corresponding domain name. Thus, rather than “cornellcs:udp”, we
would use “/usa/ny/Ithaca/cornell/cs:udp”. The core Internet coincides with the root
domain, and is thus called “/:http”.
Each domain in Astrolabe has an attribute called contacts, which contains the contacts of those agents in the domain that have been elected as representatives. (This uses
the FIRST function that was described in Section 3.) The contacts attribute of a leaf
domain contains the singleton set with the contact of the agent of that leaf domain.
We will now briefly revisit Astrolabe’s gossip protocol to show how this works in
practice. When an agent wants to gossip the table of some domain, it has to come up
with an address. First, the agent picks one of the table’s rows at random and retrieves
the contacts attribute from that row. The agent then picks one of the contacts at random.
The resulting contact is a set of extended addresses. The agent removes the addresses
of realms that it cannot reach (more on this below). If there is more than one remaining
address, the agent has to make one more choice.
In order to make intelligent choices, each Astrolabe agent maintains statistics about
addresses. This is simple to do, as each gossip message is followed by a response.
Currently, an agent maintains for each extended address the following three values:
1. outstanding: the number of gossips sent since the last response was received;
2. last sent: time of last send of a gossip;
3. last received: time of last reception of a response.
If there is more than one extended address to choose from, the agent scores each
address:
1. If there is no outstanding gossip, the score is the preference;
2. If it has been more than a minute since the last gossip was sent, the score is the
preference;
3. If there is just one outstanding gossip, the score is the preference minus one;
4. In all other cases, the score is zero.
This results in the following behavior. In the normal case, when gossips are followed
by responses, the address of the highest preference is always used. If a single response
9
got lost, the score becomes only slightly smaller. The intention is that if there is more
than one address of the same preference, the ones that are only somewhat flaky become
less preferential. If there are more losses, the score becomes such that the address is
only used as a last resort. Once a minute, the score is, for a single send operation, reset
to the original preference. This allows addresses to be occasionally re-tested.
5 Configuration
In order for Astrolabe to scale well, the domain hierarchy has to be set up with care.
Each domain in Astrolabe runs an instance of the gossip protocol among the representatives of its child domains. The first concern to think about is the size of domains, that is,
the number of child domains in a domain. If very large, the size of gossip messages, as
well as the generated gossip load, will be large as well (they both grow linearly with the
domain size). If chosen to be very small, the hierarchy becomes very deep, and latency
will suffer. In practice, we find that a size of 25-100 child domains in a domain works
well. Smaller sizes are possible too, at the cost of some additional latency, but larger
sizes make the load unacceptably large.
Locality is a second important consideration. Domains should be constructed (if
possible) so that the number of network hops between its child domains’ representatives
is minimized, and so that independent domains (one domain is not an ancestor of the
other) do not share any network links. If the Internet were a tree topology, the Astrolabe
hierarchy should be preferably identical to this tree. In reality the edges of the Internet
often resemble a tree topology, but the internal Internet is a complicated mesh of links
that defies any resemblance to a tree.
In practice, this means that there is considerable freedom in designing the higher
levels of the Astrolabe hierarchy, within the limits of the branching factor, but the lower
levels should be mapped closely to the topology of the Internet edge. If we ignore the
branching factor, this is not much different from the DNS hierarchy design. In DNS, too,
the low levels often correspond closely to the network topology, while the high levels
of the hierarchy have little correspondence to the actual network topology. Thus the
main difference between DNS and Astrolabe configuration is the constrained branching
factor of the Astrolabe hierarchy.
Astrolabe supports two forms of configuration: manual and automatic. The manual
configuration supports various notions of security, including an integrated PKI infrastructure for the Astrolabe service. The automatic configuration is not secure. In order
to foil all but the simplest forms of compromise, the communication is scrambled and
signed using secret keys. Below, we will focus on the insecure automatic configuration.
More on Astrolabe security can be found in [vRB02].
In an insecure version of Astrolabe, all an agent needs to know is
– Its domain name;
– The set of realms that it can send messages to;
– How to find peer agents to gossip with.
In the remainder of this section, we will look at the automatic configuration of
domain names and realms.
10
Currently, we generate the Astrolabe domain name of an agent from the DNS domain name of the host, and the process identifier of the agent. We will first explain how
this is done, and then provide the rational for this design. Say that the DNS domain
name is ✘✚✙✜✛✌✘✣✢✤✛✌✛✥✛✌✛✌✛✌✘✧✦ , and the process id of the agent is ★ . We use a one-way hash
function on ✘ ✢ ✛✌✛✌✛✥✘ ✦ (all but the first component of the domain name) to construct three
✩ ✢ , ✩✫✪ , and ✩✫✬ . Finally, the Astrolabe domain name is constructed to be
6-bit
integers,
✭
✭
✭
✭
✭
✭
✭
✭
✘ ✦
✩ ✢
✩✧✪
✩✫✬
✘ ✦✯✮✑✢
✛✌✛✌✛ ✘ ✙
★ .
For example, say an agent runs as process 4365 on host “rome.cs.cornell.edu”. By
hashing “cs.cornell.edu” onto three 6-bit integers, we have effectively split the “.edu”
domain up into ✝ ✢✱✰ pieces, as the “.edu” domain itself is much too large for a single Astrolabe domain. Using this construction, the Astrolabe “/edu” domain itself has at most
64 child domains. Say the three generated integers in our example are 25, 43, and 4 respectively. Then the domain name of the agent is “/edu/25/43/4/cornell/cs/rome/4365”.
(The three generated domains can be hidden from view if so desired.)
The hope is that each of the domains following “/edu/25/43/4” are of relatively
limited size that can be supported by the Astrolabe protocol, and that these domains
reflect the network topology to a close enough approximation. If in the future this turns
out to be insufficient, we can update the downloadable executable to use more levels, or
perhaps come up with an adaptive scheme. The addition of the process identifier makes
it possible to run multiple agents on the same host.
Next we have to determine the set of realms that the agent can reach. We assume
that any agent can communicate to the “/:http” realm, that is, any agent can use HTTP
to reach another agent on the core Internet. (Agents may use WPAD to determine the
existence
and✭ location
of
an✭ HTTP proxy automatically.) Furthermore, we assume that
✭
✭
✭
✭
✭
✘ ✦
✩ ✢
✩✧✪
✩✫✬
✘ ✦✯✮✑✢
✛✌✛✌✛ ✘ ✢ :udp (“/edu/25/43/4/cornell/cs:udp” in our example) is a
realm, and that all agents within this realm can communicate with one another.
Currently, these are all the assumptions we make about realms. In practice this is
sometimes conservative, and in that case agents that can communicate directly using
UDP will use an Application Level Gateway (ALG) instead. In the next section, we
describe how ALGs are configured and used.
6 Communication through an ALG
An Application Level Gateway (ALG) may be the only possibility for two agents to
communicate (see Figure 3). Significant care should be taken in deploying and configuring ALGs. The number of ALGs is likely to be small compared to the number of
agents using them, and thus they should be used judiciously in order not to overload
them or the network links that connect them. Also, care should be taken that the peerto-peer network remains tolerant of failures, and does not get partitioned when a single
ALG server crashes or otherwise becomes unavailable, and that network security is not
compromised. Finally, in order for the system to scale, it should be possible to add new
ALG servers dynamically to the system as the number of Astrolabe agents grows. These
new servers should automatically be discovered and used by the existing agents as well
as the new ones.
11
(1)
Relay Server (RS)
(2)
(3)
(4)
✲
✲
✳
✳
✴✵
Core Internet
Firewall
Firewall
Fig. 3. Application Level Gateway. (1) Receiver sends a RECEIVE request using an HTTP POST
request; (2) Sender sends the message using a SEND request using an HTTP POST request; (3)
ALG forwards the message to the receiver using an HTTP 200 response; (4) ALG sends an empty
HTTP 200 response back to the sender.
Ideally, an ALG is located on the network path between a sender and a receiver,
so that the number of hops that a message has to travel is not severely affected by the
presence of an ALG. Since many senders may send messages to the same receiver,
it follows that the ALG should be located as close to the receiver as possible. Thus,
ideally, each firewall or NAT box has a companion ALG that serves receivers behind
the firewall. In practice, we suspect that far fewer ALGs will be deployed, but it is still
important for receivers to connect to the nearest-by ALG or ALGs.
Each receiver that wishes to receive messages through an ALG has to use HTTP
requests to the ALG. In practice, this happens over a persistent TCP connection. In
order to reduce the number of such connections to an ALG, not every host behind a
firewall has to connect to the ALG. In Astrolabe, only representatives for the realm
corresponding to the firewalled site gossip beyond the firewall boundaries, and only
these agents (typically, two or three), need to receive through an ALG. The other agents
learn indirectly of updates outside the realm through gossip with the representatives
(see Figure 4).
Site 1
Source
Core Internet
Site 2
ALG Server 1
ALG Server 2
Destination
Fig. 4. The many ways gossip can travel from a source host in Site 1 to a destination host in Site 2.
Each site has four hosts, two of which are representatives, behind a firewall. The representatives
of Site 2 connect to two different ALG servers to receive messages from outside their firewall.
12
In order for agents to locate ALGs, the ALGs themselves are situated in the Astrolabe hierarchy itself. Each ALG has a companion Astrolabe agent with a configured
domain name. The relays attribute of the corresponding leaf domain is set to the singleton set containing the TCP/IP address of the ALG. This attribute is aggregated into
internal domains in the same way as the contacts attribute (i.e., using the FIRST aggregation operator).
An agent determines whether it is a representative for a firewalled site by monitoring the contacts attribute of the corresponding realm domain and noticing whether
its contact is in there. When this becomes the case, the agent finds ALGs by traveling
up the Astrolabe hierarchy starting in its realm and finding the relays attributes, stopping when it has located ALGs or when it reaches the root domain. To ensure fault
tolerance, is typically chosen to be a small integer such as 2 (as in Figure 4). If no
ALGs are found, agents resort to using a set of built-in addresses of ALG servers that
we deployed for this purpose.
For each ALG in the set, the agent generates a new extended address of the form
(“domain-name@ALG”, “/:http”, preference), and adds this address to its contact set.
The preference is chosen to be relatively low compared to its other addresses, so as to
discourage its use. Finally, the agent sends an HTTP request to the ALG to receive the
first message on this address.
✶
✶
7 Related Work
The most popular peer-to-peer systems such as Chord [SMKK95], Pastry [RD01], and
Tapestry [ZKJ01] implement distributed hash tables (DHT), then use these tables to
locate desired objects. Just as a conventional hash table maps a key to a value, a DHT
maps a key to a location in the network. The host associated with that location stores a
copy of the value associated with the key. The hosts that implement the DHT maintain
size that allow messages to be routed in
steps to
routing tables of
one of these locations.
Astrolabe also implements a distributed data structure, although it is neither a DHT
nor a distributed file system. Instead, Astrolabe most closely resembles a spreadsheet,
particularly because updates of an attribute causes other attributes to change. However,
the Astrolabe interface is more like that of a database. Athough Astrolabe uses a peer-topeer protocol with scalability properties similar to those of DHTs, it differs in important
ways from the best known DHT solutions.
In particular, Astrolabe reflects the physical organization of the hosts in its domain
hierarchy, while DHT implementations hide the hosts and present a uniform key-value
mapping. This difference is also apparent in the underlying protocols. Astrolabe exploits the domain organization so that most message exchanges are between nearby
hosts, while the DHT implementations require special mechanisms to avoid doing large
numbers of long distance exchanges. Pastry and Tapestry come closest to exploiting
proximity in their message routing, but the protocols that maintain the routing tables
themselves still require mostly long distance messages. Also, these protocols treat each
network link as equal, and are expected to suffer from significant problems with slow
modem links.
✞✠✟✷✡✌☞✸✍✺✹✻✒
✞✠✟✷✡✥☞✎✍✼✹✻✒
13
The designers of Tapestry are investigating a two-level architecture, called Brocade
[JK02], that exploits the network topology. Basically, each site (e.g., an organization or
campus) deploys a number of so-called supernodes. The supernodes are organized in
an ordinary Tapestry network. Each site then deploys another Tapestry network locally,
including its supernodes. Messages are now routed in three superhops: first to a local
supernode, then to the remote supernode, and lastly to its final destination. Simulation
shows dramatic improvements in performance.
Astrolabe is perhaps most similar to a multi-level Brocade architecture. Astrolabe’s
aggregation facilities are used to elect, for each domain, the most appropriate supernodes. Election may be done based on quality of connectivity, resource capacity, host
security, etc., and these policies may in fact be changed on the fly. Another difference
between Astrolabe and other peer-to-peer protocols is therefore that Astrolabe exploits
the available heterogeneity in the network, rather than hiding it.
Much of Astrolabe is concerned with the details of peer-to-peer communication in
the actual Internet, an environment rife with Network Address Translation and other inconveniences. Well-known technologies in this field are Groove Networks (groove.net)
and JXTA (jxta.org). Groove Networks provide a peer-to-peer communications technology for distributed collaboration. Although in theory peers in Groove can communicate
directly with one another (assuming they are not separated by firewalls or NAT), they
heavily rely on their proprietary ALG, called the Groove Relay Server [Ora01]. Unless
peers are explicitly configured to communicate directly with one another, they will use
the Relay Server for communication. They also use the Relay Server for other functions.
This includes message queuing for off-line peers and resource discovery. This makes
most Groove applications heavily dependent on the Relay Server, and direct peer-topeer interactions are rarely used.
JXTA [Gon01] is an open platform for peer-to-peer interactions in the network, intended for pervasive use from servers to workstations to PDAs and cell phones. JXTA
offers a variety of services such as Peer Discovery and Peer Membership. In order to
allow peer discovery and peer-to-peer communication, the notion of an ALG (RendezVous Server in JXTA terminology) has been proposed, but this is still an ongoing research effort.
8 Conclusions
By combining peer-to-peer and gossip protocols and using the resulting mechanism to
implement a scalable database abstraction, Astrolabe plugs a gap in the existing Internet
infrastructure. Today, far too many applications are forced to operate in the dark: they
lack a good way to sense the status of component systems and of the network, and
yet need to adapt their behavior on the basis of such status. More broadly, there is
an important need for better scalable computing tools addressing the communications
and configuration requirements of large-scale applications. Astrolabe offers such tools,
packaged in an easily used database abstraction. Because Astrolabe itself is stable under
stress and extremely scalable, it promotes the development of new kinds of applications
sharing these properties.
14
Acknowledgements
We would like to thank the following people for various contributions to the Astrolabe
design and this paper: Tim Clark, Al Demers, Terrin Eager, Johannes Gehrke, Barry
Gleeson, Indranil Gupta, Kate Jenkins, Anh Look, Yaron Minsky, Andrew Myers, Venu
Ramasubramanian, Richard Shoenhair, Emin Gun Sirer, Lidong Zhou, and the anonymous reviewers.
References
[ABKM01]
[Blo70]
[DGH 87]
✗
[Gon01]
[GWVB 01]
✗
[JK02]
[OPSS93]
[Ora01]
[RD01]
[SMKK95]
[vRB02]
[vRMH98]
[ZKJ01]
D.G. Andersen, H Balakrishnan, M.F. Kaashoek, and R. Morris. Resilient overlay networks. In Proc. of the Eighteenth ACM Symp. on Operating Systems
Principles, pages 131–145, Banff, Canada, October 2001.
B. Bloom. Space/time tradeoffs in hash coding with allowable errors. CACM,
13(7):422–426, July 1970.
A. Demers, D. Greene, C. Hauser, W. Irish, J. Larson, S. Shenker, H. Sturgis,
D. Swinehart, and D. Terry. Epidemic algorithms for replicated database maintenance. In Proc. of the Sixth ACM Symp. on Principles of Distributed Computing,
pages 1–12, Vancouver, BC, August 1987.
L. Gong. JXTA: A network programming environment. IEEE Internet Computing, 5(3):88–95, May/June 2001.
S.D. Gribble, M. Welsh, R. Von Behren, E.A. Brewer, D. Culler, N. Borisov,
S. Czerwinski, R. Gummadi, J. Hill, A. Joseph, R.H. Katz, Z.M. Mao, S. Ross,
and B. Zhao. The Ninja architecture for robust internet-scale systems and services. To appear in a Special Issue of Computer Networks on Pervasive Computing, 2001.
A.D. Joseph and J.D. Kubiatowicz. Brocade: Landmark routing on overlay networks. In Proc. of the First International Workshop on Peer-to-Peer Systems,
Cambridge, MA, March 2002.
B. M. Oki, M. Pfluegl, A. Siegel, and D. Skeen. The Information Bus—an architecture for extensible distributed systems. In Proc. of the Fourteenth ACM Symp.
on Operating Systems Principles, pages 58–68, Asheville, NC, December 1993.
A. Oram, editor. Peer-To-Peer: Harnessing the Power of Disruptive Technologies. O’Reilly, 2001.
A. Rowstron and P. Druschel. Pastry: Scalable, distributed object location and
routing for large-scale peer-to-peer systems. In Proc. of the Middleware 2001,
November 2001.
I. Stoica, R. Morris, D. Karger, and M.F. Kaashoek. Chord: A scalable peer-topeer lookup service for Internet applications. In Proc. of the ’95 Symp. on Communications Architectures & Protocols, Cambridge, MA, August 1995. ACM
SIGCOMM.
R. van Renesse and K.P. Birman. Astrolabe: A robust and scalable technology
for distributed system monitoring, management, and data mining. ACM Transactions on Computer Systems, 2002. Submitted for review.
R. van Renesse, Y. Minsky, and M. Hayden. A gossip-style failure detection
service. In Proc. of Middleware’98, pages 55–70. IFIP, September 1998.
B.Y. Zhao, J. Kubiatowicz, and A. Joseph. Tapestry: An infrastructure for faulttolerant wide-area location and routing. Technical Report UCB/CSD-01-1141,
University of California, Berkeley, Computer Science Department, 2001.