Towards Reproducing Inter-Domain AS Paths

Transkript

1 Lehrstuhl für Netzwerkarchitekturen Faktultät für Informatik Technische Universität München Diplomarbeit Towards Reproducing Inter-Domain AS Paths Wolfgang Mühlbauer 15th November 2005 Aufgabenstellerin: Prof. Anja Feldmann, Ph.D. Betreuer: Olaf Maennel

2

3 Erklärung Ich versichere, dass ich diese Diplomarbeit selbständig verfasst und nur die angegebenen Quellen und Hilfsmittel verwendet habe. Wolfgang Mühlbauer München, 15. November 2005 iii

4

5 Kurzfassung Das Internet setzt sich aus ungefähr autonomen Systemen (AS) zusammen. Diese werden in der Regel von verschiedenen Instanzen betrieben und stehen miteinander im Wettbewerb. Ungeachtet dieser Tatsache, muss die weltweite Erreichbarkeit gewährleistet sein, unter gleichzeitiger Beibehaltung der Autonomie der einzelnen Teilnetze: Jedes AS kann Verkehrslenkungsstrategien (Routing Policies) definieren, welche eigene wirtschaftliche oder technische Interessen berücksichtigen. Bedauerlicherweise mangelt es immer noch einem grundlegenden Verständnis des komplexen Zusammenspiels zwischen den einzelnen autonomen Systemen. Im Rahmen dieser Diplomarbeit stellen wir eine Methode vor, welche die in der Realität gewählten Wege möglichst genau in Simulationen nachbildet. Auf der Basis eines AS Graphen, der aus realen Wegewahl-Tabellen ermittelt wird, wird ein einfaches Simulationsmodell erstellt. Die anfängliche Vernachlässigung von Policies führt im allgemeinen dazu, dass sich die simulierten Wege von den Pfaden unterscheiden, die man in der Realität beobachten kann. Im folgenden wird die Simulation schrittweise verfeinert, mit dem Ziel die simulierten Wegewahl(Routing)-Entscheidungen mit der Realität in Einklang zu bringen. Internetdienstanbieter (ISPs) können von solchen realistischen Simulationen in vielerlei Hinsicht profitieren. Leichtere Diagnose von auftretenden Problemen, Optimierung der Netzwerkleistung, Erkennen von Peering Nachbarn, ebenso wie eine Verbesserung der Anbindung an das restliche Internet, sind nur einige wenige Beispiele. In der wissenschaftlichen Welt können solche Simulationen einen Beitrag zur Entwicklung von ausgereifteren Techniken in der Verkehrskapazitätsplanung (traffic engineering) leisten oder helfen das Zusammenspiel zwischen Routingschicht und darüberliegenden Netzwerkebenen (overlay networks) besser zu verstehen. v

6

7 Abstract Today s Internet is composed of approximately 20,000 competing autonomous systems (ASes) which have to cooperate with each other to provide global reachability. Each AS is controlled by different administrative entities, and therefore encodes various economic, business, and performance decisions in its routing policy. Unfortunately, the complex interactions arising from such a distributed system are still poorly understood. In this thesis, we present a methodology to reproduce the BGP inter-domain routing decisions as realistically as possible by means of simulation. We start with a simple simulation model, relying solely on an AS graph which is derived from BGP table dumps. Due to missing policies, the routes chosen in the simulation will be different from those observed in reality. In an iterative process, we try to refine our simulation to accommodate the simulated routing decisions with those observed in reality. Such an Internet-wide simulation technique can be very valuable for ISPs to diagnose problems, optimize network performance, identify peering partners and provide better connectivity to the rest of the Internet. In addition, researchers can benefit from such a model. The design of more sophisticated traffic engineering techniques, as well as a better understanding of the interplay between the routing layer and overlay networks are only two applications. vii

8

9 Contents 1 Introduction Outline Background Internet Routing Architecture BGP Basics BGP Path Selection BGP Data Collection Architectures Related Work Data Sets Raw Data Collection Basic Statistics on Used Data Sets General Approach Approach C-BGP Simulator Towards a Better Understanding of Inter-Domain Routing Route Diversity Size of Routing Tables Diversity of Selected Routes within ASes Impact of Inter-Domain Connectivity Interaction between Intra-Domain and Inter-Domain Routing Impact of Route Reflectors Summary Simulation Setup Inter-Domain Model Modeling Inter-Domain Policies Standard Business Relationships Intra-Domain Model IBGP Topology Models Interplay between Interior and Exterior Gateway Protocols Model Implementation ix

10 Contents 6.4 Simplifications in the Model Towards Realistic Simulations General Approach Splitting Heuristic How it works Nondeterminism of Splitting Heuristic Results and Discussion Filter Detection Heuristic Per-Prefix Heuristic How it works Results and Discussion Comparison Conclusion and Future Work 68 A Divergence Issues 70 B Perl Scripts 72 B.1 CBGPInputGenerator.pl B.2 runsim.pl B.3 runsimforprefixes.pl B.4 analyzesimresults.pl B.5 splittingheuristic.pl B.6 filteringheuristic.pl B.7 perprefixheuristic.pl Acknowledgments 77 List of Figures 78 List of Tables 79 List of Algorithms 80 Bibliography 81 x

11 Chapter 1 Introduction The Internet has become an integral part of our everyday lives. Originally developed as part of a Cold War military project (ARPANET), a wide range of people nowadays have access to this world-wide communication medium and information source. s, voice and video conversations are taking the place of snail mail and traditional phone calls. Billions of e-commerce transactions per year underline the economic importance of the Internet for Business-to-Customer (B2C) and Business-to-Business (B2B) relationships. Technically, the Internet is a complex distributed system, composed of many independent networks. Altogether, there exist some 20,000 autonomous systems (ASes) which are controlled by different administrative entities. By interconnecting with each other and exchanging traffic, they form the network of networks - the Internet. In spite of this loose agglomeration of different networks, the end-user asks for world-wide reachability and connectivity. He wants to exchange traffic, irrespective of whether the destination host is in the local network or in a different AS. Routing protocols such as BGP [35] or OSPF [29] constitute the foundation for directing messages (packets) through the networks towards their destination. Unfortunately, ensuring reachability is not the only requirement for routing protocols. ISPs need to cover the costs arising from the maintenance of their technical infrastructure. Many of them are customers, buying their connectivity to the rest of the Internet from other ASes. To save or earn money, network operators could for example try to forward all their incoming traffic to customer ASes and not to providers. This explains why routing decisions are in general not completely based on system-wide criteria (e.g. shortest AS path), but have to consider economic rules and policies, too. Disregarding of their own economic interests, ISPs have to satisfy the increasing request for services such as VPN or VoIP. However, best effort service is simply not sufficient for security- or time-critical applications. In order to get guarantees for high performance and reliability, customers can conclude Service Level Agreements (SLAs) with their providers. In the future, ISPs therefore will have to adjust their networks to fulfill tight Quality of Service (QoS) constraints. Coping with strongly varying traffic loads, robustness to link or router failures as well as balancing load 1

12 Chapter 1 Introduction evenly across the network are only some of the aspects which need to be considered for that. It seems hardly impossible to organize (inter-)domain routing in a way which accommodates all three requirements: world-wide reachability, highly-configurable policies to enforce economic goals and mechanisms for quality assurance. A better understanding of what is going on in the routing layer, is therefore indispensable. In our work, we build a model, which is supposed to reproduce the inter-domain AS paths of the Internet as realistically as possible. We start with a simple simulation model, relying on an AS graph derived from BGP table dumps (e.g., RIPE [2], Route- Views [3]). In the beginning, each AS is reproduced with a single router and AS-level peerings are set according to the edges of the AS connectivity graph. C-BGP [31], a scalable BGP decision-process simulator developed at the Université Catholique de Louvain, is used to run large-scale simulations of our models. In general, the AS paths chosen in the simulation will be different from those which we observe in BGP routing tables from RIPE [2], RouteViews [3] or other collection sources. This is due to missing policies which can overwrite the rule of preferring the shortest path towards a destination. In an iterative process, we try to refine our models to accommodate the simulated paths with the observed paths. Given a set of AS paths, heuristics set up additional policy rules with the goal of adjusting the input paths in the simulation model. Altogether, the number of matches, i.e. AS paths which are seen in the observed data as well as known by at least one router in the simulation, can be increased from 41% to 87%, even for paths which have not been used as input for the heuristics. Such an Internet-wide simulation technique can be very valuable for operators to diagnose problems, optimize network performance, identify peering partners and provide better connectivity to the rest of the Internet (e.g., shorter average AS paths due to an improved selection of upstream providers). The main advantage of simulations lies in the global view which they provide of the Internet, and in the opportunity to play with different scenarios before applying changes in reality. As a long-term goal, realistic simulations could give answers to some of the following what-if questions: impact of link failures on routing dynamics, BGP update spread and traffic shifts impact of changed status of a relationship (e.g. canceled peering) impact of changes in the connectivity of transit networks In addition to ISPs, researchers can benefit from large-scale realistic simulations. Studying routing protocol behavior and traffic shifts, imposed by different loadadaptive routing mechanisms, can help to design more sophisticated traffic engineering techniques. Furthermore, our simulations could give additional insights into the interplay between the routing layer and overlay networks. Take as an example 2

13 1.1 Outline peer-to-peer (p2p) networks, for which in general there is no direct physical connection between two neighbors of the p2p-layer. What would be the performance gain, if the routing layer and overlay topology are more correlated with each other? Could p2p neighbor discovery be extended to consider physical distances (e.g. AS path length), obtained from our simulations? 1.1 Outline This work is organized as follows. We start with some background information on the routing architecture of the Internet, followed by an overview of BGP [35], the de-facto standard protocol for inter-domain routing. Chapter 2 closes with a short summary of previous work related to this thesis. In Chapter 3 we present the data sets, which will be used throughout all this work. After describing how we collected the AS path information, some basic statistics are given. Next is a brief description of the general simulation approach in Chapter 4. This includes a short presentation of C-BGP [31], the simulator tool used in this work, and the definition of metrics to measure how good the matching between simulated and observed paths is for a given model. The aim of Chapter 5 is to obtain insights into the impact of different aspects (e.g. ibgp or inter-domain connectivity). Simple experiments are not conducted with the pretense of being realistic, but are supposed to study the importance of various factors in terms of our goal to reproduce inter-domain AS paths. Subsequent to these preparative experiments, Chapter 6 presents our modeling approach. This part explains how the real Internet world is translated into a model, which can be simulated. The main focus is placed on the discussion why certain aspects are considered in our model and why others are neglected. The main step towards more realism in terms of routing decisions is made in Chapter 7. Heuristics try to accommodate simulations with observations from reality. The result is a simulation model, which captures both aspects: the available links in the topology as well as the decisions actually taken by the routing system. Finally, we conclude and present future directions in Chapter 8. 3

14 Chapter 2 Background The remaining chapters of this work presuppose some knowledge on routing in the Internet. This chapter 1 is supposed to give some background information on routing in general as well as on BGP, the de-facto standard protocol for inter-domain routing. Additionally, we will explain some terminology used throughout this thesis and point out related work. 2.1 Internet Routing Architecture The Internet is sometimes referred to as the network of networks. Actually, it consists of more than 20,000 autonomous systems (ASes) which are administrated independently. Each AS again, can be composed of multiple networks, which are generally operated under the same authority (e.g. an enterprise or a university). We distinguish two basic types of routing protocols which together ensure global reachability. Inside an AS, an Interior Gateway Protocol (IGP) such as IS-IS [7] or OSPF [29] is used to propagate routing information. However between ASes, reachability information is exchanged relying on an Exterior Gateway Protocol (EGP). BGP [35] is the de-facto standard inter-domain routing protocol, currently used in the Internet. ASes are networks themselves, consisting of several routers which are interconnected with each other. Different technologies including Synchronous Optical Networks (SONET/SDH) and/or Ethernet can be used to implement the link layer connectivity. The already mentioned IGP [28] protocols like IS-IS [7] or OSPF [29] are responsible for the routing inside an AS: each router selects a shortest path to a destination prefix according to a metric defined by the network administrator. The routers inside the core of a network are frequently referred to as core routers, making up the backbone of a network. Such routers are generally not connected with routers in neighboring ASes. Routers which provide inter-domain connectivity are called border routers. The physical locations of these border routers are frequently referred to as Points of Presence (PoPs). 1 parts of this chapter follow the example of [25] and [34] 4

15 2.2 BGP Basics The routing between ASes is more complex than the routing inside an AS. It has to consider contractual agreements, concluded by neighboring ASes. Take as example a customer-provider relationship: many ASes buy Internet connectivity from one or more transit providers (upstream providers). Besides this customer-provider relationship, there is also a category referred to as peering relationship. In such a case, the neighboring ASes share the costs, which arise from the maintenance of the connecting link. Therefore, such a peering link is exclusively used to exchange traffic with the neighbor AS and its customers. Transit traffic is not allowed to flow through peering links [17, 15]. Note that peer is often used ambiguously. On the hand, it can denote the just described economic relationship between two ASes, on the other hand it just means any BGP neighbor with which a BGP session has been established. In the course of this thesis, we will sometimes classify ASes according to their position in the economic hierarchy. ASes that have no upstream providers are referred to as tier-1. Due to this definition, all tier-1 ASes more or less have to peer with each other 2 in order to exchange routing information with other tier-1 ASes. Therefore, we say that such ASes are in the core of the Internet, while smaller customer ASes are more at the periphery of the Internet graph. As already mentioned, BGP [38] is the de-facto standard inter-domain routing protocol of today s Internet. It has to guarantee the adherence to economic constraints (e.g., customer-provider or peering relationships). In addition to the exchange of reachability information, it allows to specify rules which implement (economic) policies. The next section discusses BGP in more details. 2.2 BGP Basics The Border Gateway Protocol (BGP) [35] was designed as a successor to the Exterior Gateway Protocol [23]. It is a variant of the class of distance-vector protocols, where neighboring routers exchange link cost information to destinations. BGP itself is a so-called path-vector protocol. A route advertisement indicates the reachability of a network. To avoid cycles and to provide a distance-metric the AS number (ASN) of each AS on the path to the destination is propagated along with the route advertisements. Reachability information is exchanged by establishing a session between two BGP speaking routers, relying on an underlying TCP connection. After a session is set up, first all best routes are announced to the neighbors. Afterwards, update messages are only sent whenever the current best route changes. Note that there are two variants of BGP. On inter-domain links - connecting routers of different ASes - ebgp is used to announce reachable prefixes. Inside ASes, ibgp ensures that the interior routers learn how to reach a destination. The difference is that ibgp does not modify the AS 2 See [14] for a nice counter-example. 5

16 Chapter 2 Background path attribute (otherwise the same AS number would be appended several times). In order to avoid routing loops, it is therefore necessary, to configure a full mesh or to use route reflection [4] inside ASes. Four types of messages can be exchanged between two BGP speakers: 1. OPEN is used to establish a BGP session between two routers. 2. UPDATE messages carry reachability information. An update either advertises a prefix or withdraws a previously announced prefix. Multiple announcements and/or withdraws can be packed into one BGP UPDATE packet. 3. NOTIFICATION is used to close a BGP session in case of an error. 4. KEEPALIVES: if no update or notification messages have been exchanged for a longer period of time, they ensure that the connection is not closed. If a router announces a prefix to its neighbor, this indicates to the receiver that it can reach the specified prefix by forwarding traffic for this destination to the sender of the announcement. Provided that the router can no longer reach the prefix or does not want to carry the traffic towards this destination any more, it sends an update message in which the route is withdrawn. Every time, a router changes its best routes, it will have to inform all neighbors to which it has announced the route, about the change. BGP allows to associate attributes with routes. These BGP attributes are considered during the BGP decision process when a best route is selected from the set of all routes learned for a certain prefix. Generally, network operators can manipulate routes by modifying or removing some of these attributes. BGP attributes can be classified into transient and non-transient. The difference is that transient attributes can be passed on to neighboring ASes while non-transient cannot. In the following, we give a short summary of the available attributes. For more information, the reader is referred to [38]. Any route advertisement includes an ordered sequence of ASes (AS path), indicating the ASes through which traffic to the announced prefix will flow (provided that this path is selected as best route). Every time, a BGP message is forwarded to a router in a different AS, the AS number (ASN) is prepended to the current AS path. Consequently, the AS that originated the prefix is at the end of the AS path. Note that the actual path, used in the reverse direction may not be the same due to policies enforced along the path between the AS and the destination prefix. The AS path information is not sufficient to forward traffic. Therefore, BGP messages contain a NEXT HOP attribute, telling to which IP address a router should send packets. Note that a router is not required to have a BGP session with the NEXT HOP router, as BGP allows third-party next hops. Inside an AS, the next hop attribute points to the selected exit router (by default the border router of the remote AS) and not to a neighboring ibgp router. However, network administrators frequently 6

17 2.3 BGP Path Selection RIB AS1 t 1 t 1 AS2 Best route s t o r e d i n F IB! " #$ % &(') +*,.-/01 2 AS3 t 3 t 2 t 3 AS4 Figure 2.1: Update processing per router Figure 2.2: Example: update propagation configure routers with the option "next-hop-self" in order to use the border router of the local AS as the exit router. This has the advantage that IGP does not have to know the links between the border routers of both ASes. Other available attributes like for example Local Preference, Multiple Exit Discriminator (MED) and Community values are mainly used to implement policies or to add state to a route. With local preference, ISPs can influence outgoing traffic (e.g. prefer customers over peers over upstreams). In addition, it can be used inside ASes to implement policies for the best exit point. The main purpose of the Multiple Exit Discriminator, is to indicate the best entry point into the own network to neighboring ASes, provided that there are multiple connections. Community attributes [9] allow to add state to a route. Contrary to other BGP attributes which have a well defined meaning, the community attribute provides a large code space which can be independently defined by network administrators. The added state can be used for signaling inside a domain and/or across domains. 2.3 BGP Path Selection Figure 2.1 illustrates the basic principle of a BGP-speaking router. Suppose that a router learns a route from one of its neighbors. For each BGP session, it is possible to define a set of ingress filter rules. When a route is received over a BGP session, the router checks whether the route is in accordance with the ingress rules, defined for this session. If yes, the router may manipulate some attributes of the route, before storing it in a BGP routing table. This BGP routing table, containing the learned routes, is referred to as Routing Information Base (RIB). The BGP decision process, which is now used to select a preferred route, will be discussed in one of the next paragraphs. Assume, that the router has decided on a best route for a certain prefix. If this new best route is different from the old one, 7

18 Chapter 2 Background the routing table is updated. Before being propagated to the neighbors, the new best route may have to pass some egress filters. Similar to the ingress filters, these rules can manipulate the BGP attributes of a route and even forbid its announcement to neighboring routers. Due to memory limitations, the routers do not propagate all the learned paths to their neighbors but only the selected best route. Figure 2.2 shows how updates spread through a network. For the remainder of this work, we will refer to the routes which are known to a router but not necessarily selected as best paths as RIB-In routes. The routes, which are actually selected, are said to be in the Loc-RIB of a router. To select the best route among the set of known routes for the same prefix in the RIB, a BGP speaker follows a set of selection criteria called the best path selection algorithm. In the following, we discuss nine of the most important rules. These criteria are applied in the specified order until only one path remains. 1. Next-hop reachable? The first rule of the BGP decision process ensures that the next-hop is actually reachable. This is done by a periodic IGP look-up of the next-hops. 2. Prefer highest local-pref: This BGP attribute is generally set by the border routers when a route is received from a neighbor AS. It remains local to the AS and is only propagated to ibgp neighbors. With local-pref it is possible to specify the preference among different routes towards a given destination (higher values are better). For example, it can be used to prefer customer over peering or provider routes. The default value is Prefer locally originated routes: Routes which are locally originated by the router or redistributed from IGP by the router are preferred over routes that are learned from other BGP speakers. 4. Prefer routes with the shortest AS path length: As already mentioned, the AS numbers are prepended to the AS paths, while the routing information propagates through the network. By preferring routes with short AS path lengths, BGP assumes, that short AS path lengths are correlated with short paths (low delay) to the destination (which is not necessarily true for the Internet). 5. Prefer the path with the lowest origin type: This rule considers how the route has been learned. Routes learned via an Exterior Gateway Protocol (EGP) are preferred over routes which have been introduced into BGP by Interior Gateway Protocols (IGP) like OSPF [29] or IS-IS [7]. In addition, routes can be learned by some other means, e.g. manual configuration (INCOMPLETE). 6. Prefer the route with the lowest MED value: Routes without a MED attribute are considered to have the lowest possible MED value. The MED attribute is used to select a particular egress point in the local domain. This can be used for cold-potato routing [40]. Cold potato routing aims at carrying the traffic 8

19 2.3 BGP Path Selection as long as possible in the own network, before handing it off to the neighbor. The input filter of the neighbor domain might override the MED value, such that the use of this attribute usually relies on a mutual understanding between two neighboring ASes. In general MED values are only compared if the routes have been learned from the same neighbor AS. However, some vendors offer to compare MED non deterministically [41]. In this case, MED values are compared, even if two routes have been learned from different neighbor ASes. Note that, we will use this feature in Section 7.4. Furthermore, MEDs can be used as an internal metric. Contrary to local-pref, MED comparison is in the decision process after the AS path evaluation. This allows to define a ranking for routes, which still considers the AS path length. 7. Prefer ebgp routes over ibgp: This rule ensures that the local AS gets rid of IP packets as soon as possible ( early-exit or hot-potato routing). 8. Prefer routes with the lowest IGP cost to the egress point: This rule enables what is commonly termed early-exit or hot-potato routing [45]. The goal is to get rid of IP packets as soon as possible. This is achieved by directing traffic to the closest border router based on the intra-domain path costs (IGP metric). Most router vendors implement some kind of scanner process, which is run periodically to see if the IGP cost has changed, and then updates the best BGP route accordingly. 9. Tie-breaking rules: Suppose, there exist still at least two alternative routes, which are equally good up to this step. The implementation of the required tie-break is vendor-specific. Some BGP implementations prefer routes received from the router with the lowest router ID (IP addresses), while others prefer the oldest route. Note, that tie-breaking introduces some randomness into the inter-domain path selection. We point out that the description of the best path selection process above is not complete. Some vendors implement slightly different variants and some even allow operators to modify the selection process. Note that routing domains do not necessarily correspond to administrative domains, i.e. routers inside the same AS can select different routes to reach a certain prefix. This is due to the fact that each router independently picks one route from a possible set of alternatives. This choice 3 affects traffic forwarding as well as route propagation. Using the standard approach, ibgp routers inside an AS have to establish a full mesh of ibgp sessions. Therefore, Route Reflection [4] was designed to improve 3 There are some efforts to allow a router to select multiple paths for load sharing purposes, see e.g. [12]. 9

20 Chapter 2 Background San Francisco Munich Beijing r61 r62 AS 6 r63 r51 AS 5 r52 r53 r21 r41 AS 2 AS 4 r22 r32 r43 r42 AS 3 r33 ibgp peer AS 1 prefix p best route customer provider r11 (traffic flows this direction) Figure 2.3: Example: Path sections of ASes 2. the scalability inside ASes. Route reflectors (RRs) are normal BGP speakers, which pick a best route based on the best path selection process as described above and announce the selected route to their neighbors. They reflect ibgp updates from RRclients to other clients as well as to other RRs and also distribute ibgp updates from RRs to clients (but not to other RRs). If appropriately configured (see [22]), routing loops cannot occur inside the AS. The idea with route reflectors is to aggregate route information and thus keep multiple alternative routes inside Points of Presence (PoPs) or with a certain geographic region. RRs can be organized hierarchical, with a full-mesh only on the top-level of the hierarchy. To avoid single points of failures, redundant RRs can be configured. Selecting the best route among a given set of alternative routes is not trivial. To illustrate typical effects of the best path selection algorithm in the context of multiple ASes, consider the example 4 shown in Figure It shows six ASes. AS 1 is multihomed to AS 2 and AS 3, which both provide transit service for AS 1. AS 2 and AS 3 are customers of AS 4. AS 3 is also a customer of AS 5 (customer-provider 4 example taken from [25] 5 Images used with permission. (Thanks to and 10

21 2.3 BGP Path Selection relationships are indicated by solid black lines). AS 4, AS 5 and AS 6 all peer with each other (peering links are indicated by dashed red lines) and therefore build the core of the AS graph, i.e. they can be considered as tier-1 ASes. Keep in mind that ASes often cover the same geographical area while still being competitors. To illustrate this we have chosen three geographically distant locations: San Francisco, Munich, and Beijing. In the example of Figure 2.3 a router that is supposed to be located in San Francisco is labeled r 1, one from Munich is labeled r 2, and one located in Beijing is labeled r 3 (where the stands for the ASN of the AS). Contrary to AS 1, AS 2 and AS 3, the ASes 4,5 and 6 provide service in all three regions and peer in multiple places but not necessarily at each location. AS 1 announces its prefix p to its upstream providers AS 2 and AS 3. As both have no alternative routes, they select the direct path. AS 2 and AS 3 propagate their selected route to AS 4, which learns the route over all four links. Routers inside AS 4 are free to choose their exit point (unless, for example, AS 3 decides to use MEDs to signal that it wants to receive incoming IP traffic over the Munich -link and AS 4 respects the MED setting). If MEDs are not used, a hot-potato -like behavior is often the default. This means that traffic flows directly to the closest egress router 6. In the case of AS 4, traffic entering the router in San Francisco (router r41) leaves the AS via the San Francisco -link to AS 2 (link r41 to r21). Traffic originated in Munich entering router r42 leaves the AS via one of the Munich -links. Note that this router can choose between two paths which are equally good up to the last step of the BGP decision process. Therefore, tie-breaking based on the remote router ID is used to pick one link. Next consider how AS 5 learns about prefix p. This situation is slightly different, because the path via Beijing has a shorter AS path length (furthermore AS 3 is a customer and customers are often preferred via local-pref). The router r52, which is located in Munich, learns the prefix p over a peering link in Munich (link to router r42), and thus could deliver the traffic locally. However, all routers in AS 5 may prefer to send the traffic for prefix p to Beijing, to be then handed-over via the customer link to AS 3. Finally consider AS 6: Traffic from router r62 (located in Munich) can be delivered via the peering link to AS 5 (to router r52), because AS 5 picked the customer link over AS 3. But then traffic detours via Beijing. If the link in Beijing fails (between r33/as3 and r53/as 5), then r52 (AS 5) will use the peering link to AS 4 and therefore withdraws the announcement given to AS 6 (because a peer does not offer transit service for another peer). Ironically, in such a case, router r62 would choose as next hop either r43 or r41. Again, we point out that routing domains are not equal to administrative domains, i.e. routers inside the same AS can select different paths to reach a certain prefix. Take as example AS 6: Router r61 uses AS path " ", while router r62 propagates AS path " " and router r63 uses AS path " " as best route. This illustrates how complex the interactions in the Internet can be. 6 Note that the closest egress router is determined based on the intra-domain path costs, i.e. the IGP metric. 11

22 Chapter 2 Background 2.4 BGP Data Collection Architectures Unfortunately, ISPs generally do not disclose topology information. BGP data collection architectures are supposed to provide operators with a view of the Internet from different locations, by collecting and archiving BGP data. This data can be useful for diagnosing network problems or for studying protocol behavior. In general, two types of data are provided: BGP table dumps and BGP updates: Whereas table dumps provide a snapshot of the Routing Information Base (RIB), updates contain a time series of the changes to the routes. Regarding table dumps, it is sufficient to log in to a router (a real router or a software BGP speaker) and to query the RIB. BGP updates can be obtained in two different ways: First, one can dump an IP packet trace of TCP port 179 on a link, which is used by the BGP sessions. Second, software pseudo-bgp speaker ( route collector ) peering with real routers can be used to record updates. Such a route collector stores the original BGP update packet, a timestamp and both end points. Note that BGP data collection is passive, i.e. the collectors do receive updates but they do not originate any prefixes, nor do they send updates to their neighbors. There exist many such route collectors in the Internet. Since September 1999, more and more collectors have been added. Due to the increasing number of route collectors, the provided view of the Internet has steadily become more representative. One example of a public BGP collection infrastructure is RIPE (Réseaux IP Européens) [2]. It maintains about 14 different Remote Route Collectors (RRC) located at different places in the world. Each RRC peers with several operational routers and therefore has a number of BGP feeds. Note that some of the feeds are partial feeds, i.e. the neighboring router does not announce all its best routes to the collector. Another public source is Oregon s RouteViews Project [3]. In addition to public collection infrastructures, there exist many proprietary BGP collectors. This can be looking glasses, trace collections, or as in the case of Akamai [1] remote collectors. Akamai has collectors in about 500 ASes. Some of these feeds are full-feeds, some are partial feeds. Frequently, there are multiple observation points inside the same AS. The location where route collectors are placed makes a difference. Economic relationships (e.g. customer-provider and peering) determine which routes are preferred or announced to other ASes. Recall that tier-1 ASes generally peer with many other tier-1s, and know a lot of alternative paths. Contrary to tier-1 ASes, the path diversity closer at the periphery is significantly restricted. An additional reason for this comes from the fact that many customer ASes are multi-homed (one primary and one backup connection). The backup path may not be visible to most of the Internet, unless a failure close to the customer AS occurs. In addition, a monitoring point would be required in the neighborhood of the failure. Therefore, Teixeira et al. [44] argue that data from BGP collection architectures is not sufficient. It is not 12

23 2.5 Related Work possible to get a complete view of the Internet by solely relying on data collection architectures. 2.5 Related Work Realistic models of the routing decisions in the Internet are important for the development and evaluation of network protocols. In addition, they could support ISPs in their decisions, facilitating for example traffic engineering. Previous work can be broadly classified in two categories: measurements to get a better understanding of the structure of the Internet and simulations which try to reproduce some of the characteristics of the real Internet. Knowledge on the Internet s topology is important for building realistic models. Previous work in this field has been focused on the interconnection structure at either AS [10, 17, 18] or router level [37]. This thesis will use measured information to obtain the connectivity between Internet domains. Since routing between ASes is controlled by BGP, a policy-based routing protocol, such AS-level connectivity does not imply reachability. Various attempts [15, 5, 39, 26] have been made to infer basic contractual agreements between ASes (e.g. customer-provider), relying on paths observed from different vantage points. Simulations allow to examine protocol dynamics and have the advantage of providing a global view on a distributed network like the Internet. A number of simulation frameworks [36, 31] have been developed to study BGP-related questions. The results from topology generators [27, 8] could be used as input for simulations. Such generators produce topologies with similar graph properties as seen in the real Internet, having the advantage of scalability. However, previous simulations of the Internet frequently failed to capture the careful design that is at the basis of real networks. Our work tries to combine simulations with observations from reality. Measured information on the AS-level connectivity is used to build an initial simulation model. Then, the simulation results are brought in accordance with observations from reality. To the best of our knowledge, no simulation framework fully captures both aspects: the available links in the topology and the decisions actually taken by the routing system. 13

24 Chapter 3 Data Sets It is the stated aim of this work to reproduce the routing decisions and the interdomain topology of the Internet as realistically as possible by means of simulation. Recent progress in simulators [31], has made it possible to run large-scale simulations for topologies in the magnitude of the Internet. Yet, the pretense of being realistic, still requires some information upon which a model and a simulation for the real word can be designed. This chapter is going to present the data sets that will be used throughout all the parts of this thesis. Of course, it would be possible to rely on synthetic data produced by topology generators such as BRITE [27] and GT-ITM [8]. However, Quoitin [32] argues that topology generators in general do not capture the careful design that is at the basis of real network topologies. Therefore, our simulations are based on real data gathered from BGP collection architectures (see Section 2.4), providing us with a view of the network from different locations inside the Internet. Even if such a view is incomplete due to the limited number of vantage points, it should allow to set up expressive models of the Internet. Another motivation for using data from BGP collection architectures is validation. Since the gathered data sets include BGP table dumps from real routers, this routing information can be compared with the simulation model for the purpose of validation. For more information, the reader is referred to Section 7.1. This chapter is arranged in two parts. The first part describes where and how the raw data was collected. The second presents some basic statistics on the gathered data sets, with a focus on the properties of the inferred AS-level topology. 3.1 Raw Data Collection Our work relies on BGP routing table dumps obtained from RIPE [2], RouteViews [3], a local ISP and Akamai Technology [1]. For all four sources, data was recorded at 07/18/05 around midnight by using the BGP data collection techniques described in Section 2.4. More precisely, table dumps from the following route collectors are considered: 14

25 3.1 Raw Data Collection RIPE: RRC00-RRC14 RouteViews: route-views.oregon-ix.net and 5 IPv4 Zebra collectors SpaceNet (a small local ISP) Akamai Technology: collectors in about 500 ASes For the remainder of this work, data from RIPE, RouteViews and SpaceNet will be referred to as public data, whereas the Akamai information is also referred to as private data. Table 3.1 gives some details on the collection of the raw data. Data Source Public Akamai Public &Akamai # BGP collectors 22 > 500 > 500 # peering sessions 1,212 1,767 2,337 # ASes with BGP feeds # ASes with multiple feeds Table 3.1: Raw data collection (broken down by Public and Akamai data) Altogether, data was collected from 2,337 different peering sessions to more than 990 ASes, including Tier1 ISPs, major European ISPs, Asian ISPs and stub ASes. It is pointed out that some ASes provide full feeds, while others only reveal parts of their routing information to the BGP collectors. For 322 out of 990 (32.5%) of the monitored ASes, there exist multiple feeds. Currently, all the results of this work are exclusively based on BGP table dumps. Starting with the AS paths from the table snapshots, peerings between Internet domains can be identified. In this way, it is possible to infer the AS graph only to a certain extent, restricted by the number of observation points. Note, that BGP update traces might give additional knowledge on the AS-level connectivity and should be considered in future work. In a nutshell, the used raw data gives insight into the basic connectivity graph (ASlevel peerings) and the prefixes originated in the Internet. For the remainder of this work, we exclude from further consideration all prefix announcements which include private address space or which are duplicate announcements. For the simulations and also for the validation process (see Chapter 7), the only parts that are actually needed are the observed AS paths. For this purpose, AS path prepending - a popular policy for traffic engineering purposes - is removed. One has to keep in mind that the observed AS paths do not reflect the complete status quo of the Internet at a fixed point in time, but possibly contain some inconsistencies. Provided the case that there exist two different BGP feeds from the same router, it could be possible to see two different best routes to the same destination. 15

26 Chapter 3 Data Sets 3.2 Basic Statistics on Used Data Sets BGP table dumps obtained from route collectors allow to infer the AS-level connectivity graph to a certain extent. This connectivity information fulfills two purposes in our work: First, it serves as a starting point for the simulation, indicating which ASes or routers share ibgp sessions. Second, it will be used for validation and verification in order to check whether the routers in the simulation select the same AS paths as those observed in reality. This section presents some basic statistics on the gathered data sets. distinct AS paths (ignoring AS prepending) 4,230,439 number of ASes 20,416 AS-level connections (peerings) 56,781 transit ASes 3,389 (16.6%) stub ASes 17,027 (83.4%) originating ASes 20,311 (99.5%) observed prefixes (public address space) 281,516 # prefixes with multiple originating ASes 1,616 Table 3.2: Basic statistics on used raw data (including Public and Akamai) Table 3.2 shows that 20,416 ASes and 57,781 AS peerings can be identified from the more than 4 million distinct AS paths, obtained when data from all our BGP collectors are put together. In the following chapters, we will sometimes distinguish between transit and stub ASes. A transit AS is a network domain, which forwards traffic from at least one of its peering ASes to one of its neighbors. Transit ASes are counted by only considering those ASes that appear at least once as an intermediate AS hop in an observed path; all the remaining ASes are regarded as stub ASes. The numbers in Table 3.2 show that only 16.6% of the ASes can be classified as transit domains based on our observed paths. Data Source Public Akamai Public & Akamai distinct AS paths (no prepending) 2,107,890 2,335,259 4,230,439 AS paths exclusively in.. 371, ,352 - observed ASes 20,306 20,315 20,416 observed AS peerings 46,084 44,980 56,781 Table 3.3: Comparison of Public and Akamai data If duplicate announcements and prefixes from the private address space are removed, 281,516 observed prefixes remain, being originated at 20,311 ASes (almost all ASes announce at least one prefix!). In terms of inconsistencies, 0.6% of the prefixes were found to be originated at multiple ASes. Continuing with statistics, we investigate 16

27 3.2 Basic Statistics on Used Data Sets how far the information gained from public data differs from that obtained from Akamai. Table 3.3 indicates that both data sets contain information which cannot be inferred from only one source. Take as example the total number of peering links (56,781), with Public data as well as Akamai data only detecting approximately 45,000 ASlevel connections. This underlines the need of multiple vantage points for an adequate view of the Internet. 17

28 Chapter 4 General Approach The preceding chapter has been an important step towards reproducing inter-domain AS paths, answering the question which AS paths are to be reproduced. Basically, the goal is to have a simulation model, where the observed AS paths obtained from BGP table dumps (see Chapter 3) are selected as best routes by the simulated routers. Now, we describe the general setup for the performed simulations. This chapter consists of two parts. First, we describe which basic input data is used to build an initial simulation model. In particular, Section 4.1 pinpoints the core problem which is supposed to be overcome by this work: mismatches between observed AS paths and simulated paths. Metrics are defined to measure the extent to which a simulation model and observed AS paths are in accordance with each other. In Section 4.2, we give a brief introduction into the used simulator tool C-BGP. The focus will be placed on the reasons which favor this simulator with regards to our goals. 4.1 Approach The goal of this work is to reproduce inter-domain AS paths in simulations. As a prerequisite, it is necessary to clarify two questions: Which AS paths are to be reproduced by the simulation and what is the general simulation setup? Concerning the first question, we decide to use AS paths obtained from BGP table dumps (see Chapter 3). As simulator, we choose C-BGP [31], relying on an AS connectivity graph as an initial model. Subsequent Figure 4.1 summarizes the approach, which serves as a basis for the rest of this work. To begin with, the initial simulation model is based on the AS connectivity graph which is constructed as follows: Observed AS paths from all available BGP table dumps are used (Public and Private). Relying on this information, ASes and peerings are identified. Note, that in the beginning all ASes are modeled with a single router, and peerings are set according to the edges of the AS graph. Implementations of simulation models will assign IP addresses to routers, such that the first 16 bits 18

29 4.1 Approach BGP table dumps (RIPE, Routeviews,...) connectivity graph model simulate observed AS paths simulated paths match Figure 4.1: Initial setup reflect the AS number, and the last 16 bits an ID unique inside an AS. Keep in mind that IP addresses can be used as a tie-break criterion in the BGP decision process (see Section 2.3). After agreeing on the initial model, we run a simulation with C-BGP [31]. The simulator C-BGP has been developed at the Université Catholique de Louvain (UCL) and scales very well for a high number of routers. Simulations are performed prefixwise, meaning that there is an extra simulation for each origin AS - ASes which are the last hop of an observed AS path. Note that this is a simplification, as one AS can originate multiple prefixes. However, slight modifications are sufficient for adapting our approach to deal with real IP prefixes. Suppose that all simulations have finished. Now, the next step consists in comparing the simulated paths with the observed AS paths. Unfortunately, the results obtained for the initial simulation model ( Shortest AS Path ) are in general not in accordance with the observed AS paths. This seems to be a contradiction, as we use the observed paths to infer the peerings between ASes and to construct the connectivity graph. The explanation lies in missing filtering and ranking policies which overwrite the rule of preferring the shortest path towards a destination. In the context of this work, reproducing inter-domain paths relates to the matching between observed and simulated paths. Heuristics, presented in Chapter 7 will refine simulation models in order to bring in line the simulation results with the observed paths. This is done by incrementally adding policy rules and new routers to the simulation. First, simulations are run for some prefixes. Then, heuristics compare the obtained simulated paths with the routes that have been observed in the Internet (data sets from Chapter 3). Relying on this comparison, the heuristics compute changes which are supposed to reduce the discrepancies between the current simulations and the observations from the real Internet. This cycle is generally repeated until there is a perfect matching between the observed paths and the simulated paths or until the heuristics do not make progress in terms of the number of matchings. 19