Hi, we have had some problems recently apparently caused by route flapping, and this inspired me to create a setup where I can log the "route flaps of the world" and then do some perl programming to analyze the debug logs. After about a day of programming I now have a handfull of scripts which are able to post-process the output of "debug ip bgp updates" on a Cisco router. The router I am using to log this with has a direct peer with nord-gw.nordu.net, that should be taken into account when looking at the results (if I'm not mistaken, to some extent we'll see more local problems than those on remote networks). Processing yesterday's log (covering the interval Sep 29 04:05 to Sep 30 04:05) produced the following "top 20" list of ASes injecting route flaps as seen from our perspective: Origin(s) flaps retr unrch dests AS AS Description 544 26070 264 266 109 544 The DataNet IP Service 719 17035 1 1 13 719 LANLINK autonomous system 1740 11880 2624 2623 186 1740 CERFNET 1717 7716 2645 2639 574 1717 RENATER 1221:297 6626 1155 1157 254 1221 AARNET-AS 1221:297 6626 1155 1157 254 297 NSN-UMD-AS 174:2149 5263 1705 1704 140 2149 PSINET-2 174:2149 5263 1705 1704 140 174 NYSERNET-AS 86 2890 489 488 113 86 SURANET-AS 2149:174 2604 769 769 178 2149 PSINET-2 2149:174 2604 769 769 178 174 NYSERNET-AS 2551 2354 318 318 174 2551 NETCOM-AS 372:297 2342 333 333 109 372 NSN-AMES-AS 372:297 2342 333 333 109 297 NSN-UMD-AS 297:1221 2168 337 346 95 1221 AARNET-AS 297:1221 2168 337 346 95 297 NSN-UMD-AS 372 2024 393 392 311 372 NSN-AMES-AS 701 1664 525 525 170 701 AlterNet 813 1589 617 615 83 813 CANET-ASN 2915 1488 391 391 83 2915 SPIN-NET Legend: "Origin(s)" are the AS or ASes seen as "origin" of routing updates (essentially the last element in the logged path). "flaps" is the sum of "retr" and the number of times the route was updated (e.g. new path) "retr" is the number of times a route was withdrawn (retracted) "unrch" is the number of withdrawals which caused loss of connectivity for a destination "dests" is the number of destinations involved in route flaps Excluded from this picture are any routes involving as-sets (there are a number of routing flaps involving them as well, I'll see about analyzing them separately...). We are (as I write) trying to sort out the nordic-internal problem involving the two on the top of the list. One thing I want to draw to your attention, thogh, is the fact that many destinations are apparently originated by more than one AS. I was under the impression that this is an illegal or highly undesireable configuration. Attached below you will find a more detailed report showing the number of times each individual path was announced for each of the entries on the "top 20" list, I can also easily (on request) produce a "per-network" report for any given AS. It would seem that even beyond AS 1800 there are severe instabilities in the routing. Comments? - Havard PS. Maybe this should be redirected somewhere else than bgpd? nanog? iepg? PPS. I am willing to give away my scripts to anyone asking (privately) for them. Note that they're quick hacks and still evolving... ------------------------------ Origin(s) flaps retr unrch dests AS AS Description 544 26070 264 266 109 544 The DataNet IP Service Count Path 244 2603 1759 544 12734 2603 1741 1759 544 12840 2603 1741 544 5 2603 1805 1800 1239 1280 701 1759 544 Origin(s) flaps retr unrch dests AS AS Description 719 17035 1 1 13 719 LANLINK autonomous system Count Path 17045 2603 1741 719 1 2603 3224 2043 1128 286 790 719 Origin(s) flaps retr unrch dests AS AS Description 1740 11880 2624 2623 186 1740 CERFNET Count Path 1029 2603 1805 1800 1133 690 1740 5071 2603 1805 1800 690 1740 937 2603 1755 1800 690 1740 1947 2603 1805 1800 701 690 1740 173 2603 1755 1800 701 690 1740 51 2603 1755 1800 1133 690 1740 82 2603 1805 1800 1239 1280 1740 15 2603 1805 1800 1133 1674 690 1740 5 2603 1755 1800 1133 1674 690 1740 15 2603 1755 1800 1239 1280 1740 Origin(s) flaps retr unrch dests AS AS Description 1717 7716 2645 2639 574 1717 RENATER Count Path 2174 2603 3224 2043 1128 1755 1717 2486 2603 1755 1717 479 2603 3224 2043 1128 1133 1800 1755 1717 Origin(s) flaps retr unrch dests AS AS Description 1221:297 6626 1155 1157 254 1221 AARNET-AS Count Path 1885 2603 1805 1800 690 372 1221 846 2603 1805 1800 701 690 372 1221 272 2603 1805 1800 1133 690 372 1221 32 2603 1755 1800 701 690 372 1221 31 2603 1755 1800 690 372 1221 4 2603 1755 1800 1133 690 372 1221 1 2603 1805 1800 1239 1240 690 372 1221 1 2603 1755 1800 1239 1240 690 372 1221 Origin(s) flaps retr unrch dests AS AS Description 1221:297 6626 1155 1157 254 297 NSN-UMD-AS Count Path 582 2603 1805 1800 690 297 1035 2603 1805 1800 701 690 297 427 2603 1805 1800 1133 690 297 347 2603 1755 1800 701 690 297 6 2603 1755 1800 690 297 2 2603 1805 1800 1239 690 297 Origin(s) flaps retr unrch dests AS AS Description 174:2149 5263 1705 1704 140 2149 PSINET-2 Count Path 332 2603 1805 1800 1133 690 2149 397 2603 1805 1800 690 2149 154 2603 1805 1800 701 690 2149 58 2603 1755 1800 701 690 2149 93 2603 1755 1800 1133 690 2149 32 2603 1805 1800 1239 1240 690 2149 5 2603 1755 1800 1239 1240 690 2149 51 2603 1755 1800 690 2149 Origin(s) flaps retr unrch dests AS AS Description 174:2149 5263 1705 1704 140 174 NYSERNET-AS Count Path 1432 2603 1805 1800 174 335 2603 1755 1800 174 229 2603 1805 1800 690 174 113 2603 1805 1800 701 690 174 108 2603 3224 2043 1128 1133 1800 174 43 2603 1755 1800 690 174 22 2603 1755 1800 701 690 174 86 2603 1805 1800 1239 1280 174 33 2603 1755 1800 1239 1280 174 25 2603 1805 1800 1133 690 174 8 2603 1805 1800 690 701 174 2 2603 1755 1800 690 701 174 1 2603 1755 1800 1133 690 174 Origin(s) flaps retr unrch dests AS AS Description 86 2890 489 488 113 86 SURANET-AS Count Path 93 2603 1755 1800 86 857 2603 1805 1800 86 25 2603 1805 1800 690 279 86 352 2603 1805 1800 690 279 3561 86 84 2603 1805 1800 1133 690 279 3561 86 72 2603 1755 1800 690 279 3561 86 80 2603 1805 1800 701 690 86 674 2603 1805 1800 690 86 44 2603 1755 1800 690 86 32 2603 1805 1800 1239 1280 1957 690 279 3561 86 28 2603 1805 1800 1239 1280 1957 690 86 41 2603 1805 1800 1133 690 86 3 2603 1755 1800 1133 690 279 3561 86 3 2603 1755 1800 701 690 279 3561 86 5 2603 1755 1800 1239 1280 1957 690 279 3561 86 3 2603 1755 1800 690 279 86 2 2603 1805 1800 1133 690 279 86 12 2603 1805 1800 1239 1280 1957 690 279 86 3 2603 1805 1800 701 690 279 86 1 2603 1755 1800 701 690 86 2 2603 1755 1800 1239 1280 1957 690 86 1 2603 1755 1800 1239 1280 1957 690 279 86 Origin(s) flaps retr unrch dests AS AS Description 2149:174 2604 769 769 178 2149 PSINET-2 Count Path 220 2603 1805 1800 1133 690 2149 35 2603 1755 1800 1133 690 2149 116 2603 1805 1800 701 690 2149 215 2603 1805 1800 690 2149 37 2603 1755 1800 690 2149 27 2603 1755 1800 701 690 2149 8 2603 1805 1800 1239 1240 690 2149 2 2603 1755 1800 1239 1240 690 2149 2 2603 1805 1800 1133 1674 690 2149 Origin(s) flaps retr unrch dests AS AS Description 2149:174 2604 769 769 178 174 NYSERNET-AS Count Path 709 2603 1805 1800 174 47 2603 1805 1800 1239 1280 174 118 2603 1805 1800 690 174 158 2603 1755 1800 174 18 2603 1755 1800 1239 1280 174 55 2603 3224 2043 1128 1133 1800 174 42 2603 1805 1800 701 690 174 10 2603 1755 1800 701 690 174 13 2603 1755 1800 690 174 4 2603 1805 1800 1133 690 174 Origin(s) flaps retr unrch dests AS AS Description 2551 2354 318 318 174 2551 NETCOM-AS Count Path 708 2603 1805 1800 690 1321 2551 886 2603 1805 1800 2551 158 2603 1805 1800 701 690 2551 91 2603 1805 1800 690 2551 22 2603 1805 1800 701 690 1321 2551 93 2603 1755 1800 2551 30 2603 3224 2043 1128 1133 2551 20 2603 3224 2043 1128 1133 1800 2551 9 2603 1805 1800 1239 1280 2551 3 2603 1755 1800 701 690 2551 7 2603 1755 1800 690 1321 2551 2 2603 1755 1800 690 2551 3 2603 1805 1800 1133 690 1321 2551 3 2603 1755 1800 701 690 1321 2551 1 2603 1755 1800 1239 1280 2551 Origin(s) flaps retr unrch dests AS AS Description 372:297 2342 333 333 109 372 NSN-AMES-AS Count Path 163 2603 1805 1800 1133 690 372 1067 2603 1805 1800 690 372 407 2603 1805 1800 701 690 372 66 2603 1755 1800 701 690 372 51 2603 1755 1800 1133 690 372 64 2603 1755 1800 690 372 Origin(s) flaps retr unrch dests AS AS Description 372:297 2342 333 333 109 297 NSN-UMD-AS Count Path 182 2603 1805 1800 690 297 3 2603 1805 1800 1133 690 297 3 2603 1755 1800 690 297 3 2603 1805 1800 701 690 297 Origin(s) flaps retr unrch dests AS AS Description 297:1221 2168 337 346 95 1221 AARNET-AS Count Path 567 2603 1805 1800 690 372 1221 253 2603 1805 1800 701 690 372 1221 97 2603 1805 1800 1133 690 372 1221 1 2603 1755 1800 701 690 372 1221 Origin(s) flaps retr unrch dests AS AS Description 297:1221 2168 337 346 95 297 NSN-UMD-AS Count Path 93 2603 1755 1800 690 297 189 2603 1805 1800 690 297 249 2603 1805 1800 701 690 297 221 2603 1805 1800 1133 690 297 161 2603 1755 1800 701 690 297 Origin(s) flaps retr unrch dests AS AS Description 372 2024 393 392 311 372 NSN-AMES-AS Count Path 315 2603 1805 1800 1133 690 372 515 2603 1755 1800 690 372 614 2603 1805 1800 690 372 136 2603 1805 1800 701 690 372 50 2603 1755 1800 701 690 372 1 2603 1755 1800 1133 690 372 Origin(s) flaps retr unrch dests AS AS Description 701 1664 525 525 170 701 AlterNet Count Path 20 2603 1805 1800 1239 1280 701 590 2603 1805 1800 701 47 2603 3224 2043 1128 1133 1800 701 102 2603 1755 1800 701 59 2603 3224 2043 1128 1133 701 283 2603 1805 1800 690 701 39 2603 1805 1800 1133 1674 690 701 Origin(s) flaps retr unrch dests AS AS Description 813 1589 617 615 83 813 CANET-ASN Count Path 588 2603 1805 1800 701 813 232 2603 1755 1800 701 813 39 2603 1805 1800 1239 1280 701 813 47 2603 3224 2043 1128 1133 1800 701 813 48 2603 3224 2043 1128 1133 701 813 10 2603 1805 1800 690 701 813 8 2603 1755 1800 1239 1280 701 813 1 2603 1755 1800 690 701 813 2 2603 1805 1800 1133 690 701 813 Origin(s) flaps retr unrch dests AS AS Description 2915 1488 391 391 83 2915 SPIN-NET Count Path 121 2603 1805 1800 1133 690 2149 174 2713 2915 121 2603 1755 1800 1133 690 2149 174 2713 2915 375 2603 1755 1800 174 2713 2915 379 2603 1805 1800 174 2713 2915 12 2603 3224 2043 1128 1133 1800 174 2713 2915 44 2603 1805 1800 690 2149 174 2713 2915 39 2603 1805 1800 701 690 2149 174 2713 2915 1 2603 1805 1800 1239 1280 174 2713 2915 2 2603 1755 1800 1239 1280 174 2713 2915 3 2603 1755 1800 690 2149 174 2713 2915