Attached please find the 2nd draft for the DB-WG minutes (v4). A big thank you to Shane for taking notes, and to Markos, Holger and Carsten for submitting corrections regarding missing names. We are still working on collecting the URLs for the presentations, and on aggregating the open actions - before putting the minutes up on the web as well. Comments and corrections still welcome - in particular if you feel mis-quoted or represented as name??. Thank you, regards, Wilfried. _________________________________:_____________________________________ Wilfried Woeber : e-mail: Woeber@CC.UniVie.ac.at UniVie Computer Center - ACOnet : Tel: +43 1 4277 - 140 33 Universitaetsstrasse 7 : Fax: +43 1 4277 - 9 140 A-1010 Vienna, Austria, Europe : RIPE-DB: WW144, PGP keyID 0xF0ACB369 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~:~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Draft Minutes, Database Working Group v4, 08-May-2001 RIPE39 @ Bologna A. Admin stuff - list of participants, scribe Scribe: Shane Kerr - agenda bashing Draft agenda accepted as final - minutes approved - content-wise, action on Wilfried and webmaster@ripe.net to insert links to presentations B. Actions From previous meetings - On Joao to present "unique handle proposal" @ ARIN meeting Joao: still needs more work, so it wasn't presented. Randy Bush is going to submit an IETF draft. Wilfried: can the draft be submitted to DB-WG mailing list? Joao: still needs some brush up... Wilfried: can you keep track of this? Joao: yes, Action on Joao to track development of draft and circulate the draft (or a pointer to the draft) on the DB-WG mailing list - On NCC DB Group (Joao) implement "lir-allocated" attribute fo inetnum objects Wilfried: still on the to-do-list, for obvious reasons (DB software reimplementation and RPSL transition) - On Wilfried Start discussion about legacy object migration on the list [ on the agenda ] - On Carsten Specify the "identity override" [ on the agenda ] C. DB Operations Update (Andrei R., RIPE NCC) "RIPE Database Operations Update" Andrei Robachevsky <andrei@ripe.net> Outline - Whois Service before migration - Whois Service after the migration - Related service RIPE Whois Service - Database Statistics - Database Operations - <ripe-dbm@ripe.net> 3.8 million objects [ graph ] sharp decrease caused by migration of German domains since then # objects grows slight slowdown caused by Austrian and Italian domain migration Distribution by object type [ chart ] Most objects were domain objects Now most objects are person objects (about 85%) Data protection [ graph ] Small number of unmaintained aut-nums Still maintained, but legacy objects that noone can change directly [ graph ] Person object protection - almost 30% unmaintained Wilfried: what about percentage? A: don't know what happened at RIPE 37 to cause numbers so low Wilfried: wasn't there a time when DENIC was requested to withdraw their maintainers? Ulf: needed to protect person objects to prevent someone else from changing them Andrei: *increase* was caused by German migration Ulf: DENIC has CRYPT-PW that every German ISP is supposed to know [ graph ] PGP deployment 2% of objects are PGP protected Domain migration - Domain objects migrated .at February 2001 .it April 2001 - Top 5 529956 .arpa 34334 .sk 11791 .hu 11132 .lt 8682 .lv - CENTR migration information Unreferenced person objects grow to 88% 28 June 2000 only 12% (before German domain migration) Today 88% Two factors: ccTLD's still use RIPE registry as nic-hdl generator a lot of person objects were left behind Personal Data in the DB 3,2 M person objects 66% unreferenced and maintained 22% unreferenced and unmaintained 12% referenced Legal consequences (details in Joao's presentation) Operations - Performance - Queries - Updates - Uptime Queries About 8/sec average [ graph ] Spike in March for Italian domains Spike in January for Austrian domains Needed to find all references Distribution of Queries 48% domains (before domain migration) 32% domains (after domain migration - almost half now IP numbers though) Updates About 5/min average [ graph ] Min/Max values since RIPE-38 Queries: 13 q/sec max, 6 q/sec min Updates: 50427/day max, 10/day min (caused by server slowdown) Update rate by object type [ chart ] Was 54% person [ chart ] Now 65% person after migration 56% are new persons Downtime and Actions Taken - No downtime - Main server performance slowdown 9 March hardware was upgraded to cope with load - Migration 2001-04-23 Deliberate query/update outage for 6 hours Communication with Database Users - Database status http://www.ripe.net/ripencc/pub-services/db/DBstatus.html - Database operational graphs http://www.ripe.net/ripencc/pub-services/db/mrtg/whois.html - FAQ http://www.ripe.net/ripencc/faq/database/ - RIPE-DBM RIPE DBM The Human Face of the Database - Engin Gündüz - Magnus Karlsson - Shane Kerr - Ambrose Magee - Denis Walker Magnus and Denis are new. Unfortunately Ambrose decided to leave RIPE NCC - May is his last month. RIPE-DBM - Activities - User support <ripe-dbm@ripe.net> - DB usage monitoring - AUP - NRTM - DB Server Performance Monitoring - Mntner object requests - PGP license requests - proposal to discontinue this service two reasons: 1. licenses are for (old) PGPi version with known bugs 2. new database uses GnuPG, which is open source and compatible <ripe-dbm@ripe.net> Typical week 1700 messages - maintainer creation requests (25-36/week) - PGP license requests (1-2/week) - general questions (50/week) etc. lots of country-to-ip mapping questions Whois Service After the Migration - Migration timeline - New Server Operatoipn - Presentation given at Joint DB/Routing WG session: http://www.ripe.net/ripe/meetings/current/ripe-39/presentations.html Migration Timeline - 2 April, ripe-db 3.0 release - 12 April, Advised Mirror switchover 4 servers mirror the new database - 19 April, Migration of the TEST database Dress rehersal of the migration - 23 April, Migration of the RIPE database Migratiopn night that lasted 12 hours 8-page migration manual no real problems - 14 May, default updates switch to RPSL - 15 October, RIPE-181 updates will no longer be accepted New Version of the RIPE Database - Supports RPSL (RFC2622) - extended syntax (for all object types) - new objects and attributes - Supports RPSS (RFC2725) - new authorization rules - Supports RAToolSet (4.7.0) RtConfig -protocol ripe - Code is completely rewritten Server Architecture [ diagram ] Queries per second weeks 14-17 No surprises, even though there were no surprises! Slight decrease, but no explanation [ graph ] Updates per minutes weeks 14-17 Again, no surprises [ graph ] Migration TODO list - still need to resolve minor issues - publish consistent documentation - make ripe-dbase-3.0.1 release - start next generation of DB consistency project - host and support development of RAToolSet Related RIPE Database Services - Database consistency project - Will be postponed until autumn when new version is available - Routing Reality Check - Not much progress since last RIPE meeting - TEST database - now in RPSL format - <test-dbm@ripe.net>, RIPE-181 format - <test-rpsl@ripe.net>, RPSL format - test-whois.ripe.net, port 43 Summary - The database was successfully migrated to v3.0 - domain migration is almost complete - still 2.5M unreferenced person objects left - things to do - DB consistency project - RRCC project - ripe-dbase-3.1.0 release - DBToolSet (incl. RAToolSet) Questions? Holger Münx (UUnet DE): as a multinational, really concerned about the RFC2725 issue. Cannot use it because some objects are in other registries, but they are still in the RIPE database, and security not enabled. Because they have to go to other objects as well, it would be really helpful to have mirroring between the registries. Andrei: we have a workaround to create foreign object - but you're right that security is low another problem is that even if we have a perfect solution, other registries don't implement RFC2725 (yet), so we can't guarantee security, so this depends on collaboration with other registries - we can't do it alone Holger: is this high priority? Andrei: we think this is very important - and it breaks the advantage of RFC2725. We're going to discuss this issue in more detail at NANOG, and probably we can find some kind of solution. Holger: it would be helpful if you could post some progress to the database working group list APNIC: going to migrate to the database (RIPE v3 code), but about 6 to 9 months behind the RIPE NCC Wilfried: does it make a difference to have the same software deployed in two registries? Joao: the software is not relavent, but right now the new RIPE database software is the only code that implements RFC725. We could solve certain authentication problems, but we would still have one problem with ARIN's authentication/authorization model, which is quite different. ** Wilfried: I'm going to put this on the plate as the Next Big Project. The NCC should find out what the problems are, so we can start working on it. ** Gert Döring from spacenet: Would like to see some sort of synchronous update mechnanism for the database. It might be useful for others as well. Andrei: we appreciate this kind of input. This is on the list for "DBToolSet", so maybe this kind of thing can be done there. Wilfried: thanks to RIPE NCC Andrei: if you have problems, contact us Wilfried's short list of questions: - How many have digital signature technology available everyday? less than half answered - How many use commercial PGP implementation? some - How many use GPG? less - Who would be interested in a workshop or training in PGP? some We'll try to organize some sort of BOF or training for RIPE40 Ulf: who is still using MAIL-FROM? [ laughter ] D. Report from the ReImp/Mig TF (Ulf K., and NCC DB Group) Ulf Kieber: Thanks to RIPE NCC for smooth transition! Basically a short revamp of presentation yesterday 4 different types of problems PGP Update problems - took about 1 day to resolve - 181 translation path does not process PGP correctly Acknowledgement problems - no ack for some updates Conversion problems - minor indentation issues If you have had problems with some updates, please double-check and/or resend updates. NCC should probably discard old update messages, as re-processing them might rol back objects to an older state Wilfried: proposes to close down task force, agreed. E. "Orphaned" person object (Joao) How many attendees are LIR? A fair number Problem with several million unreferenced person objects is that we have a potential legal problem. Certain European Union directives about what you can store in databases. Basically you should not store information about anyone that is not required and directly related to your operations. Not only do we do this, but it is publicly accessible. It's not necessary. There for historical reasons, but not directly related to current operations. This is becoming a big problem. Afraid that the RIPE NCC will be held liable for holding information that it shouldn't. Is this something we should try to fix? If so, when? How? Ulf: internal attribute that marks with timestamp "unused since", after a certain period of time purge the object Joao: but do people think that we want to proceed with deleting these objects? Ulf: usually maintainer objects, and this person would be held responsible ??1: DE registry is planning on deleting all of ours, and they should be deleted. The ccTLD domain registries should pull their person objects. Carsten Schiefner: Kind of "echo" of DE domains, should be made clear that this status is a transitional state. A quick fix would not be very helpful. Wilfried: how do we overlap between unreferenced but maintained and those necessary for DENIC? Joao: let's not just concentrate on DENIC Andrei: 600000 by DENIC, 400000 by (next highest) Marcos Sanz: DENIC does not accept registration if person object is unmaintained. Joao: problem is not that it is maintained or not, but rather that it is there and not related to other objects in the database. If DENIC is a problem, should we put a deadline? In the meantime, we need to solve the problem. ??4: If person object is unreferenced, we have to remove them ASAP. Send a note to the maintainer? Carsten: RIPE NCC should be aware of the problem. Has the RIPE NCC already been the target of data protection authorities? Is this really a problem now or is it just to be watched carefully? Joao: not worried about authorities, but rather individuals complaining Ulf: need an automatic timeout ???: As soon as object remains unreferenced, delete it. Wilfried: concensus that should only contain data related to the purpose of the database. We have watched the domain name thing for quite a while, (setting a deadline) is helpful. Propose to delete unreferenced objects by June 1, and inform maintainers beforehand. If DENIC (or whoever) can provide documentation of requirement, then RIPE NCC should be safe. Andrei: are we trying to solve only DENIC problem? Joao: no, no, no. We need a clear statement for ALL sources of information like this. Andrei: although DENIC contains most numbers, when DENIC removes objects, we don't expect complaints. For *other* objects, we DO expect complaints. ** Wilfried: put an action on the NCC about what we intend to do -> Joao will generate problem statement, proposals, and target date of June 1. <- ??5: proposes to send e-mail to people, like ARIN did it Ray Plzk (ARIN): we did this, then deleted ???: not useful for generic case where objects are maintained. Do we want to handle the 0.5 million questions about this, and how? F. Enhancements to IRRd to support SQL, Jake K. and Karen W. "A Relational DB Approach to the IRR" Jake Khuon, Global Crossing Gerald Winters, Global Crossing Karen Williams, TimesTen "Yet Another Relational DB Approach to the IRR" :) Gerald did most of coding at Global Crossing Goals - Replacement of the radix tree data structures used by IRRd to store objects with relational tables and an SQL backend database. - Maintain comparable performance Motivations - Flexible query - "Power" of SQL - Can make development of tools simpler - Flexible development platform - hard coded data structures required some hard coded query routines - can make extensions to the database simpler - Some data best represented as relational tables Concerns/Requirements - Performance - Need to compete with speed of in-memory radix structures - Dual datastore philosophy - "Fast path" (IRRd native radix) - "Slow path" (SQL tables) - Some data would have to exist in both - Synchronization problems - Path decision maker routine would add latency - SQL tables only philosophy - Single datastore: data exists in only one place - Would need to be in-memory for speed - Scalability - Needs to handle large number of objects (3+ million) - Needs to model multiple registries - Robustness - Transaction logging - replication System Characteristics - IRRd is now a query/update core server - Both RPSL and RIPE-181 compliant will not handle mixed-datatypes - all in a source must be alike - SQL pass-thru interface - Service front-ends implemented as modules (e.g. email) - TimesTen is the database manager. - No IRRd indexing - A 100% TimesTen relational DB backend What is TimesTen? - An in-memory relational database - 1000% faster than disk-based databases - Standard database features: - SQL2 language and full SQL transaction semantics - Fully durable (always two database copies on disk) - Replication for high availability - Thread-safe General Schema Design - Keep a copy of each object referenced by unique 'ID' Misc_Tbl - All the data to search on kept in auxiliary index tables Pre-Compiled vs. User-Defined Queries - Use pre-compiled queries where feasible for increased performance - IRRd ! queries - ripewhois queries - Allow the user to define queries (SQL select) - select * from mntner_tbl where ... Query Path Overview - Pre-compiled path already bound (msec faster) [ no number brought though ] Whois Query Processing - Use knowledge of object types to speed up lookups by reducing the number of object tabes that must be searched e.g. "rs-..." can only be a route set object - Need radix tree '1-level-less-specific' search capability to support whois queries Radix Lookups - four types - 1 level less specific - all levels less specific - 1 level more specific - all levels more specific - use numeric representation suitable for relational tables first, last, and middle address Radix Lookup Examples - select route from Route_Tbl where FA >= FAr and LA >= LAr and source = '..' Whois Query Processing (continued) - Need secondary indexing to support inverse lookups and person lookups - build an inverse table as index into main table Joao: Database is 1000% faster than disk-based? What would comparison look like if using a RAMdisk or a big cache? Karen: we compare with full caching, we know where our data is. We're still 10x faster! I've seen between 20 and 40x faster at customer sites. Joao: How to implement more or less specific queries. We use a special index external to database. What are the advantages of using full relational model? Jake: When doing updates - with a radix tree, you need to synchronize tree and databases. This was simpler. Cengiz: More specific required radix tree, less specific was no problem. How do you do more specific? Most of the time I just want the keys. Cengiz: Request to implement more specific queries. Antony Antony: to x10, how much memory for 3 million objects Karen: less than 0.5 Gbyte in memory G. Something like a "force:" (or force-delete) attribute. Unable to change a person's *name* on a person object. If not referenced, it is easy to delete, but otherwise have to go through RIPE-DBM mailbox. Want a "force" attribute to specify "I really know what I'm doing" when changing a person name. Is this on the "wish-list" or not? ???: Pretty important to have a way to change a name. Why do we need a special attribute? While I like the attribute, perhaps it is the wrong way. Should just do it! Joao: internally currently not possible. Still some objects with references by name exist, and this prevents it. Something like 4000 objects with references by name. If we can clean up these objects, we can release this artificial constraints. David Kessens: thought was that a typo in the nic-hdl could overwrite the information in a person object Wilfried: if reference by name, change contact to to all nic-hdl list Joao: if by name, kind of undefined, but if by nic-hdl seems like an assignment ???: Add a remark if we do this! [Speaker]: Is this a priority? Santos: Allow to change name of person? If change initial shouldn't handle change as well? Wilfried: that's what we have now! Andrei: two issues: 1. reference by name and cleaning up is important and a good idea 2. releasing primary key constraint - concern is unintentional hijacking of person objects Also from RIPE-DBM point of view, don't see a lot of requests to change these kinds of typos. If really an issue, then we can do it, other wise we can have a more clear procedure. Ulf: 3 persons were left wrong because it was easier than sending to RIPE-DBM. If I could do it myself, then would do it. ???: really not something that should go to a human Joao: I actually support users doing it themselves. Would prefer fixing references and removing restrictions rather than adding a new feature. Wilfried: after unreferenced objects, how many are left? Carsten: many objects that reference person by name and not unique Joao: *all* name-based references left are non-unique Wilfried: replace name with however many handles are necessary, plus put in a remarks line Gert: force thing is not that bad - because of typos, just a gut feeling because I'll be the one to fix it H. Legacy Object Migration, Richard J. "Early Registration Record Transfers" Richard Jimmerson, ARIN Background 1980's - All IP and AS registrations made by entities under a US Department of Defense contract 1990's - APNIC and RIPE NCC began registering IP and AS numbers for their respective regions Current Situation - Many registrations not maintained by appropriate regional registry - Many organizations have to interact with more than one RIR to modify their registration records - Have difficulty making timely in-addr.arpa updates - The in-addr.arpa zone contains delegations longer than a /8 (/16's and /24's) Goals - Registration records will be maintained in the appropriate RIR database - Organizations will be able to: - interact with a single RIR - make more timely in-addr.arpa updates - in-addr.arpa zone will contain only /8 delegations Tasks - transfer all IPv4 registrations to appropriate RIR - transfer all AS registration to appropriate RIR - establish in-addr.arpa domains Maintaining in-addr.arpa - each RIR will maintain a suite of in-addr.arpa servers - APNIC/RIPE NCC have already deployed - ARIN is testing - non-shared /8 maintained by appropriate RIR - shared /8 maintained by majority record holder RIR coordination efforts - sample dump provided by ARIN in early 2000 - candidate list provided by ARIN 2000-08-22 - companion dump of all network data provided 2000-08-23 - updated/additional data provided 2001-03 "Pre-RIR address space" Mirjam Kuehne, RIPE NCC Related Items - Transfer of DB objects - technically easy - conflicts need to be resolved - Merger algorithm for reverse delegation - techincally easy - ARIN setting up reverse DNS server - start testing in Q2/2001 Current Issues - What determines transfer? - location of network - Conflicts in inetnum objects - Conflicts in person/role objects - What if contact person is not approachable - If in doubt, keep as is? Possible inetnum conflicts 1. Exact match in ARIN and RIPE DB (4325) - override one with the other Wilfried: what do you mean by exact match? Mirjam: address range exact match, not whole object 2. More specific network in RIPE DB (e.g. allocation in ARIN, assignements in RIPE) - keep both? 3. Minor mismatch - override one with the other (latest data?) - human intervention - contact maintainer 4. More specific network in ARIN DB (3060) Next Steps - publish list of objects to be transferred - operators encouraged to actively transfer data - clear procedure needed - try to resolve easy conflicts - try to alert all contacts affected by transfer - find out what operators need to prepare Joao: exact match - solution is override one with the other, but which one? we want suggestions! David Kessens: how much data? Joao: numbers in parenthesis David Kessens: contact those people first, case by case Mirjam: what if can't find? David: then leave as is Wilfried: could label space as "up for reclaim" (joke) David: might be a small group of people Wilfried: what about person objects? It could be messy. Ray Plzak: suggestion: for exact matches, make a synthesis and timestamp the effective data of the merger, and timestamp when the record was created in the ARIN database Ulf: first ask which database do I need to hold my objects Joao: now maintained in ARIN, will be in RIPE NCC Wilfried: got community start thinking about it - please keep us informed! Y. Input from other WG's (From Local-IR) Try to make database transactions more accessible and easy to understand Make it more transparent for the end users Nurani: we spend a lot of time answering questions on how to interact with the RIPE database suggestion of improving documentation perhaps also easier tools (web end or other tools) Wilfried: also not relying on NCC staff for easy-to-answer questions also, can we recruit a small team of individuals to subscribe to a db-help mailing list? Andrei: planning on publishing user manual Wilfried: database working group will take this task on before the next meeting See you again in Prague! ______________________________________________________________________