Hi Denis, Noted! I appreciate the historic context. Kind regards, Job On Wed, May 25, 2016 at 01:53:11PM +0000, ripedenis@yahoo.co.uk wrote:

Just to clarify, I agree there is a problem with the current implementation of showing history of operational data. At the time the decision not to show deleted data was arbitrary. As no history had been shown at all before a limited history was offered with the intention to review as more experience was gained on the usefulness of this information. My previous comments were intended as an improvement and clarification of the problem statement.

From: "ripedenis@yahoo.co.uk" <ripedenis@yahoo.co.uk> To: Job Snijders <job@instituut.net>; Database WG <db-wg@ripe.net> Sent: Wednesday, 25 May 2016, 15:31 Subject: Re: [db-wg] NWI-2 - displaying history for DB objects where available

Hi Job With the current transfer policy an address block may be split into 2 or more parts. As the address range is the primary key defining the object, each of these parts is a new object and none relate to the old parent object. The deleted parent object is currently not available as history as it has been deleted.

Also if an address range is reassigned it may have been deleted and recreated by the LIR. Although this still has the same primary key, internally it has a different unique object id within the database. The current history feature does not cross boundaries of object ids. So you can only see the history of an object back to the most recent deletion point.

So if anyone has a legitimate interest in who held an address range in the past you need to be able to see the history of deleted and recreated objects. The focus MUST only be on resource objects. You should never show the history of personal or security data. cheersdenis

From: Job Snijders <job@instituut.net> To: db-wg@ripe.net Sent: Wednesday, 25 May 2016, 14:57 Subject: [db-wg] NWI-2 - displaying history for DB objects where available

Dear Working Group,

This is the second Numbered Work Item.

(You can review https://www.ripe.net/ripe/mail/archives/db-wg/2016-April/005190.html to ensure you have an overview of the next steps.)

--------- NWI-2:

    The RIPE NCC was tasked with the following action point: AP69.2     [RIPE NCC] Come up with some straw man proposals for displaying     history for objects where available.

    The RIPE Database already provides a mechanism for object history.     In our understanding the problem statement that motivated the action     point is that some people are interested in seeing the history of     *deleted* objects. It is our understanding that this interest is     mainly focussed on primary (resource) objects.

    However, it is not entirely clear to the RIPE NCC why this is     useful. We also have some concerns, mainly non-technical, and     therefore we believe that the problem statement and motivation needs     to be better defined in the working group. ---------

The next step is to help refine the problem statement. I would appreciate it if working group participants with an interest in the history of objects speak up and elaborate on the issue.

Kind regards,

Job

Hi Sascha On 25/05/2016 22:48, Sascha Luck [ml] wrote:

On Wed, May 25, 2016 at 02:57:15PM +0200, Job Snijders wrote:

...

--------- NWI-2:

...

However, it is not entirely clear to the RIPE NCC why this is useful. We also have some concerns, mainly non-technical, and therefore we believe that the problem statement and motivation needs to be better defined in the working group.

Indeed, I should like to see some use cases that go beyond simple curiosity or "intelligence collection".

Also, how does this affect objects referred to by the "historical" objects? Does this mean that a deleted personal or organisational object will now have to be retrievable forever for historical purposes? Does a "right to be forgotten" exist for ripedb entries?

The current implementation does not show history of personal data. Apart from the legal issues, it is technically not possible to provide reliable personal history because of the way Nic-handles were reused in the past. You cannot query for a MNTNER object so history is also not available for those objects. Not sure if history of ORGANISATION objects is allowed now, but personally I don't think that should be available to anyone outside of the organisation. cheers denis

rgds, Sascha Luck

...

---------

The next step is to help refine the problem statement. I would appreciate it if working group participants with an interest in the history of objects speak up and elaborate on the issue.

Kind regards,

Job

Hi George On 11/07/2016 05:43, George Michaelson wrote:

(this may have been fixed, but this is how I understand things are today)

As currently implemented, the internal DB history of an object is tied to the fundamental DB object ID, which is (I believe) a monotonically rising/unique identifier.

correct

If an object is DEL and then ADD rather than UPD, the associated history of that object is disconnected: Not lost, it exists inside the RIPE SQL model but it has detached from the apparent resource, because the DEL/ADD sequence creates a new object with a new fundamental ID.

yes and no. DEL/ADD does create a new 'object' with a new (internal) object id (and other internal flags like version number are also reset). But these separate physical objects are still connected through the primary key. I would see this more as an event in the life of a logical object, identified by the pkey, rather than a disconnect.

I say fundamental ID, distinct from the apparent unique primary key. The primary key of an INETNUM is not what drives the history mechanism: its the specific object ID which is internal.

'mechanism' is an interesting choice of word here. If you construct a mechanism you can make it do whatever your requirements are. If you actually want to be driven by the internal object id then that is what you build. The current implementation of object history (unless it has changed recently) was arbitrarily built on this object id. You can only look at the history of a currently existing object. This object has an internal object id. You are presented with all the versions of the object that has this object id. So you get the history back to the most recent deletion/(re)creation point. But this was a 'first iteration' design constraint. If you want the full history of, say, an inetnum with a particular address range (the pkey of the object) the 'mechanism' can be built to traverse the internal object ids (deletion points) and follow the full trail of all objects with the same pkey. This is quite easy to do. The community just needs to agree on what they want.

This is one aspect of history-in-whois which is leading APNIC to consider a design for a JSON based approach, which presents the history of objects over time, irrespective of the ADD/DEL/ADD gaps.

This can be done with a very easy tweak to the current implementation.

I believe this needs to be part of your problem statement because as it stands, the history mechanism implies something about an object which may not be complete.

The mechanism does not imply anything. It is what it was designed to be (as stated above). If the community wants something different then specify different requirements.

Additionally, we are considering a model which ties history to the start-end pairs of a range, alongside start-end pairs of dates, This is because (unlike prefix/len) its reasonably simple to construct canonical lists of the entire set of objects which span a time-window, and have an end of range bigger than the start, and a start of range less than the end, which encompasses the overlaps, splits and joins of that range.

I believe the RIPE NCC may already have looked at this.

A large amount of the data is affected by changing referenced objects. As others have noted, the RIPE NCC region has a significant issue in current data privacy legislation which guides limits on what can be seen. Noting that, we feel that the use cases of the history probably do need to reflect the history of changes of associated person, role and maintainer objects. Luckily the DB internal history mechanism tracks this, so we think we can take our basic tabular approach for start-end date and range pairs, and augment it to make the complete (JSON) block of a resource, and the specific state of associated records at that time. Most of this is pre-computable data, which will occupy disk (or database) but is not inherently complex in itself. There may be a lot of repetition (denormalized) but disk (and within limits memory) is cheap.

I believe the RIPE NCC may already have looked at this as well. But let me add the caveat that privacy laws will prevent making any historical personal data available from the RIPE Database even if it is still held internally for the legitimate purposes of the database. However, a lot of 'information' can still be provided about related objects without breaking the privacy laws. But then we are into my favourite topic of the data model and services that provide interpreted information from the database rather than raw data that can be misleading. Services like RIPEstat for example. cheers denis

-George