Hi Cynthia I understand your concerns but feel I need to clarify the points you made. Most people don't understand the internal workings of the database. That is not surprising as it is quite complex. I spent 14 years with the RIPE NCC as a holistic engineer and analyst working with the database. I know it inside out from all possible angles. I wrote the spec for historical queries in 2013. We had no requirements, it was just an idea that someone thought might be useful. To get something up and running quickly I went for the low hanging fruit. Because of the way objects are managed internally within the database, stopping at the most recent deletion point for an object history was simpler. It was a completely arbitrary, technical constraint. It had no privacy reasoning behind it. At the time we released this service in 2013, it was reviewed by the legal team and they had not asked for this constraint for any reason. (The legal team is currently doing a new review of the issue.) So lets look at exactly what this constraint means. Currently we provide some history of operational and corporate data objects. These objects are all heavily redacted to remove anything that is considered to be personal data. We do not provide any history of objects that are considered to be mostly personal or security data. Any object can have multiple versions and each version can have multiple instances. An object may be created (v1), updated several times, deleted, re-created (v2), updated, deleted, re-created (v3) and updated again. The arbitrary constraint means only v3 data will be available with all it's updates. None of the data for v1 and v2 is made available. By dropping this arbitrary constraint all the history of this object, v1, v2 and v3, will be available. All the data for v1 and v2 will be redacted in the same way as v3 to remove any personal data. There is no difference in the data for v1, v2 and v3. It is all operational and corporate data with personal data removed. As we have generally accepted that it is ok to release the historical, non personal, data for v3, it should also be ok to release the historical, non personal, data for v1 and v2. If there is any privacy concerns over the v1, v2 data, then we would have exactly the same privacy concerns over the v3 data that we currently provide. This technical change does not change the object types that we allow historical data to be queried for. Nor does it change any of the attribute types within those objects that we redact. We simply provide a complete set rather than a partial set of the operational and corporate data we currently offer. Lets look at some examples of what this means. Many objects have never been deleted. Some of these have been around since this version of the RIPE Database model was released in 2001. If you query for the history of these objects you will get the full 21 years of history. Many assignments are given out to end users, then deleted and re-assigned to another end user. This can happen multiple times. This prefix will have many versions v1...vn. Only the history of vn is currently available, and only if this version still exists in the 'live' database. For many researchers of address space usage or routing issues and abuse investigators and brokers, the history beyond vn is useful data. Also when a resource is transferred or consolidated the allocation object is deleted and re-created by the RIPE NCC. So for an object which last week you could see 21 years of history, which has just been transferred, you will only see 1 week of history now. There is also occasionally a case of an operator who accidentally deletes the wrong assignment and does not have an up-to-date copy of the object as it was just before it was deleted. They currently have to ask the RIPE NCC to supply them with the most recent version. Without this arbitrary constraint they could just recreate it, then look up the history themselves. There is also the issue of NRTM. Some users have been operating an NRTM stream for years, even decades. They have the un-redacted versions of the history of all the resource objects since they started streaming the updates. Redacting personal data only started a few years ago. In many cases they have the full history, regardless of how many times the object has been deleted and re-created. Anyone who starts an NRTM stream now will start to build up their own history of redacted, operational data objects that will remain in their own database after objects are deleted and re-created. That is an accepted practice. This constraint really is totally arbitrary. It is rooted in technical expedience in getting the service started. It has nothing to do with privacy. There really is no good reason to keep this arbitrary constraint. Either we provide historical operational data or we don't. Offering a partial data set based on random events makes no sense. cheers denis co-chair DB-WG On Mon, 4 Apr 2022 at 23:33, Cynthia Revström <me@cynthia.re> wrote:

Hi,

I am sorry for the delayed response but I object to removing that constraint.

It feels problematic to me from a privacy perspective, and it feels like the last deletion point is a fair balance between providing useful info and not providing too much info.

I don't think this restriction should be removed, at least not without a very good reason to do so.

-Cynthia

On Thu, Mar 31, 2022 at 1:32 PM denis walker via db-wg <db-wg@ripe.net> wrote:

...

Colleagues

When I wrote the spec for historical queries, almost 10 years ago, I included an arbitrary constraint to only show history back to the most recent deletion point. This was to get something in production quickly and see how useful it was. Over the years several people have asked for this arbitrary constraint to be removed. No one has objected to removing it. The co-chairs therefore recommend that this arbitrary constraint be removed. The co-chairs now ask the RIPE NCC to produce an impact analysis and implementation plan to remove it. We will then seek a final approval from the community on the plan.

cheers denis co-chair DB-WG

--

To unsubscribe from this mailing list, get a password reminder, or change your subscription options, please visit: https://lists.ripe.net/mailman/listinfo/db-wg

Colleagues Although I am replying to your email Cynthia, my comments are generally expressed to all. Let me try again to explain history. I still think there is a misunderstanding about what the RIPE Database is and what it means to publish information in this database. This is a public database. It is available globally to anyone who has an internet connection. Once you publish any information in this database it is public data. The full details of that data will be downloaded and copied by many people. This has almost certainly been done for many years if not decades. There may well be some people out there who have more historical data than the RIPE NCC holds. Anyone who is concerned about privacy should not allow their personal data to be published in this database. Once it is published it is too late to worry about privacy. It is already out there, it is public and it has been copied. You cannot take it back. You may have the right to be forgotten, but you don't have the means to be forgotten once you have broadcast your personal details in public. That is the simple reality of the internet. So what data might some people have? I am not going to explain the full technical details of how to do this, for obvious security reasons. But there are at least two ways that anyone can create their own private copy of the whole RIPE Database including full details of every update and including all personal data. It is quite likely that some people have been doing this for the last 20+ years. Even if you haven't done this in the past, you could start doing it today and create a private copy with all future updates. So any object that is deleted tomorrow will still exist in your private database. The RIPE Database has no privacy protection for personal data once it has been published. It is a PUBLIC database accessible by everyone. This is one of the many reasons why the RIPE Database should not contain any personal data. So back to historical queries. By providing this service for operational data going back to the start of the database, without any arbitrary restrictions, we remove the need for most people to go to the trouble of building up their own local, private database with a full copy of the data. By restricting access to past operational data on some arbitrary condition we actually monetise the data that some people already hold. We could create a market for some people to sell that data. Just like IPv4, where there is a demand for something, with few options for supply, some people will make money from it. So again I will say that there is no valid reason to restrict access to full operational data through an official interface that attempts to protect personal information. cheers denis co-chair DB-WG On Sat, 16 Apr 2022 at 18:39, Cynthia Revström <me@cynthia.re> wrote:

Hi,

Sorry for the late reply, I didn't see that you had replied.

The reason that I think the last deletion should be the cut-off point is that it feels quite natural, you deleted the object, it should no longer exist.

Also given how a decent number of individuals now have resources from the NCC (either directly or through a sponsoring LIR) I feel like we should consider if organisation objects should really have history. This feels even more important given how full legal names are required (afaik), which will then be published.

This might be a separate discussion but I think it is important to consider the pros and cons of this.

-Cynthia

On Tue, Apr 5, 2022 at 10:50 AM denis walker <ripedenis@gmail.com> wrote:

...

Hi Cynthia

I understand your concerns but feel I need to clarify the points you made. Most people don't understand the internal workings of the database. That is not surprising as it is quite complex. I spent 14 years with the RIPE NCC as a holistic engineer and analyst working with the database. I know it inside out from all possible angles. I wrote the spec for historical queries in 2013. We had no requirements, it was just an idea that someone thought might be useful. To get something up and running quickly I went for the low hanging fruit. Because of the way objects are managed internally within the database, stopping at the most recent deletion point for an object history was simpler. It was a completely arbitrary, technical constraint. It had no privacy reasoning behind it. At the time we released this service in 2013, it was reviewed by the legal team and they had not asked for this constraint for any reason. (The legal team is currently doing a new review of the issue.)

So lets look at exactly what this constraint means. Currently we provide some history of operational and corporate data objects. These objects are all heavily redacted to remove anything that is considered to be personal data. We do not provide any history of objects that are considered to be mostly personal or security data. Any object can have multiple versions and each version can have multiple instances. An object may be created (v1), updated several times, deleted, re-created (v2), updated, deleted, re-created (v3) and updated again. The arbitrary constraint means only v3 data will be available with all it's updates. None of the data for v1 and v2 is made available. By dropping this arbitrary constraint all the history of this object, v1, v2 and v3, will be available. All the data for v1 and v2 will be redacted in the same way as v3 to remove any personal data. There is no difference in the data for v1, v2 and v3. It is all operational and corporate data with personal data removed. As we have generally accepted that it is ok to release the historical, non personal, data for v3, it should also be ok to release the historical, non personal, data for v1 and v2. If there is any privacy concerns over the v1, v2 data, then we would have exactly the same privacy concerns over the v3 data that we currently provide.

This technical change does not change the object types that we allow historical data to be queried for. Nor does it change any of the attribute types within those objects that we redact. We simply provide a complete set rather than a partial set of the operational and corporate data we currently offer.

Lets look at some examples of what this means. Many objects have never been deleted. Some of these have been around since this version of the RIPE Database model was released in 2001. If you query for the history of these objects you will get the full 21 years of history. Many assignments are given out to end users, then deleted and re-assigned to another end user. This can happen multiple times. This prefix will have many versions v1...vn. Only the history of vn is currently available, and only if this version still exists in the 'live' database. For many researchers of address space usage or routing issues and abuse investigators and brokers, the history beyond vn is useful data. Also when a resource is transferred or consolidated the allocation object is deleted and re-created by the RIPE NCC. So for an object which last week you could see 21 years of history, which has just been transferred, you will only see 1 week of history now. There is also occasionally a case of an operator who accidentally deletes the wrong assignment and does not have an up-to-date copy of the object as it was just before it was deleted. They currently have to ask the RIPE NCC to supply them with the most recent version. Without this arbitrary constraint they could just recreate it, then look up the history themselves.

There is also the issue of NRTM. Some users have been operating an NRTM stream for years, even decades. They have the un-redacted versions of the history of all the resource objects since they started streaming the updates. Redacting personal data only started a few years ago. In many cases they have the full history, regardless of how many times the object has been deleted and re-created. Anyone who starts an NRTM stream now will start to build up their own history of redacted, operational data objects that will remain in their own database after objects are deleted and re-created. That is an accepted practice.

This constraint really is totally arbitrary. It is rooted in technical expedience in getting the service started. It has nothing to do with privacy. There really is no good reason to keep this arbitrary constraint. Either we provide historical operational data or we don't. Offering a partial data set based on random events makes no sense.

cheers denis co-chair DB-WG

On Mon, 4 Apr 2022 at 23:33, Cynthia Revström <me@cynthia.re> wrote:

...

Hi,

I am sorry for the delayed response but I object to removing that constraint.

It feels problematic to me from a privacy perspective, and it feels like the last deletion point is a fair balance between providing useful info and not providing too much info.

I don't think this restriction should be removed, at least not without a very good reason to do so.

-Cynthia

On Thu, Mar 31, 2022 at 1:32 PM denis walker via db-wg <db-wg@ripe.net> wrote:

...

Colleagues

When I wrote the spec for historical queries, almost 10 years ago, I included an arbitrary constraint to only show history back to the most recent deletion point. This was to get something in production quickly and see how useful it was. Over the years several people have asked for this arbitrary constraint to be removed. No one has objected to removing it. The co-chairs therefore recommend that this arbitrary constraint be removed. The co-chairs now ask the RIPE NCC to produce an impact analysis and implementation plan to remove it. We will then seek a final approval from the community on the plan.

cheers denis co-chair DB-WG

--

To unsubscribe from this mailing list, get a password reminder, or change your subscription options, please visit: https://lists.ripe.net/mailman/listinfo/db-wg

Hi Sebastian Can we clarify what historical data you are referring to? Are you suggesting that absolutely no historical data should ever be made available from the RIPE Database? Do you think it is wrong to provide details of what organisation held a specific address block last year or the year before? History of personal data held in the database is not available. Personal data contained in operational object types is removed from the data returned for a historical query. As far as possible only historical operational (non personal) data is available through the historical queries. cheers denis co-chair DB-WG On Wed, 18 May 2022 at 10:05, Sebastian Wiesinger via db-wg <db-wg@ripe.net> wrote:

* denis walker via db-wg <db-wg@ripe.net> [2022-04-22 01:33]:

...

Colleagues

Although I am replying to your email Cynthia, my comments are generally expressed to all. Let me try again to explain history. I still think there is a misunderstanding about what the RIPE Database is and what it means to publish information in this database. This is a public database. It is available globally to anyone who has an internet connection. Once you publish any information in this database it is public data. The full details of that data will be downloaded and copied by many people. This has almost certainly been done for many years if not decades. There may well be some people out there who have more historical data than the RIPE NCC holds. Anyone who is concerned about privacy should not allow their personal data to be published in this database. Once it is published it is too late to worry about privacy. It is already out there, it is public and it has been copied. You cannot take it back. You may have the right to be forgotten, but you don't have the means to be forgotten once you have broadcast your personal details in public. That is the simple reality of the internet.

I think Cynthia and most people on this mailinglist understand what the RIPE database is. Even though you might have entered private information in the past it does not mean the database should publish these records for ever after they are deleted. You HAVE the right to have your information removed from the RIPE database. It's completely irrelevant where else that data may have been copied to.

Of course the information might be copied and downloaded by many persons but that is no reason to make it easily available to do so after this information is deleted.

The database is not "the Internet" and I find this argument highly fatalistic. So even when there are places on "the Internet" where this data still exists, it doesn't have to exist in the RIPE DB, readily available.

...

So back to historical queries. By providing this service for operational data going back to the start of the database, without any arbitrary restrictions, we remove the need for most people to go to the trouble of building up their own local, private database with a full copy of the data. By restricting access to past operational data on some arbitrary condition we actually monetise the data that some people already hold. We could create a market for some people to sell that data. Just like IPv4, where there is a demand for something, with few options for supply, some people will make money from it.

I don't agree with this argument. "We better give people full access to the data or bad people will make money of it." Why? Why would people pay money to get old data from the RIPE DB. Also this would be a black market as it would violate privacy laws. And there are way better data sources already available than the RIPE DB data.

Best Regards

Sebastian

-- GPG Key: 0x58A2D94A93A0B9CE (F4F6 B1A3 866B 26E9 450A 9D82 58A2 D94A 93A0 B9CE) 'Are you Death?' ... IT'S THE SCYTHE, ISN'T IT? PEOPLE ALWAYS NOTICE THE SCYTHE. -- Terry Pratchett, The Fifth Elephant

--

To unsubscribe from this mailing list, get a password reminder, or change your subscription options, please visit: https://lists.ripe.net/mailman/listinfo/db-wg

I can't think of a good reason to ever have history for attributes like descr and remarks. And I totally agree with Sebastian that just because the data might have been copied by others on the internet doesn't mean that the NCC should keep redistributing it. -Cynthia On Wed, May 18, 2022, 11:42 Sebastian Wiesinger via db-wg <db-wg@ripe.net> wrote:

* denis walker via db-wg <db-wg@ripe.net> [2022-05-18 11:01]:

...

Hi Sebastian

Can we clarify what historical data you are referring to? Are you suggesting that absolutely no historical data should ever be made available from the RIPE Database? Do you think it is wrong to provide details of what organisation held a specific address block last year or the year before?

History of personal data held in the database is not available. Personal data contained in operational object types is removed from the data returned for a historical query. As far as possible only historical operational (non personal) data is available through the historical queries.

I think there should at most only be the lastest data be available for deleted objects or none at all. There are fields like descr or remarks that can contain information that should no longer be present in the database after the object is deleted.

In general I'm for removing information from a public database after it is supposed to be deleted. I'm not sure who is using information for deleted objects for anything, I at least never have.

If there is a valid request for data on deleted objects it could always be handled by the NCC.

Best Regards

Sebastian

-- GPG Key: 0x58A2D94A93A0B9CE (F4F6 B1A3 866B 26E9 450A 9D82 58A2 D94A 93A0 B9CE) 'Are you Death?' ... IT'S THE SCYTHE, ISN'T IT? PEOPLE ALWAYS NOTICE THE SCYTHE. -- Terry Pratchett, The Fifth Elephant

--

To unsubscribe from this mailing list, get a password reminder, or change your subscription options, please visit: https://lists.ripe.net/mailman/listinfo/db-wg