proposal: new attribute 'geofeed:'
Dear DB-WG, Some colleagues are working to address the never-ending-story of 'where the heck are IPs geographically?'. This problem space has been brought up numerous times in the Database Working Group, but we never managed to solve it. As with all compsci problems adding a layer of indirection can help ;-) This current draft suggests overloading the RPSL 'remarks:' field with a structured attribute value, however I suspect we would do ourselves a disservice to overload a 'remarks:' field. Instead it would be better to add a 'geofeed:' attribute to the RPSL inetnum/inetnum6 class dictionary, which as value contains a URL with http or https scheme. The draft: https://tools.ietf.org/html/draft-ymbk-opsawg-finding-geofeeds The value of the attribute could be validated using something like "org.apache.commons.validator.UrlValidator", the attribute would look like this, only valid in the inetnum/inet6num: "geofeed: [optional] [single] [ ]" Example object: inetnum: 192.0.2.0 - 192.0.2.255 netname: EXAMPLE country: NL geofeed: https://example.com/geofeed.csv ... snip ... source: RIPE What do others think? Kind regards, Job ps. In IRRd v4.2 support for the 'geofeed:' attribute will be added https://github.com/irrdnet/irrd/issues/396
Hi Job, all
On 6 Oct 2020, at 18:28, Job Snijders via db-wg <db-wg@ripe.net> wrote:
Dear DB-WG,
Some colleagues are working to address the never-ending-story of 'where the heck are IPs geographically?'. This problem space has been brought up numerous times in the Database Working Group, but we never managed to solve it. As with all compsci problems adding a layer of indirection can help ;-)
This current draft suggests overloading the RPSL 'remarks:' field with a structured attribute value, however I suspect we would do ourselves a disservice to overload a 'remarks:' field.
Instead it would be better to add a 'geofeed:' attribute to the RPSL inetnum/inetnum6 class dictionary, which as value contains a URL with http or https scheme.
The draft: https://tools.ietf.org/html/draft-ymbk-opsawg-finding-geofeeds
The value of the attribute could be validated using something like "org.apache.commons.validator.UrlValidator", the attribute would look like this, only valid in the inetnum/inet6num:
"geofeed: [optional] [single] [ ]"
Example object:
inetnum: 192.0.2.0 - 192.0.2.255 netname: EXAMPLE country: NL geofeed: https://example.com/geofeed.csv ... snip ... source: RIPE
What do others think?
Yes I find this much more reasonable instead of overloading the remarks field. By default the remarks are being ignored by the parsers as it doesn’t contain any usable information that can be used in the router config generation (or other type of config). They contain information for the operator/human that reads the file on his screen. By enriching the RPSL dictionary and having a “geofeed” RPSL attribute (which by the way should not be mandatory) will be easier for someone to extend his parser to use that field without overloading the parser with many “if” and regex expressions. Plus the upcoming RFC specifies that "The format MUST be as in this example,“ so a verification needs to be applied later on. Of course it’s weird to talk about enriching RPSL on 2020 but putting this apart, I believe it’s more correct to implement it in this way.
Kind regards,
Job
ps. In IRRd v4.2 support for the 'geofeed:' attribute will be added https://github.com/irrdnet/irrd/issues/396
Best regards, Stavros Konstantaras | Sr. Network Engineer | AMS-IX M +31 (0) 620 89 51 04 | T +31 20 305 8999 ams-ix.net
By enriching the RPSL dictionary and having a “geofeed” RPSL attribute (which by the way should not be mandatory) will be easier for someone to extend his parser to use that field without overloading the parser with many “if” and regex expressions. Plus the upcoming RFC specifies A that "The format MUST be as in this example,“ so a verification needs to be applied later on.
I second this.
Of course it’s weird to talk about enriching RPSL on 2020 but putting this apart, I believe it’s more correct to implement it in this way.
If nobody ever adapts RPSL to the new requirements, RPSL is dead. So the question is: Should we throw away the RRDB in favour of something else, or do we extend RPSL in the way it was designed?
On Wed, Oct 07, 2020 at 06:31:11AM +0000, Lutz Donnerhacke via db-wg wrote:
By enriching the RPSL dictionary and having a “geofeed” RPSL attribute (which by the way should not be mandatory)
Indeed, it is not mandatory.
will be easier for someone to extend his parser to use that field without overloading the parser with many “if” and regex expressions. Plus the upcoming RFC specifies A that "The format MUST be as in this example,“ so a verification needs to be applied later on.
I second this.
Of course it’s weird to talk about enriching RPSL on 2020 but putting this apart, I believe it’s more correct to implement it in this way.
If nobody ever adapts RPSL to the new requirements, RPSL is dead. So the question is: Should we throw away the RRDB in favour of something else, or do we extend RPSL in the way it was designed?
Given that the implementation cost of adding a 'geofeed:' attribute in WHOIS servers is almost negligible, and it merely is a pointer to a the 'geofeed' data file in a format which can be used in different contexts (perhaps linked from peeringdb, perhaps referenced from RPKI objects), it seems a durable investment regardless of RPSL's future. Kind regards, Job
If nobody ever adapts RPSL to the new requirements, RPSL is dead. So the question is: Should we throw away the RRDB in favour of something else, or do we extend RPSL in the way it was designed?
we have a lot of rpsl-based toolchains out there. this does not mean we should not work on next gen. but, while we do so, keep up the old. appreciate job moving this forward in the ripe db community. i doubt anyone likes the remarks: hack. but it (or its kin) works today across all regions. e.g. see what massimo has done with it, https://github.com/massimocandela/geofeed-finder randy
+1 this is a much more reasonable solution than remarks imo - Cynthia On Tue, 6 Oct 2020, 18:29 Job Snijders via db-wg, <db-wg@ripe.net> wrote:
Dear DB-WG,
Some colleagues are working to address the never-ending-story of 'where the heck are IPs geographically?'. This problem space has been brought up numerous times in the Database Working Group, but we never managed to solve it. As with all compsci problems adding a layer of indirection can help ;-)
This current draft suggests overloading the RPSL 'remarks:' field with a structured attribute value, however I suspect we would do ourselves a disservice to overload a 'remarks:' field.
Instead it would be better to add a 'geofeed:' attribute to the RPSL inetnum/inetnum6 class dictionary, which as value contains a URL with http or https scheme.
The draft: https://tools.ietf.org/html/draft-ymbk-opsawg-finding-geofeeds
The value of the attribute could be validated using something like "org.apache.commons.validator.UrlValidator", the attribute would look like this, only valid in the inetnum/inet6num:
"geofeed: [optional] [single] [ ]"
Example object:
inetnum: 192.0.2.0 - 192.0.2.255 netname: EXAMPLE country: NL geofeed: https://example.com/geofeed.csv ... snip ... source: RIPE
What do others think?
Kind regards,
Job
ps. In IRRd v4.2 support for the 'geofeed:' attribute will be added https://github.com/irrdnet/irrd/issues/396
+1 Matthias Merkel Executive Vice President Staclar, Inc. [cid:image001.png@01D69CB4.4D061960] +1-628-213-1141 | +49 15678 585608 [cid:image002.png@01D69CB4.4D061960] matthias.merkel@staclar.com<mailto:matthias.merkel@staclar.com> [cid:image007.png@01D69CB4.4D0EA4E0] staclar.com<https://staclar.com/> [cid:image008.png@01D69CB4.4D0EA4E0] Munich, Germany [cid:image009.png@01D69CB4.4D0EA4E0] [linkedin]<https://www.linkedin.com/in/matthias-merkel/> From: db-wg <db-wg-bounces@ripe.net> On Behalf Of Cynthia Revström via db-wg Sent: Wednesday, 7 October 2020 14:14 To: Job Snijders <job@instituut.net> Cc: DB-WG <db-wg@ripe.net> Subject: Re: [db-wg] proposal: new attribute 'geofeed:' +1 this is a much more reasonable solution than remarks imo - Cynthia On Tue, 6 Oct 2020, 18:29 Job Snijders via db-wg, <db-wg@ripe.net<mailto:db-wg@ripe.net>> wrote: Dear DB-WG, Some colleagues are working to address the never-ending-story of 'where the heck are IPs geographically?'. This problem space has been brought up numerous times in the Database Working Group, but we never managed to solve it. As with all compsci problems adding a layer of indirection can help ;-) This current draft suggests overloading the RPSL 'remarks:' field with a structured attribute value, however I suspect we would do ourselves a disservice to overload a 'remarks:' field. Instead it would be better to add a 'geofeed:' attribute to the RPSL inetnum/inetnum6 class dictionary, which as value contains a URL with http or https scheme. The draft: https://tools.ietf.org/html/draft-ymbk-opsawg-finding-geofeeds The value of the attribute could be validated using something like "org.apache.commons.validator.UrlValidator", the attribute would look like this, only valid in the inetnum/inet6num: "geofeed: [optional] [single] [ ]" Example object: inetnum: 192.0.2.0 - 192.0.2.255 netname: EXAMPLE country: NL geofeed: https://example.com/geofeed.csv ... snip ... source: RIPE What do others think? Kind regards, Job ps. In IRRd v4.2 support for the 'geofeed:' attribute will be added https://github.com/irrdnet/irrd/issues/396
An interesting proposal, but merging an external data set with RIPE Database arises some questions: - RIPE Database is set up to contain hierarchical data already. With this proposal, we would take some of this data outside the database in a manner that does not guarantee consistency with the database itself. For example, 192.168/16 specifies a geofeed file that has different prefixes in it than the children of said inetnum (or cover a range that is not even listed in the RIPE Database). - Or, if the maintainers of 192.168/16 are different from the maintainer of its children, the owner of 192.168/16 has rights to override whatever is below it, if e.g. the owner of 192.168.1/24 doesn't set geofeed: on its inetnum. - It's also possible that ranges in the RIPE Database partially overlap with the prefixes in the geofeed file. E.g. you could specify an inetnum range '192.168.0.17-192.168.0.25' in the RIPE Database, while the geofeed csv allows prefixes only. That would make for some interesting choices when dealing with inconsistency in client code. - It's repeated multiple times in the proposal that "many RPSL repositories have weak if any authentication". But it also says: "When using data from a geofeed file, one MUST ignore data outside of the inetnum: object's inetnum: attribute's address range". These statements seem contradictory to me. On one hand, don't trust authentication for inet[6]num objects, on the other hand, verify to only accept ranges that fall into said inetnum range? - Although it's mentioned that clients should not request the geofeed file too often, it's not specified what constitutes as such. Lacking clear specification will lead to fragmentation in this area. - A clear specification of the geofeed: property is necessary to forego abuse. For example, what stops me from specifying a geofeed URL " https://google.com/?q=pr0n", or anything similar? Furthermore, when geofeed: attribute is presented in HTML, for example on the RIPE Database whois query form, it can be abused targeting the user's browser and cookies directly. - The RIPE Database can be downloaded as text dumps from a single location, and many clients make use of this functionality. There is no such thing existing for geofeed: attribute, so each client have to visit each of the URLs presented in the RIPE Database text dump. That is N clients, M attributes, doesn't seem like a scalable solution. This would effectively scatter geolocation data across the whole internet instead of keeping it in one place, with all the advantages and disadvantages that come with that. My personal impression of the proposal is quite mixed. On one hand, it's nice to see initiatives to extend usability of the RIPE Database and, within reasonable limits, fix the current situation around geolocation. On the other hand, the proposed solution would achieve this in a decentralized manner, and then sort of (ab)use the RIPE Database as a main anchor point, without adding any new features to the database itself. For example, the geofeed data could just as well be put in in-addr.arpa dns responses; it is not strictly tied to data already available in inet[6]num, merely has a 1:1 relationship to it. Personally, I think this data could be integrated into the RIPE Database itself, which would fix all of the consistency issues and would offer a clear, single entry point to all data. Cheers, Agoston On Tue, Oct 6, 2020 at 6:28 PM Job Snijders via db-wg <db-wg@ripe.net> wrote:
Dear DB-WG,
Some colleagues are working to address the never-ending-story of 'where the heck are IPs geographically?'. This problem space has been brought up numerous times in the Database Working Group, but we never managed to solve it. As with all compsci problems adding a layer of indirection can help ;-)
This current draft suggests overloading the RPSL 'remarks:' field with a structured attribute value, however I suspect we would do ourselves a disservice to overload a 'remarks:' field.
Instead it would be better to add a 'geofeed:' attribute to the RPSL inetnum/inetnum6 class dictionary, which as value contains a URL with http or https scheme.
The draft: https://tools.ietf.org/html/draft-ymbk-opsawg-finding-geofeeds
The value of the attribute could be validated using something like "org.apache.commons.validator.UrlValidator", the attribute would look like this, only valid in the inetnum/inet6num:
"geofeed: [optional] [single] [ ]"
Example object:
inetnum: 192.0.2.0 - 192.0.2.255 netname: EXAMPLE country: NL geofeed: https://example.com/geofeed.csv ... snip ... source: RIPE
What do others think?
Kind regards,
Job
ps. In IRRd v4.2 support for the 'geofeed:' attribute will be added https://github.com/irrdnet/irrd/issues/396
- RIPE Database is set up to contain hierarchical data already. With this proposal, we would take some of this data outside the database in a manner that does not guarantee consistency with the database itself. For example, 192.168/16 specifies a geofeed file that has different prefixes in it than the children of said inetnum (or cover a range that is not even listed in the RIPE Database).
from the i-d When using data from a geofeed file, one MUST ignore data outside of the inetnum: object's inetnum: attribute's address range. please read the draft randy
You are aware you ignored 90% of my points and picked a single one out of my email? But let me answer this with an example. Say, 192.168/16 has 5 children, each a /24. You put in the geofeed file for a single 192.168/16 a /20, covering four of these. The /20 does not exist in the RIPE DB. So a client has to be smart enough to match the prefixes from the RIPE DB, while also checking the children of 192.168/16 for more specific geofeed: attributes. Doable, of course, but this is left open in the draft. And there is the issue with inetnum ranges. What if 192.168/16 has a child 192.168.0.7-192.168.0.25. You make a geofeed prefix for 192.168.0.0/27? Or /28? What should the client do when it encounters prefixes 192.168.0.0/30 and 192.168.0.24/30 in the geofeed file ? Again, open question. Read my email. Agoston On Wed, Oct 7, 2020 at 4:03 PM Randy Bush via db-wg <db-wg@ripe.net> wrote:
- RIPE Database is set up to contain hierarchical data already. With this proposal, we would take some of this data outside the database in a manner that does not guarantee consistency with the database itself. For example, 192.168/16 specifies a geofeed file that has different prefixes in it than the children of said inetnum (or cover a range that is not even listed in the RIPE Database).
from the i-d
When using data from a geofeed file, one MUST ignore data outside of the inetnum: object's inetnum: attribute's address range.
please read the draft
randy
On Wed, Oct 07, 2020 at 04:50:43PM +0200, Horváth Ágoston János via db-wg wrote:
You are aware you ignored 90% of my points and picked a single one out of my email?
Correct. The rest of the email is interesting but directly related to the contents of a geofeed file, which is a discussion more suitable for OPSAWG in IETF. https://datatracker.ietf.org/wg/opsawg/about/ Some of your remarks were redundant, for instance "A clear specification of the geofeed: property is necessary to forego abuse." - in the initial email I pointed out that the [red: whois server side] attribute value should be validated using for example "org.apache.commons.validator.UrlValidator" Inserting HTML code in the attribute value would not consitute a valid URL.
But let me answer this with an example. Say, 192.168/16 has 5 children, each a /24. You put in the geofeed file for a single 192.168/16 a /20, covering four of these. The /20 does not exist in the RIPE DB. So a client has to be smart enough to match the prefixes from the RIPE DB, while also checking the children of 192.168/16 for more specific geofeed: attributes.
Doable, of course, but this is left open in the draft.
And there is the issue with inetnum ranges. What if 192.168/16 has a child 192.168.0.7-192.168.0.25. You make a geofeed prefix for 192.168.0.0/27? Or /28? What should the client do when it encounters prefixes 192.168.0.0/30 and 192.168.0.24/30 in the geofeed file ?
Again, open question.
The proposal is to make it possible to include a structured reference to a geofeed file. Geofeed file consumption & parsing is something I consider outside the scope of the RIPE database working group. More discussion is helpful, but the topic at hand for RIPE DB-WG really is no more than "can a new optional attribute be introduced ("geofeed:") whose attribute value will contain a URL. Kind regards, Job
On Wed, Oct 07, 2020 at 03:43:33PM +0200, Horváth Ágoston János wrote:
An interesting proposal, but merging an external data set with RIPE Database arises some questions:
- RIPE Database is set up to contain hierarchical data already. With this proposal, we would take some of this data outside the database in a manner that does not guarantee consistency with the database itself.
I don't think that is what is happening, this is more analogous to "abuse-mailbox:". The value contains an email address (which is an entrypoint outside the database, and interacting with the entrypoint is outside the scope of this working group). The "geofeed:" attribute is a pointer to elsewhere, and what a data consumer wants to do with that information is up to them. Kind regards, Job
+1 and I would like to suggest an adjustment: Not everyone will add the "geofeed" value and the geolocations in the csv will not validated by a 3rd party and may also be not up to date. I suggest to make the csv creation (of each INETNUM object) automatic and hosted by RIPE NCC, RIPE NCC can use all the current ATLAS Probes in order to check the physical location of each ip (using latencies from many probes to each ip, and using latencies from many probes to each router in the routing path to each ip, other ICMP queries can be used as well to query the routers in the routing path for physical information such as timezone as additional measures, etc). Algorithem will be efficient meaning at first only small number of probes will check each ip and then more probes physically near the first probe with the smallest latency. Kind Regards, Elad ________________________________ From: db-wg <db-wg-bounces@ripe.net> on behalf of Job Snijders via db-wg <db-wg@ripe.net> Sent: Wednesday, October 7, 2020 5:19 PM To: Horváth Ágoston János <horvath.agoston@gmail.com> Cc: db-wg@ripe.net >> Database WG <db-wg@ripe.net> Subject: Re: [db-wg] proposal: new attribute 'geofeed:' On Wed, Oct 07, 2020 at 03:43:33PM +0200, Horváth Ágoston János wrote:
An interesting proposal, but merging an external data set with RIPE Database arises some questions:
- RIPE Database is set up to contain hierarchical data already. With this proposal, we would take some of this data outside the database in a manner that does not guarantee consistency with the database itself.
I don't think that is what is happening, this is more analogous to "abuse-mailbox:". The value contains an email address (which is an entrypoint outside the database, and interacting with the entrypoint is outside the scope of this working group). The "geofeed:" attribute is a pointer to elsewhere, and what a data consumer wants to do with that information is up to them. Kind regards, Job
A RIPE hosted geolocation feed may indeed provide some advantages, however in my opinion this should instead be based on data supplied by the LIR (or end user for PI assignments), as it would be using an external geolocation feed, instead of automatically via Atlas. There are various reasons for why a measurement approach may return bad data (i.e. routing inconsistencies). Matthias Merkel Executive Vice President Staclar, Inc. [cid:image001.png@01D69CED.A2E46E70] +1-628-213-1141 | +49 15678 585608 [cid:image002.png@01D69CED.A2E46E70] matthias.merkel@staclar.com<mailto:matthias.merkel@staclar.com> [cid:image007.png@01D69CED.A2F7F670] staclar.com<https://staclar.com/> [cid:image008.png@01D69CED.A2F7F670] Munich, Germany [cid:image009.png@01D69CED.A2F7F670] [linkedin]<https://www.linkedin.com/in/matthias-merkel/> From: db-wg <db-wg-bounces@ripe.net> On Behalf Of Elad Cohen via db-wg Sent: Wednesday, 7 October 2020 17:01 To: Horváth Ágoston János <horvath.agoston@gmail.com>; Job Snijders <job@instituut.net> Cc: db-wg@ripe.net >> Database WG <db-wg@ripe.net> Subject: Re: [db-wg] proposal: new attribute 'geofeed:' +1 and I would like to suggest an adjustment: Not everyone will add the "geofeed" value and the geolocations in the csv will not validated by a 3rd party and may also be not up to date. I suggest to make the csv creation (of each INETNUM object) automatic and hosted by RIPE NCC, RIPE NCC can use all the current ATLAS Probes in order to check the physical location of each ip (using latencies from many probes to each ip, and using latencies from many probes to each router in the routing path to each ip, other ICMP queries can be used as well to query the routers in the routing path for physical information such as timezone as additional measures, etc). Algorithem will be efficient meaning at first only small number of probes will check each ip and then more probes physically near the first probe with the smallest latency. Kind Regards, Elad ________________________________ From: db-wg <db-wg-bounces@ripe.net<mailto:db-wg-bounces@ripe.net>> on behalf of Job Snijders via db-wg <db-wg@ripe.net<mailto:db-wg@ripe.net>> Sent: Wednesday, October 7, 2020 5:19 PM To: Horváth Ágoston János <horvath.agoston@gmail.com<mailto:horvath.agoston@gmail.com>> Cc: db-wg@ripe.net<mailto:db-wg@ripe.net> >> Database WG <db-wg@ripe.net<mailto:db-wg@ripe.net>> Subject: Re: [db-wg] proposal: new attribute 'geofeed:' On Wed, Oct 07, 2020 at 03:43:33PM +0200, Horváth Ágoston János wrote:
An interesting proposal, but merging an external data set with RIPE Database arises some questions:
- RIPE Database is set up to contain hierarchical data already. With this proposal, we would take some of this data outside the database in a manner that does not guarantee consistency with the database itself.
I don't think that is what is happening, this is more analogous to "abuse-mailbox:". The value contains an email address (which is an entrypoint outside the database, and interacting with the entrypoint is outside the scope of this working group). The "geofeed:" attribute is a pointer to elsewhere, and what a data consumer wants to do with that information is up to them. Kind regards, Job
Hi, I would just like to remind everyone that the issue that what the draft RFC is attempting to solve is purely a unified way for resource holders to say where the addresses are located. That is it, it is not attempting to actually check where the addresses are, just that it is actually the resource holder that is giving you this information. Additionally, the only thing being discussed here which is relevant is "should we have a geofeed attribute that accepts a URL, or just a remarks attribute with a value prefixed with Geofeed?" "remarks: Geofeed https://example.com/geofeed.csv" vs "geofeed: https://example.com/geofeed.csv" That is it, other things than that are not relevant to the db-wg, and should go on other relevant mailing lists at the IETF or whoever is coordinating this. - Cynthia On Wed, Oct 7, 2020 at 9:06 PM Matthias Merkel via db-wg <db-wg@ripe.net> wrote:
A RIPE hosted geolocation feed may indeed provide some advantages, however in my opinion this should instead be based on data supplied by the LIR (or end user for PI assignments), as it would be using an external geolocation feed, instead of automatically via Atlas. There are various reasons for why a measurement approach may return bad data (i.e. routing inconsistencies).
*Matthias Merkel*
Executive Vice President
Staclar, Inc.
+1-628-213-1141 | +49 15678 585608
matthias.merkel@staclar.com
staclar.com
Munich, Germany
[image: linkedin] <https://www.linkedin.com/in/matthias-merkel/>
*From:* db-wg <db-wg-bounces@ripe.net> *On Behalf Of *Elad Cohen via db-wg *Sent:* Wednesday, 7 October 2020 17:01 *To:* Horváth Ágoston János <horvath.agoston@gmail.com>; Job Snijders < job@instituut.net> *Cc:* db-wg@ripe.net >> Database WG <db-wg@ripe.net> *Subject:* Re: [db-wg] proposal: new attribute 'geofeed:'
+1
and I would like to suggest an adjustment:
Not everyone will add the "geofeed" value and the geolocations in the csv will not validated by a 3rd party and may also be not up to date.
I suggest to make the csv creation (of each INETNUM object) automatic and hosted by RIPE NCC, RIPE NCC can use all the current ATLAS Probes in order to check the physical location of each ip (using latencies from many probes to each ip, and using latencies from many probes to each router in the routing path to each ip, other ICMP queries can be used as well to query the routers in the routing path for physical information such as timezone as additional measures, etc). Algorithem will be efficient meaning at first only small number of probes will check each ip and then more probes physically near the first probe with the smallest latency.
Kind Regards,
Elad ------------------------------
*From:* db-wg <db-wg-bounces@ripe.net> on behalf of Job Snijders via db-wg <db-wg@ripe.net> *Sent:* Wednesday, October 7, 2020 5:19 PM *To:* Horváth Ágoston János <horvath.agoston@gmail.com> *Cc:* db-wg@ripe.net >> Database WG <db-wg@ripe.net> *Subject:* Re: [db-wg] proposal: new attribute 'geofeed:'
On Wed, Oct 07, 2020 at 03:43:33PM +0200, Horváth Ágoston János wrote:
An interesting proposal, but merging an external data set with RIPE Database arises some questions:
- RIPE Database is set up to contain hierarchical data already. With this proposal, we would take some of this data outside the database in a manner that does not guarantee consistency with the database itself.
I don't think that is what is happening, this is more analogous to "abuse-mailbox:". The value contains an email address (which is an entrypoint outside the database, and interacting with the entrypoint is outside the scope of this working group).
The "geofeed:" attribute is a pointer to elsewhere, and what a data consumer wants to do with that information is up to them.
Kind regards,
Job
"should we have a geofeed attribute that accepts a URL, or just a remarks attribute with a value prefixed with Geofeed?"
"remarks: Geofeed https://example.com/geofeed.csv" vs "geofeed: https://example.com/geofeed.csv"
and we're currently trying to bridge two tensions. there will be a long transition from remarks: to geofeed:. the next draft will try and finesse that with words along the vein of Ideally, RPSL would be augmented to define a new RPSL geofeed: attribute in the inetnum: class. Until such time, this document defines the syntax of a Geofeed remarks: attribute which contains a URL of a geofeed file. The format MUST be as in this example, "remarks: Geofeed " followed by a URL which will vary. inetnum: 192.0.2.0/24 # example remarks: Geofeed https://example.com/geofeed.csv Any particular inetnum: object MAY have, at most, one geofeed reference. the other tension is that this all needs to work across all five regions and their non-homogenous syntaxes and services. randy
Well, not really. Say, if geofeed: is accepted, that also sets the way how to store geolocation data in the RIPE Database. I mean, allowing geofeed: and then adding country:, city: attributes in inetnum would be inconsistent and confusing. So it is much more than just an optional attribute. I do think discussion about externalizing data (that also used to be part of inetnum via the country: attribute back in the day) belongs here. Cheers, Agoston On Wed, Oct 7, 2020 at 4:20 PM Job Snijders <job@instituut.net> wrote:
On Wed, Oct 07, 2020 at 03:43:33PM +0200, Horváth Ágoston János wrote:
An interesting proposal, but merging an external data set with RIPE Database arises some questions:
- RIPE Database is set up to contain hierarchical data already. With this proposal, we would take some of this data outside the database in a manner that does not guarantee consistency with the database itself.
I don't think that is what is happening, this is more analogous to "abuse-mailbox:". The value contains an email address (which is an entrypoint outside the database, and interacting with the entrypoint is outside the scope of this working group).
The "geofeed:" attribute is a pointer to elsewhere, and what a data consumer wants to do with that information is up to them.
Kind regards,
Job
Purely as a point of information, Randy also asked for APNIC to consider this field, and it has been put into the Opportunity process inside Registry Product, of which I am the manager. I am following this discussion. My primary concerns are the effects on tools, of a new type:value assertion appearing in the RPSL. If this is not a high barrier to deployment it has consequences beyond geofeed: marker for us. Separately, I think in order to ensure RDAP and Whois stay in alignment, some thought needs to be given to how this value projects into JSON in RDAP records. And, lastly, I would like this field, because it would relieve some of the tension around the country: marker in inetnum and inet6num and the economycode field of the delegated statistics file, which is by definition about the entity, not the addresses. Widespread acceptance of a geofeed mark would allow registry to return to a normative use of the economycode for the domicile of the holding entity. cheers -George On Wed, Oct 7, 2020 at 2:28 AM Job Snijders via db-wg <db-wg@ripe.net> wrote:
Dear DB-WG,
Some colleagues are working to address the never-ending-story of 'where the heck are IPs geographically?'. This problem space has been brought up numerous times in the Database Working Group, but we never managed to solve it. As with all compsci problems adding a layer of indirection can help ;-)
This current draft suggests overloading the RPSL 'remarks:' field with a structured attribute value, however I suspect we would do ourselves a disservice to overload a 'remarks:' field.
Instead it would be better to add a 'geofeed:' attribute to the RPSL inetnum/inetnum6 class dictionary, which as value contains a URL with http or https scheme.
The draft: https://tools.ietf.org/html/draft-ymbk-opsawg-finding-geofeeds
The value of the attribute could be validated using something like "org.apache.commons.validator.UrlValidator", the attribute would look like this, only valid in the inetnum/inet6num:
"geofeed: [optional] [single] [ ]"
Example object:
inetnum: 192.0.2.0 - 192.0.2.255 netname: EXAMPLE country: NL geofeed: https://example.com/geofeed.csv ... snip ... source: RIPE
What do others think?
Kind regards,
Job
ps. In IRRd v4.2 support for the 'geofeed:' attribute will be added https://github.com/irrdnet/irrd/issues/396
Hello, On 8 Oct 2020, at 00:35, George Michaelson via db-wg <db-wg@ripe.net> wrote:
I am following this discussion. My primary concerns are the effects on tools, of a new type:value assertion appearing in the RPSL. If this is not a high barrier to deployment it has consequences beyond geofeed: marker for us.
I doubt this is a significant issue, because there are already several custom different RPSL attributes used by various RPSL databases. RIPE has sponsoring-org on inetnums, some have last-modified on their objects, and I think there are several other arbitrary attributes already. Tools should be capable of encountering attributes they don’t understand. I am in favour of this proposal. Encoding structured information into a free-form remarks field seems sloppy and error prone. I don’t see any downsides to adding the new optional attribute, and it’s fairly simple to implement. Sasha
Dear DB-WG chairs, Today an acquaintance (who works at an ISP) reached out to me describing some annoying Geo IP location issues they were facing. I thought to myself "didn't we have a solution to such issues?" and was reminded of this thread. Chairs, how do we proceed to work towards the deployment of this new RPSL 'geofeed:' attribute? What is the next step? strawman problem statement for NWI process: "Associating an approximate physical location with an IP address has proven to be a challenge to solve within the current constraints of the RIPE database. Over the years the community has choosen to consider addresses in the RIPE database to relate to entities in the assignment process itself, not the subsequent actual use of IP addresses after assignment. The working group is asked to consider whether the RIPE database can be used as a springboard for parties wishing to correlate geographical information with IP addresses by allowing structured references in the RIPE database towards information outside the RIPE database which potentially helps answer Geo IP Location queries." Outstanding questions to the group & authors of the geofeed draft: - What would the RDAP integration look like? (as per George: ideally WHOIS and RDAP stay aligned) - Why not use the RPKI to distribute geofeed information? - What downsides/risks/challenges down the road might exist? Will this be the final and full solution to GeoIP challenges? - Have any of the GeoIP aggregators/vendors responded to the 'geofeed:' initiative? What do the likes of amazon/google/maxmind think of the format? Kind regards, Job
Hi Job and db-wg, I just want to start out by saying that I think this is a very worthwhile endeavour IMO, and thank you Job for reminding me of this thread. :) I have also written some opinions about your questions and ideas below, and I would like to hear some input from people on the WG as this is sort of my initial thoughts but have not been thoroughly considered.
Why not use the RPKI to distribute geofeed information?
I think using RPKI for this will make it needlessly complicated for people to use. I think we want to make this as simple as possible for someone to find. To me it seems like it might be a good idea to add it as an attribute in the RIPE DB and then additionally having some like the HTTP API for finding abuse contacts but for geofeed ( https://github.com/RIPE-NCC/whois/wiki/WHOIS-REST-API-abuse-contact ).
What downsides/risks/challenges down the road might exist?
As I noted in the comment above, I think the main challenge will be to make it simple and basic enough to implement for people wanting to publish geofeed URLs and people wanting to fetch geofeed URLs. I think having a basic HTTP JSON API to get this URL would really help with implementation as it would be really easy for data consumers to query for the URL and not have to parse WHOIS data or read RDAP RFCs or whatever. Additionally, WHOIS is not encrypted/authenticated (which HTTPS/TLS is) and as such while the risk for malicious activity here might be low, I think having it authenticated would help (as in making sure you get the URL that the resource holder provided, not that the data in there is necessarily correct). I think that if this is implemented, only HTTPS urls should be allowed, as in no plain text HTTP. (I don't have very strong feelings but it seems reasonable to me) And on the note of the main challenge, I also think a part of making it simple to implement is to try to have the same API for this between all of the RIRs, because I don't think this will really succeed if only RIPE NCC has it. I think that if the RIPE NCC implements this, the next step will be to try to coordinate this with other RIRs so it can be a workable system for all RIRs.
Will this be the final and full solution to GeoIP challenges?
I would say that this is probably not something that can easily be figured out other than by implementing it and waiting for a few years.
Have any of the GeoIP aggregators/vendors responded to the 'geofeed:' initiative? What do the likes of amazon/google/maxmind think of the format?
I am also very curious about this, in particular to what the people wanting to use this data think of it (aka google, amazon, etc). I don't think maxmind is likely to approve of it as it basically destroys their product if successful. Additionally I think this would be quite useful if it succeeded as if I recall correctly, the BBC uses the delegated list files on ftp.ripe.net for GeoIP. This sort of became problematic when the whole delegated list file country code has to correspond to legal address thing happened, as this broke for some cases. - Cynthia On Sun, Jan 3, 2021 at 7:30 PM Job Snijders via db-wg <db-wg@ripe.net> wrote:
Dear DB-WG chairs,
Today an acquaintance (who works at an ISP) reached out to me describing some annoying Geo IP location issues they were facing.
I thought to myself "didn't we have a solution to such issues?" and was reminded of this thread.
Chairs, how do we proceed to work towards the deployment of this new RPSL 'geofeed:' attribute? What is the next step?
strawman problem statement for NWI process:
"Associating an approximate physical location with an IP address has proven to be a challenge to solve within the current constraints of the RIPE database. Over the years the community has choosen to consider addresses in the RIPE database to relate to entities in the assignment process itself, not the subsequent actual use of IP addresses after assignment.
The working group is asked to consider whether the RIPE database can be used as a springboard for parties wishing to correlate geographical information with IP addresses by allowing structured references in the RIPE database towards information outside the RIPE database which potentially helps answer Geo IP Location queries."
Outstanding questions to the group & authors of the geofeed draft:
- What would the RDAP integration look like? (as per George: ideally WHOIS and RDAP stay aligned) - Why not use the RPKI to distribute geofeed information? - What downsides/risks/challenges down the road might exist? Will this be the final and full solution to GeoIP challenges? - Have any of the GeoIP aggregators/vendors responded to the 'geofeed:' initiative? What do the likes of amazon/google/maxmind think of the format?
Kind regards,
Job
hi cynthia
Why not use the RPKI to distribute geofeed information?
I think using RPKI for this will make it needlessly complicated for people to use. I think we want to make this as simple as possible for someone to find.
as i said to job over on the dark side of the force, the ietf, hammer seeks nails, eh? :) geoseekers already mine whois. so we are putting the keys under the streetlight (excuse what may be an american-only idiom). asking someone who wants to publish their geofeed location to first deal with rpki is an artifical barrier to deployment. asking someone who wants to find geofeed data to first deal with rpki is an artificial barrier to deployment. is the rpki to be the next dns/bgp, a place to dump data? have we all looked at https://datatracker.ietf.org/doc/draft-ietf-opsawg-finding-geofeeds/ it's all pretty much layed out there, i hope. improvements solicited, of course. randy
Would it make sense to wait with implementation until the the ietf draft comes out of draft status? Seems like jumping into beta waters to me. Agoston On Sun, Jan 3, 2021 at 8:22 PM Randy Bush via db-wg <db-wg@ripe.net> wrote:
hi cynthia
Why not use the RPKI to distribute geofeed information?
I think using RPKI for this will make it needlessly complicated for people to use. I think we want to make this as simple as possible for someone to find.
as i said to job over on the dark side of the force, the ietf,
hammer seeks nails, eh? :)
geoseekers already mine whois. so we are putting the keys under the streetlight (excuse what may be an american-only idiom).
asking someone who wants to publish their geofeed location to first deal with rpki is an artifical barrier to deployment.
asking someone who wants to find geofeed data to first deal with rpki is an artificial barrier to deployment.
is the rpki to be the next dns/bgp, a place to dump data?
have we all looked at
https://datatracker.ietf.org/doc/draft-ietf-opsawg-finding-geofeeds/
it's all pretty much layed out there, i hope. improvements solicited, of course.
randy
Agoston, I think that it's fine to help push adoption ahead of the IETF process. The proposal is not super complicated (which is good). While possibly the authentication details within the feed will change I don't think the RPSL attribute will need to be modified from the current draft. Cheers, -- Shane On 06/01/2021 17.59, Horváth Ágoston János via db-wg wrote:
Would it make sense to wait with implementation until the the ietf draft comes out of draft status? Seems like jumping into beta waters to me.
Agoston
On Sun, Jan 3, 2021 at 8:22 PM Randy Bush via db-wg <db-wg@ripe.net <mailto:db-wg@ripe.net>> wrote:
hi cynthia
>> Why not use the RPKI to distribute geofeed information? > > I think using RPKI for this will make it needlessly complicated for people > to use. I think we want to make this as simple as possible for someone to > find.
as i said to job over on the dark side of the force, the ietf,
hammer seeks nails, eh? :)
geoseekers already mine whois. so we are putting the keys under the streetlight (excuse what may be an american-only idiom).
asking someone who wants to publish their geofeed location to first deal with rpki is an artifical barrier to deployment.
asking someone who wants to find geofeed data to first deal with rpki is an artificial barrier to deployment.
is the rpki to be the next dns/bgp, a place to dump data?
have we all looked at
https://datatracker.ietf.org/doc/draft-ietf-opsawg-finding-geofeeds/ <https://datatracker.ietf.org/doc/draft-ietf-opsawg-finding-geofeeds/>
it's all pretty much layed out there, i hope. improvements solicited, of course.
randy
And on the note of the main challenge, I also think a part of making it simple to implement is to try to have the same API for this between all of the RIRs
https://github.com/massimocandela/geofeed-finder what remains is to go from a remarks: attribute to a first class geofeed: attribute, which i took as the core of job's post. randy
Hi Job So if we were to create a new NWI with your suggested Problem statement: Associating an approximate physical location with an IP address has proven to be a challenge to solve within the current constraints of the RIPE database. Over the years the community has chosen to consider addresses in the RIPE database to relate to entities in the assignment process itself, not the subsequent actual use of IP addresses after assignment. The working group is asked to consider whether the RIPE database can be used as a springboard for parties wishing to correlate geographical information with IP addresses by allowing structured references in the RIPE database towards information outside the RIPE database which potentially helps answer Geo IP Location queries. Would you say a possible Solution statement could be as simple as: Implement a new "geofeed:" attribute in the INETNUM and INET6NUM objects based on the IETF draft https://datatracker.ietf.org/doc/draft-ietf-opsawg-finding-geofeeds/ On deployment update any objects in the RIPE Database containing a "remarks: Geofeed" with a correctly formatted url into a "geofeed:" attribute. cheers denis co-chair DB-WG On Sun, 3 Jan 2021 at 19:11, Job Snijders via db-wg <db-wg@ripe.net> wrote:
Dear DB-WG chairs,
Today an acquaintance (who works at an ISP) reached out to me describing some annoying Geo IP location issues they were facing.
I thought to myself "didn't we have a solution to such issues?" and was reminded of this thread.
Chairs, how do we proceed to work towards the deployment of this new RPSL 'geofeed:' attribute? What is the next step?
strawman problem statement for NWI process:
"Associating an approximate physical location with an IP address has proven to be a challenge to solve within the current constraints of the RIPE database. Over the years the community has choosen to consider addresses in the RIPE database to relate to entities in the assignment process itself, not the subsequent actual use of IP addresses after assignment.
The working group is asked to consider whether the RIPE database can be used as a springboard for parties wishing to correlate geographical information with IP addresses by allowing structured references in the RIPE database towards information outside the RIPE database which potentially helps answer Geo IP Location queries."
Outstanding questions to the group & authors of the geofeed draft:
- What would the RDAP integration look like? (as per George: ideally WHOIS and RDAP stay aligned) - Why not use the RPKI to distribute geofeed information? - What downsides/risks/challenges down the road might exist? Will this be the final and full solution to GeoIP challenges? - Have any of the GeoIP aggregators/vendors responded to the 'geofeed:' initiative? What do the likes of amazon/google/maxmind think of the format?
Kind regards,
Job
Hi Denis,
Would you say a possible Solution statement could be as simple as:
Implement a new "geofeed:" attribute in the INETNUM and INET6NUM objects based on the IETF draft https://datatracker.ietf.org/doc/draft-ietf-opsawg-finding-geofeeds/ On deployment update any objects in the RIPE Database containing a "remarks: Geofeed" with a correctly formatted url into a "geofeed:" attribute.
Sounds simple enough 🙂 Let's implement! Sander
One more question after coffee:
On deployment update any objects in the RIPE Database containing a "remarks: Geofeed" with a correctly formatted url into a "geofeed:" attribute.
Sounds simple enough 🙂
What happens when someone (or someone's automated system) pushes an object with a "remarks: Geofeed" again? Do an automatic conversion on update? How many scripts will that break (or worse: how many scripts will get stuck in a loop because the object in the database isn't what they believe it should be, so they update it again and again and …) So maybe not do an automatic update after all, and maybe update the web based editor with a helpful "This looks like a Geofeed remark, click here to change it to a geofeed attribute" button 🙂 Cheers! Sander
What happens when someone (or someone's automated system) pushes an object with a "remarks: Geofeed" again?
from draft-ietf-opsawg-finding-geofeeds Any particular inetnum: object MAY have, at most, one geofeed reference, whether a remarks: or a proper geofeed: attribute when one is defined. now all we need is for the someone or someone's automated system to read the spec :) randy
On 2021-01-04 21:41, Randy Bush via db-wg wrote:
What happens when someone (or someone's automated system) pushes an object with a "remarks: Geofeed" again?
from draft-ietf-opsawg-finding-geofeeds
Any particular inetnum: object MAY have, at most, one geofeed reference, whether a remarks: or a proper geofeed: attribute when one is defined.
now all we need is for the someone or someone's automated system to read the spec :)
Of course the question is "what should happen in the bad case", eg. what's the expected behaviour if there are 2 or more "remarks: geofeed" attributes? Should the DB reject such an update? Should the geofeed client pick one at random? Should it pick the first one? Or the last one? Or none? Robert (speaking for myself)
from draft-ietf-opsawg-finding-geofeeds
Any particular inetnum: object MAY have, at most, one geofeed reference, whether a remarks: or a proper geofeed: attribute when one is defined.
now all we need is for the someone or someone's automated system to read the spec :)
Of course the question is "what should happen in the bad case", eg. what's the expected behaviour if there are 2 or more "remarks: geofeed" attributes?
Should the DB reject such an update? Should the geofeed client pick one at random? Should it pick the first one? Or the last one? Or none?
my personal opinion would be, as the spec says only one, an attempt to add a second should fail. as should an initial attempt to create with more than one. but i have faith that we can make it more complicated :) randy
Should the DB reject such an update? Should the geofeed client pick one at random? Should it pick the first one? Or the last one? Or none?
my personal opinion would be, as the spec says only one, an attempt to add a second should fail. as should an initial attempt to create with more than one.
Interesting. I was under the impression that the point for allowing the alternative representation of using remarks: is to fit in the current RPSL scheme, thereby not needing to change implementation. Rejection of remarks:geofeed duplicates would violate that. In other words, once an implementation enforces the single-instance remarks:geofeed entry, it may as well enforce the geofeed: key instead... To me that means that the enforcement of single-instance remarks:geofeed entries is unlikely to happen. Cheers, Robert
Hi Ed Can you do a quick count on INETNUM and INET6NUM objects and see how many currently have a "remarks: Geofeed" attribute and if any have more than one. An interesting question Robert. Perhaps an even more interesting one is, if we implement the "geofeed:" attribute, should we allow anyone to add any "remarks: Geofeed" or should we flag that as a syntax error? Or will any tool that accesses this data ignore any "remarks:" attributes if there is a "geofeed:" attribute? cheers denis co-chair DB-WG ---------- Forwarded message --------- From: Robert Kisteleki via db-wg <db-wg@ripe.net> Date: Tue, 5 Jan 2021 at 09:45 Subject: Re: [db-wg] proposal: new attribute 'geofeed:' To: <db-wg@ripe.net> On 2021-01-04 21:41, Randy Bush via db-wg wrote:
What happens when someone (or someone's automated system) pushes an object with a "remarks: Geofeed" again?
from draft-ietf-opsawg-finding-geofeeds
Any particular inetnum: object MAY have, at most, one geofeed reference, whether a remarks: or a proper geofeed: attribute when one is defined.
now all we need is for the someone or someone's automated system to read the spec :)
Of course the question is "what should happen in the bad case", eg. what's the expected behaviour if there are 2 or more "remarks: geofeed" attributes? Should the DB reject such an update? Should the geofeed client pick one at random? Should it pick the first one? Or the last one? Or none? Robert (speaking for myself)
An interesting question Robert. Perhaps an even more interesting one is, if we implement the "geofeed:" attribute, should we allow anyone to add any "remarks: Geofeed" or should we flag that as a syntax error? Or will any tool that accesses this data ignore any "remarks:" attributes if there is a "geofeed:" attribute?
after more coffee, i have misgivings about any process on remarks: other than the temporary hack. it seems analogous to prescribing the content of comments in a programming language. randy
Hi Denis, Colleagues,
On 5 Jan 2021, at 14:30, denis walker <ripedenis@gmail.com> wrote:
Hi Ed
Can you do a quick count on INETNUM and INET6NUM objects and see how many currently have a "remarks: Geofeed" attribute and if any have more than one.
I found 47 inetnums, 17 inet6nums and 2 aut-num objects with a "remarks: Geofeed" attribute. I didn't find any with more than one. Regards Ed Shryane RIPE NCC
Hello all, On 5 Jan 2021, at 19:29, Edward Shryane via db-wg <db-wg@ripe.net> wrote:
On 5 Jan 2021, at 14:30, denis walker <ripedenis@gmail.com> wrote:
Hi Ed
Can you do a quick count on INETNUM and INET6NUM objects and see how many currently have a "remarks: Geofeed" attribute and if any have more than one.
I found 47 inetnums, 17 inet6nums and 2 aut-num objects with a "remarks: Geofeed" attribute. I didn't find any with more than one.
For what it’s worth, in IRRd the geofeed attribute will be supported in the next release. It’s optional, permitted once, and must be a URL. We don’t validate against additional “remarks: Geofeed” attributes. Remarks are for humans from IRRd’s perspective, and nothing is done with them except reproducing them in queries. Sasha
participants (15)
-
Cynthia Revström -
denis walker -
Edward Shryane -
Elad Cohen -
George Michaelson -
Horváth Ágoston János -
Job Snijders -
Lutz Donnerhacke -
Matthias Merkel -
Randy Bush -
Robert Kisteleki -
Sander Steffann -
Sasha Romijn -
Shane Kerr -
Stavros Konstantaras