More even earth distribution of measurement probes
I'd like to check worldwide connectivity to my site. I create a new measurement via web-interface and add, say, 300 probes, expecting them to cover most world countries. However, looking at the map <https://atlas.ripe.net/measurements/7861872/?filter=#map> I notice that most are selected from Europe: there're two dozens in Netherlands (many with same ASNs), but only two in Australia, one in Japan and nothing in Brazil. Obviously, most ATLAS probes are from Europe, and selection algorithm does not take probe density into account. Wouldn't it be nice to have the following features in web-interface and API: * even distribution of probes by country or continent based on population, area, and other user-defined weight numbers; * avoiding probes with same ASNs ? I don't know how current algorithm works, but easiest modification would be two-step process: (1) randomly select country based on specified weights; (2) randomly select probe within country, avoiding earlier selected probes or ASNs and failing if none are left available; repeat from step 1 until necessary number of probes is selected. P.S. I tried to manually request 1 probe from each country, but stumbled on unrecognised country codes, countries with no probes etc. before even trying population-weighted distribution. Without direct access to probes database nice solution can be difficult to achieve via current API, although crude one looks possible. P.P.S. I understand this option will create more load on probes in less-covered countries, but since it's possible to limit selection to specific countries anyway this is hardly a strong argument against adding this feature. -- With Best Regards, Marat Khalili
On Fri, Mar 10, 2017 at 11:47:12AM +0300, Marat Khalili <mkh@rqc.ru> wrote a message of 94 lines which said:
300 probes, expecting them to cover most world countries.
Most people seem to expect some sort of random selection in the complete set of probes, but it does not work that way. Some probes are favored over others (anchors, for instance) and there is a priority for the less loaded probes.
Wouldn't it be nice to have the following features in web-interface and API: * even distribution of probes by country or continent based on population, area, and other user-defined weight numbers; * avoiding probes with same ASNs
Currently, you have to do it yourself: retrieve the list of probes, apply your criteria, and submit a measurement with the explicit IDs.
Without direct access to probes database
But you have access ftp://ftp.ripe.net/ripe/atlas/probes/archive/2017/03/
On 10/03/17 13:07, Stephane Bortzmeyer wrote:
But you have access ftp://ftp.ripe.net/ripe/atlas/probes/archive/2017/03/ Aw, great! Didn't know about this "interface" :) Thank you!
-- With Best Regards, Marat Khalili
Dear Marat, On 10/03/17 09:47, Marat Khalili wrote:
I'd like to check worldwide connectivity to my site. I create a new measurement via web-interface and add, say, 300 probes, expecting them to cover most world countries.
There was a hackathon project that made this possible in some way, called "Spatial Bucketing of RIPE Atlas Probes on Map Projection", by Julian Hammer: "[he] wanted to solve the following problem: when scheduling measurements from lots of probes, requesting a "random worldwide selection" currently results in a choice that is very biased towards Western/Northern Europe and the USA (see image without Sbucket on the left). Julian's solution makes a more equal selection, by using the grid to divide the globe and choose probes based on that division (see image on the right after applying Sbucket)." Images: https://labs.ripe.net/Members/becha/ripe-atlas-hackathon-results Code: https://github.com/cod3monk/RIPE-Atlas-sbucket I hope you find this useful! Cheers, Vesna
However, looking at the map <https://atlas.ripe.net/measurements/7861872/?filter=#map> I notice that most are selected from Europe: there're two dozens in Netherlands (many with same ASNs), but only two in Australia, one in Japan and nothing in Brazil. Obviously, most ATLAS probes are from Europe, and selection algorithm does not take probe density into account.
Wouldn't it be nice to have the following features in web-interface and API: * even distribution of probes by country or continent based on population, area, and other user-defined weight numbers; * avoiding probes with same ASNs ?
I don't know how current algorithm works, but easiest modification would be two-step process: (1) randomly select country based on specified weights; (2) randomly select probe within country, avoiding earlier selected probes or ASNs and failing if none are left available; repeat from step 1 until necessary number of probes is selected.
P.S. I tried to manually request 1 probe from each country, but stumbled on unrecognised country codes, countries with no probes etc. before even trying population-weighted distribution. Without direct access to probes database nice solution can be difficult to achieve via current API, although crude one looks possible.
P.P.S. I understand this option will create more load on probes in less-covered countries, but since it's possible to limit selection to specific countries anyway this is hardly a strong argument against adding this feature.
--
With Best Regards, Marat Khalili
Cool, will try it! I almost wrote my own tool already, but I didn't try to make even spatial distribution, it is obviously more difficult than by-country one. -- With Best Regards, Marat Khalili On 10/03/17 14:56, Vesna Manojlovic wrote:
Dear Marat,
On 10/03/17 09:47, Marat Khalili wrote:
I'd like to check worldwide connectivity to my site. I create a new measurement via web-interface and add, say, 300 probes, expecting them to cover most world countries.
There was a hackathon project that made this possible in some way, called "Spatial Bucketing of RIPE Atlas Probes on Map Projection", by Julian Hammer:
"[he] wanted to solve the following problem: when scheduling measurements from lots of probes, requesting a "random worldwide selection" currently results in a choice that is very biased towards Western/Northern Europe and the USA (see image without Sbucket on the left).
Julian's solution makes a more equal selection, by using the grid to divide the globe and choose probes based on that division (see image on the right after applying Sbucket)."
Images: https://labs.ripe.net/Members/becha/ripe-atlas-hackathon-results
Code: https://github.com/cod3monk/RIPE-Atlas-sbucket
I hope you find this useful!
Cheers, Vesna
However, looking at the map <https://atlas.ripe.net/measurements/7861872/?filter=#map> I notice that most are selected from Europe: there're two dozens in Netherlands (many with same ASNs), but only two in Australia, one in Japan and nothing in Brazil. Obviously, most ATLAS probes are from Europe, and selection algorithm does not take probe density into account.
Wouldn't it be nice to have the following features in web-interface and API: * even distribution of probes by country or continent based on population, area, and other user-defined weight numbers; * avoiding probes with same ASNs ?
I don't know how current algorithm works, but easiest modification would be two-step process: (1) randomly select country based on specified weights; (2) randomly select probe within country, avoiding earlier selected probes or ASNs and failing if none are left available; repeat from step 1 until necessary number of probes is selected.
P.S. I tried to manually request 1 probe from each country, but stumbled on unrecognised country codes, countries with no probes etc. before even trying population-weighted distribution. Without direct access to probes database nice solution can be difficult to achieve via current API, although crude one looks possible.
P.P.S. I understand this option will create more load on probes in less-covered countries, but since it's possible to limit selection to specific countries anyway this is hardly a strong argument against adding this feature.
--
With Best Regards, Marat Khalili
My solution, for the record: https://gist.github.com/qm2k/ea5f67fcf4e3dd2208ab3bc4ee233560 -- With Best Regards, Marat Khalili On 10/03/17 11:47, Marat Khalili wrote:
I'd like to check worldwide connectivity to my site. I create a new measurement via web-interface and add, say, 300 probes, expecting them to cover most world countries. However, looking at the map <https://atlas.ripe.net/measurements/7861872/?filter=#map> I notice that most are selected from Europe: there're two dozens in Netherlands (many with same ASNs), but only two in Australia, one in Japan and nothing in Brazil. Obviously, most ATLAS probes are from Europe, and selection algorithm does not take probe density into account.
Wouldn't it be nice to have the following features in web-interface and API: * even distribution of probes by country or continent based on population, area, and other user-defined weight numbers; * avoiding probes with same ASNs ?
I don't know how current algorithm works, but easiest modification would be two-step process: (1) randomly select country based on specified weights; (2) randomly select probe within country, avoiding earlier selected probes or ASNs and failing if none are left available; repeat from step 1 until necessary number of probes is selected.
P.S. I tried to manually request 1 probe from each country, but stumbled on unrecognised country codes, countries with no probes etc. before even trying population-weighted distribution. Without direct access to probes database nice solution can be difficult to achieve via current API, although crude one looks possible.
P.P.S. I understand this option will create more load on probes in less-covered countries, but since it's possible to limit selection to specific countries anyway this is hardly a strong argument against adding this feature.
--
With Best Regards, Marat Khalili
participants (3)
-
Marat Khalili -
Stephane Bortzmeyer -
Vesna Manojlovic