Best way to download bulk data older than a month?
I'm trying to download the RIPE Atlas traceroutes for a day that is older than one month. Given that the FTP repository publishes only results from the last month, I need to use the API. So, what I'm doing is to list of measurement IDs for a given data and then for each ID I do a GET for the results of that ID. The code is here: https://github.com/vgiotsas/RIPEAtlas_Queries/blob/main/download_atlas_trace... However, often some downloads time out and when I'm trying to resume it using the Range header it seems that the API doesn't support it and the download starts over until it fails again halfway through the download. Is there a better way to do such bulk downloads? Or at least to resume failed downloads? Thanks!
Hi again folks. For anyone having similar issues, I managed to address most issues using curl and forcing HTTP version 1.0. I also fixed some bug I had in my API query parameters (the `start` parameter for `measurements/{pk}/results` has the same effect as the `start_time__lt` for `measurements/` queries). The updated script is here, maybe it can be useful to others too: https://github.com/vgiotsas/RIPEAtlas_Queries/blob/main/download_atlas_trace... ________________________________ From: Vasileios Giotsas <giotsas@hotmail.com> Sent: Saturday, May 31, 2025 8:06 PM To: Giovane C. M. Moura via ripe-atlas <ripe-atlas@ripe.net> Subject: [atlas] Best way to download bulk data older than a month? I'm trying to download the RIPE Atlas traceroutes for a day that is older than one month. Given that the FTP repository publishes only results from the last month, I need to use the API. So, what I'm doing is to list of measurement IDs for a given data and then for each ID I do a GET for the results of that ID. The code is here: https://github.com/vgiotsas/RIPEAtlas_Queries/blob/main/download_atlas_trace... However, often some downloads time out and when I'm trying to resume it using the Range header it seems that the API doesn't support it and the download starts over until it fails again halfway through the download. Is there a better way to do such bulk downloads? Or at least to resume failed downloads? Thanks!
Hi Vasilesios we are glad that you found a solution, in cases like this we can assist you in various ways, for example providing a bulk archive to be downloaded. There would be some limitations, as the bulk archive could only be generated by measurement ID, or by measurement type in a specific date range, but still fully comprehensive and much more efficient that querying the API. Please let us if you need further help kind regards Francesco On 02/06/2025 01:33, Vasileios Giotsas wrote:
Hi again folks.
For anyone having similar issues, I managed to address most issues using curl and forcing HTTP version 1.0. I also fixed some bug I had in my API query parameters (the `start` parameter for `measurements/{pk}/results` has the same effect as the `start_time__lt` for `measurements/` queries). The updated script is here, maybe it can be useful to others too:
https://github.com/vgiotsas/RIPEAtlas_Queries/blob/main/download_atlas_trace... <https://github.com/vgiotsas/RIPEAtlas_Queries/blob/main/download_atlas_traceroutes.py>
------------------------------------------------------------------------ *From:* Vasileios Giotsas <giotsas@hotmail.com> *Sent:* Saturday, May 31, 2025 8:06 PM *To:* Giovane C. M. Moura via ripe-atlas <ripe-atlas@ripe.net> *Subject:* [atlas] Best way to download bulk data older than a month? I'm trying to download the RIPE Atlas traceroutes for a day that is older than one month.
Given that the FTP repository publishes only results from the last month, I need to use the API. So, what I'm doing is to list of measurement IDs for a given data and then for each ID I do a GET for the results of that ID.
The code is here: https://github.com/vgiotsas/RIPEAtlas_Queries/blob/main/download_atlas_trace... <https://github.com/vgiotsas/RIPEAtlas_Queries/blob/main/download_atlas_traceroutes.py>
However, often some downloads time out and when I'm trying to resume it using the Range header it seems that the API doesn't support it and the download starts over until it fails again halfway through the download.
Is there a better way to do such bulk downloads? Or at least to resume failed downloads?
Thanks!
----- To unsubscribe from this mailing list or change your subscription options, please visit:https://mailman.ripe.net/mailman3/lists/ripe-atlas.ripe.net/ As we have migrated to Mailman 3, you will need to create an account with the email matching your subscription before you can change your settings. More details at:https://www.ripe.net/membership/mail/mailman-3-migration/
Hi all, I wanted to raise a related question (apologies if that is not the right place to do so): I’ve found that one of the most effective ways to extract data for a specific date is through the BigQuery table containing the traceroute results. However, in its current form, the table isn’t partitioned by date (or at all), which makes it quite cumbersome to query, as each query scans the entire dataset (currently over 66 TB). Would it be possible to make the RIPE Atlas BigQuery tables partitioned by date (e.g., using start_time or a similar field)? Partitioning would greatly reduce query costs and make it much easier for users to work with the data, especially when focusing on specific time ranges. For example, M-Lab partitions its BigQuery tables by date, which makes their dataset much easier to work with efficiently. Thanks again for your support and for making this data accessible! Best, Loqman
On Jun 2, 2025, at 11:20 AM, Francesco Iannuzzelli <fiannuzzelli@ripe.net> wrote:
Hi Vasilesios we are glad that you found a solution, in cases like this we can assist you in various ways, for example providing a bulk archive to be downloaded. There would be some limitations, as the bulk archive could only be generated by measurement ID, or by measurement type in a specific date range, but still fully comprehensive and much more efficient that querying the API. Please let us if you need further help kind regards Francesco
On 02/06/2025 01:33, Vasileios Giotsas wrote:
Hi again folks.
For anyone having similar issues, I managed to address most issues using curl and forcing HTTP version 1.0. I also fixed some bug I had in my API query parameters (the `start` parameter for `measurements/{pk}/results` has the same effect as the `start_time__lt` for `measurements/` queries). The updated script is here, maybe it can be useful to others too:
https://github.com/vgiotsas/RIPEAtlas_Queries/blob/main/download_atlas_trace...
From: Vasileios Giotsas <giotsas@hotmail.com> <mailto:giotsas@hotmail.com> Sent: Saturday, May 31, 2025 8:06 PM To: Giovane C. M. Moura via ripe-atlas <ripe-atlas@ripe.net> <mailto:ripe-atlas@ripe.net> Subject: [atlas] Best way to download bulk data older than a month?
I'm trying to download the RIPE Atlas traceroutes for a day that is older than one month.
Given that the FTP repository publishes only results from the last month, I need to use the API. So, what I'm doing is to list of measurement IDs for a given data and then for each ID I do a GET for the results of that ID.
The code is here: https://github.com/vgiotsas/RIPEAtlas_Queries/blob/main/download_atlas_trace...
However, often some downloads time out and when I'm trying to resume it using the Range header it seems that the API doesn't support it and the download starts over until it fails again halfway through the download.
Is there a better way to do such bulk downloads? Or at least to resume failed downloads?
Thanks!
----- To unsubscribe from this mailing list or change your subscription options, please visit: https://mailman.ripe.net/mailman3/lists/ripe-atlas.ripe.net/ As we have migrated to Mailman 3, you will need to create an account with the email matching your subscription before you can change your settings. More details at: https://www.ripe.net/membership/mail/mailman-3-migration/
To unsubscribe from this mailing list or change your subscription options, please visit: https://mailman.ripe.net/mailman3/lists/ripe-atlas.ripe.net/ As we have migrated to Mailman 3, you will need to create an account with the email matching your subscription before you can change your settings. More details at: https://www.ripe.net/membership/mail/mailman-3-migration/
Hello Loqman, Glad to hear you're using this dataset. The BigQuery traceroute results are exposed to the public through a *view* (ripencc-atlas.measurements.traceroute). As such, it has no partitioning. However, this view is backed by a real table (you can inspect the view definition under the "Details" tab). This table does have partitioning by start_time. Queries using the public view and filtering on start_time will touch a smaller set of (daily) partitions in the backing table, thus keeping query costs manageable. For example: -- This query will process 18.73 GB when run. SELECT * FROM `ripencc-atlas.measurements.traceroute` WHERE start_time = '2025-06-01' -- This query will process 66.62 TB when run. SELECT * FROM `ripencc-atlas.measurements.traceroute` We have a Getting Started guide here: https://github.com/RIPE-NCC/ripe-atlas-bigquery/blob/main/docs/gettingstarte... Let us know if you still perceive the issue and we will investigate further. Any other comment or suggestion, please send it through! We also have atlas-bq@ripe.net for BigQuery-specific inquiries. Cheers, -- Agustín RIPE NCC
On 2 Jun 2025, at 18:43, Loqman Salamatian <salamatianloqman@gmail.com> wrote:
Hi all,
I wanted to raise a related question (apologies if that is not the right place to do so): I’ve found that one of the most effective ways to extract data for a specific date is through the BigQuery table containing the traceroute results. However, in its current form, the table isn’t partitioned by date (or at all), which makes it quite cumbersome to query, as each query scans the entire dataset (currently over 66 TB).
Would it be possible to make the RIPE Atlas BigQuery tables partitioned by date (e.g., using start_time or a similar field)? Partitioning would greatly reduce query costs and make it much easier for users to work with the data, especially when focusing on specific time ranges. For example, M-Lab partitions its BigQuery tables by date, which makes their dataset much easier to work with efficiently.
Thanks again for your support and for making this data accessible!
Best, Loqman
On Jun 2, 2025, at 11:20 AM, Francesco Iannuzzelli <fiannuzzelli@ripe.net> wrote:
Hi Vasilesios we are glad that you found a solution, in cases like this we can assist you in various ways, for example providing a bulk archive to be downloaded. There would be some limitations, as the bulk archive could only be generated by measurement ID, or by measurement type in a specific date range, but still fully comprehensive and much more efficient that querying the API. Please let us if you need further help kind regards Francesco
On 02/06/2025 01:33, Vasileios Giotsas wrote:
Hi again folks.
For anyone having similar issues, I managed to address most issues using curl and forcing HTTP version 1.0. I also fixed some bug I had in my API query parameters (the `start` parameter for `measurements/{pk}/results` has the same effect as the `start_time__lt` for `measurements/` queries). The updated script is here, maybe it can be useful to others too:
https://github.com/vgiotsas/RIPEAtlas_Queries/blob/main/download_atlas_trace...
From: Vasileios Giotsas <giotsas@hotmail.com> Sent: Saturday, May 31, 2025 8:06 PM To: Giovane C. M. Moura via ripe-atlas <ripe-atlas@ripe.net> Subject: [atlas] Best way to download bulk data older than a month? I'm trying to download the RIPE Atlas traceroutes for a day that is older than one month.
Given that the FTP repository publishes only results from the last month, I need to use the API. So, what I'm doing is to list of measurement IDs for a given data and then for each ID I do a GET for the results of that ID.
The code is here: https://github.com/vgiotsas/RIPEAtlas_Queries/blob/main/download_atlas_trace...
However, often some downloads time out and when I'm trying to resume it using the Range header it seems that the API doesn't support it and the download starts over until it fails again halfway through the download.
Is there a better way to do such bulk downloads? Or at least to resume failed downloads?
Thanks!
----- To unsubscribe from this mailing list or change your subscription options, please visit: https://mailman.ripe.net/mailman3/lists/ripe-atlas.ripe.net/ As we have migrated to Mailman 3, you will need to create an account with the email matching your subscription before you can change your settings. More details at: https://www.ripe.net/membership/mail/mailman-3-migration/
To unsubscribe from this mailing list or change your subscription options, please visit: https://mailman.ripe.net/mailman3/lists/ripe-atlas.ripe.net/ As we have migrated to Mailman 3, you will need to create an account with the email matching your subscription before you can change your settings. More details at: https://www.ripe.net/membership/mail/mailman-3-migration/
----- To unsubscribe from this mailing list or change your subscription options, please visit: https://mailman.ripe.net/mailman3/lists/ripe-atlas.ripe.net/ As we have migrated to Mailman 3, you will need to create an account with the email matching your subscription before you can change your settings. More details at: https://www.ripe.net/membership/mail/mailman-3-migration/
Hi, Ah, I must have missed that while experimenting but that behavior is exactly what I was hoping for, so that’s great. Quick follow-up: how far back does the data go? The earliest date I’ve found with data is 2024-12-04, which is essentially six months ago. Is there a way to access older data with these tables? Thanks, Loqman
On Jun 2, 2025, at 1:05 PM, Agustín Formoso <aformoso@ripe.net> wrote:
Hello Loqman,
Glad to hear you're using this dataset.
The BigQuery traceroute results are exposed to the public through a *view* (ripencc-atlas.measurements.traceroute). As such, it has no partitioning.
However, this view is backed by a real table (you can inspect the view definition under the "Details" tab). This table does have partitioning by start_time. Queries using the public view and filtering on start_time will touch a smaller set of (daily) partitions in the backing table, thus keeping query costs manageable. For example:
-- This query will process 18.73 GB when run. SELECT * FROM `ripencc-atlas.measurements.traceroute` WHERE start_time = '2025-06-01'
-- This query will process 66.62 TB when run. SELECT * FROM `ripencc-atlas.measurements.traceroute`
We have a Getting Started guide here: https://github.com/RIPE-NCC/ripe-atlas-bigquery/blob/main/docs/gettingstarte...
Let us know if you still perceive the issue and we will investigate further.
Any other comment or suggestion, please send it through! We also have atlas-bq@ripe.net for BigQuery-specific inquiries.
Cheers,
-- Agustín RIPE NCC
On 2 Jun 2025, at 18:43, Loqman Salamatian <salamatianloqman@gmail.com> wrote:
Hi all,
I wanted to raise a related question (apologies if that is not the right place to do so): I’ve found that one of the most effective ways to extract data for a specific date is through the BigQuery table containing the traceroute results. However, in its current form, the table isn’t partitioned by date (or at all), which makes it quite cumbersome to query, as each query scans the entire dataset (currently over 66 TB).
Would it be possible to make the RIPE Atlas BigQuery tables partitioned by date (e.g., using start_time or a similar field)? Partitioning would greatly reduce query costs and make it much easier for users to work with the data, especially when focusing on specific time ranges. For example, M-Lab partitions its BigQuery tables by date, which makes their dataset much easier to work with efficiently.
Thanks again for your support and for making this data accessible!
Best, Loqman
On Jun 2, 2025, at 11:20 AM, Francesco Iannuzzelli <fiannuzzelli@ripe.net> wrote:
Hi Vasilesios we are glad that you found a solution, in cases like this we can assist you in various ways, for example providing a bulk archive to be downloaded. There would be some limitations, as the bulk archive could only be generated by measurement ID, or by measurement type in a specific date range, but still fully comprehensive and much more efficient that querying the API. Please let us if you need further help kind regards Francesco
On 02/06/2025 01:33, Vasileios Giotsas wrote:
Hi again folks.
For anyone having similar issues, I managed to address most issues using curl and forcing HTTP version 1.0. I also fixed some bug I had in my API query parameters (the `start` parameter for `measurements/{pk}/results` has the same effect as the `start_time__lt` for `measurements/` queries). The updated script is here, maybe it can be useful to others too:
https://github.com/vgiotsas/RIPEAtlas_Queries/blob/main/download_atlas_trace...
From: Vasileios Giotsas <giotsas@hotmail.com> Sent: Saturday, May 31, 2025 8:06 PM To: Giovane C. M. Moura via ripe-atlas <ripe-atlas@ripe.net> Subject: [atlas] Best way to download bulk data older than a month? I'm trying to download the RIPE Atlas traceroutes for a day that is older than one month.
Given that the FTP repository publishes only results from the last month, I need to use the API. So, what I'm doing is to list of measurement IDs for a given data and then for each ID I do a GET for the results of that ID.
The code is here: https://github.com/vgiotsas/RIPEAtlas_Queries/blob/main/download_atlas_trace...
However, often some downloads time out and when I'm trying to resume it using the Range header it seems that the API doesn't support it and the download starts over until it fails again halfway through the download.
Is there a better way to do such bulk downloads? Or at least to resume failed downloads?
Thanks!
----- To unsubscribe from this mailing list or change your subscription options, please visit: https://mailman.ripe.net/mailman3/lists/ripe-atlas.ripe.net/ As we have migrated to Mailman 3, you will need to create an account with the email matching your subscription before you can change your settings. More details at: https://www.ripe.net/membership/mail/mailman-3-migration/
To unsubscribe from this mailing list or change your subscription options, please visit: https://mailman.ripe.net/mailman3/lists/ripe-atlas.ripe.net/ As we have migrated to Mailman 3, you will need to create an account with the email matching your subscription before you can change your settings. More details at: https://www.ripe.net/membership/mail/mailman-3-migration/
----- To unsubscribe from this mailing list or change your subscription options, please visit: https://mailman.ripe.net/mailman3/lists/ripe-atlas.ripe.net/ As we have migrated to Mailman 3, you will need to create an account with the email matching your subscription before you can change your settings. More details at: https://www.ripe.net/membership/mail/mailman-3-migration/
Indeed at the moment all measurement tables' partitions expire after 180d, and there's no way to access expired partitions. You can always create a personal copy of the data that's currently available, and set your own expiration rules. You could filter heavily and keep only the data of interest for your use case. Hope this helps, -- Agustín RIPE NCC
On 2 Jun 2025, at 20:53, Loqman Salamatian <salamatianloqman@gmail.com> wrote:
Hi,
Ah, I must have missed that while experimenting but that behavior is exactly what I was hoping for, so that’s great.
Quick follow-up: how far back does the data go? The earliest date I’ve found with data is 2024-12-04, which is essentially six months ago. Is there a way to access older data with these tables?
Thanks, Loqman
On Jun 2, 2025, at 1:05 PM, Agustín Formoso <aformoso@ripe.net> wrote:
Hello Loqman,
Glad to hear you're using this dataset.
The BigQuery traceroute results are exposed to the public through a *view* (ripencc-atlas.measurements.traceroute). As such, it has no partitioning.
However, this view is backed by a real table (you can inspect the view definition under the "Details" tab). This table does have partitioning by start_time. Queries using the public view and filtering on start_time will touch a smaller set of (daily) partitions in the backing table, thus keeping query costs manageable. For example:
-- This query will process 18.73 GB when run. SELECT * FROM `ripencc-atlas.measurements.traceroute` WHERE start_time = '2025-06-01'
-- This query will process 66.62 TB when run. SELECT * FROM `ripencc-atlas.measurements.traceroute`
We have a Getting Started guide here: https://github.com/RIPE-NCC/ripe-atlas-bigquery/blob/main/docs/gettingstarte...
Let us know if you still perceive the issue and we will investigate further.
Any other comment or suggestion, please send it through! We also have atlas-bq@ripe.net for BigQuery-specific inquiries.
Cheers,
-- Agustín RIPE NCC
On 2 Jun 2025, at 18:43, Loqman Salamatian <salamatianloqman@gmail.com> wrote:
Hi all,
I wanted to raise a related question (apologies if that is not the right place to do so): I’ve found that one of the most effective ways to extract data for a specific date is through the BigQuery table containing the traceroute results. However, in its current form, the table isn’t partitioned by date (or at all), which makes it quite cumbersome to query, as each query scans the entire dataset (currently over 66 TB).
Would it be possible to make the RIPE Atlas BigQuery tables partitioned by date (e.g., using start_time or a similar field)? Partitioning would greatly reduce query costs and make it much easier for users to work with the data, especially when focusing on specific time ranges. For example, M-Lab partitions its BigQuery tables by date, which makes their dataset much easier to work with efficiently.
Thanks again for your support and for making this data accessible!
Best, Loqman
On Jun 2, 2025, at 11:20 AM, Francesco Iannuzzelli <fiannuzzelli@ripe.net> wrote:
Hi Vasilesios we are glad that you found a solution, in cases like this we can assist you in various ways, for example providing a bulk archive to be downloaded. There would be some limitations, as the bulk archive could only be generated by measurement ID, or by measurement type in a specific date range, but still fully comprehensive and much more efficient that querying the API. Please let us if you need further help kind regards Francesco
On 02/06/2025 01:33, Vasileios Giotsas wrote:
Hi again folks.
For anyone having similar issues, I managed to address most issues using curl and forcing HTTP version 1.0. I also fixed some bug I had in my API query parameters (the `start` parameter for `measurements/{pk}/results` has the same effect as the `start_time__lt` for `measurements/` queries). The updated script is here, maybe it can be useful to others too:
https://github.com/vgiotsas/RIPEAtlas_Queries/blob/main/download_atlas_trace...
From: Vasileios Giotsas <giotsas@hotmail.com> Sent: Saturday, May 31, 2025 8:06 PM To: Giovane C. M. Moura via ripe-atlas <ripe-atlas@ripe.net> Subject: [atlas] Best way to download bulk data older than a month? I'm trying to download the RIPE Atlas traceroutes for a day that is older than one month.
Given that the FTP repository publishes only results from the last month, I need to use the API. So, what I'm doing is to list of measurement IDs for a given data and then for each ID I do a GET for the results of that ID.
The code is here: https://github.com/vgiotsas/RIPEAtlas_Queries/blob/main/download_atlas_trace...
However, often some downloads time out and when I'm trying to resume it using the Range header it seems that the API doesn't support it and the download starts over until it fails again halfway through the download.
Is there a better way to do such bulk downloads? Or at least to resume failed downloads?
Thanks!
----- To unsubscribe from this mailing list or change your subscription options, please visit: https://mailman.ripe.net/mailman3/lists/ripe-atlas.ripe.net/ As we have migrated to Mailman 3, you will need to create an account with the email matching your subscription before you can change your settings. More details at: https://www.ripe.net/membership/mail/mailman-3-migration/
To unsubscribe from this mailing list or change your subscription options, please visit: https://mailman.ripe.net/mailman3/lists/ripe-atlas.ripe.net/ As we have migrated to Mailman 3, you will need to create an account with the email matching your subscription before you can change your settings. More details at: https://www.ripe.net/membership/mail/mailman-3-migration/
----- To unsubscribe from this mailing list or change your subscription options, please visit: https://mailman.ripe.net/mailman3/lists/ripe-atlas.ripe.net/ As we have migrated to Mailman 3, you will need to create an account with the email matching your subscription before you can change your settings. More details at: https://www.ripe.net/membership/mail/mailman-3-migration/
----- To unsubscribe from this mailing list or change your subscription options, please visit: https://mailman.ripe.net/mailman3/lists/ripe-atlas.ripe.net/ As we have migrated to Mailman 3, you will need to create an account with the email matching your subscription before you can change your settings. More details at: https://www.ripe.net/membership/mail/mailman-3-migration/
participants (4)
-
Agustín Formoso -
Francesco Iannuzzelli -
Loqman Salamatian -
Vasileios Giotsas