- dns-wg - mailman.ripe.net

post-mortem for ripe.net DNSSEC problem on 1 November 2023
by Paul de Weerd 04 Nov '23

04 Nov '23

Dear colleagues, Please find below the post mortem for the DNSSEC problem that caused most of RIPE NCC's services to become unavailable yesterday. Please reach out if you have any questions or feedback. Thanks, Paul de Weerd Manager Global Information Infrastructure team RIPE NCC Summary On 1 November, from 10:45 to 12:15 UTC, most names in the ripe.net zone were bogus due to expired DNSSEC signatures being served. This rendered most of the RIPE NCC’s services unreachable. After investigating the issue, we found a typo in a change to our zone where a record had a TTL that was longer (864,000 seconds instead of 86,400) than the refresh interval for RRSIGs (seven days). This caused our signer to stop refreshing signatures and only sign changes to the zone. We are talking to the vendor of our DNSSEC signing solution about this case to see what can be improved on that end, have implemented a pre-commit check to prevent TTLs longer than a day in the ripe.net zone and are looking at improving monitoring for stale signatures to spot issues like this before they cause problems. Impact DNSSEC signatures in the ripe.net zone are valid for 14 days, with our signers configured to resign them after half that time (seven days). On 1 November at 10:45 UTC the signature on several records in the ripe.net zone expired. These records had last been signed on 18 October and were due to be re-signed on the 25th. However, due to a problem with the TTL on one record, our signer stopped re-signing records in the zone on 25 October. This resulted in the expiry of 11,026 out of 11,389 records on 1 November. New or changed records were still properly signed (363 of them), which meant that our monitoring, which checks the signature validity of the SOA record at the zone apex, missed this issue. Because our internal resolvers are configured for DNSSEC validation, the impact was rather immediate for staff, as many internal services broke due to this issue. After first dismissing some alternative causes, we quickly found the problem was with expired signatures in the ripe.net zone, so we turned our attention to our signers. At the same time, we temporarily disabled DNSSEC validation on our internal resolvers so we could more easily access our own systems while troubleshooting. Resolution While debugging, we found that the rrsig-refresh option that we configured to seven days (half the value of the rrsig-lifetime option of 14 days) was likely involved, logs showed: info: [ripe.net.] DNSSEC, signing zone error: [ripe.net.] DNSSEC, rrsig-refresh too low to prevent expired RRSIGs in resolver caches info: [ripe.net.] DNSSEC, next signing at 2023-10-25T10:02:02+0000 error: [ripe.net.] zone event 're-sign' failed (invalid parameter) At 12:14 UTC we removed that option from our configuration and we could sign the zone again. The freshly signed zone was pushed out and went live a little bit later, which meant that at 12:15 UTC our services were available again for most users. Unfortunately, some users kept seeing problems for several hours after we restored the signatures. Root cause After further investigation we found that the change that triggered this problem introduced a record in the ripe.net zone with a TTL of 864,000 (ten days). Because this TTL is longer than our rrsig-refresh configuration, this could lead to cases where a resolver’s cache contains the record with an expired signature. The signer software rightfully complained about this. We were surprised to find it then stopped refreshing signatures for all records in the zone that didn’t change. Future steps During the incident and the aftermath we identified a few changes that we want to make to improve the resiliency of our setup and allow us to find cases like these before they become problems. Our current RRSIG freshness monitoring did not catch this case, because the records we monitor still had valid and recent signatures, so we are considering what we can do to cover this situation. We have also improved our zone-editing pipeline to catch typos or misconfigurations for TTL values. Next to that, the problem also affected our ability to communicate internally, as our internal chat system was unresolvable too. We have some means of out-of-band communication, but will review how we can improve that. Additionally, while the status.ripe.net website is hosted on separate infrastructure, the fact that it is also in the ripe.net domain meant that it was just as unreachable as our other services. We will evaluate this approach and see how we can improve on it. Timeline (times in UTC) 25 October 08:52 a record was added to the ripe.net zone with a TTL of 10 days 08:53 knot incrementally signs ripe.net successfully 09:02 knot fails to sign the ripe.net zone for the first time 1 November 10:45 ripe.net signatures expire and many records go bogus 11:27 DNSSEC validation on internal resolvers was disabled 12:14 changed configuration and manually re-signed zone 12:15 ripe.net zone has new valid signatures 12:38 DNSSEC validation on internal resolvers is re-enabled 2 November 08:39 typo in TTL fixed, bringing it back to 86,400 seconds as intended 08:39 added check in pipeline to detect too large TTL values

4 3

NET zone operational announcement: DNSSEC algorithm rollover
by Wessels, Duane 31 Oct '23

31 Oct '23

Verisign will soon begin the transition to DNSSEC algorithm 13 (ECDSA) for the NET zone. Over the next few days, algorithm 13 signatures will start to appear in the zone, followed by the algorithm 13 DNSKEY records. We expect the DS record for the NET zone to change from algorithm 8 to algorithm 13 on or around November 8th. Additional information can be found at our blog post and DNS-OARC presentation URLs below: https://blog.verisign.com/security/dnssec-algorithm-update/ https://indico.dns-oarc.net/event/47/contributions/1012/ We expect no operational impacts for end users, but please feel free to reach out to us with concerns or any observations of anomalous behavior related to this algorithm rollover. DW

1 0

DNS and K-root Quarterly Planning Q4 2023
by Paul de Weerd 28 Sep '23

28 Sep '23

Dear colleagues, The DNS Services quarterly plans, including K-root, are published on our website: https://www.ripe.net/support/documentation/quarterly-planning/dns-and-k-root In this quarter, we'll continue with our day-to-day operations and update the operating system of some of our DNS servers. We also plan on deploying a new AuthDNS core site in Asia. We hope that by providing these plans, we can better inform the community of the ongoing developments on the DNS services we operate and get feedback on our work. If you have any input on these plans or suggestions for future work, please reach out to us. Kind regards, Paul de Weerd RIPE NCC

1 0

Reminder: RIPE NCC Open House Event on Community Projects Fund Updates - 27 September, 10:00 - 11:30 UTC
by Alastair Strachan 25 Sep '23

25 Sep '23

Dear colleagues, I hope all is well with you. Just a quick note to remind you about the RIPE NCC Open House event happening on Wednesday, 27th September, from 10:00 to 11:30 UTC. It's an excellent chance to hear firsthand updates from the recipients of last year’s Community Projects Fund and to engage in meaningful dialogue with project leaders. The agenda includes presentations where each project will provide an overview of their journey and accomplishments so far. Following the presentations, there will be a Q&A segment for any questions you may have. The projects and speakers lined up for the event are: 1. iNGO European Media Platform Project: Internet Identifiers in Time of War Speaker: Oksana Prykhodko 2. Aristotle University of Thessaloniki Project: AI4NetMon 2.0 Speaker: Pavlos Sermpezis 3. deSEC e.V. Project: deSEC: Secure DNSSEC Hosting Speaker: Peter Thomassen For more details and to register for the event, please visit the following link: https://www.ripe.net/participate/meetings/open-house/ripe-ncc-open-house-se… We hope to see you there! Kind Regards, Alastair Strachan RIPE NCC

1 0

Reminder: Tomorrow's RIPE NCC Open House Event on Community Projects Fund Updates - 21 September, 10:00 - 11:30 UTC
by Alastair Strachan 20 Sep '23

20 Sep '23

Dear colleagues, I hope all is well with you. Just a quick note to remind you about the RIPE NCC Open House event happening tomorrow, Thursday, 21 September 2023, from 10:00 to 11:30 UTC. It's an excellent chance to hear firsthand updates from the recipients of last year’s Community Projects Fund and to engage in meaningful dialogue with project leaders. The agenda includes presentations where each project will provide an overview of their journey and accomplishments so far. Following the presentations, there will be a Q&A segment for any questions you may have. The projects and speakers lined up for tomorrow are: 1. Foundation for Research and Technology - Hellas (FORTH) Project: ARTEMIS Lite Speaker: George Nomikos 2. Grenoble Alps University Project: fpdns2: DNS Fingerprinting Tool Speaker: Yevheniya Nosyk 3. University of Aberdeen Project: Can We Extend IPv6? - A Measurement Project Speaker: Ana Custura For more details and to register for the event, please visit the following link: https://www.ripe.net/participate/meetings/open-house/ripe-ncc-open-house-se… https://www.ripe.net/participate/meetings/open-house/ripe-ncc-open-house-se… We hope to see you there! Kind Regards, Alastair Strachan RIPE NCC

1 0

Root zone operational announcement: introducing ZONEMD for the root zone
by Wessels, Duane 12 Sep '23

12 Sep '23

I am pleased to announce that Message Digests for DNS Zones, also known as ZONEMD, will be added to the root zone later this year. This feature, specified in RFC 8976, adds cryptographic data protections to the zone as a whole, allowing the recipient to verify the authenticity of the zone’s contents. ZONEMD will be added to the root zone using a phased approach. On September 13, 2023, a ZONEMD record will make its first appearance in the root zone. At this time the Hash Algorithm field will be set to a private use algorithm number, making the ZONEMD record deliberately unverifiable. On December 6, 2023, the ZONEMD record will be published with the SHA-384 Hash Algorithm, thereby making it verifiable. We expect no operational impacts for end users. ZONEMD does not affect root zone queries and responses. The root server operators have agreed to not alter their zone ingestion processes for at least a year after ZONEMD is first introduced. Anyone that downloads the root zone file from www.internic.net or rs.internic.net should be aware that it will include the new ZONEMD resource record in its native presentation format starting on September 6th. Please feel free to follow up with any questions or concerns. References and further reading: [1] RFC 8976: “Message Digest for DNS Zones”, https://www.rfc-editor.org/rfc/rfc8976 [2] Root Server Operators Statement on adding ZONEMD to the root zone, https://root-servers.org/media/news/2022-08-Statement_on_ZONEMD.pdf [3] RZERC003: “Adding Zone Data Protections to the Root Zone”, https://www.icann.org/uploads/ckeditor/rzerc-003-en.pdf [4] Verisign Blog: “Adding ZONEMD Protections to the Root Zone”, https://blog.verisign.com/security/root-zone-zonemd/ [5] APNIC Ping Podcast episode “Adding ZONEMD protections to the root zone”, https://blubrry.com/ping_podcast/108940688/adding-zonemd-protections-to-the… DW

1 1

EDU zone operational announcement: DNSSEC algorithm rollover
by Wessels, Duane 12 Sep '23

12 Sep '23

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 Verisign is pleased to announce that an algorithm 13 (ECDSA) DS record has been published for the EDU zone, and the algorithm 8 record has been removed. Over the next few days, the algorithm 8 DNSKEY records will be removed from the EDU zone, followed by the removal of algorithm 8 signatures. We continue to expect no operational impacts for end users, but please feel free to reach out to us with concerns or any observations of anomalous behavior related to this algorithm rollover. DW -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (GNU/Linux) iQEcBAEBCAAGBQJlAIwcAAoJEGyZpGmowJiNEj4H/R07bC4outa2znVIh7BIsolu mamlG2+NlOBmdAOTzk8vdmoUMy11RHeG0W+6o6oQS9gALUgj4shxPF5z+Je0R6Us Y54HDnE4mi1KAGKOtg/AE5RYCZeA/rhLk/qxRnLiUBtlmuRwMpjkJ5k+viCEFVX+ IyZPWEJyDSC8bFx4LThYt7AP093r74vjKHa2wMffa+3VHZgrEpHReL6Na149+u3L Idb1In9cCs6nYFp/DuMopdVHy3EqK6psPSCt7jCG27qAtM4+ormsCPYYo4KI7RUo lF7X2SdDmVUfoL4lzrLUzl7MFBxJxNYCt/yPvN/VBLw2Yby2AhPHVQt7lpjIQXo= =/uAl -----END PGP SIGNATURE-----

1 0

Invitation: RIPE NCC Open House Series on Community Projects Fund Updates - September 2023
by Alastair Strachan 07 Sep '23

07 Sep '23

Dear colleagues, For over 30 years, the RIPE NCC has been an essential part of the Internet technical community. Seven years ago, we took our commitment a step further by introducing the Community Projects Fund. This initiative was designed to tackle the funding challenges faced by many noteworthy projects in our community. We're thrilled to bring you updates on the incredible progress these initiatives have made. We warmly invite you to attend our upcoming RIPE NCC Open House Series, focusing on the advancements made by last year’s Community Projects Fund recipients. In each session, the project teams will offer presentations summarising their journey and current status, followed by a Q&A session during which you can engage directly with them. Wednesday, 13 September 2023, 13:30 - 14:30 UTC 1. Elevating Demand-side Cybersecurity Through Business Procurement - Wout de Natris 2. DNS Research Federation - Internet Standards Observatory - Carolina Caeiro Thursday, 21 September 2023, 10:00 - 11:30 UTC 1. Foundation for Research and Technology - Hellas (FORTH) - ARTEMIS Lite - George Nomikos 2. Grenoble Alps University - fpdns2: DNS Fingerprinting Tool - Maciej Korczyński, Ph.D. 3. University of Aberdeen - Can We Extend IPv6? - A Measurement Project - Ana Custura Wednesday, 27 September 2023, 10:00 - 11:30 UTC 1. iNGO European Media Platform - Internet Identifiers in Time of War - Oksana Prykhodko 2. Aristotle University of Thessaloniki - AI4NetMon 2.0 - Pavlos Sermpezis 3. deSEC e.V. - deSEC: Secure DNSSEC Hosting - Peter Thomassen For more information on these upcoming events and how to register, please see here: https://www.ripe.net/participate/meetings/open-house/ripe-ncc-open-house-se… Kind Regards Alastair Strachan RIPE NCC

1 0

EDU zone operational announcement: DNSSEC algorithm rollover
by Wessels, Duane 05 Sep '23

05 Sep '23

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 Verisign will soon begin the transition to DNSSEC algorithm 13 (ECDSA) for the EDU zone. Over the next few days, algorithm 13 signatures will start to appear in the zone, followed by the algorithm 13 DNSKEY records. We expect the DS record for the EDU zone to change from algorithm 8 to algorithm 13 between September 12-15. Additional information can be found at our blog post and DNS-OARC presentation URLs below: https://blog.verisign.com/security/dnssec-algorithm-update/ https://indico.dns-oarc.net/event/47/contributions/1012/ We expect no operational impacts for end users, but please feel free to reach out to us with concerns or any observations of anomalous behavior related to this algorithm rollover. DW -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (GNU/Linux) iQEcBAEBCAAGBQJk90tSAAoJEGyZpGmowJiNynsIAIWBx6Gc8CyOl0pcr65a3aUh FnrrteufEs1KvNk+HtexDs8opUGz8sKf4zG/ABj4NtTQFRwsvD/nxh8q+fnu4Vci niSPbJLZcadY+344FioGNE4/jE28jKmItx7fFmTBoaBcF5iy1aS5xSh4vOA6fcSy snJeYd6VsnrezK6IujgmiqAg7iLwl+YHvPnu8lJuXEpUPqBmO/EEHZPn7oEps8Xf A8duusfKIvP6zrIF0vnnqwS0TgUpTdam5MGtVAkv+JUlpzRmqAYNcQd872nYUVIp 5qZ1R+pjHsngwe+8pf/LQtpYwKjaiP06CUbZH8hlg3ioDWSLEAjs0NEgb45Wmhw= =Wi2m -----END PGP SIGNATURE-----

1 0

Call for nominations/volunteers to co-chair the RIPE DNS working group.
by Joao Luis Silva Damas 05 Sep '23

05 Sep '23

Dear Colleagues, This is a call for nominations or volunteers to co-chair the RIPE DNS working group. ************************************************************************************* Please DO NOT reply to this message if you wish to propose a volunteer. ************************************************************************************* Call for RIPE DNS Working Group Chair ------------------------------------------------------ In accordance with the chair selection process we announce that the term of one of our co-chairs (Joao Damas) will expire at the upcoming RIPE Meeting (RIPE 87, near Rome) and we would like to invite volunteers, or nominations for someone to become a co-chair of the DNS working group. Anyone may volunteer, and anyone may be nominated by someone else (we will check with the person for acceptance). Note that having served two terms, in accordance with the selection process Joao is not eligible. To do this, please send a mail to <dns-wg-chairs AT ripe.net <mailto:dns-wg-chairs@ripe.net>> before end of Sunday 5 November 2023 (UTC). Again, please DO NOT send your volunteer/nomination to the DNS WG mailing but rather the dns-wg-chairs mail alias. It would be helpful for candidates to give some words as to why they think they ought to be a co-chair, what new ideas, or improvements they might bring, and so on. We will publish the list of volunteers, along with anything the volunteers wish to include with their nomination, to this mailing list on Monday 6 November 2023. Discussion and support (or opposition) for each candidate can then be offered on the list. At the DNS working group session at RIPE 87, Moritz and Wilhelm hope to announce consensus based on the mailing list discussion. Timeline ———— Tuesday, 5 September 2023 This call for volunteers posted to the dns-wg mail list Sunday, 5 November 2023 Deadline for volunteers/nominations Monday, 6 November 2023 List of candidates and call for support posted to the mail list Monday, 27 November 2023 Deadline for showing support for candidates/volunteers on the mail list Tuesday, 28 November 2023 DNS WG session at RIPE 87 References ---------- The RIPE working group chair job description is included in ripe-692: https://www.ripe.net/publications/docs/ripe-692 The DNS working group chair selection process is documented here: https://www.ripe.net/participate/ripe/wg/dns/dns-wg-chair-selection-process We have 3 co-chairs so the term is 3 years. The current approach was announced on this mailing list on 2018-07-17: https://www.ripe.net/ripe/mail/archives/dns-wg/2018-July/003566.html As always, please feel free to reach out to any of the chairs directly, to us as a group at dns-wg-chairs AT ripe.net <mailto:dns-wg-chairs@ripe.net>, or discuss this or any other any relevant topic on this mailing list.

1 0