IPv6 Multihoming with Load Balancing
My goal is to use multiple uplinks, but not only for redundancy. Most of the time, all (in my case 2) uplinks are available and then the question is how to make use of both of them. With IPv4, NAT is common and thus the solution is quite simple. In my case, I am using the mwan3 package from OpenWrt. It uses iptables rules to add firewall marks to connections. If multiple uplinks are available, then the mark/uplink is chosen randomly and assigned to this (e.g. TCP) connection. This firewalls marks are used during a policy based routing. With a masquerade/source NAT, the right source address for the used route is picked and everything just works. In case of IPv6, everything is different. NAT is uncommon. One solution is to enable NAT and then everything works as with IPv4. Alternatively, RFC 8678 describes that clients can be informed about multiple uplinks. The limitation: I do not see any option for load balancing. RFC 8678 references other solutions. Shim6 seems to be not widely implemented. The Multipath Transports look like a solution for the future with Mulitpath TCP. The last solution is NPTv6. RFC 8678 does not like the solution. It is no NAT, but it still rewrites the addresses. The disadvantage: Stateless address rewriting seems only usable if there is only one prefix known to the network. If this is the global prefix of one uplink, then all connections are interrupted if the prefix of this uplink is changed. If this is the local prefix, then the clients do not know their public addresses. I tried to use a stateful source address rewriting instead. With nftables, this is easy to implement and it works if the prefix length of the uplink is longer (smaller subnet) than the internal network: Just keep the prefix and replace the bits after it with the original source address. With this, I can use local addresses in the local network and additionally provide the public address/es of one or more uplinks. I am using this in production at one location since multiple years and thus know that this works. I am interested in other approaches, experiences and feedback for this method.
Hello Jonas, I would say the right way to go is get an ASN and do proper BGP routing. Everything you describe, is imho basically a textbook example of an autonomous system. Maria On March 6, 2025 1:47:29 PM GMT+01:00, Jonas Lochmann <ripe-ipv6-wg@jonaslochmann.de> wrote:
My goal is to use multiple uplinks, but not only for redundancy. Most of the time, all (in my case 2) uplinks are available and then the question is how to make use of both of them.
With IPv4, NAT is common and thus the solution is quite simple. In my case, I am using the mwan3 package from OpenWrt. It uses iptables rules to add firewall marks to connections. If multiple uplinks are available, then the mark/uplink is chosen randomly and assigned to this (e.g. TCP) connection. This firewalls marks are used during a policy based routing. With a masquerade/source NAT, the right source address for the used route is picked and everything just works.
In case of IPv6, everything is different. NAT is uncommon. One solution is to enable NAT and then everything works as with IPv4. Alternatively, RFC 8678 describes that clients can be informed about multiple uplinks. The limitation: I do not see any option for load balancing.
RFC 8678 references other solutions. Shim6 seems to be not widely implemented. The Multipath Transports look like a solution for the future with Mulitpath TCP. The last solution is NPTv6. RFC 8678 does not like the solution. It is no NAT, but it still rewrites the addresses.
The disadvantage: Stateless address rewriting seems only usable if there is only one prefix known to the network. If this is the global prefix of one uplink, then all connections are interrupted if the prefix of this uplink is changed. If this is the local prefix, then the clients do not know their public addresses.
I tried to use a stateful source address rewriting instead. With nftables, this is easy to implement and it works if the prefix length of the uplink is longer (smaller subnet) than the internal network: Just keep the prefix and replace the bits after it with the original source address. With this, I can use local addresses in the local network and additionally provide the public address/es of one or more uplinks.
I am using this in production at one location since multiple years and thus know that this works. I am interested in other approaches, experiences and feedback for this method. ----- To unsubscribe from this mailing list or change your subscription options, please visit: https://mailman.ripe.net/mailman3/lists/ipv6-wg.ripe.net/ As we have migrated to Mailman 3, you will need to create an account with the email matching your subscription before you can change your settings. More details at: https://www.ripe.net/membership/mail/mailman-3-migration/
-- Maria Matejka (she/her) | BIRD Team Leader | CZ.NIC, z.s.p.o.
Hi all, the immediate problem with BGP (ignoring the logistic and legal aspects) would be utilizing both uplinks in an active/active fashion. Or rather, ensuring the return traffic honors the original uplink choice of the request traffic. Paolo On Thu, Mar 6, 2025, 13:59 Maria Matejka via ipv6-wg <ipv6-wg@ripe.net> wrote:
Hello Jonas,
I would say the right way to go is get an ASN and do proper BGP routing. Everything you describe, is imho basically a textbook example of an autonomous system.
Maria
On March 6, 2025 1:47:29 PM GMT+01:00, Jonas Lochmann < ripe-ipv6-wg@jonaslochmann.de> wrote:
My goal is to use multiple uplinks, but not only for redundancy. Most of the time, all (in my case 2) uplinks are available and then the question is how to make use of both of them.
With IPv4, NAT is common and thus the solution is quite simple. In my case, I am using the mwan3 package from OpenWrt. It uses iptables rules to add firewall marks to connections. If multiple uplinks are available, then the mark/uplink is chosen randomly and assigned to this (e.g. TCP) connection. This firewalls marks are used during a policy based routing. With a masquerade/source NAT, the right source address for the used route is picked and everything just works.
In case of IPv6, everything is different. NAT is uncommon. One solution is to enable NAT and then everything works as with IPv4. Alternatively, RFC 8678 describes that clients can be informed about multiple uplinks. The limitation: I do not see any option for load balancing.
RFC 8678 references other solutions. Shim6 seems to be not widely implemented. The Multipath Transports look like a solution for the future with Mulitpath TCP. The last solution is NPTv6. RFC 8678 does not like the solution. It is no NAT, but it still rewrites the addresses.
The disadvantage: Stateless address rewriting seems only usable if there is only one prefix known to the network. If this is the global prefix of one uplink, then all connections are interrupted if the prefix of this uplink is changed. If this is the local prefix, then the clients do not know their public addresses.
I tried to use a stateful source address rewriting instead. With nftables, this is easy to implement and it works if the prefix length of the uplink is longer (smaller subnet) than the internal network: Just keep the prefix and replace the bits after it with the original source address. With this, I can use local addresses in the local network and additionally provide the public address/es of one or more uplinks.
I am using this in production at one location since multiple years and thus know that this works. I am interested in other approaches, experiences and feedback for this method. ------------------------------ To unsubscribe from this mailing list or change your subscription options, please visit: https://mailman.ripe.net/mailman3/lists/ipv6-wg.ripe.net/ As we have migrated to Mailman 3, you will need to create an account with the email matching your subscription before you can change your settings. More details at: https://www.ripe.net/membership/mail/mailman-3-migration/
-- Maria Matejka (she/her) | BIRD Team Leader | CZ.NIC, z.s.p.o.
To unsubscribe from this mailing list or change your subscription options, please visit: https://mailman.ripe.net/mailman3/lists/ipv6-wg.ripe.net/ As we have migrated to Mailman 3, you will need to create an account with the email matching your subscription before you can change your settings. More details at: https://www.ripe.net/membership/mail/mailman-3-migration/
Am 06.03.2025 um 14:38:13 Uhr schrieb Paolo Nero:
the immediate problem with BGP (ignoring the logistic and legal aspects) would be utilizing both uplinks in an active/active fashion. Or rather, ensuring the return traffic honors the original uplink choice of the request traffic.
Why is that relevant? -- Gruß Marco Send unsolicited bulk mail to 1741268293muell@cartoonies.org
the immediate problem with BGP (ignoring the logistic and legal aspects) would be utilizing both uplinks in an active/active fashion. Or rather, ensuring the return traffic honors the original uplink choice of the request traffic.
Why is that relevant?
Imagine for example a network admin who wishes to use a "fat pipe" uplink for large file downloads: the intuitive approach to accomplish this would be for him to configure the gateway so that it forwarded e.g. Dropbox traffic over that link. If the return flow came back from the other uplink, perhaps having a much smaller capacity, that would quickly saturate it and render the steering pointless. The same considerations apply to uplinks with different delay/jitter characteristics, or even privacy (private circuits vs a shared medium like GPON) and the jurisdictions the data flows through. Simply put, it is intuitive for admins to assume that the return traffic will come back from the same link (of course, it is incorrect to assume so). An asymmetric return may also cause issues when the gateway is a stateful device such as a firewall. Paolo On Thu, Mar 6, 2025, 17:49 Marco Moock <mm@dorfdsl.de> wrote:
Am 06.03.2025 um 14:38:13 Uhr schrieb Paolo Nero:
the immediate problem with BGP (ignoring the logistic and legal aspects) would be utilizing both uplinks in an active/active fashion. Or rather, ensuring the return traffic honors the original uplink choice of the request traffic.
Why is that relevant?
-- Gruß Marco
Send unsolicited bulk mail to 1741268293muell@cartoonies.org ----- To unsubscribe from this mailing list or change your subscription options, please visit: https://mailman.ripe.net/mailman3/lists/ipv6-wg.ripe.net/ As we have migrated to Mailman 3, you will need to create an account with the email matching your subscription before you can change your settings. More details at: https://www.ripe.net/membership/mail/mailman-3-migration/
Hi, Jonas Lochmann wrote:
I tried to use a stateful source address rewriting instead. With nftables, this is easy to implement and it works if the prefix length of the uplink is longer (smaller subnet) than the internal network: Just keep the prefix and replace the bits after it with the original source address. With this, I can use local addresses in the local network and additionally provide the public address/es of one or more uplinks. I am using this in production at one location since multiple years and thus know that this works. I am interested in other approaches, experiences and feedback for this method.
Maria’s comment about BGP multihoming is correct and reasonable if you have one location/few locations and use access circuits that providers are willing to run BGP over. It doesn’t help if you are trying to arrange low-cost resilient internet access over low cost FTTx/cellular to, say, hundreds or thousands of branch offices. It’s one use-case for v4 NAT which, even this NAT denier, agrees works well. Is your solution based on any published standard, Jonas, or has it been implemented as a feature on any commercial small router? Best wishes, Andy Davidson (AJBD-RIPE)
Hi, On Thu, Mar 06, 2025 at 01:47:27PM +0000, Andy Davidson wrote:
Maria’s comment about BGP multihoming is correct and reasonable if you have one location/few locations and use access circuits that providers are willing to run BGP over. It doesn’t help if you are trying to arrange low-cost resilient internet access over low cost FTTx/cellular to, say, hundreds or thousands of branch offices. It’s one use-case for v4 NAT which, even this NAT denier, agrees works well.
The goal is to make it low-cost. Otherwise BGP could help, but load balancing incoming traffic does not sound trivial. Another perspective: Instead of two cheap uplinks with speed x, I expect two uplinks with speed x*2 to be cheaper than two uplinks with speed x and BGP.
Is your solution based on any published standard, Jonas, or has it been implemented as a feature on any commercial small router?
No, this is no standard. I submitted a patch to OpenWrt (Router Linux Distribution that can run at x86 hardware and consumer routers) to implement this - the patch that I am using right now. One developer complained about the fact that this is not based on any standard. Another one suggested discussing this concept here.
Jonas, There’s been a lot of work and discussions on IPv6 multihoming. You touch on a number of considerations in your mail. Apart from PI and BGP unfortunately the only pragmatically available answer right now is to do what you do for IPv4. You can try stateless NPTv6 or do full NAPT66. And load balance per flow depending on implementation support. Pick a PA prefix from one of the providers or number your network from doc space if you want stable addresses. (The IETF has depreferenced ULA in source address selection so if using that you may get less ipv6 preference over v4). Cheers, Ole
On 6 Mar 2025, at 13:47, Jonas Lochmann <ripe-ipv6-wg@jonaslochmann.de> wrote:
My goal is to use multiple uplinks, but not only for redundancy. Most of the time, all (in my case 2) uplinks are available and then the question is how to make use of both of them.
With IPv4, NAT is common and thus the solution is quite simple. In my case, I am using the mwan3 package from OpenWrt. It uses iptables rules to add firewall marks to connections. If multiple uplinks are available, then the mark/uplink is chosen randomly and assigned to this (e.g. TCP) connection. This firewalls marks are used during a policy based routing. With a masquerade/source NAT, the right source address for the used route is picked and everything just works.
In case of IPv6, everything is different. NAT is uncommon. One solution is to enable NAT and then everything works as with IPv4. Alternatively, RFC 8678 describes that clients can be informed about multiple uplinks. The limitation: I do not see any option for load balancing.
RFC 8678 references other solutions. Shim6 seems to be not widely implemented. The Multipath Transports look like a solution for the future with Mulitpath TCP. The last solution is NPTv6. RFC 8678 does not like the solution. It is no NAT, but it still rewrites the addresses.
The disadvantage: Stateless address rewriting seems only usable if there is only one prefix known to the network. If this is the global prefix of one uplink, then all connections are interrupted if the prefix of this uplink is changed. If this is the local prefix, then the clients do not know their public addresses.
I tried to use a stateful source address rewriting instead. With nftables, this is easy to implement and it works if the prefix length of the uplink is longer (smaller subnet) than the internal network: Just keep the prefix and replace the bits after it with the original source address. With this, I can use local addresses in the local network and additionally provide the public address/es of one or more uplinks.
I am using this in production at one location since multiple years and thus know that this works. I am interested in other approaches, experiences and feedback for this method. ----- To unsubscribe from this mailing list or change your subscription options, please visit: https://mailman.ripe.net/mailman3/lists/ipv6-wg.ripe.net/ As we have migrated to Mailman 3, you will need to create an account with the email matching your subscription before you can change your settings. More details at: https://www.ripe.net/membership/mail/mailman-3-migration/
Hi, On 06/03/2025 13:47, Jonas Lochmann wrote:
I tried to use a stateful source address rewriting instead. With nftables, this is easy to implement and it works if the prefix length of the uplink is longer (smaller subnet) than the internal network: Just keep the prefix and replace the bits after it with the original source address. With this, I can use local addresses in the local network and additionally provide the public address/es of one or more uplinks.
I am using this in production at one location since multiple years and thus know that this works. I am interested in other approaches, experiences and feedback for this method.
Can you please be more specific about this solution? Which IPv6 addresses do you use in your network? Is it a prefix of one of the providers, ULA or something else? Can you more elaborate on why the provider's prefix has to be longer? If internal prefix is fd12:dead:beef::/48 Provider A is using 2001:db8:a::/56 Provider B is using 2001:db8:b::/56 The translator receives packet from fe12:dead:beef:1234::1 and chooses provider A, will it translate its source address to 2001:db8:a:0034::1? If yes, what then happens with packets from fe12:dead:beef:ab34::1? Also, can you link the repository/PR regarding the patch you use? -- Best regards, Ondřej Caletka
On Thu, Mar 06, 2025 at 04:41:46PM +0100, Ondřej Caletka wrote:
Can you please be more specific about this solution? Which IPv6 addresses do you use in your network? Is it a prefix of one of the providers, ULA or something else?
Right now ULA + prefix from one provider. I used prefixes from both providers in the past. I see no limit regarding the combinations.
Can you more elaborate on why the provider's prefix has to be longer?
It does not have to. My statement was misleading. The point was that it can be longer and this method continues working.
If internal prefix is fd12:dead:beef::/48 Provider A is using 2001:db8:a::/56 Provider B is using 2001:db8:b::/56
The translator receives packet from fe12:dead:beef:1234::1 and chooses provider A, will it translate its source address to 2001:db8:a:0034::1?
Yes
If yes, what then happens with packets from fe12:dead:beef:ab34::1?
The same: It is rewritten to 2001:db8:a:0034::1
Also, can you link the repository/PR regarding the patch you use?
https://patchwork.ozlabs.org/project/openwrt/patch/20250112131635.8660-1-ope... The core aspect is this nftables snippet that is filled according to the current prefix delegation: "snat ip6 to ip6 saddr and " + suffix_mask + " or " + base_addr This shows how powerful the expressions in nftables are: Logical operations with IPs are possible. In the example, the generated action would be: snat ip6 to ip6 saddr and ::ff:ffff:ffff:ffff:ffff or 2001:db8:a:: Before that, an accept rule is generated if the source address already matches the uplink IP so that this snat action is skipped.
Hi Jonas, You can find much more information and comparison here: https://datatracker.ietf.org/doc/html/draft-fbnvv-v6ops-site-multihoming-03 This draft is blocked in IETF because some participants (Google is the most vocal but not exclusive) are strongly against mentioning any NAT (including NPT) - they believe that such information should be blocked from dissemination - you could listen to video from any IETF meeting discussed this draft. Hence, pay attention, that the current IETF consensus is to create as many problems as possible for people using NAT/NPT in the IPv6 world. The MHMP problem will not be resolved very soon because there is PVD technology (provisioning domains) that is patented and pushed by some influential individuals in the IETF. I could not comprehend why they believed that PVD could resolve MHMP. Because PVD is the router (and host) link virtualization: if a router plays the role of a few virtual routers on the link then the MHMP problem immediately aggravates, it is not becoming easy in any way. Nobody steps down from the throne to explain to me why - I did ask many times. Probably because money likes silence. But we have what we have: MHMP is left for PVD. You did mention RFC 8678. Then probably you have a multi-hop routing site because this RFC concentrates only on this aspect of the MHMP problem (all other problems are out of the scope). Then look to section 6 of our draft (comparison table) - you will have big challenges with the provider's addresses - this option is probably blocked for you. Actually, RFC 8676 is pretty useless yet, because there is no way to propagate ISP uplink loss through the site (to withdraw the particular carrier IPv6 PA address) - the blackholing is guaranteed. Then the advice to get your own address space and become the full BGP speaker is probably a good one. Actually, the problem is very complex (you could look at the draft) - IPv6 flexibility on the 1st hop always translates to tremendous complexity. Are you sure that you need multi-homing in IPv6? This rabbit hole is very deep. You could stay with multi-homing in IPv4. PS: Shim6 was NOT a solution for resiliency, because the address space of one carrier was used for tunnel identification. If this particular carrier is lost, all tunnels should be down (and everything should be started from scratch-election of the new carrier as the basis for tunnel addressing). Moreover, the mechanism to inform hosts about the loss of a particular carrier (and respective PA address space) is pretty broken, even now, not to say 10 years ago. Hence instead of adopting a new carrier for the identification, we would get blackholing with a timer measured in a week. Shim6 was useful only for load balancing. No wonder why it did not fly. The people who developed shim6 are very respected in IETF - it is taboo to talk about a deficiency of any solution they developed. (Actually, I respect very much the leading person on shim6. Just I do not agree that a solution is prohibited from discussion). Shim6 is mentioned everywhere just for politeness and respect to the people who did a lot for IETF. (they really did!) If you mention shim6 then you would just have more chances for document publication. Does not matter that it does not work at all, even theoretically. Hence, you would still see shim6 in some documents. It is pure politics. Eduard -----Original Message----- From: Jonas Lochmann <ripe-ipv6-wg@jonaslochmann.de> Sent: Thursday, March 6, 2025 15:47 To: ipv6-wg@ripe.net Subject: [ipv6-wg] IPv6 Multihoming with Load Balancing My goal is to use multiple uplinks, but not only for redundancy. Most of the time, all (in my case 2) uplinks are available and then the question is how to make use of both of them. With IPv4, NAT is common and thus the solution is quite simple. In my case, I am using the mwan3 package from OpenWrt. It uses iptables rules to add firewall marks to connections. If multiple uplinks are available, then the mark/uplink is chosen randomly and assigned to this (e.g. TCP) connection. This firewalls marks are used during a policy based routing. With a masquerade/source NAT, the right source address for the used route is picked and everything just works. In case of IPv6, everything is different. NAT is uncommon. One solution is to enable NAT and then everything works as with IPv4. Alternatively, RFC 8678 describes that clients can be informed about multiple uplinks. The limitation: I do not see any option for load balancing. RFC 8678 references other solutions. Shim6 seems to be not widely implemented. The Multipath Transports look like a solution for the future with Mulitpath TCP. The last solution is NPTv6. RFC 8678 does not like the solution. It is no NAT, but it still rewrites the addresses. The disadvantage: Stateless address rewriting seems only usable if there is only one prefix known to the network. If this is the global prefix of one uplink, then all connections are interrupted if the prefix of this uplink is changed. If this is the local prefix, then the clients do not know their public addresses. I tried to use a stateful source address rewriting instead. With nftables, this is easy to implement and it works if the prefix length of the uplink is longer (smaller subnet) than the internal network: Just keep the prefix and replace the bits after it with the original source address. With this, I can use local addresses in the local network and additionally provide the public address/es of one or more uplinks. I am using this in production at one location since multiple years and thus know that this works. I am interested in other approaches, experiences and feedback for this method. ----- To unsubscribe from this mailing list or change your subscription options, please visit: https://mailman.ripe.net/mailman3/lists/ipv6-wg.ripe.net/ As we have migrated to Mailman 3, you will need to create an account with the email matching your subscription before you can change your settings. More details at: https://www.ripe.net/membership/mail/mailman-3-migration/
Hi, On Fri, Mar 07, 2025 at 07:20:44AM +0000, Vasilenko Eduard via ipv6-wg wrote:
You did mention RFC 8678. Then probably you have a multi-hop routing site because this RFC concentrates only on this aspect of the MHMP problem (all other problems are out of the scope). Then look to section 6 of our draft (comparison table) - you will have big challenges with the provider's addresses - this option is probably blocked for you. Actually, RFC 8676 is pretty useless yet, because there is no way to propagate ISP uplink loss through the site (to withdraw the particular carrier IPv6 PA address) - the blackholing is guaranteed. Then the advice to get your own address space and become the full BGP speaker is probably a good one.
I only have one router/one "next-hop" that itself is connected to multiple uplinks using multiple interfaces. With source address based routing, this step works. An uplink loss is detected using the mwan3 software. The result is that further connections are rewritten using my stateful prefix rewriting and redirected to another uplink.
Actually, the problem is very complex (you could look at the draft) - IPv6 flexibility on the 1st hop always translates to tremendous complexity. Are you sure that you need multi-homing in IPv6? This rabbit hole is very deep. You could stay with multi-homing in IPv4.
Then I get fast IPv4 and slow IPv6. Then I could disable IPv6 for a better user experience. Nothing that I want to do.
I only have one router/one "next-hop" that itself is connected to multiple uplinks using multiple interfaces. With source address based routing, this step works. An uplink loss is detected using the mwan3 software. The result is that further connections are rewritten using my stateful prefix rewriting and redirected to another uplink.
If the host is not warned that some prefix is deprecated (reminder, it is not possible now through a few hops of routers), then the host would use the disconnected IP address space for the packet source header. Such packet may enter only the respective Carrier. Any other Carrier must drop it for spoofing protection (a BCP is requesting it). Source routing just does not make sense because the mode fundamental problem was not resolved. If you would do NAT in this situation, then do not create a problem for yourself: do NAT for all traffic, and preserve the current IPv4 design. IPv6 first hop is a really complex matter. For example: I did complain many times in 6man that they have an extraordinary choice for SASA (RFC 6724). IPv6 decides about the packet structure (for example: source IP address field) only after the next hop is decided. This field in the packet would be populated with the IP prefix advertised by the next hop router. People, not familiar with IPv6 may not believe in this. Eduard -----Original Message----- From: Jonas Lochmann <ripe-ipv6-wg@jonaslochmann.de> Sent: Friday, March 7, 2025 16:46 To: Vasilenko Eduard <vasilenko.eduard@huawei.com> Cc: ipv6-wg@ripe.net Subject: Re: [ipv6-wg] Re: IPv6 Multihoming with Load Balancing Hi, On Fri, Mar 07, 2025 at 07:20:44AM +0000, Vasilenko Eduard via ipv6-wg wrote:
You did mention RFC 8678. Then probably you have a multi-hop routing site because this RFC concentrates only on this aspect of the MHMP problem (all other problems are out of the scope). Then look to section 6 of our draft (comparison table) - you will have big challenges with the provider's addresses - this option is probably blocked for you. Actually, RFC 8676 is pretty useless yet, because there is no way to propagate ISP uplink loss through the site (to withdraw the particular carrier IPv6 PA address) - the blackholing is guaranteed. Then the advice to get your own address space and become the full BGP speaker is probably a good one.
I only have one router/one "next-hop" that itself is connected to multiple uplinks using multiple interfaces. With source address based routing, this step works. An uplink loss is detected using the mwan3 software. The result is that further connections are rewritten using my stateful prefix rewriting and redirected to another uplink.
Actually, the problem is very complex (you could look at the draft) - IPv6 flexibility on the 1st hop always translates to tremendous complexity. Are you sure that you need multi-homing in IPv6? This rabbit hole is very deep. You could stay with multi-homing in IPv4.
Then I get fast IPv4 and slow IPv6. Then I could disable IPv6 for a better user experience. Nothing that I want to do.
Hi, Jonas, On 6/3/25 09:47, Jonas Lochmann wrote:
My goal is to use multiple uplinks, but not only for redundancy. Most of the time, all (in my case 2) uplinks are available and then the question is how to make use of both of them.
With IPv4, NAT is common and thus the solution is quite simple. In my case, I am using the mwan3 package from OpenWrt. It uses iptables rules to add firewall marks to connections. If multiple uplinks are available, then the mark/uplink is chosen randomly and assigned to this (e.g. TCP) connection. This firewalls marks are used during a policy based routing. With a masquerade/source NAT, the right source address for the used route is picked and everything just works.
In case of IPv6, everything is different. NAT is uncommon. One solution is to enable NAT and then everything works as with IPv4. Alternatively, RFC 8678 describes that clients can be informed about multiple uplinks. The limitation: I do not see any option for load balancing.
To put it bluntly, multi-router/multi-prefix is currently broken. This is not the first time someone raises/notes this, but this is probably the last instance of it: * URL: https://www.ietf.org/archive/id/draft-gont-v6ops-multi-ipv6-02.txt * HTMLized: https://datatracker.ietf.org/doc/html/draft-gont-v6ops-multi-ipv6 (as noted, it's not just about the source address, but also about using the right combination of source DNS resolver, source address, and next hop). Once *that* problem was addressed. one might come up with something for load *sharing* (whether that's having hosts select the source address/prefix randomly (or other things being equal), or other options). In a lot of scenarios -- despite rather religious claims against that direction -- you may solve the problem as suggested doing NAT for IPv6. (particularly if this is one of the many problems you have on your table to solve, as is the case for many organizations)). Thanks, -- Fernando Gont SI6 Networks e-mail: fgont@si6networks.com PGP Fingerprint: F242 FF0E A804 AF81 EB10 2F07 7CA1 321D 663B B494
On 07/03/2025 11:30, Fernando Gont wrote:
In a lot of scenarios -- despite rather religious claims against that direction -- you may solve the problem as suggested doing NAT for IPv6. (particularly if this is one of the many problems you have on your table to solve, as is the case for many organizations)).
I think it is also worth mentioning that this is by no means a problem with IPv6. Solving the same issue on IPv4 would be exactly as complex as any solution on IPv6. The only difference is that on IPv4 we generally don't care about end-to-end principle, while in IPv6 we try to preserve it as much as possible. Which is probably a good thing, but if this prevent us from deploying IPv6, then I would say it is still better to have some reachability to IPv6-only resources, be it via a proxy server (these ancient protocols still exist!) or some sort of NAT than to have no IPv6 reachability at all. -- Ondřej
On 7/3/25 07:59, Ondřej Caletka wrote:
On 07/03/2025 11:30, Fernando Gont wrote:
In a lot of scenarios -- despite rather religious claims against that direction -- you may solve the problem as suggested doing NAT for IPv6. (particularly if this is one of the many problems you have on your table to solve, as is the case for many organizations)).
I think it is also worth mentioning that this is by no means a problem with IPv6. Solving the same issue on IPv4 would be exactly as complex as any solution on IPv6.
The only difference is that on IPv4 we generally don't care about end-to-end principle, while in IPv6 we try to preserve it as much as possible. Which is probably a good thing, but if this prevent us from deploying IPv6, then I would say it is still better to have some reachability to IPv6-only resources, be it via a proxy server (these ancient protocols still exist!) or some sort of NAT than to have no IPv6 reachability at all.
Agreed on that. One of the interesting bits is that in IPv4, NATs have been embraced as a solution to the problem at hand. Whereas in the IPv6 world NATs are generally demonized, while we don't bite the bullet to address the problem at hand (without employing NATs or the like). Thanks, -- Fernando Gont SI6 Networks e-mail: fgont@si6networks.com PGP Fingerprint: F242 FF0E A804 AF81 EB10 2F07 7CA1 321D 663B B494
Strongly disagree with you both. IPv6 has enormous complexity (compared to IPv4) on the 1st hop. Root causes for this: - many IP addresses per host - blocked DHCP on the most popular OS and all smartphones - strong push against NAT (that resolves many problems for connection to Carriers) - address resolution that is on L3 not on L2 IPv6 was a political compromise between people pushing IP, IPX, AppleTalk, DecNet, and a few other funny networking technologies. It inherited a lot of complexity from them. Ed/ -----Original Message----- From: Fernando Gont <fgont@si6networks.com> Sent: Friday, March 7, 2025 16:01 To: Ondrej.Caletka@ripe.net; ipv6-wg@ripe.net Subject: [ipv6-wg] Re: IPv6 Multihoming with Load Balancing On 7/3/25 07:59, Ondřej Caletka wrote:
On 07/03/2025 11:30, Fernando Gont wrote:
In a lot of scenarios -- despite rather religious claims against that direction -- you may solve the problem as suggested doing NAT for IPv6. (particularly if this is one of the many problems you have on your table to solve, as is the case for many organizations)).
I think it is also worth mentioning that this is by no means a problem with IPv6. Solving the same issue on IPv4 would be exactly as complex as any solution on IPv6.
The only difference is that on IPv4 we generally don't care about end-to-end principle, while in IPv6 we try to preserve it as much as possible. Which is probably a good thing, but if this prevent us from deploying IPv6, then I would say it is still better to have some reachability to IPv6-only resources, be it via a proxy server (these ancient protocols still exist!) or some sort of NAT than to have no IPv6 reachability at all.
Agreed on that. One of the interesting bits is that in IPv4, NATs have been embraced as a solution to the problem at hand. Whereas in the IPv6 world NATs are generally demonized, while we don't bite the bullet to address the problem at hand (without employing NATs or the like). Thanks, -- Fernando Gont SI6 Networks e-mail: fgont@si6networks.com PGP Fingerprint: F242 FF0E A804 AF81 EB10 2F07 7CA1 321D 663B B494 ----- To unsubscribe from this mailing list or change your subscription options, please visit: https://mailman.ripe.net/mailman3/lists/ipv6-wg.ripe.net/ As we have migrated to Mailman 3, you will need to create an account with the email matching your subscription before you can change your settings. More details at: https://www.ripe.net/membership/mail/mailman-3-migration/
On 8/3/25 14:07, Vasilenko Eduard wrote:
Strongly disagree with you both. IPv6 has enormous complexity (compared to IPv4) on the 1st hop. Root causes for this: - many IP addresses per host - blocked DHCP on the most popular OS and all smartphones - strong push against NAT (that resolves many problems for connection to Carriers) - address resolution that is on L3 not on L2
What does that have to do with the problem at hand, or what we said? Let me go one by one:
- many IP addresses per host
This, by itself, has nothing to do with the problem at hand. IN IPv4, if you had implementation configure multiple addresses via DHCPv4, or you were to manually configure multiple addresses manually, you'd get the same. The problem is not the multiple addresses, but rather than IP hosts routes packets on the destination (vs more complex policy routing, e.g. considering the source prefix).
- blocked DHCP on the most popular OS and all smartphones
Same as above. If every host had DHCPv6 support, but you didn't have policy routing (e.g., RFC8028), the situation wouldn't change a single millimeter.
- strong push against NAT (that resolves many problems for connection to Carriers)
I explicitly noted that "NAT is particularly demonized in IPv6 world", yes. -- but, again, NAT for IPv6 solves the same problem as IPv4 NAT -- so nothing different here.
- address resolution that is on L3 not on L2
What does this have to do with the problem at hand?
IPv6 was a political compromise between people pushing IP, IPX, AppleTalk, DecNet, and a few other funny networking technologies. It inherited a lot of complexity from them.
It inherited a lot of complexity, created additional one, and was deployed 20 years or so after the original design. Nobody is arguing otherwise. The discussion here was on the specific topic of multi-ipv6: In that regard, it boils down to: IPv4 has a de-facto constraint where each host configures a single address via IPv4, and this is a private address (RFC1918). NAT is widely deployed, so the end host is not exposed to the public address that is used to communicate on the public Internet. So while the theoretical problems are the same, in virtually all practical cases you are shielded from them, thanks to NATs (same as for RFC8978). It is also the case that a lot of folks have demonized NATs, while at the same time deluded themselves that some of the issues that NATs shield you from either doesn't exist, will somehow disappear, or will solve themselves. slaac-renum is one of such examples (RFC8978 and friends). multi-ipv6 is yet another. (In that regards, <https://datatracker.ietf.org/doc/draft-gont-v6ops-multi-ipv6/> is essentially a document that says "this is a problem that needs a native solution", and these are the core scenarios that such a solution should address.) Thanks, -- Fernando Gont SI6 Networks e-mail: fgont@si6networks.com PGP Fingerprint: F242 FF0E A804 AF81 EB10 2F07 7CA1 321D 663B B494
There was a comment that IPv6 in general is the same complexity as IPv4. Hence, my reaction: it is not. OK. Returning to the problem scope: - Missing NAT is the biggest challenge for MHMP. - The second challenge is many IP addresses per host. It is possible to position it as a feature, but this feature creates a lot of complexity. IPv4 did not mandate that every host should support it. IPv6 does. It is a big difference that everybody in IPv6 is involved in it (like he or not). Anybody is welcome to read RFC 6724 (SASA) to see the consequences. How many people worldwide understand it well? One hundred or two? Of course, it is a value for these people. Ed/ -----Original Message----- From: Fernando Gont <fgont@si6networks.com> Sent: Sunday, March 9, 2025 19:00 To: Vasilenko Eduard <vasilenko.eduard@huawei.com>; Ondrej.Caletka@ripe.net; ipv6-wg@ripe.net Subject: Re: [ipv6-wg] Re: IPv6 Multihoming with Load Balancing On 8/3/25 14:07, Vasilenko Eduard wrote:
Strongly disagree with you both. IPv6 has enormous complexity (compared to IPv4) on the 1st hop. Root causes for this: - many IP addresses per host - blocked DHCP on the most popular OS and all smartphones - strong push against NAT (that resolves many problems for connection to Carriers) - address resolution that is on L3 not on L2
What does that have to do with the problem at hand, or what we said? Let me go one by one:
- many IP addresses per host
This, by itself, has nothing to do with the problem at hand. IN IPv4, if you had implementation configure multiple addresses via DHCPv4, or you were to manually configure multiple addresses manually, you'd get the same. The problem is not the multiple addresses, but rather than IP hosts routes packets on the destination (vs more complex policy routing, e.g. considering the source prefix).
- blocked DHCP on the most popular OS and all smartphones
Same as above. If every host had DHCPv6 support, but you didn't have policy routing (e.g., RFC8028), the situation wouldn't change a single millimeter.
- strong push against NAT (that resolves many problems for connection to Carriers)
I explicitly noted that "NAT is particularly demonized in IPv6 world", yes. -- but, again, NAT for IPv6 solves the same problem as IPv4 NAT -- so nothing different here.
- address resolution that is on L3 not on L2
What does this have to do with the problem at hand?
IPv6 was a political compromise between people pushing IP, IPX, AppleTalk, DecNet, and a few other funny networking technologies. It inherited a lot of complexity from them.
It inherited a lot of complexity, created additional one, and was deployed 20 years or so after the original design. Nobody is arguing otherwise. The discussion here was on the specific topic of multi-ipv6: In that regard, it boils down to: IPv4 has a de-facto constraint where each host configures a single address via IPv4, and this is a private address (RFC1918). NAT is widely deployed, so the end host is not exposed to the public address that is used to communicate on the public Internet. So while the theoretical problems are the same, in virtually all practical cases you are shielded from them, thanks to NATs (same as for RFC8978). It is also the case that a lot of folks have demonized NATs, while at the same time deluded themselves that some of the issues that NATs shield you from either doesn't exist, will somehow disappear, or will solve themselves. slaac-renum is one of such examples (RFC8978 and friends). multi-ipv6 is yet another. (In that regards, <https://datatracker.ietf.org/doc/draft-gont-v6ops-multi-ipv6/> is essentially a document that says "this is a problem that needs a native solution", and these are the core scenarios that such a solution should address.) Thanks, -- Fernando Gont SI6 Networks e-mail: fgont@si6networks.com PGP Fingerprint: F242 FF0E A804 AF81 EB10 2F07 7CA1 321D 663B B494
On Fri, Mar 7, 2025 at 11:59 AM Ondřej Caletka <ocaletka@ripe.net> wrote:
On 07/03/2025 11:30, Fernando Gont wrote:
In a lot of scenarios -- despite rather religious claims against that direction -- you may solve the problem as suggested doing NAT for IPv6. (particularly if this is one of the many problems you have on your table to solve, as is the case for many organizations)).
I think it is also worth mentioning that this is by no means a problem with IPv6. Solving the same issue on IPv4 would be exactly as complex as any solution on IPv6.
The only difference is that on IPv4 we generally don't care about end-to-end principle, while in IPv6 we try to preserve it as much as possible. Which is probably a good thing, but if this prevent us from deploying IPv6, then I would say it is still better to have some reachability to IPv6-only resources, be it via a proxy server (these ancient protocols still exist!) or some sort of NAT than to have no IPv6 reachability at all.
I also agree with this, very good point. Multihoming is a layer 3 issue, not an IPv6 issue. And in a small/medium site, it is aggravated by the fact that the tools traditionally used to tackle the issue, such as BGP and PI addressing, are not available or it is not scalable to make them available for all. IPv4 got NAT with the excuse of address scarcity, and then it was turned into a crude routing tool. IPv6 has the experimental NPTv6 which is simply NAT but with all the non-routing aspects removed -- it has the luxury of only changing the "locator" bits (if we ignore the checksum workaround) since there is no scarcity on either side of the translator and thus no need to overload. Now... with a good stretch of the imagination you could see NPT as a way of "forcing" the return packet to be segment routed, albeit without a SRH and without encapsulation. The traffic path is anchored to the node performing the translation. But for some reason SRv6 is currently less controversial than NPTv6. Paolo
On 6. 3. 25 13:47, Jonas Lochmann wrote:
My goal is to use multiple uplinks, but not only for redundancy. Most of the time, all (in my case 2) uplinks are available and then the question is how to make use of both of them. As soon as you leave the AS+PI+BGP area in this case you start getting into a deep, dark, complex well of magic and witchcraft.
I wouldn't :) Cheers, Jan
Hi, On Sat, Mar 08, 2025 at 03:31:50PM +0100, Jan Zorz - Go6 wrote:
On 6. 3. 25 13:47, Jonas Lochmann wrote:
My goal is to use multiple uplinks, but not only for redundancy. Most of the time, all (in my case 2) uplinks are available and then the question is how to make use of both of them. As soon as you leave the AS+PI+BGP area in this case you start getting into a deep, dark, complex well of magic and witchcraft.
As if AS+PI+BGP is *not* dark & full of witchcraft :-) (Figuring out why and where your packets are coming *back* to you - or not! - and getting balancing across multiple upstreams about the way you want/need that balanced... - outgoing is the trivial part) Also, AS+PI+BGP just does not scale to millions of endusers or "barbershop scale" small enterprises. Gert Doering -- NetMaster -- have you enabled IPv6 on something today...? SpaceNet AG Vorstand: Sebastian v. Bomhard, Ingo Lalla, Karin Schuler, Sebastian Cler Joseph-Dollinger-Bogen 14 Aufsichtsratsvors.: A. Grundner-Culemann D-80807 Muenchen HRB: 136055 (AG Muenchen) Tel: +49 (0)89/32356-444 USt-IdNr.: DE813185279
Gert Doering <gert@space.net> writes:
Hi,
On Sat, Mar 08, 2025 at 03:31:50PM +0100, Jan Zorz - Go6 wrote:
On 6. 3. 25 13:47, Jonas Lochmann wrote:
My goal is to use multiple uplinks, but not only for redundancy. Most of the time, all (in my case 2) uplinks are available and then the question is how to make use of both of them. As soon as you leave the AS+PI+BGP area in this case you start getting into a deep, dark, complex well of magic and witchcraft.
As if AS+PI+BGP is *not* dark & full of witchcraft :-)
(Figuring out why and where your packets are coming *back* to you - or not! - and getting balancing across multiple upstreams about the way you want/need that balanced... - outgoing is the trivial part)
Also, AS+PI+BGP just does not scale to millions of endusers or "barbershop scale" small enterprises.
While I agree with you Gert, maybe it *should*. The whole point of IPv6 is end-to-end connectivity and re-enabling users. Imho something like multi link applications *should* be something trivial to solve in the IPv6 world. Whether that's PI/PA+LIR space or multiple addresses per node, that's debatable, but generally speaking, IPv6 should be there to *solve* problems, not to add. Just my 200 KRW, Nico -- Sustainable and modern Infrastructures by ungleich.ch
Hi, On Sun, Mar 09, 2025 at 04:59:10PM +0900, Nico Schottelius wrote:
The whole point of IPv6 is end-to-end connectivity and re-enabling users. Imho something like multi link applications *should* be something trivial to solve in the IPv6 world.
Whether that's PI/PA+LIR space or multiple addresses per node, that's debatable, but generally speaking, IPv6 should be there to *solve* problems, not to add.
I agree :-) "Multiple address per node" + a decent API so *users* can easily control what they want applications to do ("web surfing via DSL, bittorrent on the cable Internet") + proper source address failover would be bliss... Especially "proper source address failover" would be much more robust against remote failures than "BGP" - think "packets going out via ISP A, and there being a blackhole [think broken MPLS tunnel] between ISP A and ISP X, so packets disappear" - with BGP you basically need to fumble on your route policies then. With "mmh, ISP A source does not work, let's try ISP B source" things can be fixed from the application/client OS level. Alas... nobody really interested IETF-side or OS/browser-side in working on that. Much more important to waste energy with stuff like DoH... Gert Doering -- NetMaster -- have you enabled IPv6 on something today...? SpaceNet AG Vorstand: Sebastian v. Bomhard, Ingo Lalla, Karin Schuler, Sebastian Cler Joseph-Dollinger-Bogen 14 Aufsichtsratsvors.: A. Grundner-Culemann D-80807 Muenchen HRB: 136055 (AG Muenchen) Tel: +49 (0)89/32356-444 USt-IdNr.: DE813185279
On Sun, Mar 9, 2025 at 9:10 AM Gert Doering <gert@space.net> wrote:
Hi,
On Sun, Mar 09, 2025 at 04:59:10PM +0900, Nico Schottelius wrote:
The whole point of IPv6 is end-to-end connectivity and re-enabling users. Imho something like multi link applications *should* be something trivial to solve in the IPv6 world.
Whether that's PI/PA+LIR space or multiple addresses per node, that's debatable, but generally speaking, IPv6 should be there to *solve* problems, not to add.
I agree :-)
"Multiple address per node" + a decent API so *users* can easily control what they want applications to do ("web surfing via DSL, bittorrent on the cable Internet") + proper source address failover would be bliss...
I'm now imagining opening up the Windows Settings->Apps and selecting "DSL line" for Teams/Slack and "LEO Satellite" for Youtube. Yeah, I'd REALLY like this... Paolo
On Sun, Mar 09, 2025 at 09:10:36AM +0100, Gert Doering wrote:
"Multiple address per node" + a decent API so *users* can easily control what they want applications to do ("web surfing via DSL, bittorrent on the cable Internet") + proper source address failover would be bliss...
Especially "proper source address failover" would be much more robust against remote failures than "BGP" - think "packets going out via ISP A, and there being a blackhole [think broken MPLS tunnel] between ISP A and ISP X, so packets disappear" - with BGP you basically need to fumble on your route policies then. With "mmh, ISP A source does not work, let's try ISP B source" things can be fixed from the application/client OS level.
Actually, I implemented something related once: https://codeberg.org/jonas-l/socksbalance This is a socks proxy server (can be configured in applications like Browsers) that selects source IPs for the connections. It's implementation with round robin is quite simple. However, just retrying to connect results in a certain chance that the application picks the other source IP. There is of course the possibility to make it more advanced. This proxy server can run at the end user machine itself.
participants (13)
-
Andy Davidson -
Fernando Gont -
Gert Doering -
Jan Zorz - Go6 -
Jonas Lochmann -
Marco Moock -
Maria Matejka -
Nico Schottelius -
Ole Trøan -
Ondřej Caletka -
Paolo Nero -
ripe-ipv6-wg＠jonaslochmann.de
-
Vasilenko Eduard