linux - How to make iproute2 multiple uplinks work with masquerading -

i've following problem routing + nat: if i've 2 isp , i'm using 2 nexthop in default route masquerade on both isp links, see routing cache regenerated, packets sent new link (after cache regeneration) uses wrong source address masquerading.

here config.

i've 2 links outside via 2 different providers: eth1 , eth2 eth0 lan

$ ip (part of output, since have 3 more interfaces disabled)
2: eth1: mtu 1500 qdisc pfifo_fast qlen 1000
inet 192.168.1.254/24 brd 192.168.1.255 scope global eth1
3: eth2: mtu 1500 qdisc pfifo_fast qlen 1000
inet 192.168.2.254/24 brd 192.168.2.255 scope global eth2
6: eth0: mtu 1500 qdisc pfifo_fast qlen 1000
inet 192.168.5.1/24 brd 192.168.5.255 scope global eth0

roting tables:

$ ip r 192.168.5.0/24 dev eth0 proto kernel scope link src 192.168.5.1
192.168.2.0/24 dev eth2 proto kernel scope link src 192.168.2.254
192.168.1.0/24 dev eth1 proto kernel scope link src 192.168.1.254
default nexthop via 192.168.1.1 dev eth1 weight 1
nexthop via 192.168.2.1 dev eth2 weight 1

$ ip r s t eth1
default via 192.168.1.1 dev eth1

$ ip r s t eth2
default via 192.168.2.1 dev eth2

rules:

$ ip ru
0: lookup local
32450: 192.168.2.254 lookup eth2
32717: 192.168.5.124 lookup eth1
32766: lookup main
32767: lookup default

q1: if pings 2 pc in lan: 5.137 , 5.147, same ip (195.60.1.1) how can go via different links (ping 195.60.1.1 run on both computers)?

$ ip r g 195.60.1.1 192.168.5.137 iif eth0
195.60.169.6 192.168.5.137 via 192.168.1.1 dev eth1 src 192.168.5.1
cache mtu 1500 advmss 1460 hoplimit 128 iif eth0

$ ip r g 195.60.1.1 192.168.5.147 iif eth0
195.60.169.6 192.168.5.147 via 192.168.2.1 dev eth2 src 192.168.5.1
cache mtu 1500 advmss 1460 hoplimit 128 iif eth0

the routing in case should same users. should send packets same destination via same link (even if source ip different). isn't it?

q2: see in tcpdump on external interfaces routing cache regenerated. can forced ip r f t cache. results in change of link pings. 1 of 2 machines looses connection. tcpdump found happens because routing has decided use link, masquerade not updated according:

$ tcpdump -i eth1
ip 192.168.2.254 > 195.60.1.1: icmp echo request, id 10677, seq 242, length 64
ip 192.168.1.254 > 195.60.1.1: icmp echo request, id 37387, seq 244, length 64 ip 195.60.1.1 > 192.168.1.254: icmp echo reply, id 37387, seq 244, length 64

the second , third packets request-reply from/to 5.137

the first packet request .5.147 wrong source address on interface due masquerade not updated after routing cache purge - hence, no reply, since source address of masqueraded packet wrong.

here masquerade setting

$ iptables -l -t nat
chain postrouting (policy accept 752k packets, 48m bytes)
pkts bytes target prot opt in out source destination
2840k 256m masquerade -- eth1 192.168.5.0/24 anywhere
2491k 229m masquerade -- eth2 192.168.5.0/24 anywhere

i understand can use conntrack mark packets, little bit more complicated. prefer use destination ip key routing. wrong in scenario? why routing cache purges not notify nat-engine changes in routing?

ok, answer found using search engines.

this particular behaviour bug in linux kernel known @ least since 2005. julian anastasov has written patch workaround error (see http://www.ssi.bg/~ja/#routes)

anyway, found chosen scenario load balancing , nat not good, since may break authorization on sites , makes jabber , skype flicker due recache of routes resulting in changes of routes each destination (since use nat, external ip changes , skype , other services sees logged in computer).

much better way share multiple links on big office split users channels. so, assigned preferred channel each computer in our network , if channel not choose other channel computer. strategy keeps same external ip (after nating) every computer in our network. setting preferred channel allows send critical employee via faster channels, while employee dealing big files on low cos wide slow channels. use 4 channels, since isp in our region goes down @ least 2 times week several hours.