nixos.org domain resolution failures

[rnhmjoj]

Affected service

Nameserver authoritative for nixos.org (ns1.gandi.net).

Describe the issue
On some networks, the nameserver authoritative for nixos.org returns SERVFAIL in reply to queries of nixos.org.
Subdomains like cache.nixos.org and hydra.nixos.org are unaffected, though.

System information

I tested from 3 different ISPs, 2/3 are affected:

1. Iliad/Free SAS, affected

   $ mtr ns1.gandi.net
   wes -> ns1.gandi.net (173.246.100.2)
                                              Packets               Pings
    Host                                    Loss%   Snt   Last   Avg  Best  Wrst StDev
    1. 194.149.186.30                        2.7%   297    2.3   2.9   2.1   4.3   0.4
    2. i3dnet.mix-it.net                     0.0%   297    3.0   4.0   2.2  44.1   3.5
    3. ns-1-a.gandi.net                      0.0%   297    2.9   3.0   2.1   3.9   0.4
   
   $ host -4 -t A nixos.org ns1.gandi.net
   Using domain server:
   Name: ns1.gandi.net
   Address: 173.246.100.2#53
   Aliases: 
   
   Host nixos.org not found: 2(SERVFAIL)
   

2. Vodafone IT, affected

   $ mtr ns1.gandi.net
   maxwell -> ns1.gandi.net (173.246.100.2)
                                             Packets               Pings
    Host                                   Loss%   Snt   Last   Avg  Best  Wrst StDev
    1. net-x-y-z-t.cust.vodafonedsl.it      0.0%     5   12.6  14.0  12.6  16.3   1.9
    2. (waiting for reply)
    3. (waiting for reply)
    4. vodafone-it-gw-mlk.cw.net            0.0%     5   16.7  17.1  16.7  17.4   0.3
    5. 195.59.1.25                          0.0%     5   16.2  16.5  15.4  19.6   1.8
    6. ae12-xcr1.mlu.cw.net                 0.0%     5   16.8  19.7  16.5  29.4   5.5
    7. ae15-xcr1.mlu.cw.net                 0.0%     5   15.6  16.8  15.5  20.5   2.1
    8. ns-1-a.gandi.net                     0.0%     5   16.8  16.9  16.8  17.3   0.2
   
   $ host -4 -t A nixos.org ns1.gandi.net
   Using domain server:
   Name: ns1.gandi.net
   Address: 173.246.100.2#53
   Aliases: 
   
   Host nixos.org not found: 2(SERVFAIL)
   

3. Telecom Italia, unaffected

   $ mtr ns1.gandi.net
   edge -> ns1.gandi.net (173.246.100.2)
                                            Packets               Pings
    Host                                  Loss%   Snt   Last   Avg  Best  Wrst StDev
    1. (waiting for reply)
    2. 172.17.185.16                       0.0%    15    7.3   7.1   6.9   7.7   0.2
    3. 172.17.185.26                       0.0%    15    8.0   7.5   7.1   8.0   0.3
    4. 172.19.184.86                       0.0%    15    8.8   8.6   8.3   8.9   0.2
    5. 172.19.177.20                       0.0%    14    8.5  11.9   8.1  55.6  12.6
    6. 195.22.192.144                      0.0%    14    8.8  10.8   8.1  40.7   8.6
    7. 195.22.210.120                      0.0%    14   25.5  28.3  25.4  60.2   9.2
    8. 195.22.210.69                       0.0%    14   26.6  26.3  26.0  26.6   0.2
    9. et20-1.core-2.csd5.gandi.net        0.0%    14   25.9  25.8  25.6  26.1   0.2
   10. ns-1-a.gandi.net                    0.0%    14   26.6  26.3  26.1  26.6   0.2
   
   $ host -4 -t A nixos.org ns1.gandi.net
   Using domain server:
   Name: ns1.gandi.net
   Address: 173.246.100.2#53
   Aliases: 
   
   nixos.org has address 63.176.8.218
   nixos.org has address 35.157.26.135
   

This is a trace from my recursive nameserver for a nixos.org A query:

259 nixos.org: Wants DNSSEC processing, auth data required by query for A
288 nixos.org: No valid SOA found for ., which is the best match for nixos.org
291 QM nixos.org: Step0 Not cached
293 QM nixos.org: Checking if we have NS in cache for 'nixos.org'
299 QM nixos.org: NS (with ip, or non-glue) in cache for 'nixos.org' -> 'ns-161-c.gandi.net', within bailiwick: 0, not in cache / did not look at cache
305 QM nixos.org: NS (with ip, or non-glue) in cache for 'nixos.org' -> 'ns-61-b.gandi.net', within bailiwick: 0, not in cache / did not look at cache
309 QM nixos.org: NS (with ip, or non-glue) in cache for 'nixos.org' -> 'ns-177-a.gandi.net', within bailiwick: 0, not in cache / did not look at cache
319 QM nixos.org: We have NS in cache for 'nixos.org' (flawedNSSet=0)
322 QM nixos.org: Step1 Ancestor from cache is nixos.org
325 QM nixos.org: Step2 New child nixos.org
327 QM nixos.org: Step3 Going to do final resolve
328 nixos.org: Wants DNSSEC processing, auth data required by query for A
337 nixos.org: No valid SOA found for ., which is the best match for nixos.org
341 nixos.org: No cache hit for 'nixos.org|A', trying to find an appropriate NS record
343 nixos.org: Checking if we have NS in cache for 'nixos.org'
346 nixos.org: NS (with ip, or non-glue) in cache for 'nixos.org' -> 'ns-161-c.gandi.net', within bailiwick: 0, not in cache / did not look at cache
350 nixos.org: NS (with ip, or non-glue) in cache for 'nixos.org' -> 'ns-61-b.gandi.net', within bailiwick: 0, not in cache / did not look at cache
354 nixos.org: NS (with ip, or non-glue) in cache for 'nixos.org' -> 'ns-177-a.gandi.net', within bailiwick: 0, not in cache / did not look at cache
359 nixos.org: We have NS in cache for 'nixos.org' (flawedNSSet=0)
367 nixos.org: Cache consultations done, have 3 NS to contact
391 nixos.org: Nameservers: ns-177-a.gandi.net(0.01ms), ns-61-b.gandi.net(0.01ms), ns-161-c.gandi.net(6.49ms)
426 nixos.org: Trying to resolve NS 'ns-177-a.gandi.net' (1/3)
436 ns-177-a.gandi.net: Nameserver ns-177-a.gandi.net IPs: 173.246.100.178(0.01ms)
456 nixos.org: Resolved 'nixos.org' NS ns-177-a.gandi.net to: 173.246.100.178
462 nixos.org: Trying IP 173.246.100.178:53, asking 'nixos.org|A'
210481 nixos.org: ns-177-a.gandi.net (173.246.100.178) returned a Server Failure, trying sibling IP or NS
210497 nixos.org: Trying to resolve NS 'ns-61-b.gandi.net' (2/3)
210507 ns-61-b.gandi.net: Nameserver ns-61-b.gandi.net IPs: 213.167.230.62(0.01ms)
210515 nixos.org: Resolved 'nixos.org' NS ns-61-b.gandi.net to: 213.167.230.62
210520 nixos.org: Trying IP 213.167.230.62:53, asking 'nixos.org|A'
416681 nixos.org: ns-61-b.gandi.net (213.167.230.62) returned a Server Failure, trying sibling IP or NS
416694 nixos.org: Trying to resolve NS 'ns-161-c.gandi.net' (3/3)
416703 ns-161-c.gandi.net: Nameserver ns-161-c.gandi.net IPs: 217.70.187.162(6.44ms)
416711 nixos.org: Resolved 'nixos.org' NS ns-161-c.gandi.net to: 217.70.187.162
416716 nixos.org: Trying IP 217.70.187.162:53, asking 'nixos.org|A'
740944 nixos.org: ns-161-c.gandi.net (217.70.187.162) returned a Server Failure, trying sibling IP or NS
740956 nixos.org: Failed to resolve via any of the 3 offered NS at level 'nixos.org'
740963 nixos.org: Failed (res=-1)
740967 QM nixos.org: Step3 Final resolve: Server Failure/0

Note: IPv6 and AAAA are similarly affected.

[mweinelt]

https://atlas.ripe.net/measurements/132952403/overview

Failing nodes

All nodes (within that same area)

[rnhmjoj]

Any idea?

[mweinelt]

I suppose they get routed to the same LiveDNS instance, but hard to say.

[mweinelt]

I filed an issue with our DNS hoster. I think the issue might be that their nameserver failed to expand the ALIAS record we use on the domain apex.

[vcunat]

Right, it makes sense that ALIAS would be more fragile than the rest.

[mweinelt]

Temporarily migrated the domain apex for nixos.org and nix.dev to the IPv4-only netlify loadbalancer.

85a21b5

[mweinelt]

Approached the marketing team to migrate the website to a provider that can better accommodate hosting on the domain apex, which means providing static IP addresses, and they are looking into that.