Operations 8 min read

Troubleshooting DNS Resolution Failure of goofy.app in Singapore Office Due to DNSSEC Misconfiguration

After users in Singapore reported inability to resolve the internal domain goofy.app, a systematic investigation revealed that DNSSEC misconfiguration—specifically an incorrect DS record—caused DNS resolution failures globally, while Chinese DNS servers succeeded due to disabled DNSSEC validation, and removing the faulty key resolved the issue.

ByteFE

Oct 20, 2021

Troubleshooting DNS Resolution Failure of goofy.app in Singapore Office Due to DNSSEC Misconfiguration

Background

Colleagues reported that the internal domain goofy.app and its subdomains were inaccessible from the Singapore office, while they worked from other locations and via VPN.

Investigation

Initial curl test showed DNS resolution failure: $> curl https://foo.goofy.app/ curl: (6) Could not resolve host: foo.goofy.app . Direct IP access with host header succeeded, confirming network connectivity.

Observations:

Chinese office network could resolve the domain.

Singapore VPN could resolve.

Singapore office could not resolve.

Further checks with online DNS tools showed that the domain resolves publicly only in mainland China.

Hypothesis 0 – Internal DNS misconfiguration

Checked that the domain is not a pure internal name; it resolves publicly but points to internal IPs.

Hypothesis 1 – GeoDNS

Tested with dig foo.goofy.app @8.8.8.8 +subnet=114.114.114.0/24 +short which returned no result, indicating possible GeoDNS, but the DNS team confirmed no such setting.

Hypothesis 2 – Cross‑border link issue

Performed dig +trace and observed that the .app TLD name servers are located outside China, so the problem is not a cross‑border network failure.

Hypothesis 3 – .app TLD NS failure

Tested another .app domain ( get.app ) which resolved correctly, ruling out a TLD‑wide outage.

Hypothesis 4 – DNSSEC configuration error

Online Google DNS query showed “DNSSEC validation failure”. Disabling DNSSEC validation allowed the domain to resolve. The DS record for goofy.app contained an incorrect SHA‑256 key, breaking the trust chain.

Resolution

The erroneous DS record was removed from the DNS management console, after which resolution succeeded worldwide.

Post‑mortem

Chinese resolvers typically disable DNSSEC validation, so they continued to resolve the domain despite the bad DS record, whereas most external resolvers enforce DNSSEC and therefore failed.

Key takeaway: a single malformed DS record can cause global DNS outages when DNSSEC is enabled.