I’ve been evaluating two Netgear FVX538 as possible el’cheapo VPN concentrator at work. The designers of this model have decided that you must have NTP running. They’ve taken this to the point that you can’t even configure the ‘WAN’ (external) interfaces without setting up both a nameserver and a default gateway. I found this requirement puzzling until I started looking at the traffic coming out of the system.

16:00:46.950661 arp who-has 192.168.100.1 tell 192.168.100.100
16:00:46.950675 arp reply 192.168.100.1 is-at 00:11:43:6c:81:79 (oui Unknown)
16:00:47.516093 IP 192.168.100.200.65098 > 10.10.10.10.domain: 41016+ A? time-h.netgear.com. (36)
16:00:47.516182 IP 192.168.100.200.65098 > 10.10.10.10.domain: 41016+ MX? time-h.netgear.com. (36)
16:00:47.972148 IP 192.168.100.100.65098 > 10.10.10.10.domain: 39168+ A? time-g.netgear.com. (36)
16:00:47.972236 IP 192.168.100.100.65098 > 10.10.10.10.domain: 39168+ MX? time-g.netgear.com. (36)
16:00:50.546054 IP 192.168.100.200.65098 > 10.10.10.10.domain: 44016+ A? time-h.netgear.com. (36)
16:00:50.546145 IP 192.168.100.200.65098 > 10.10.10.10.domain: 44016+ MX? time-h.netgear.com. (36)
16:00:53.576117 IP 192.168.100.200.65098 > 10.10.10.10.domain: 47016+ A? time-h.netgear.com. (36)
16:00:53.576205 IP 192.168.100.200.65098 > 10.10.10.10.domain: 47016+ MX? time-h.netgear.com. (36)

And the systems are incredibly aggressive about it as well. Firing off a batch of domain queries every few seconds. Maybe they would be better behaved if 10.10.10.10 actual existed and was a resolver but still…

So I set about trying to turn NTP off. Can this been done via the web interface? Of course not. Does the manual tell you how to turn it off? Of course not. Since these little boxes run some variation of Montavista embedded Linux (and all you have to do to get a root shell is connect to the serial console… authentication? WE DON’T NEED NO STINKING AUTHENTICATION) I decided to attempt to shut it off behind the management interfaces back.

Finding the process was easy enough:

# ps -eaf | grep ntp
253 root 1312 S /mnt/cramfs/igateway/sntpc

And killing it seemed easy enough:

# kill 253

And sure enough the resolver queries stop for about 45s.

# Could not open /proc/298/cmdline
MONITORD : sntpc HAS DIED !!!!!!! , pid = 253
Bringing down interfaces
ixp0.12: del 01:00:5e:00:00:01 mcast address from master interface
ixp0.12: del 01:00:5e:00:00:01 mcast address from vlan interface
Removing driver modules
Removing Cavium driver
rmmod: pkp_drv: Device or resource busy

The system is going down NOW !!
Sending SIGTERM to all processes.
Sending SIGKILL to alligatewaySMP thread terminated.
umount: /dev/mtdblock2 busy – remounted read-only
Please stand by while rebooting the system.
Enabling flash read mode
Restarting system.

Argh!

Evern worse, you can’t even change the NTP server(s) as they are hardcoded into the sntpc binary.

# grep netgear.com /mnt/cramfs/igateway/sntpc
time-a.netgear.com
time-b.netgear.com
time-c.netgear.com
time-d.netgear.com
time-e.netgear.com
time-f.netgear.com
time-g.netgear.com
time-h.netgear.com

There are no editing tools in the system image (not even ed), and the image itself seems to be a hacked version of cramfs so I can’t even modify the image on another system and reflash the firmware. I’d settle for just being able to remove sntpc from the monitord configuration, or stopping monitord from starting at all but the managment interface doesn’t support that either. Needless to say Netgear got a support email. I’m sure they’ll get right on fixing this. At least they were only $359 each through Dell and setting up an ipsec tunel (what I purchased them for) is pretty strait forward. Stay tuned for a rant in the future about these untis shipping with a broken TLS implimentation.

Update #3:

The original email to Netgear technical support (for context):

Hello,

I just purchased two FVX538 for hardware validation purposes and I have a couple of questions.

Why isn’t there any sort of login/authentication on the serial console (fw V1.6.38 & V1.6.49)?

Why wasn’t busybox built with ping or traceroute support? It should be pretty obvious that an admin might want ping support from the CLI.

These systems will never be able to access the Internet. Yet they are constantly attemping to to resolve the hostnames time-h.netgear.com, time-g.netgear.com, etc., presumably for NTP syncronization. Is it possible to disable NTP updates or reconfigure the NTP server list?

Are there any plans to ssh to the CLI? The arm processor in this box is certainly capable of supporting it.

Thanks,

-J

Update #1:

It turns out this unit has a funky IOS like interface that you can get at by either telneting into the unit or running ‘cli’ from the root shell attached to the serial interface.

Connected to 192.168.110.1.
Escape character is ‘^]’.

login: admin
Password:

**************************************************************
Welcome To The Netgear FVX538 Command Line Interface
**************************************************************
FVX538: />

This interface appears to have the ability to configure ntp, including setting the ntp servers and enabling/disabling the service.

FVX538: /config/sntpc>show
Sntpc Status : Disabled
FVX538: /config/sntpc>save
Saving The Configuration. This may take some time.

Except that not only does this NOT change the running config all configuration changes are lost after a reboot.

FVX538: /config/sntpc>show
Sntpc Status : Enabled
Timezone : GMT , 00:00 Hrs From GMT
Sntpc Server usage : Secondary server
Operational Status : Entered in Delayed-Wait state after DNS_RESOLVE retri

This is that same story that I got from Netgear’s technical support.

Case # 2834811
Problem Presupport
Cause Online request
Status Open
Notes
2/25/2006 12:07:00 AM

Thank you for contacting NETGEAR Support my name is Jacquelin and I will be your support agent for this case.

Currently your case is at Level 1. We will be working to assist you in resolving the issue you described. Because we are doing this online it may require a few contacts before we can resolve the issue, but please be assured we are here to assist.

I understand your concern and appreciate the opportunity to assist you.

Log in to the router configuration page.
Open Internet Explorer or any other browser and access the site : http://192.168.0.1 or http://192.168.1.1
By default, the username is admin, the password is password.

Click on Diagnostics under Management,where you can see both Ping and Trace route.
Updating or Reconfiguring NTP server list is not possible,since the NTP server is configured once the router is configured for the first time.

I hope this should resolve your concern.

Kindly revert back for further assistance.

Regards,
Jacquelin.

Sigh. Perhaps level 2 technical support will file a bug fix/feature request for me.

Update #2:

My response to Netgear technical support:

It appears that your supposed to be able to configure the NTP service by either telneting into the unit or running ‘cli’ from the root shell attached to the serial interface.

FVX538: /config/sntpc>show
Sntpc Status : Disabled
FVX538: /config/sntpc>save
Saving The Configuration. This may take some time.

Except that not only does this NOT change the running config all configuration changes are lost after a reboot.

FVX538: /config/sntpc>show
Sntpc Status : Enabled
Timezone : GMT , 00:00 Hrs From GMT
Sntpc Server usage : Secondary server
Operational Status : Entered in Delayed-Wait state after DNS_RESOLVE retri

Can you please either:

a) escalate this bug to tech support level N+1
b) file a bug with your development group to the effect that the CLI interface on this model is broken.
c) file a feature request to add NTP configuration to the web management interface
d) publish the source code to sntpc daemon and your modified cramfs firmware construction tools so I can fix this myself.

Thanks.

Update #5 (2006-03-06):

It seems that Netgear support did actually respond to my last email (they don’t send you the actual response, just an email notification that your case status has changed) and I missed it.

2/26/2006 7:03:00 AM
Thank you for following up with the information I requested. Upon reviewing the data you have provided I believe that this issue is much more extensive than I can assist you with here at L1. I am going to escalate your case to L2 for further review and response. Please allow them 24 hours to review and respond to you.

Once you have reviewed my response here, when prompted, please select no to resolving your case and no to respond. The L2 agent will update the case once it is received.

Again, I thank you for the opportunity to assist you and THANK YOU for choosing NETGEAR.

Regards,
Naveen
NETGEAR SUPPORT

Except that when I attempt to follow those instructions there is no “no reply” option so I put in some simple text:

Please escalate this issue to “L2” support.

And I get this lovely response:

A fatal server error occured and has been emailed to support personel.