vyatta

While working on Interwoven OpenDeploy I came across the following problem:

Large deployments or file-pushes spanning a WAN or a continent used to sometimes time-out or roll back. The problem was noted where there was a significant difference of size between file lists.

This is what happens:

1. OD starts n threads based on the n lists of files to be deployed.
2. Thread 1 finishes and the remaining n-1 threads continue file transfer.
3. After exactly 5 minutes, thread 1 times out (shows a TCP packet with RST flag set on tcpdump) and after all threads finish, the deployment fails and rolls back the transaction.

Root cause:
Some network device on the way times out TCP idle sessions more than 300 seconds and sends an RST flag, dropping the connection essentially. When this happens, OpenDeploy considers the transaction corrupt and rolls it back.

Fix

1. Get the firewall to extend the timeout to a more reasonable time (perhaps similar to the default tcp_keepalive_time of 7200 seconds?) – not practically possible if a number of teams are involved.
2. Change tcp_keepalive_time to ~200 seconds
3. If the keepalive change does not help alone, try http://libkeepalive.sourceforge.net . Works like a charm!

Generally speaking, and not being ‘opendeploy-centric’, I did learn the importance of keepalive packets and how the default value of 7200 seconds might not be practical when an application talks to servers across network borders.

Thanks to my colleague Prajal Sutaria for working on this!

Anyone who has worked with SAML knows very well how effective and simple it is to achieve federated services with your own authentication mechanism. What needs to be remembered, though, is that end-users might very well be behind firewalls. And with that come proxies; and those proxies open up the Pandora’s box aka cache.

Proxies can cache POST response from the authentication user agent and make user1 see a page which says ‘Welcome user2′. Do a forced-refresh (Ctrl-F5, Cmd-R) on the browser, and you can see your own ID again.

Fixes:
1. Ensure proxies don’t cache any content for your authentication domain.
2. Pass a ‘random’ value like the timestamp using Javascript to the URL (to make it unqiue)
3. Force the content-provider’s web server, and the user agent web server to set Cache-Control to max-age=0 and proxy-revalidate.
4. Make sure you’re sending an invalidation string in the packet as well.

Clearing proxies in a company with about ~100 proxy servers might not be the right choice. The onus should lie on the development and the sysadmin team to make sure important pages are non-cacheable. Never trust proxy servers is the motto here.

Problem Statement: Ability to ping a user-defined hostname with a valid IP address
Solution: Simple, put it in the /etc/hosts file and you’re done.

You still can’t do it; did you check nsswitch.conf? This is what should be there: hosts: files dns .
So, with the right /etc/nsswitch.conf and /etc/hosts, should it work?

root@treebeard:~# cat /etc/hosts
127.0.0.1 localhost
127.0.1.1 treebeard
192.168.2.2 mithrandir



root@treebeard:~# ping mithrandir
PING mithrandir (192.168.2.2) 56(84) bytes of data.
64 bytes from mithrandir (192.168.2.2): icmp_seq=1 ttl=64 time=0.092 ms
64 bytes from mithrandir (192.168.2.2): icmp_seq=2 ttl=64 time=0.067 ms


It works!
But…

root@treebeard:~#sudo su - mohit
mohit@treebeard:~$ ping mithrandir
ping: unknown host


It seems when I switch to a non-root user, entries in /etc/hosts fail to take effect.
Why?

The problem is with the read attributes on /etc/nsswitch.conf. I hadn’t noticed that it was world-unreadable.

root@treebeard:~# chmod o+r /etc/nsswitch.conf
root@treebeard:~#sudo su - mohit
mohit@treebeard:~$ ping mithrandir
mohit@treebeard:~$ ping mithrandir
PING mithrandir (192.168.2.2) 56(84) bytes of data.
64 bytes from mithrandir (192.168.2.2): icmp_seq=1 ttl=64 time=0.092 ms
64 bytes from mithrandir (192.168.2.2): icmp_seq=2 ttl=64 time=0.067 ms


Worked, finally. The weird thing is I would have assumed ping to complain that it wasn’t able to read a file or something, but there was nothing of that sort. This means you can actually force a user to stick to DNS resolution and all daemons and root-owned processes to leverage /etc/hosts.

Bad idea I’d say. This might be a ticking time-bomb. I faced this problem when configuring two nodes for a 10g RAC cluster. The DB runs as a user, and the DBA had a tough time getting the private interconnect working – thanks to nsswitch.conf.

Lesson learnt.

After having built mod_python.so and doing a LoadModule modules/mod_python.so I expected everything to work fine, assuming I did a SetHandler python-program and a PythonHandler helloworld within an Apache virtualhost.

I could get Hello World! on the page then, with this following snippet:

$cat helloworld.py
from mod_python import apache
def handler(req):
req.content_type = “text/plain”
req.write(”Hello World!”)
return apache.OK

That’s when the problems began.

1. I couldn’t put in another .py file in the same directory that could run successfully – 404 returned
2. All HTML files returned a 404 as well
3. There was nothing on the error logs

After much reading around, here’s the final configuration that worked:

--snip--
LoadModule modules/mod_python.so
--snip--
PythonDebug On
AddHandler python-program .py
PythonHandler mod_python.publisher
PythonPath "['/appl/python2.5/lib/','/appl/webdocs/','/appl/python2.5/bin/python/','/appl/python2.5/lib/python2.5/site-packages/mod_python/','/appl/python2.5/lib/python2.5/site-packages/'] + sys.path"
--snip--


And the URL that worked was http://myhost/helloworld.py/handler

Apparently, mod_python (also) has three kinds of default handlers – cgi, publisher and PSP.

cgihandler – to work with existing CGI scripts by creating a false environment
psp – Python Server Pages, just like JSP
publisher – The preferred way to handle python pages, using page/method

publisher is recommended for newer applications, and as you move forward and your application increases in complexity and entry points, you can make your own handlers.

Like I said, 101 and basics, but what the heck, I get to write something here. Itchy fingers.

More later.

Finally, I have a home for Automator on Google Code at sysadmin-automator. I will start uploading existing scripts tomorrow.

I have finally been able to create a project on sourceforge for that pet project of mine – Automator.

It all started with my irritation at having to execute the same commands over a number of servers, most of the times during the day. Unfortunately, webmin was not much of an option – I don’t prefer it too much because of it’s highly invasive nature. Moreover, I should have the ability to run a script over a group of servers that I know belong to an application or a tier. For that matter, if today, I wanted to sync all my linux servers from a specific NTP server *manually*, I should ideally execute a command whose pseudocode might look like:

execute on all servers that have OS as linux

So, beginning with this concept, I realized this was not much of a technical feat. It actually was the cumbersome task of ‘documenting’ what each server constituted right from the very beginning and then using this information during my day-to-day tasks.

Let’s say there’s an application called WORDPRESS which has three tiers – DB, WEB and APP and also has three separate set of servers each for DEVelopment, STAGE and PRODuction (that’s how big companies work, you know). To add to it, there are servers across the world, but this particular application actually is in a data-center in NYC. Now, using this information (and much more that can be collected automatically from each server, like, OS, architecture), I should be able to know what servers to work on.

In a nutshell, this is what automator might work like:

mohit@test$auto-command -g WORDPRESS-PROD-WEB -c ntpdate ntp.timeserver.com

This will run ntpdate on all WordPress production web servers. Neat! But how do you bell the cat? The only way I have been able to figure out is during initial handover of the box. Every organization that manages more than a hundred servers has a process in place which has a team ‘audit’ the server and run scripts in order to configure default services before it is handed over to the customer. This is when we can run our own ‘automator service registration’. All you do is connect to the remote server, push your script, run it remotely and then grab some data automatically and of course manually (like environment, name of application etc) and store it in a MySQL database.

I have this working using a system of bash shell scripts. But with time, modifications and the sheer number of servers that I manage, I am moving this over to Python/MySQL. But yes, I’d like to keep it simple – just as a command line interface with a very basic framework that other SAs can use to build their own scripts when they include the package.

Let’s see how far this idea goes.