In order to brush up some elementary statistics, I decided to read more about standard deviation as a measure of variability. What I couldn’t understand initially was the difference between

as ’standard deviation of a sample’, and …

which is ’sample standard deviation’.The difference here lies in the denominator – n versus n-1.

The wikipedia article on SD calls it Bessel’s correction and the wiki entry on that is equally impenetrable, only for qualified mathematicians. Here’s a more plausible (in my own words) explanation of the reason behind reducing n to n-1 that I read in the book ‘How to think about statistics’:

Enter Degrees of Freedom.

They are essentially values of data, or scores, that are free to vary. Consider a set of 5 numbers. If you were asked to guess those numbers, you could theoretically be free to think of any number. Let’s say that the first number is 3. What’s the second number? Again, you could think of anything. Let’s say you come up with 4 (let’s keep it simple); and so on (3, 4, 5 and 7). Now, assuming you know the mean (5 for our example) of the scores, what could be the missing value? There’s only one possible number – 6.

So, if you know the mean of a set of scores with a single value missing, you are no longer free to select its value and the only way to determine value for the 5th score is by using the remaining 4 scores. That’s the n-1 right there.

When you calculate the mean of a set of scores, all scores were free to have whichever values they were. So you divide them by n and get a mean for that set of scores. Remember, when you calculate the mean for score(s), your degrees of freedom remains n.

When you calculate standard deviation, you are not using those independent values, but the mean as well! If you had a score missing, using the mean would have been fine. However, in order to ensure our standard deviation isn’t biased and is based on ‘truly independent’ scores, you have to make corrections for the loss of a single degree of freedom.

Which is why an ‘accurate’ formula for standard deviation will have n-1 and not n. The reason this is usually ignored is that the effect on standard deviation for a large set of scores doesn’t really change that much. A sample of 100 scores will only have 1% difference on the number of samples to be used.

I know I sound like a complete novice at this – I am. Knowing mathematics and actually writing about it are two completely different things. Let’s see if I can learn both (wink).

It is standard practice to use the Insert Cookie mechanism to enable persistence on a virtual server on a BigIP LTM. However, that can be an expensive task for the load-balancer; inserting cookies takes CPU. Another way is to use an existing cookie that already marks your session and use cookie hashing. One example is the JSESSIONID cookie set by application servers such as Tomcat.

All you need to do is:

1. Go to Profiles -> Persistence -> Create New…
2. Using the cookie profile as the base, create one and mark it as Custom
3. Change the type to Cookie Hash and put in the cookie name
4. You can put in values for offset and length (for example, 1 and 32)
5. Go to Virtual Servers -> Your Virtual Server
6. Make sure client profile is HTTP
7. Under the Resources tab change the default persistence mechanism to the one you just created.

Viola!

But, you’ll soon notice it doesn’t really work as expected, especially when your clients are behind a proxy; why? If the source address of the client requests are the same and there’s already a TCP connection running, the LTM will forward that request there without even looking at the cookie. This is done to help performance. However, since you naturally need to accommodate proxies and such connections, make sure the HTTP Profile on your virtual server is OneConnect. That will make sure each of your requests get scanned and forwarded based on the cookie value and not the source IP address.

This is well-documented in the manual, but doesn’t seem to appear when I google/bing around. And for trigger-happy folks like me who don’t RTFM and start experimenting, this just might save 5 or 10 minutes.

After a long hiatus from my activity here, I have decided on two things:

1. Blog entries can be short

2. They need to be more frequent – for my own sanity as well.

I plan to read and take cues from the TCP/IP Drinking Game and learn the nuances of the protocol a bit more. There are some interesting things I learnt which I will write about, later.

That’s it for now. Ciao!

This is the first time I started a file transfer, and on hindsight, struck my head and said “Wish I’d started this with nohup or screen…”; I could have left home on time with this laptop tagging along with me.

What I didn’t know *then* (now I know) that there’s a beautiful bash built-in called disown which can attch any/all running jobs to the init process. Yay!


[root@linux-test data]# scp RHEL4-U5-i386-AS-disc4.iso suleym@192.168.1.1:/appl/RHEL-AS4/
suleym@192.168.1.1's password:
RHEL4-U5-i386-AS-disc4.iso 0% 1016KB 1.0MB/s 06:21 ETA
[1]+ Stopped scp RHEL4-U5-i386-AS-disc4.iso suleym@192.168.1.1:/appl/RHEL-AS4/
[root@linux-test data]# bg
[1]+ scp RHEL4-U5-i386-AS-disc4.iso suleym@3.122.220.169:/appl/RHEL-AS4/ &
[root@linux-test data]# disown -h


More about disown here and in man bash.

Writing this post for the sake of not reverse-engineering searches on Google, and so that people see disown *related* to nohup.

I came across a simple but intriguing problem – apachectl restart will work and restart apache processes, and in my case restart the CA/Netegrity Siteminder agent as well. However, the server didn’t respond, and neither were there any messages on the error log. SM logs said the agent initialized successfully.

When I remove mod_sm.so and restart apache (after removing environment variables related to SM), everything worked just fine. I naturally assumed that the problem was with this module that I just removed.

It turned out that the problem was with this particular semaphore which didn’t release since about the last 24 hours, and was somehow linked to the siteminder agent module. After I did an ipcrm -s ID, everything was working fine as before.

I always thought semaphores/shared memory segments not freeing up will result in apache not restarting successfully. This is the first time apache didn’t complain on a restart, no logs displayed any errors, ‘removing’ a module rectified the error, and putting it back actually made the issue recur!

Need to learn more about semaphore allocation in linux.

For those who not have heard about Berkeley DB (called BDB): it is a transactional storage engine with basic key/value pairs, very agile and highly performance-oriented, with no SQL-like engine. Compared to it’s native version, the Java Edition has quite a few differences and is useful when it is to be integrated with a basic Java application.

The aim of the database is to be available in RAM all the time as much as possible, so that all query responses are fast. Based on this, here’s my take on tuning the JVM that hosts the BDB:

• JVM heap size should be around the same size as the data store
• Use the Concurrent Mark/Sweep GC algorithm to have low-pause GC times
• Since most of the objects are going to be living ‘forever’, it’ll make sense to have a huge tenured generation
• If the DB size can vary, refrain from giving Xmx and Xms the same values. Give a huge difference so that the JVM can manage it as your data grows

This is what CATALINA_OPTS might look like (includes a lot of debug flags as well):

CATALINA_OPTS="-server -Xms1024m -Xmx4096m -XX:+UseMembar -XX:+PrintGCDetails -
XX:+PrintGCApplicationStoppedTime -XX:NewRatio=4 -XX:+UseConcMarkSweepGC -verbos
e:gc -Xloggc:/appl/tomcat/logs/gcdata.txt"


-XX+UseMemBar is there to accomodate for the high IO waits I had been seeing – I think there’s a problem in linux with the JDK using memory barriers. I read about the bug here.

BDB Java Edition is not a replacement for a traditional database, but is a means to have almost immediate results for things like look-up data, subscriptions and most frequently-used information. There are quite a number of on-line resources available to help you set it up and use it – native or Java, whichever your flavor is.

memcached is another such tool that is useful when improving performance for an application-database connection.  More on it in another post some other time.

Cheers!

If you need to parse SSI from a JSP response, there are two simple ways to do it:

1. Use the SSIServlet and handle it within tomcat
2. If you have a separate web server like Apache in front of tomcat, and you want that web server to do it, the plot thickens.

If you ask, ‘why, when you are already using Java? You can do all that you can do with SSI in a JSP, right?‘, you might be surprised. Let’s just say the reason is out-of-scope for this post.

So, you have a three-tier architecture with web servers spread across the world and app/DB servers local to certain data-centers. Naturally, you might want to ‘assimilate’ content on the web servers (closest to local users based on 3DNS/similar) where it’s already present instead of shuttling bytes back-and-forth between the web and app layers. That’s the reason. And did I say earlier it was out of scope? My bad.

The way you would do it is set up Apache on a specific Location to grab for, put an AddOutputFilterByType statement with the MIME type as text/x-server-parsed-html and finally, on the JSP itself, you will set the MIME type using setContentHeader for the response.

Your Location section might look like this:

<Location /application/ssiparser >
Options +Includes
AddOutputFilterByType INCLUDES;DEFLATE text/x-server-parsed-html
</Location>

In an ideal scenario, everything should have been hunky-dory, but life isn’t so simple. At least it didn’t happen so easily for me.

What I had done earlier was, in order to make certain performance improvements, added a CompressionFilter on tomcat to gzip all responses from it so that the app-web performance improves as well. This meant that once the response reached Apache it would already be gzipped and SSI parsing would not be possible. Mind you, this is Apache 2.0.x and not 2.2.x where you can actually set up FilterDeclare and such.

There are two ways to get around this problem:

1. Get the CompressionFilter to exclude the Location you have on for SSI, and then pass on INCLUDES;DEFLATE to AddOutputFilterByType.
2. Or, unset the Accept-Encoding header on the request first so that it doesn’t take gzip and the CompressionFilter doesn’t compress it at all. If I try to deflate it again now, it doesn’t happen.

The problem with (2) is that you end up sending decompressed data across. Option (1) would be the right way to go.

(1) will entail a change on the web.xml for your application.

(2) will look like this:

<Location /application/ssiparser >
Options +Includes
RequestHeader unset Accept-Encoding
RequestHeader set Accept-Encoding deflate
AddOutputFilterByType INCLUDES;DEFLATE text/x-server-parsed-html
</Location>


The JSP will start with:

<%
response.setHeader("Content-Type","text/x-server-parsed-html");
%>
<!--#include virtual="/static/content/news.html"-->
<!--#include virtual="/static/content/weather.html"-->
<!--#include virtual="/static/content/media.html"-->


Most folks do not upgrade Apache as they do with other kinds of software, just because it's so damn stable and fulfills your requirements very well. However, I feel if you need to work with filters and play around with them, 2.2 will be the way to go.

]]>