Eric

Glassfish 2.1.1 on Linux: Performance Tuning Essentials

Glassfish may be the best J2EE Application Server out there. It is stable, it is fully j2ee 1.5 compatible, it runs all technologies and it is open source! I love this application server.
Here are some tips that will make it respond faster and increase its capacity in requests per second.

1. Disable application auto-deployment and dynamic class reloading :
Stand-Alone Instances > server (Admin Server) > Advanced tab
-> Auto Deploy --> uncheck
-> Reload  --> uncheck

2 Disable dynamic JSP reloading :
Edit "default-web.xml" inside the config directory of each instance and change the init-param development to false for the org.apache.jasper.servlet.JspServlet (instance restart is required):

3. Minimize logging.
Logger writes to disk and that is very expensive. So you can rearrange logging levels to "SEVERE" for all the loggers (since this setting is dynamic and you can change it to diagnose without restarts, Cool!). 
Admin Console > Logger Settings page > Log Levels tab

Further log minimizing:
Tail your server.log at peak hours and watch carefully at the messages. Catch the loggers that print unnecessary messages and either speak to the programmer to minimize the messages of their applications or you can immediately adjust the logging level for that logger subsystem!
Add a new property with the logger name and set the level to SEVERE or even OFF!!! 

4. Tuning HTTP File Caching in memory for faster response to static resources:
Configurations > config-name > HTTP Service (HTTP File Cache)
Globbally : true 
Max Files Count : 128 - 512 (is a good start for small applications but it really depends on the applications static resources)
Max Age : 86400 (1 day) - 604800 (7 days) is a good start for non dynamic reloading applications (see tip No 3)

5. JVM Parameters:
Configurations > config-name > JVM Settings (JVM Options). 

-server [maximum program execution speed by advanced optimized compilation]
-XX:+UseConcMarkSweepGC [using the Concurrent Mark Sweep garbage collector can cause a drop in throughput for heavily utilized systems, because it is running more or less constantly, but it prevents long pauses, so it is best for real time applications]
-XX:+DisableExplicitGC [Disable explicit full gc collections (System.gc() calls) since it would only interfere with the garbage collection algorithms and create big pause times]
-Xms=-Xmx [having the same starting and maximum heap memory will avoid spending time on any kind of unnecessary resizing of the heap memory]
-Xmn [Set at most half of the heap memory, since the garbage collection here should faster, more often and contains short lived objects. A good start is 1/5 of the heap size.]
-Xss128k [128k Stack Size is a very good start. If you get Stack Overflow error increase it by 128k at a time until you reach a point where you no longer get the error. You might even lower it to 64k (or lower) if your application is really lightweight, and then you will be able to serve more concurrent clients ]
-XX:SurvivorRatio=8 [survivor space and eden ratio will be 1:8. If survivor spaces are too small, copying collection overflows directly into the old generation. If survivor spaces are too large, they will be empty.]
-XX:MaxPermSize=-XX:PermSize [If you get an "java.lang.OutOfMemoryError: PermGen space", you need to increase this value, since the default is 64MB. If you set the initial size and maximum size to equal values you may be able to avoid some full garbage collections that may occur if/when the permanent generation needs to be resized.]

-XX:+CMSClassUnloadingEnabled [Enables the CMS Garbage collector (if you use it) to cleanup the PermGen space too. ]
-XX:+UseParNewGC [this parallel young generation collector can be used with the concurrent low pause collector that collects the tenured generation.]
-XX:ParallelGCThreads [If number of cpus is less than 8 then put the number of cpus else add (3 + (5/8) * (number of cpus)) ]
-XX:TargetSurvivorRatio=90 [Allows 90% of the survivor spaces to be occupied instead of the default 50%, allowing better utilization of the survivor space memory. ]

-XX:MaxTenuringThreshold=30 

-Djava.awt.headless=true
-Dcom.sun.enterprise.server.ss.ASQuickStartup=false

JVM Parameters example:
-server -Xmx2g -Xms2g -Xmn800m -Xss128k -XX:SurvivorRatio=8 -XX:+UseConcMarkSweepGC -XX:+DisableExplicitGC -XX:+UseParNewGC -XX:TargetSurvivorRatio=90 -XX:MaxTenuringThreshold=30 -Djava.awt.headless=true -Dcom.sun.enterprise.server.ss.ASQuickStartup=false



6. Tuning Linux :
Start by checking system limits for file descriptors with this command:
$ cat /proc/sys/fs/file-max
8192
The current limit shown is 8192. To increase it to 65535, use the following command (as root):
$ echo "65535" > /proc/sys/fs/file-max
To make this value to survive a system reboot, add it to /etc/sysctl.conf and specify the maximum number of open files permitted:
fs.file-max = 65535
Note: The parameter is not proc.sys.fs.file-max, as one might expect.
To list the available parameters that can be modified using sysctl:
$ sysctl -a
To load new values from the sysctl.conf file:
$ sysctl -p /etc/sysctl.conf
To check and modify limits per shell, use the following command:
$ limit
The output will look something like this:
cputime         unlimited
filesize        unlimited
datasize        unlimited
stacksize       8192 kbytes
coredumpsize    0 kbytes
memoryuse       unlimited
descriptors     1024
memorylocked    unlimited
maxproc         8146
openfiles       1024
The openfiles and descriptors show a limit of 1024. To increase the limit to 65535 for all users, edit /etc/security/limits.conf as root, and modify or add the nofile setting (number of file) entries:
*         soft    nofile                     65535
*         hard    nofile                     65535
The character “*” is a wildcard that identifies all users. You could also specify a user ID instead.
Then edit /etc/pam.d/login and add the line:
session required /lib/security/pam_limits.so
On Red Hat, you also need to edit /etc/pam.d/sshd and add the following line:
session required /lib/security/pam_limits.so
On many systems, this procedure will be sufficient. Log in as a regular user and try it before doing the remaining steps. The remaining steps might not be required, depending on how pluggable authentication modules (PAM) and secure shell (SSH) are configured.

Tune the TCP/IP settings :

Add the following entry to /etc/rc.local
echo 30 > /proc/sys/net/ipv4/tcp_fin_timeout
echo 60000 > /proc/sys/net/ipv4/tcp_keepalive_time
echo 15000 > /proc/sys/net/ipv4/tcp_keepalive_intvl
echo 0 > /proc/sys/net/ipv4/tcp_window_scaling

Add the following to /etc/sysctl.conf

# Disables packet forwarding
net.ipv4.ip_forward = 0
# Enables source route verification
net.ipv4.conf.default.rp_filter = 1
# Disables the magic-sysrq key
kernel.sysrq = 0
net.ipv4.ip_local_port_range = 1204 65000
net.core.rmem_max = 262140
net.core.rmem_default = 262140
net.ipv4.tcp_rmem = 4096 131072 262140
net.ipv4.tcp_wmem = 4096 131072 262140
net.ipv4.tcp_sack = 0
net.ipv4.tcp_timestamps = 0
net.ipv4.tcp_window_scaling = 0
net.ipv4.tcp_keepalive_time = 60000
net.ipv4.tcp_keepalive_intvl = 15000
net.ipv4.tcp_fin_timeout = 30
Add the following as the last entry in /etc/rc.local
sysctl -p /etc/sysctl.conf
Reboot the system.

Use this command to increase the size of the transmit buffer:

tcp_recv_hiwat ndd /dev/tcp 8129 32768

Make the OS to use swap file only on emergencies:
swappiness=0
swap off; swap on

7. Disable the Security Manager only if your application server is inside an intranet or inside a very well protected environment (If you are sure that no malicious code will be run on the server and you do not use authentication within your application, then you can disable the security manager). It is generally not recommended but it could provide a significant performance boost (since the security manager has expensive calls).
Configurations > config-name > JVM Settings (JVM Options)
delete the option that contains the following text: 
-Djava.security.manager


8. Performance monitoring tools:
jconsole : jmx instance monitoring
jvisualvm : will give you a clear view of the cpu utilization, the garbage collections, perm gen size and jvm options. It can also profile and create thread dumps.
Applications Manager : is a nice commercial monitoring tool. Has good support for GlassFish (There is a Free Licence for up to 5 Glassfish instances through jmx).
nmon : nice open source linux monitoring command line gui.

9. Disable monitoring if you have no problems to detect and resolve.
Configurations > config-name > Monitoring
Change all levels to Low if you still need a little monitoring, or Off if your applications are running smoothly. 

10. Update your software as often as your infrastructure availability/downtime permits. Update your jdk, since bug fixes and optimizations may increase the performance of your application server and make some weird errors go away! The same goes for the operating system, as well as the jdbc drivers or any other libraries your applications use.

11. Patch Glassfish 2.1.x with the latest Grizzly 1.0.x releases.
The Grizzly thread manager library is the heart of GlassFish. Since oracle is not planning to release another v2 release, this kind of patch is significant and relatively easy. Unless you decide to move on to v3 release!

First Check the current Grizzly version by setting the JVM property: "-Dcom.sun.enterprise.web.connector.grizzly.displayConfiguration=true".
Make sure the web-container log level is INFO. Restart Glassfish instance and check server.log. You should see an output like this :

Grizzly 1.0.30 running on Mac OS X-10.5.8 under JDK version: 1.6.0_15-Apple Inc.
port: 8080
maxThreads: 5
ByteBuffer size: 4096
useDirectByteBuffer: 8192
maxKeepAliveRequests: 250
keepAliveTimeoutInSeconds: 30
Static File Cache enabled: false
Pipeline : com.sun.enterprise.web.portunif.PortUnificationPipeline
Round Robin Selector Algorithm enabled: false
Round Robin Selector pool size: 1
Asynchronous Request Processing enabled: true|#]

where Grizzly version is 1.0.30. Please note, that if you'll see a similar output, but without Grizzly version in it, it means that your version is older than 1.0.30, so a Grizzly upgrade is recommended. Download latest Grizzly 1.0.x binary file from:
http://download.java.net/maven/2/com/sun/grizzly/grizzly-framework-http/
and save it to a directory, for example :
/home/gfuser/grizzly/grizzly-framework-http-1.0.30.jar

Then set Glassfish prefix-classpath to "/home/gfuser/grizzly/grizzly-framework-http-1.0.30.jar" to force Glassfish use the latest Grizzly classes instead of the embedded ones. Restart Glassfish and check the server.log again to confirm the success of the patch. Reset the web-container log level to SEVERE.


Further reading. 

Java Tuning White Paper :
http://java.sun.com/performance/reference/whitepapers/tuning.html

Frequently Asked Questions about Garbage Collection :
http://www.oracle.com/technetwork/java/faq-140837.html

Tuning Garbage Collection :
http://www.oracle.com/technetwork/java/gc-tuning-5-138395.html
  
Java HotSpot VM Options :
http://www.oracle.com/technetwork/java/javase/tech/vmoptions-jsp-140102.html

Sun Java System Application Server 9.1 Performance Tuning Guide :
http://download.oracle.com/docs/cd/E19159-01/819-3681/ 


Any comments, suggestions, problems or requests will be warmly welcomed.

Posted at 10:10PM sept. 13, 2011 by Éric in Glassfish  | 

Installation de roller 4.0.1 sous Glassfish 2.1

Installation de roller 4.0.1 sous Glassfish 2.1 En rédaction.

Posted at 12:58AM avr. 03, 2009 by Éric in Glassfish  | 

GlassFish Enterprise Server v2.1 installation

Exemple d'installation sous Solaris

• Téléchargé Glassfish.

# export JAVA_HOME=/usr/jdk/latest
# cd /opt
# mkdir glassfish
# /usr/sfw/bin/wget http://download.java.net/javaee5/v2.1_branch/promoted/SunOS_X86/glassfish-installer-v2.1-b60e-sunos_x86.jar
# /usr/jdk/latest/bin/java -client -Xmx128m -jar glassfish-installer-v2.1-b60e-sunos_x86.jar
# cd glassfish

• (optionnel) Vous pouvez changer les ports 80/443. Il faut modifier le fichier setup.xml.

# vi setup.xml

...

# grep -i property setup.xml  |egrep "https.port|instance.port"
<property name="instance.port" value="80"/>
<property name="https.port" value="443"/>

• Install.

# chmod -R +x lib/ant/bin
# export JAVA_OPTS="-Xms128m -Xmx256m"
# lib/ant/bin/ant -f setup.xml

...
create.domain:
     [exec] Using port 4848 for Admin.
     [exec] Using port 80 for HTTP Instance.
     [exec] Using port 7676 for JMS.
     [exec] Using port 3700 for IIOP.
     [exec] Using port 443 for HTTP_SSL.
     [exec] Using default port 3820 for IIOP_SSL.
     [exec] Using default port 3920 for IIOP_MUTUALAUTH.
     [exec] Using default port 8686 for JMX_ADMIN.
     [exec] Domain being created with profile:developer, as specified by variable AS_ADMIN_PROFILE in configuration file.
     [exec] Security Store uses: JKS
     [exec] Domain domain1 created.
     [exec] Login information relevant to admin user name [admin] for this domain [domain1] stored at [/.asadminpass] successfully.
     [exec] Make sure that this file remains protected. Information stored in this file will be used by asadmin commands to manage this domain.
   [delete] Deleting: /opt/glassfish/passfile

BUILD SUCCESSFUL
Total time: 29 seconds

• Changer le JDK pour la bonne version.

# grep AS_JAVA /opt/glassfish/config/asenv.conf 
AS_JAVA="/usr/jdk/latest"

• Ajouter Glassfish au service SMF. Nous devrons créer un fichier passfile pour l'intégrer.

# vi /opt/glassfish/passfile
AS_ADMIN_USER=admin
AS_ADMIN_PASSWORD=<mot de passe>
AS_ADMIN_MASTERPASSWORD=<mot de passe>

# chmod 400 /opt/glassfish/passfile

# bin/asadmin create-service --passwordfile passfile domains/domain1

The Service was created successfully. Here are the details:
Name of the service:application/SUNWappserver/domain1
Type of the service:Domain
Configuration location of the service:/opt/glassfish/domains
Manifest file location on the system:/var/svc/manifest/application/SUNWappserver/domain1_usr_local_glassfish_domains/Domain-service-smf.xml.
The service could be enabled using svcadm command.
Command create-service executed successfully.

Il faut mettre le mot de passe de AS_ADMIN_MASTERPASSWORD=<mot de passe> en commentaire.

# svcs domain1
STATE          STIME    FMRI
disabled       10:24:48 svc:/application/SUNWappserver/domain1:default

# svcadm enable svc:/application/SUNWappserver/domain1:default

# svcs domain1
STATE          STIME    FMRI
online         10:29:21 svc:/application/SUNWappserver/domain1:default

• Suppression des fichier d'installation.

  # rm ../glassfish-installer-v2.1-b60e-sunos_x86.jar
  

Cluster Load Balancer

Configuring the Cluster/Load Balancer with GlassFish V2

GassFish LB Cluster

# bin/asadmin 
asadmin> start-domain

asadmin> delete-jvm-options "\-client"

asadmin> create-jvm-options "\-server"

asadmin> delete-jvm-options "\-XX\:MaxPermSize=192m"

asadmin> create-jvm-options "\-XX\:MaxPermSize=512m"

asadmin> delete-jvm-options "\-Xmx512m"

asadmin> create-jvm-options "\-Xmx2g"

asadmin> create-jvm-options "\-Xms2g"

asadmin> create-jvm-options "\-XX\:+CMSClassUnloadingEnabled"

asadmin> create-jvm-options "\-XX\:+CMSPermGenSweepingEnabled"

asadmin> create-cluster --host localhost --port 4848 cluster1
asadmin> create-node-agent --host localhost --port 4848 agent1
libre gestion par glassfish
asadmin> create-instance --host localhost --port 4848 --nodeagent agent1 --cluster cluster1 instance1
asadmin> create-instance --host localhost --port 4848 --nodeagent agent1 --cluster cluster1 instance2

reboot

asadmin> start-node-agent agent1
Please enter the admin user name>admin
Please enter the admin password> <mot de passe>

Please enter the master password [Enter to accept the default]:> <appuie sur enter>
asadmin> start-cluster --host localhost --port 4848 cluster1

asadmin> create-http-lb --devicehost 10.53.229.223 --deviceport 4443 --autoapplyenabled=true --interactive=true --lbenableallinstances=true --lbenableallapplications=true lb-cluster1
asadmin> enable-http-lb-server lb-cluster1
asadmin> enable-http-lb-application clusterjsp

Installation apache Coolstack

Configuration du module LB aslb.
Telecharger le module aslb.

# /usr/jdk/jdk1.6.0_11/bin/jar -xf aslb-9.1.1-b9.jar
# cd /opt/glassfish/lib
# mkdir plugin
# cp *.zip /opt/glassfish/lib/plugin
# unzip "*.zip"
# rm *.zip
# chmod -R 755 lib

# cd /opt/coolstack/apache2
# mkdir sec_db_files modules/resource modules/errorpages
# cp /opt/glassfish/lib/plugin/lib/webserver-plugin/solaris/apache2.2/LBPlugin*.res /opt/coolstack/apache2/modules/resource/
# cp /opt/glassfish/lib/plugin/lib/webserver-plugin/solaris/apache2.2/mod_loadbalancer.so /opt/coolstack/apache2/modules/
# cp /opt/glassfish/lib/plugin/lib/webserver-plugin/solaris/apache2.2/errorpages/* /opt/coolstack/apache2/modules/errorpages/
# cp /opt/glassfish/lib/plugin/lib/install/templates/loadbalancer.xml.example /opt/coolstack/apache2/conf/
# cp /opt/glassfish/lib/plugin/lib/dtds/sun-loadbalancer* /opt/coolstack/apache2/conf/
# cp /opt/glassfish/lib/plugin/lib/webserver-plugin/solaris/apache2.2/*.db /opt/coolstack/apache2/sec_db_files/

# vi /opt/coolstack/apache2/conf/loadbalancer.xml

<!DOCTYPE loadbalancer PUBLIC "-//Sun Microsystems Inc.//DTD Sun ONE Application Server 7.1//EN" "sun-loadbalancer_1_2.dtd">
<loadbalancer>
    <cluster name="cluster1"  policy="round-robin">
        <instance  name="instance1" enabled="true" disable-timeout-in-minutes="60" listeners="http://10.53.229.223:38080 https://10.53.229.223:38181"  weight="100"/>

        <instance  name="instance2" enabled="true" disable-timeout-in-minutes="60" listeners="http://10.53.229.223:38081 https://10.53.229.223:38182"  weight="100"/>
        <instance  name="instance3" enabled="true" disable-timeout-in-minutes="60" listeners="http://10.53.229.223:38082 https://10.53.229.223:38183"  weight="100"/>
        <web-module context-root="clusterjsp" enabled="true" disable-timeout-in-minutes="60" error-url="sun-http-lberror.html" />
        <health-checker url="/" interval-in-seconds="10" timeout-in-seconds="30" />
    </cluster>
    <property name="reload-poll-interval-in-seconds" value="60"/>

    <property name="response-timeout-in-seconds" value="30"/>
    <property name="https-routing" value="true"/>
    <property name="require-monitor-data" value="false"/>
    <property name="active-healthcheck-enabled" value="false"/>
    <property name="number-healthcheck-retries" value="3"/>
    <property name="rewrite-location" value="true"/>

</loadbalancer>

# vi /opt/coolstack/apache2/conf/extra/httpd-aslb.conf

##BEGIN EE LB Plugin Parameters
LoadFile /usr/lib/libCstd.so.1
LoadModule apachelbplugin_module /opt/coolstack/apache2/modules/mod_loadbalancer.so
<IfModule mod_apache2lbplugin.cpp>
config-file "/opt/coolstack/apache2/conf/loadbalancer.xml"
locale en
</IfModule>

<VirtualHost 10.53.229.223>
DocumentRoot "/opt/coolstack/apache2/htdocs"
ServerName cluster1.dgeq.qc.ca
</VirtualHost>
##END EE LB Plugin Parameters

# vi /opt/coolstack/apache2/conf/httpd.conf
ajouter a la fin.

# LoadBalancer aslb pour glassfish
Include conf/extra/httpd-aslb.conf

# vi .profile
# vi .bash_profile
sous LD_LIBRARY_PATH
LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:/usr/lib:/usr/lib/mps:/usr/share/lib

LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:/opt/glassfish/lib # glassfish aslb
LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:/opt/glassfish/lib/plugin/lib # glassfish aslb
LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:/opt/coolstack/lib #  coolstack pour glassfish aslb

# vi /opt/coolstack/apache2/bin/apachectl
ajouter au début du fichier sous #!/bin/sh
###
### LD_LIBRARY_PATH
###
LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:/usr/lib:/usr/lib/mps:/usr/share/lib
LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:/usr/sfw/lib
LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:/opt/glassfish/lib # glassfish aslb
LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:/opt/glassfish/lib/plugin/lib # glassfish aslb
LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:/opt/coolstack/lib #  coolstack pour glassfish aslb
export LD_LIBRARY_PATH

Posted at 07:45PM nov. 29, 2008 by Éric in Glassfish  |