Troubleshooting Guide

Contents

Please contact our Forums or our users mailing list if your query is not addressed either on this page or in our documentation section.

Exceptions, Issues...

On Windows, I get the following error message right after the first "preparing to install" dialog box finishes: Windows error 3 occurred while loading the Java VM. What does this mean and how do I get around this?

You probably have several versions of the Java SDK and runtime installed and something in the Windows registry entries on your machine is confusing the installer. (Even if the JAVA_HOME environment variable set correctly). To get past this issue, clean up your environment or add the "LAX_VM" parameter and the location of the correct Java executable to the command line, like this:

terracotta-windows-2.6.0.exe LAX_VM "c:/jdk1511/bin/java.exe"

I am getting a message that says "The DSO boot JAR you need for this platform does not exist. You may need to run the 'make-boot-jar' script." What does that mean?

You need to create the boot-jar. Terracotta ships with some standard boot-jars - this means though that the shipped boot jar does not match with the JDK version and/or OS your environment is running. The following command will create the appropriate boot-jar for your environment to fix this problem. You should have to run this only once, until something relevant in the environment changes (e.g. upgrade the JVM).

%TC_INSTALL_DIR%\dso\bin\make-boot-jar.bat

$TC_INSTALL_DIR/dso/bin/make-boot-jar.sh

Make sure to delete the existing boot-jar files under TC_INSTALL_DIR/lib/dso-boot as the make-boot-jar script is sometimes unable to replace existing files.

The -o option (output) tells the script where to create the boot-jar and $TC_INSTALL_DIR%/common/lib/dso-boot/ is the default location. If you omit the -o option, the boot JAR will be generated in the default location.

See the Concept and Architecture Guide for more information on what the boot JAR is.

So I need to add a certain class to the boot-jar. How do I do it?

See Q3 to create the boot-jar. The difference here is that we will provide the configuration file with the additional classes that need to be added to the boot-jar as a command line parameter.

First, add the class in the configuration file in the <additional-boot-jar-classes> section:

<additional-boot-jar-classes>
    <include>java.awt.datatransfer.Clipboard</include>
</additional-boot-jar-classes>

Next, provide the configuration file with the above entry as a parameter to the make-boot-jar script:

%TC_INSTALL_DIR%\dso\bin\make-boot-jar.bat -f tc-config.xml

$TC_INSTALL_DIR/dso/bin/make-boot-jar.sh -f tc-config.xml

Make sure to delete the existing boot-jar files under TC_INSTALL_DIR/lib/dso-boot as the make-boot-jar script is sometimes unable to replace existing files.

I did everything I was supposed to do and nothing is being clustered or shared. What next?

Is the Terracotta Server running? Look at the client logs for exceptions. Chances are there was some exception which is most likely covered in this document or in our documentation section. If there are no exceptions and you see something similar to the following:

Terracotta, version 2.2 as of 20061203-151234

line in the client logs, check if the clients show up in the Admin Console (see point 2 in Q6). If clients are showing up but the roots are not, make sure that the fully qualified name of the root is accurate (e.g. org.system.services.foo). If you still have problems, please contact our mailing list (tc-users@lists.terracotta.org).

I am getting com.tc.exception.TCNonPortableObjectError. What next?

For a detailed guide on how to configure Terracotta DSO for your application, including what to do with TCNonPortableObjectErrors, read the Configuring Terracotta DSO for Your Application document.

There can be a number of reasons to encounter this exception. This exception basically means that one (or more) of the objects in the shared object graph reference a Non-Portable object. A Non-Portable object is an object that Terracotta cannot cluster or share. An object can be non-portable in one of the following scenarios:

1. Object in question is implicitly non-portable, i.e. cannot be clustered. In other words, it doesn't make sense to share these objects across JVMs - examples include objects that hold a resource native to the OS/Hardware (File Handle, Sockets etc.). If a JVM is holding on to a physical socket on one machine, this socket does not hold any significance to another JVM on another machine, and hence it does not make sense to share it.

• Solution: Mark these fields as "transient" in the Terracotta config. A transient field is an instance variable that is referenced by a shared object, but is NOT shared by Terracotta.

There are two ways to do this. If the field in question is already marked transient in the containing class, you can turn on transient behavior for the class (note that you can only do so for the entire class. You may want to fill in the on-load section as well - it behaves similarly to the readObject() Java Serialization API).

<include>
   <class-expression>com.mycompany.pkga.MyClassOne</class-expression>
   <honor-transient>true</honor-transient>
   <!-- optional -->
   <on-load>
     <execute><![CDATA[...]]></execute>
   </on-load>  
</include>

If the field in question is not already marked transient, or you need more fine grained control for the fields in a particular class, you can name individual fields to be treated as transient:

<transient-fields>
  <field-name>com.mycompany.pkga.MyClassOne.fieldA</field-name>
  <field-name>com.mycompany.pkgb.subpkg2.fieldB</field-name>
</transient-fields>

Tip Marking a field transient can lead to a NullPointerException in your clustered application. Let's look at the following scenario: An object, foo, is a shared object. Foo references an instance of java.io.File called file. You decide to mark file as transient. Now, in a clustered environment, when foo materializes on multiple JVMs, since file is not shared it is going to be null unless there is code to initialize file on each shared JVM. This can be done in the constructor of foo or by using the Bean Shell script in the Terracotta config.

Common Declared Transients Here are some common classes whose instances, if referenced by shared objects, will most likely need to be declared transient: Loggers, File Handlers, Sockets, java.io., java.net., java.nio., java.sql.Connection, javax.net., javax.print.*, java.lang.reflect, java.lang.Thread, java.lang.Runtime.

See the Concept and Architecture Guide for details on portability and transience.

2. The class that implements the field has not been instrumented.

• Solution: Include the class in the instrumentation section of the tc-config.xml file, as in the following example.

<instrumented-classes>
  <include>
    <class-expression>com.mycompany.pkga.*</class-expression>
  </include>
</instrumented-classes>

3. The class that implements the field has not been instrumented and requires to be in the boot-jar.

• Solution: Add the class to the <additional-boot-jar-classes> section of the configuration file and regenerate the boot-jar (see Q9).

Please note that in some cases, Terracotta needs to add explicit support for a core Java class - simply adding to the boot-jar would not work and you should escalate to Terracotta Support.

There is a list of [Unsupported Classes] published.

I am getting com.tc.object.tx.UnlockedSharedObjectException. What next?

For a detailed guide on how to configure Terracotta DSO for your application, including what to do with UnlockedSharedObjectExceptions, read the Configuring Terracotta DSO for Your Application document.

This exception means that some shared object is being mutated or modified without being protected. This ensures that the cluster state is always coherent. In Terracotta, data integrity and cluster coherency is maintained by ensuring atomic operations on shared objects. This is achieved through plain and natural Java concurrency constructs - every mutation to a shared object must be done within a synchronized block of code. This synchronized block of code is the Terracotta transaction boundary. In coding terms, the following two steps will fix this exception:

1. Simply define a synchronization boundary around the said object mutation. e.g. If the following code block is the culprit for the exception (foo is the object being mutated in the method mutate),

public void mutate(Object foo) {
    foo.setProperty(property);
}

just add the following:

public void mutate(Object foo) {
    synchronized(foo) {
        foo.setProperty(property);
    }  
}

2. The next step is to guide Terracotta to use an appropriate lock for the above mutate transaction by adding the following in the locks section of the Terracotta Config file:

<locks>
  <autolock>
    <method-expression>void com.system.services.MutateService.mutate(Object)</method-expression>
    <lock-level>write</lock-level>
  </autolock>
</locks>

Locks are declared at the method level. Make sure that you use write locks for write operations (e.g. set()) and read locks for read operations (e.g. get()) for better performance.

I get com.tc.object.tx.UnlockedSharedObjectException on a third party jar and do not have access to code. Is Terracotta any good for me?

Use Sparingly There can be a performance penalty to using named locks, so it highly recommended that you use named locks very sparingly and only where absolutely necessary.

Yes. You can use "named" locks instead of autolocks demonstrated in the previous query. Named locks will create one global lock across the cluster for the named lock. This can cause contention, and can, in some cases, decrease throughput. When using named locks, you do not need to use the synchronize keyword to isolate the transaction (step 1 in the previous query is not needed) because the method becomes the point of synchronization, and the name of the method becomes the name of the lock. Due to there only being one named lock per method, ALL object instances will serialize access through the single named lock.

Here is an example:

<locks>  
  <named-lock>
    <lock-name>mutateFoo</lock-name>
    <method-expression>void com.system.services.MutateService.mutate(Object)</method-expression>
    <lock-level>write</lock-level>
  </named-lock>
</locks>

See the Concept and Architecture Guide for more information about locks.
See the Configuration Guide and Reference for more information on how to configure locks.

I am getting java.io.IOException: No locks available, what next?

This is a problem with NFS lock mgr.

• Solution: modify tc-config.xml to use different log files

I am having trouble getting failover to work properly with 2 nodes. Especially after pulling the ethernet cable (plug). What is wrong?

Please see the High Availability With 2 Nodes document.

I am getting a COMMUNICATION ERROR when configuring Network Active/Passive on Mac OS X.

This is a known issue with the Tribes implementation on Mac OS X. The error that is thrown is here:

NodeID : NodeID[tcp://10.0.0.144:9530] Error Type : COMMUNICATION ERROR Details 
: Error publishing states to NodeID[tcp://10.0.0.144:9530] : Exception : 
com.tc.net.groups.GroupException: org.apache.catalina.tribes.ChannelException: Send failed, attempt:2 max:1; 
Faulty members:tcp://10.0.0.144:9530; 
        at com.tc.net.groups.TribesGroupManager.sendTo(TribesGroupManager.java:468) 
        at com.tc.l2.state.StateManagerImpl.publishActiveState(StateManagerImpl.java:269) 
        at com.tc.l2.ha.L2HACoordinator.nodeJoined(L2HACoordinator.java:256) 
        at com.tc.l2.handler.GroupEventsDispatchHandler.handleEvent(GroupEventsDispatchHandler.java:24) 
        at com.tc.async.impl.StageImpl$WorkerThread.run(StageImpl.java:142) 
Caused by: org.apache.catalina.tribes.ChannelException: Send failed, attempt:2 max:1; Faulty members:tcp://10.0.0.144:9530; 
        at org.apache.catalina.tribes.transport.nio.ParallelNioSender.doLoop(ParallelNioSender.java:172) 
        at org.apache.catalina.tribes.transport.nio.ParallelNioSender.sendMessage(ParallelNioSender.java:78) 
        at org.apache.catalina.tribes.transport.nio.PooledParallelSender.sendMessage(PooledParallelSender.java:48) 
        at org.apache.catalina.tribes.transport.ReplicationTransmitter.sendMessage(ReplicationTransmitter.java:80) 
        at org.apache.catalina.tribes.group.ChannelCoordinator.sendMessage(ChannelCoordinator.java:78) 
        at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:75) 
        at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:75) 
        at org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.sendMessage(TcpFailureDetector.java:87) 
        at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:75) 
        at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:75) 
        at org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:216) 
        at org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:175) 
        at com.tc.net.groups.TribesGroupManager.sendTo(TribesGroupManager.java:466) 
        ... 4 more 
Caused by: java.net.SocketException: Invalid argument 
        at sun.nio.ch.Net.setIntOption0(Native Method) 
        at sun.nio.ch.Net.setIntOption(Net.java:152) 
        at sun.nio.ch.SocketChannelImpl$1.setInt(SocketChannelImpl.java:372) 
        at sun.nio.ch.SocketOptsImpl.setInt(SocketOptsImpl.java:46) 
        at sun.nio.ch.SocketOptsImpl$IP.typeOfService(SocketOptsImpl.java:249) 
        at sun.nio.ch.OptionAdaptor.setTrafficClass(OptionAdaptor.java:158) 
        at sun.nio.ch.SocketAdaptor.setTrafficClass(SocketAdaptor.java:330) 
        at org.apache.catalina.tribes.transport.nio.NioSender.completeConnect(NioSender.java:147) 
        at org.apache.catalina.tribes.transport.nio.NioSender.process(NioSender.java:89) 
        at org.apache.catalina.tribes.transport.nio.ParallelNioSender.doLoop(ParallelNioSender.java:130) 
        ... 16 more

To fix this issue, run your Terracotta Server process with the following Java option (note: you will have to modify the Terracotta Start Server script):

-Djava.net.preferIPv4Stack=true

Terracotta for Spring...

How logical identity for the Spring application context is maintained?

Terracotta for Spring creates DSO root for each Spring application context. Because Spring framework
does not provide unique Id for the application context, matching patterns are used to select web
application name and to specify matching config names to identify application contexts
that need to be clustered.

Name of the DSO root is calculated based on the web application name and names of all config
resources used to initialize application context. If any name of the Spring configs changes or
if new configs been added, the generated id will be different.
See the following method for more details com.tcspring.DistributableBeanFactoryMixin.addLocation(String location)

When different configs are used but contexts need to have the same identity, the root name can be
assigned manually using the following configuration element:

<application>
   <spring>
     <jee-application>
       <application-contexts>
         <application-context>
           <root-name>myClusteredSpringContext</root-name>
           ...

Spring application context is not distributed

Terracotta console or a web server logs shows that Spring application context is not distributed:

INFO [com.tcspring.ApplicationHelper] - Application name Tomcat.shared
INFO [com.tcspring.DistributableBeanFactoryMixin] - 09080A0C0905030C0E05070F0805050A Context is NOT distributed

To debug that you can enable DEBUG logging for org.springframework and or.tcspring categories using either
log4j settings for your web application, or if running on Tomcat global logging settings at <tomcat home>\conf\logging.properties

Then you can check the actual names for the web application and config resources. For example:

2007-12-27 18:33:43,507 INFO [com.tcspring.ApplicationHelper] - Application name Tomcat.context:/foo
 2007-12-27 18:33:43,804 INFO [org.springframework.beans.factory.xml.XmlBeanDefinitionReader] - 
Loading XML bean definitions from ServletContext resource [/WEB-INF/spring/servlet.xml]
 2007-12-27 18:33:45,710 INFO [org.springframework.beans.factory.xml.XmlBeanDefinitionReader] - 
Loading XML bean definitions from ServletContext resource [/WEB-INF/spring/dao.xml]
 2007-12-27 18:33:45,804 INFO [org.springframework.beans.factory.xml.XmlBeanDefinitionReader] - 
Loading XML bean definitions from ServletContext resource [/WEB-INF/spring/services.xml]
 2007-12-27 18:33:46,273 INFO [org.springframework.beans.factory.xml.XmlBeanDefinitionReader] - 
Loading XML bean definitions from ServletContext resource [/WEB-INF/spring/datasource.xml]
 2007-12-27 18:33:46,335 INFO [org.springframework.beans.factory.xml.XmlBeanDefinitionReader] - 
Loading XML bean definitions from ServletContext resource [/WEB-INF/spring/cache.xml]
 2007-12-27 18:33:47,257 INFO [org.springframework.beans.factory.xml.XmlBeanDefinitionReader] - 
Loading XML bean definitions from ServletContext resource [/WEB-INF/spring/captcha.xml]
 2007-12-27 18:33:47,320 INFO [org.springframework.beans.factory.xml.XmlBeanDefinitionReader] - 
Loading XML bean definitions from ServletContext resource [/WEB-INF/spring/validators.xml]
 2007-12-27 18:33:47,351 INFO [org.springframework.beans.factory.xml.XmlBeanDefinitionReader] - 
Loading XML bean definitions from ServletContext resource [/WEB-INF/spring/storedprocs.xml]
 2007-12-27 18:33:47,648 INFO [org.springframework.beans.factory.xml.XmlBeanDefinitionReader] - 
Loading XML bean definitions from ServletContext resource [/WEB-INF/spring/pagers.xml]
 2007-12-27 18:33:47,695 INFO [com.tcspring.DistributableBeanFactoryMixin] - 09080A0C0905030C0E05070F0805050A Matching 
locations:[/WEB-INF/spring/servlet.xml, /WEB-INF/spring/dao.xml, /WEB-INF/spring/services.xml, 
/WEB-INF/spring/datasource.xml, /WEB-INF/spring/cache.xml, /WEB-INF/spring/captcha.xml, WEB-INF/spring/validators.xml, 
/WEB-INF/spring/storedprocs.xml, /WEB-INF/spring/pagers.xml]
 2007-12-27 18:33:47,695 INFO [com.tcspring.DistributableBeanFactoryMixin] 
- 09080A0C0905030C0E05070F0805050A Context is distributed
 2007-12-27 18:33:47,695 INFO [com.tcspring.ClassHierarchyWalker] - 09080A0C0905030C0E05070F0805050A 
registering include for com.planetpope.beans.SystemStatus
 2007-12-27 18:33:47,695 INFO [com.tcspring.DistributableBeanFactoryMixin] - 09080A0C0905030C0E05070F0805050A 
registering transient fields for systemStatus com.planetpope.beans.SystemStatus

In the above example application name is "foo" and corresponding configuration would look like this:

<application>
    <spring>
      <jee-application name="foo">
        <application-contexts>
          <application-context>
            <paths>
              <path>*/spring/servlet.xml
              <path>*/spring/dao.xml</path>
              <path>*/spring/services.xml</path>
              <path>*/spring/datasource.xml</path>
              <path>*/spring/cache.xml</path>
              <path>*/spring/captcha.xml</path>
              <path>*/spring/validators.xml</path>
              <path>*/spring/storedprocs.xml</path>
              <path>*/spring/pagers.xml</path>
            </paths>
            ...

When context is distributed Terracotta admin console will also show special spring_info root for each clustered
context where you could see the web application name, matching config names and also clustered Spring bean names.
For example:

tc:spring_info:05000C00060B090F090D040B0D0D0501 (java.util.ArrayList) [4/4]
   0 (com.tcspring.ApplicationContextEventProtocol)
       com.tcspring.ApplicationContextEventProtocol.appName (java.lang.String)=jmx
       com.tcspring.ApplicationContextEventProtocol.beanNames (java.util.ArrayList) [2/2]
         0 (java.lang.String)=clusteredHistory
         0 (java.lang.String)=clusteredCounder
       com.tcspring.ApplicationContextEventProtocol.configs (java.util.ArrayList) [1/1]
         0 (java.lang.String)=demo/jmx/applicationContext.xml
       com.tcspring.ApplicationContextEventProtocol.eventTypes (java.util.ArrayList) [0/0]

Share data between standalone application and web application

This is not supported out of the box, but you could use an application wrapper to make
classloader names the same between standalone application and the web app.

TODO

Best Practices...

Is there a faster way to declare configurations?

There is a very useful [Eclipse plugin] shipped with Terracotta that makes the task of creating the Terracotta configuration extremely seamless. Also, take full advantage of the AspectWerkz pattern language. e.g. To define read locks around all getters in a Class:

<locks>
  <autolock>
    <method-expression>* com.system.services.MutateService.get*(..)</method-expression>
    <lock-level>read</lock-level>
  </autolock>
</locks>

My code uses the singleton pattern. I get exceptions on restart.

In such a case please make sure that the root is the field in the Singleton and not a reference to the Singleton elsewhere in your code. Static fields are not replicated by Terracotta unless you explicitly mark them as roots (given that static fields belong to a "class" and not an "object" instance).

How do I...

How do I use my JRE instead of the one shipped with Terracotta?

Just point the TC_JAVA_HOME environment variable to a supported JDK or JRE. You should point it to the top level installation directory of the JDK i.e. the directory that contains the bin directory.

How do I know if my POJOs are being shared or not?

• The starting point would be that the client logs should display a line similar to this:
  Log Output

  Terracotta, version 2.2 as of 20061203-151234

• A quick way to verify is to monitor your cluster using the Terracotta Admin Console (TC_INSTALL_DIR/dso/bin/tc-admin.bat(sh). In the frame on the left, you will see a tree display with one of the nodes being clients and another one being roots. When you bring up the client JVMs, your should see the IP of your client JVMs under clients and the root under roots.

How do I reduce planned downtime with Terracotta?

Terracotta persists objects in a new format (field name/value pairings), that does not use Java serialization. It is decoupled from specifics of class versioning. For example - adding/deleting a field to a class will require no flush of Terracotta caches. Additional use-cases in the table below.

How do I use TC Sessions with Tomcat?

It is basically three steps:

• Install Terracotta on all nodes. Set the TC_INSTALL_DIR environment variable to your installation directory and make sure that JAVA_HOME is set to the appriprate JDK/JRE that is being used for your application.

1. Declare your web application (war) name in the configuration xml file:

<web-applications>
  <web-application>your_web_app1</web-application>
  <web-application>your_web_app2</web-application>
</web-applications>

2. Start the terracotta server on hostname:

$TC_INSTALL_DIR/sessions/bin/start-tc-server.sh -f path/to/config/xml/file

3. Start the clients (Tomcat instances) using the follwing script:

TERRACOTTA_SERVER=localhost
TC_CONFIG_PATH="${TERRACOTTA_SERVER}:9510"

. $TC_HOME/bin/dso-env.sh -q

export JAVA_OPTS="${TC_JAVA_OPTS}"
$CATALINA_HOME/bin/startup.sh

setlocal

set TERRACOTTA_SERVER=localhost
set TC_CONFIG_PATH=%TERRACOTTA_SERVER%:9510 -q

call "%TC_INSTALL_DIR%\bin\dso-env.bat"

set JAVA_OPTS=%TC_JAVA_OPTS%
call %CATALINA_HOME%\bin\startup.bat

endlocal

• Replace the localhost in TERRACOTTA_SERVER=localhost with the appropriate ip or hostname where the terracottaserver is running in step 2 (for production mode) or set the TC_CONFIG_PATH with the physical path to the terracotta config file used to start the server in step 2 (for development mode).

• If it complains about the dso-boot-jar, create one (see Q3).

• Alternatively, you can use the configurator (see Q15) that ships with the sessions product.

How do I use TC Sessions with JBoss Application Server?

It is basically three steps:

• Install Terracotta on all nodes. Set the TC_INSTALL_DIR environment variable to your installation directory and make sure that JAVA_HOME is set to the appriprate JDK/JRE that is being used for your application.

1. Declare your web application (war) name in the configuration xml file:

<web-applications>
  <web-application>your_web_app1</web-application>
  <web-application>your_web_app2</web-application>
</web-applications>

2. Start the terracotta server on hostname:

$TC_INSTALL_DIR/sessions/bin/start-tc-server.sh -f path/to/config/xml/file

3. Start the clients (JBoss instances) using the follwing script:

TERRACOTTA_SERVER=localhost
TC_CONFIG_PATH="${TERRACOTTA_SERVER}:9510"

. $TC_HOME/bin/dso-env.sh -q

export JAVA_OPTS="${TC_JAVA_OPTS}"
$JBOSS_HOME/bin/run.sh

setlocal

set TERRACOTTA_SERVER=localhost
set TC_CONFIG_PATH=%TERRACOTTA_SERVER%:9510 -q

call "%TC_INSTALL_DIR%\bin\dso-env.bat"

set JAVA_OPTS=%TC_JAVA_OPTS%
call %JBOSS_HOME%\bin\run.bat

endlocal

• Replace the localhost in TERRACOTTA_SERVER=localhost with the appropriate ip or hostname where the terracottaserver is running in step 2 (for production mode) or set the TC_CONFIG_PATH with the physical path to the terracotta config file used to start the server in step 2 (for development mode).

• If it complains about the dso-boot-jar, create one (see Q3).

• Alternatively, you can use the configurator (see Q15) that ships with the sessions product.

How do I use TC Sessions with WebLogic Application Server?

It is basically three steps:

• Install Terracotta on all nodes. Set the TC_INSTALL_DIR environment variable to your installation directory and make sure that JAVA_HOME is set to the appriprate JDK/JRE that is being used for your application.

1. Declare your web application (war) name in the configuration xml file:

<web-applications>
  <web-application>your_web_app1</web-application>
  <web-application>your_web_app2</web-application>
</web-applications>

2. Start the terracotta server on hostname:

$TC_INSTALL_DIR/sessions/bin/start-tc-server.sh -f path/to/config/xml/file

3. Start the clients (Weblogic instances) using the follwing script:

TERRACOTTA_SERVER=localhost
TC_CONFIG_PATH="${TERRACOTTA_SERVER}:9510"

. $TC_HOME/bin/dso-env.sh -q

export JAVA_OPTS="${TC_JAVA_OPTS}"
export WLS_USER=admin
export WLS_PW=admin
export PRODUCT_MODE=false

${BEA_HOME}/weblogic81/server/bin/startWLS.sh

setlocal

set TERRACOTTA_SERVER=localhost
set TC_CONFIG_PATH=%TERRACOTTA_SERVER%:9510 -q

call "%TC_INSTALL_DIR%\bin\dso-env.bat"

set JAVA_OPTS=%TC_JAVA_OPTS%
set WLS_USER=admin
set WLS_PW=admin
set PRODUCT_MODE=false

call %BEA_HOME%\weblogic81\server\bin\startWLS.cmd

endlocal

• WLS_USER, WLS_PW and PRODUCT_MODE environment variables in the above script are optional and may vary. You can start your individual domain instance instead of

  ${BEA_HOME}/weblogic81/server/bin/startWLS.sh

  or

  %BEA_HOME%\weblogic81\server\bin\startWLS.cmd

• Replace the localhost in TERRACOTTA_SERVER=localhost with the appropriate ip or hostname where the terracottaserver is running in step 2 (for production mode) or set the TC_CONFIG_PATH with the physical path to the terracotta config file used to start the server in step 2 (for development mode).

• If it complains about the dso-boot-jar, create one (see Q3).

• Alternatively, you can use the configurator (see Q15) that ships with the sessions product.

How do I use the TC Sessions Configurator for any container/app server?

Terracotta Sessions Configurator is preconfigured to use a bundled Apache Tomcat web container. Other web containers are available from the File | Servers menu option. Each web container generally requires two properties for successful operation. The first is the location of the product home directory (such as CATALINA_HOME or BEA_HOME) and the second is the location of the Java Development Kit (JDK) to use when running the web container (speciffied by the TC_JAVA_HOME property).

Apache Tomcat 5.5
If CATALINA_HOME is not set it will default to the bundled version. If TC_JAVA_HOME is not set it will default to the bundled version.

Apache Tomcat 5.0:
CATALINA_HOME must be set to a valid Tomcat 5.0 installation directory. TC_JAVA_HOME must be set to a valid JDK installation directory. Note that a JDK is required for Tomcat 5.0, as opposed to a JRE.

BEA WebLogic 8.1:
BEA_HOME must be set to a valid BEA 8.1 installation directory. TC_JAVA_HOME must be set to a valid JDK 1.4 installation directory. Note that WLS 8.1 does not work with Java 5.

Web applications that you import into the Configurator are deployed to the currently selected web container. The Terracotta provided sample web applications are pre-deployed to each of the supported web containers.

Performance Tuning

This section is under development

My application runs very slowly on Terracotta. What's going on?

One of the main reasons applications run slowly on Terracotta is due to lock configuration. You must protect critical sections of your code with distributed locks (i.e., synchronization or named locks). If you hold a lock around more code than you need, then you will have high contention for that lock and reduced concurrency. If that is the case, try to minimize the scope of the lock.

If you have too much synchronization or grab more locks than you need to, you will suffer from the overhead of grabbing distributed locks over the network. If that is the case, try to synchronize only where you need to to reduce the number of times you request distributed locks.

General Questions...

For an exhaustive list, check the Frequently Asked Questions link.

Is my platform supported?

Check here.

How do I use Terracotta to cluster my POJOs? Is there an API?

All the configuration in Terracotta happens through a single XML configuration file. For more information, see the [Configuration Guide and Reference].

What is a root?

Terracotta shares POJOs across multiple JVMs within the context of a root. This is the top level object that is shared and can be any POJO. Any references made by the root are automatically shared. Typically, the root is an instance of a collection (e.g. a Map). All POJOs held in this collection and the references made by the POJOs and so on (the entire object graph) are automatically shared.

See the Concept and Architecture Guide for more information on roots and clustered objects

How can I deploy terracotta in my clustered environment to cluster my plain java application, my application server, servlet container?

Version 2.3 and higher This answer assumes that the Terracotta version being used is 2.3 and higher.

1. Install Terracotta (on all nodes). Set the TC_INSTALL_DIR environment variable to your installation directory and make sure that JAVA_HOME is set to the appriprate JDK/JRE that is being used for your application.

2. Create the boot-jar for your environment by running the make-boot-jar script. Make sure that the JAVA_HOME environment variable is pointing to the correct JDK/JRE.

$TERRACOTTA_INSTALL_DIR/bin/make-boot-jar.sh

%TERRACOTTA_INSTALL_DIR%\bin\make-boot-jar.bat

3. Create a Terracotta configuration (tc-config.xml).

• You can use the tc-config-*.xml samples located in TC_INSTALL_DIR\config-examples directory as a good starting point. Here is a good documentation of different properies in the tc-config.xml: Configuration Guide and Reference.

• You can also refer to the tc-config-reference.xml located in the docs directory of the Terracotta installation.

Make sure that the server host property is set to the correct hostname/ip if you are running the Terracotta Server on a seprate box than your application.

<servers>
    <!-- Tell DSO where the Terracotta server can be found. -->
    <server host="localhost">
      <data>%(user.home)/terracotta/server-data</data>
      <logs>%(user.home)/terracotta/server-logs</logs>
    </server>
</servers>

<servers>
    <!-- Tell DSO where the Terracotta server can be found. -->
    <server host="host1">
      <data>%(user.home)/terracotta/server-data</data>
      <logs>%(user.home)/terracotta/server-logs</logs>
    </server>
</servers>

You can also use the %h wildcard which always uses the the local hostname of the machine on which the server started:

<servers>
    <!-- Tell DSO where the Terracotta server can be found. -->
    <server host="%h">
      <data>%(user.home)/terracotta/server-data</data>
      <logs>%(user.home)/terracotta/server-logs</logs>
    </server>
</servers>

4. Start the Terracotta Server on some machine (e.g. host1) passing the configuration created in step 3 like this:

$TC_INSTALL_DIR/bin/start-tc-server.sh -f tc-config.xml

%TC_INSTALL_DIR%\bin\start-tc-server.bat -f tc-config.xml

5. Set the environment/system properties and java options necessary to integrate Terracotta in your application. You can use the dso-env script that will set the necessary environment variables/java options for you (in the TC_JAVA_OPTS variable):

TERRACOTTA_SERVER=localhost
TC_CONFIG_PATH="${TERRACOTTA_SERVER}:9510"
. $TC_HOME/bin/dso-env.sh -q
export JAVA_OPTS="${TC_JAVA_OPTS}"
java $JAVA_OPTS MainClass

** Replace localhost above with the hostname/ip of the machine
** where the Terracotta Server is started (Step 4).

setlocal
set TERRACOTTA_SERVER_HOST=localhost
set TC_CONFIG_PATH=%TERRACOTTA_SERVER_HOST%:9510
call "%TC_INSTALL_DIR%\bin\dso-env.bat" -q
set JAVA_OPTS=%TC_JAVA_OPTS%
java %JAVA_OPTS% MainClass
endlocal

** Replace localhost above with the hostname/ip of the machine 
** where the Terracotta Server is started (Step 4).

TERRACOTTA_SERVER=localhost
TC_CONFIG_PATH="${TERRACOTTA_SERVER}:9510"
. $TC_HOME/bin/dso-env.sh -q
export JAVA_OPTS="${TC_JAVA_OPTS}"
startup.sh

** Replace localhost above with the hostname/ip of the machine where the 
** Terracotta Server is started (Step 4).

setlocal
set TERRACOTTA_SERVER_HOST=localhost
set TC_CONFIG_PATH=%TERRACOTTA_SERVER_HOST%:9510
call "%TC_INSTALL_DIR%\bin\dso-env.bat" -q
set JAVA_OPTS=%TC_JAVA_OPTS%
call startup.bat
endlocal

** Replace localhost above with the hostname/ip of the machine 
** where the Terracotta Server is started (Step 4).

That,s it.

Extra Information for the curious:

Alternatively, if you are someone who likes to know what is being set by the dso-env script to Terracotize your application, you can set it up manually also.

You will need to add the following three environment variables:

i. -Xbootclasspath/p:"%TC_HOME%\lib\dso-boot\dso-boot-hotspot_win32_150_08.jar" 

** Replace the name of the file above (dso-boot-hotspot_win32_150_08.jar)
**  with the file created in step 2.

** Make sure that the boot-jar file name reflects the same
** JDK/JRE major_minor version as the JDK/JRE you are 
** going to use to run the application (JAVA_HOME or in your PATH). 
** You can verify your java version by running  'java -version' on the command line.

ii. -Dtc.install-root="%TC_HOME%" 

** Assuming TC_HOME is set to your installation path **

iii. -Dtc.config="localhost:9510"

** Replace localhost with the hostname/ip of the machine where the 
** Terracotta Server was started in step 4 e.g. host1 or localhost if
** running on the same machine.

So, if your java invocation looks like this:

java -cp c:\lib\somejar.jar com.mypackage.MainClass

And it should look similar to this with Terracotta:

java -Xbootclasspath/p:"%TC_HOME%\lib\dso-boot\dso-boot-hotspot_win32_150_08.jar"
-Dtc.install-root="%TC_HOME%" 
-Dtc.config="localhost:9510" 
-cp c:\lib\somejar.jarcom.mypackage.MainClass

For your AppServer, you can just add the above java options/system properties to the JAVA_OPTS variable in the AppServer start script:

set JAVA_OPTS=-Xbootclasspath/p:"%TC_HOME%\lib\dso-boot\dso-boot-hotspot_win32_150_08.jar"
-Dtc.install-root="%TC_HOME%" 
-Dtc.config="localhost:9510";%JAVA_OPTS%

export JAVA_OPTS="-Xbootclasspath/p:$TC_HOME/lib/dso-boot/dso-boot-hotspot_win32_150_08.jar 
-Dtc.install-root=$TC_HOME -Dtc.config=localhost:9510":$JAVA_OPTS

TODO ConcurrentModificationException during dehydrate Hashtable and Vector aren't autolocked Null pointer exceptions and transience bean-shell on-load methods

How do I improve the performance of my application which is using clustered EHCache with Terracotta?

Add the following property in the tc.properties file under $TC_INSTALL_DIR/lib (just create one if it doesn't exist):

ehcache.concurrency = 32

This property sets the concurrency (number of striped locks per cache instance) vis-a-vis java.util.concurrent.ConcurrentHashMap. Depending on your application, you will want to play around with this number. E.g. If you have just one cache instance with 128 threads, you will want to use a big number like 64 (or even 128). But if you have hundreds of caches, you have to be careful as too high a number (200 (caches) * 200 (locks) = 40000 ) can add more overhead in terms of memory and lock management. 32 or 64 should be good starting point for most applications. The default value is 1.

How do I know if EHCache data is being shared? I can see a bunch of EHCache objects in the admin console but cannot find my application data.

First thing to look for is the terracotta root objects. Connect the TC Admin Console ($TC_INSTALL_DIR/bin/admin.sh) to the TC Server instance and look under the Roots. If you see the following roots, EHCache (Hibernate 2nd level cache) is being clustered properly:

net.sf.ehcache.CacheManager.ALL_CACHE_MANAGERS (java.util.Collections$SynchronizedRandomAccessList)
net.sf.ehcache.store.MemoryStoreEvictionPolicy.LRU (net.sf.ehcache.store.MemoryStoreEvictionPolicy)
net.sf.ehcache.store.MemoryStoreEvictionPolicy.LFU (net.sf.ehcache.store.MemoryStoreEvictionPolicy)
net.sf.ehcache.store.MemoryStoreEvictionPolicy.FIFO (net.sf.ehcache.store.MemoryStoreEvictionPolicy)
net.sf.ehcache.store.MemoryStoreEvictionPolicy.DSO (net.sf.ehcache.store.MemoryStoreEvictionPolicy)
net.sf.ehcache.CacheManager.singleton (net.sf.ehcache.CacheManager)

EHCache breaks down the key-value pair and stores the value object wrapped by Element objects in a fairly deep object graph, as it does a lot of bookkeeping with timestamps etc. I am breaking down below exactly where you should be looking for the actual application data stored in the cache:

net.sf.ehcache.CacheManager.ALL_CACHE_MANAGERS (java.util.Collections$SynchronizedRandomAccessList)
|
|_____ java.util.Collections$SynchronizedCollection.c (java.util.ArrayList) [1/1]
	   |
	   |_____ 0 (net.sf.ehcache.CacheManager)
	         |
	         |_____ net.sf.ehcache.CacheManager.caches (java.util.HashMap) [3/3]
	               |
	               |_____ 0 (MapEntry)
	                     |
	                     |_____ value (net.sf.ehcache.Cache)
	                           |
	                           |_____ net.sf.ehcache.Cache.memoryStore (net.sf.ehcache.store.TimeExpiryMemoryStore)
(continued)
|
|_____ net.sf.ehcache.store.MemoryStore.map (net.sf.ehcache.store.TimeExpiryMemoryStore$SpoolingTimeExpiryMap)
       |
       |_____ com.tcclient.ehcache.TimeExpiryMap.timeExpiryDataStore (com.tcclient.cache.CacheDataStore)
              |
              |_____ com.tcclient.cache.CacheDataStore.store (java.util.Map[])
                     |
                     |_____ 0 (java.util.HashMap) [1/1]
                            |
                            |_____ 0 (MapEntry)
                                  |
                                  |_____ key=XML_0K_1551
                                         value (com.tcclient.cache.CacheData)
                                         |
                                         |_____ com.tcclient.cache.CacheData.value (net.sf.ehcache.Element)
                                                |
	                                        |_____ net.sf.ehcache.Element.value (Value object type)
	                                               |
                                                       |_____ The value object put in the cache by the app

Unable to locate bundle dependency: I get the following exception which indicates one or more Terracotta modules or dependencies are missing. How do I get past this scenario?

Missing Terracotta Module 2008-01-18 09:47:55,240 FATAL - Unable to locate bundle dependency: 'jdk15_preinst_config', version '2.5.0', group-id 'org.terracotta.modules'; Tried to resolve the location using the following repositories: 'file:/www/terracotta/terracotta-2.5.0/bin/../modules/'

On Solaris platform:

This is a result of the way the default tar utility on Solaris works. Terracotta loads a few modules located in $TC_INSTALL_DIR/modules directory on startup and if you look in that directory, you should see a collection of .jar files. The default tar utility causes some of these file names to be truncated which results in the above scenario as Terracotta is unable to load these modules.

To get past this, use an alternative tar utility like gtar or download the patch from Sun which fixes this issue.

Links The patch for Solaris tar is Solaris 2.6 patch 105792-03. The SunSoft patch page is at: http://sunsolve.sun.com/pub-cgi/retrieve.pl?doc=fpatches/105792 You can download the Solaris version of GNUtar from: http://www.ibiblio.org/pub/packages/solaris/sparc/

More generic solution:

If it is not an unpacking issue and the kit is installed properly, check the following:

1. The user has read access to all files in the Terracotta installation (specifically the modules directory).
2. Make sure that the module specified in the tc-config.xml is actually located and read-accessible in the appropriate repository (default is $TC_INSTALL_DIR/modules).
3. Make sure that the module version number is specified appropriately in the tc-config.xml and there are no typos.