TDIing out loud, ok SDIing as well

Ramblings on the paradigm-shift that is TDI.

Saturday, February 6, 2010

What Is The Solution Directory?

The Solution Directory is the current working directory for an  SDI/TDI Server.

Each TDI Server running on the same machine should have it's own Solution Directory (or SolDir for short). If you don't know where your SolDir is pointing, simply open defaultSolDir.sh (or .bat) found in the bin sub-folder of your installation directory (e.g. V7.2). This script sets the TDI_SOLDIR environment variable that controls Server behavior.

     export/set TDI_SOLDIR="some/place/on/your/disk"

Both the ibmditk script to launch the CE, and ibmdisrv that starts a Server call defaultSolDir, along with other scripts in this folder. Like javaHome.

The Solution Directory holds configuration settings, including Java settings, for a Server. These settings are found in a file named solution.properties.

After reading solution.properties file, the Server then reads the global.properties file in the etc sub-folder of the TDI installation directory. Settings in solution.properties override those in global.properties.

If you open solution.properties in a text editor you will find two important properties that are personal to any running TDI Server:
  • The port that the Server listens to for RMI API calls: api.remote.naming.port
  • The port that the Server listens to web calls: web.server.port (dashboard, FDS, CURI and REST APIs)
This is also where you provide the paths to keystores and truststores, as well as credentials to access these.

By default, the one for the TDI Server's APIs is named testserver.jks and is found in the SolDir. Certificates used for SSL connections in your solutions are under the serverapi sub-folder and are named testadmin.jks.

NOTE: The default password for testserver.jks is server, and for testadmin.jks is administrator. To import your own certificates use keytool (found under the jvm/jre/bin sub-folder of your TDI installation directory) or the Key Manager button in the TDI CE (Configuration Editor).

NOTE II: If you remove all properties in solution.properties that you are not explicitly setting, then a TDI version upgrade might change global.properties and could then affect settings for your Server.

Leveraging the Solution Properties folder to make your solutions more portable

As mentioned above, the SolDir is the current working directory for TDI, both the CE and the Server. Any Config xml files placed in the configs sub-folder of the SolDir will be automatically loaded when the Server starts up. If they have Schedulers, then these are activated as well.

Another approach is to make a Project folder in the SolDir. Solution filepaths should point in this directory using simply the name of the folder. For example, if you project is named "MyProject" then the path to the Property Store file would be:

    MyProject/MyProject.properties

The next step is to instruct the CE where to write the compiled Config xml, which is written each time you save a change. By default this file is written as:

    MyProject/Runtime-MyProject/MyProject.xml

You can change this easily to have the Config written to your Project folder. Simply right-click on the name of your Project in the CE Navigator View and select Properties.




Choose Tivoli Directory Integrator properties at the bottom and set the Linked File parameter to a relative path that points to a config file under this same folder, like MyProjext/MyProject.xml.

NOTE III: If you use forward-slashes for paths instead of backslashes, your solution runs on Windows and all flavors of *nix (including mainframe).

Make sure you also put any other support files in this folder so you can easily move the solution by simply compressing this directory and then uncompressing it in the SolDir of the target TDI installation.

The ReadMe.txt file you make in your project directory could look something like this:

To install this solution:

1) Unzip QRTrigger.zip to your Solution Directory;
2) Edit QRTrigger.properties to configure connections
3) Run the solution with this commandline:

ibmdisrv -c QRTrigger/QRTrigger.xml -r QRListener

Violá, you have something that you can hand to the operations team for deployment, pass on to a colleague, and even publish it online (and sending me the link :)

Pointing to a specific Solution Directory

Stephen Swann pointed out to me that I should mention the scripts used to start up the CE (ibmdisrv) and the one for the Server (ibmditk). Each accepts the -s argument for specifying the SolDir to use:

    ibmdisrv -s /MyOtherSolDir -c MyOtherProject/MyOtherProject.xml -d

NOTE IV: If the SolDir does not exist then the Server will set it up the first time it starts.

Your Eclipse workspace

So the SolDir is a tool for making solutions portable, and it is independent of where you put your TDI workspace. The workspace is where the Eclipse framework housing TDI organizes your source files, and the directory structure there mirrors what you see in the CE Navigator. It is not recommended that you work directly with source files, apart from tying them to SCM like git.

What if..?

But what if you chose the program directory or "current location" when you installed, or want to move the Solution Directory for some other reason? No problem. Simply edit defaultSolDir.

Changing TDI_SOLDIR is only half the job. You also have to instruct the CE where the new SolDir is when it automatically starts up your local TDI Server. This is visible in the Server view at the bottom left of the CE, and is called 'Default'. Can change the Server Launcher settingsby right-clicking on 'Default' in the Server view and selecting Properties and making the change here too.


Final note about the Servers view

The view is not called Servers for nothing. TDI lets you define new Servers, allowing you to start multiple Servers on your PC, or connect to those running on other platforms. You can point a Project at any Server by editing its settings (right-click > Properties). This lets you test and debug the ALs there. Each Server has its own Solution Directory settings, so using relative paths means you can build your ALs on your laptop and then run them elsewhere. You will also want to use forward slashes for filepaths, since backslash only works on Windows and TDI runs a lot of other places as well.

You can also right-click on a Server and select Debug Server. This opens a blank debugger tab which then catches an ALs that are started on this Server.

And while I'm on the soapbox here are some additional tips: Define Logging for where your task.logmsg() messages are written; Code Error Hooks and log enough info for manual intervention; Write status messages to the command line by using main.logmsg(); And set up Connector Auto-Reconnect (Connection Errors).

21 comments:

Anonymous said...

Interesting article you got here. I'd like to read something more about that topic. Thank you for giving this information.

Eddie Hartman said...

Anything in particular? I'm taking requests :)

Unknown said...

I would love to have more information on passwords sync! Specially MSAD passwords.

Use the force!

Eddie Hartman said...

What would you like to know? TDI Identity Edition includes password change interceptor plugins for (among other systems) AD. This plugin grabs password changes and securely sends them to an AssemblyLine you prepare for writing the credential updates to your targets.

The transport for sending the password change message is either a queue (MQe, MQ) that your 'SyncPassword' AL reads; Or by having the plugin write password change entries to a special container/branch you set up in an LDAP directory where you pick them up using a Changelog Connector. Easy as spittin' :)

More here:
http://publib.boulder.ibm.com/infocenter/tivihelp/v2r1/topic/com.ibm.IBMDI.doc_7.0/pluginsguide.htm

Anonymous said...

Good brief and this enter helped me alot in my college assignement. Gratefulness you seeking your information.

Stephen Swann said...

Eddie,
It seems to me that many enterprises will have a need to deploy TDI using automated build scripts and may have a need to create (and populate) a solution directory from the command line rather than firing up the CE. For those people in that situation, the "ibmdisrv -s {soldir} -g" is your friend.

Eddie Hartman said...

@Stephen Swann
Absolutely. Much of what you make in TDI will be left behind for others, with perhaps less TDI-savvy, to run/automate/manage. So you need all the friends you can get :)

Unknown said...

Hi Eddie. When multiple solution directories are required, what is the best approach? I understand how to set the solution directory at startup time, but how should they be organized? Is it acceptable to have a parent directory as the default solution directory, and then subdirectories that are setup as their own solution directories? Also, it seems that in TDI 7 there is a separate Derby Database created for each solution directory, is that correct? What are the ramifications of having multiple Derby databases running on a single server? I assume they would all need unique ports, is that correct? Seems like it could be complicated to shutdown and startup all dbs for server patching, etc.

Eddie Hartman said...

@Dan Rolander
You are right that it can get messy. And there should be a way to have multiple TDI Servers share the same Solution Directory so that you can use relative paths to stuff like Properties and other support files. However, I just discovered that there is a bug - which will get fixed :)

But in the meantime, your idea of having a parent folder which contains your support files (e.g. props) - or perhaps a sub-folder that is shared with these files - and then a separate sub-directory for each TDI Server's Solution Directory, so they each get their own solution.properties with port addresses. Then your solution relative paths can all be like this:

../MyProject/MyProject.properties

Etc. And you can put a single System Store here as well which all the TDI Servers share. Just make sure you run Derby in Networked Mode. This is default now for TDI 7, but you will need to configure TDI 6 according to the docs. I would definitely not run multiple Derby instances if you can avoid it. And with networked mode, then the first TDI Server that needs the SysStore will start it - the others will just connect.

Or if you are using TDI 7.1 or later then you could opt to use Solid DB instead. This is a memory DB which is considerably faster than Derby - and more robust. It can also be easily setup for HA. Or you could opt to use an RDBMS that you have running in your infrastructure already: DB2, Informix, Oracle, SQL Server, etc. The SysStore can be pointed at any JDBC compliant db server.

Hope this helps!

Unknown said...

Hi Eddie

In a TDI 7.1.x Linux environment, would you see any complications with having one ibmdisrv process running (with, -d &) and having only one TDI solution directory, and then running all required TDI configs from it's configs folder?
My view of this would be using tdisrvctl to start/stop any of multiple TDI configs involved; though I can see the ibmdi.log being overloaded with detail due to one ibmdisrv process running. Also, would this then keep the TDI message queue/store work to a minimum?
I know the ITIM Adapter starts it's service with ibmdisrv (via ITIMAd) and not using tdisrvctl, so I would think that if the ITIM Adapter is to be run from the scenario I mentioned before, then I would create a new script (using tdisrvctl) to start the ITIM_RMI.xml TDI config.
I believe the scenario I mention would also lead to the tdisrvctl command showing a complete list of loaded configs and their assembly lines.

In summary, I'm essentially trying to run a single server TDI environment (and single TDI instance) with many configs being able to run effectively.

Interested to hear your thoughts.
Thank you.

Unknown said...

Hi Eddie

In a TDI 7.1.x Linux environment, would you see any complications with having one ibmdisrv process running (with, -d &) and having only one TDI solution directory, and then running all required TDI configs from it's configs folder?
My view of this would be using tdisrvctl to start/stop any of multiple TDI configs involved; though I can see the ibmdi.log being overloaded with detail due to one ibmdisrv process running. Also, would this then keep the TDI message queue/store work to a minimum?
I know the ITIM Adapter starts it's service with ibmdisrv (via ITIMAd) and not using tdisrvctl, so I would think that if the ITIM Adapter is to be run from the scenario I mentioned before, then I would create a new script (using tdisrvctl) to start the ITIM_RMI.xml TDI config.
I believe the scenario I mention would also lead to the tdisrvctl command showing a complete list of loaded configs and their assembly lines.

In summary, I'm essentially trying to run a single server TDI environment (and single TDI instance) with many configs being able to run effectively.

Interested to hear your thoughts.
Thank you.

Eddie Hartman said...

@Brent
I would not combine a TDI instance being used by TIM with other TDI work. TIM has its own Dispatcher engine running inside the TDI Server, and I don't know enough about this guy to speak to safe co-existence.

Use a single TDI instance to power several other TDI solutions should be no problem. You can adjust the server log settings in the /etc/log4j.properties file.

Unknown said...

Hey man, I'm tryin to migrate AL from v6.1.1 to V7.1.1 but all my Als are creating the TDISysStore under V7.1.1 path and not in solution dir, i have checked solution and global porps under al folder, they have this:
com.ibm.di.store.database=TDISysStore

But not sure why they keep trying to build TDISysStore under /opt/IBM/TDI/V7.1.1

Thanks.

Unknown said...

Hey man, I'm tryin to migrate AL from v6.1.1 to V7.1.1 but all my Als are creating the TDISysStore under V7.1.1 path and not in solution dir, i have checked solution and global porps under al folder, they have this:
com.ibm.di.store.database=TDISysStore

But not sure why they keep trying to build TDISysStore under /opt/IBM/TDI/V7.1.1

Thanks.

Eddie Hartman said...

@Unknown

In TDI 7.1.1 you can right-click on the Default server and choose 'Edit system store settings'. In the resulting editor you will notice a tiny arrowhead to the right of the editor title: Server System Store. Click on this to select one of the out-of-the-box support SysStore settings. Yes, this will update your solution.properties, which you've said you have done already. However, it will allow you to compare what you set with what the system expects to see.

Hope this helps!

Unknown said...

Hey,

Sadly i have no editor installed on my laptop, i work against TDI running on AIX.

What I did is:

com.ibm.di.store.database=$soldir/TDISysStore

and that worked, the issue now is, derby.log is being created still under /opt/IBM/TDI/V7.1.1, SO I'm assuming this ne version is skipping some of the properties I have already on v6. Will keep working.

Eddie Hartman said...

@Unknown

Depending on which TDI version you have, you can define the TDI Server running on AIX in the Servers view of the CE running on your laptop. Then you can access all the server functions.

Unknown said...

Hey Eddie, It's Ivan again, These days have been TDI learning days, managed to fix everything but one issue left. You may have the solution.

On old logs I used to see this:

2014-07-03 05:35:27,393 DEBUG [AssemblyLine.AssemblyLines/SSOQ2TAM.1962308854] - [TAM_LDAP_Lookup_1] CTGDIS064I Loading Attribute Map.


So I can see logger is recognized as AssemblyLine.AssemblyLines/SSOQ2TAM.1962308854


But since migration, I see this way:

2014-07-04 01:27:42,439 DEBUG [com.ibm.di.log.FileRollerAppender.4c2cb50c-9550-41cc-94d1-ffd9d1abc144] - CTGDIS082I Load: DisplayData2.

So logger is now recognized as com.ibm.di.log.FileRollerAppender.4c2cb50c-9550-41cc-94d1-ffd9d1abc144

which is uglier, nastier, and it's pissing me off to see my logs that way.

Do you have any idea where that is coming from?
I have to add that the format of the first line is also shown in ibmdi.log after migration:

2014-07-04 02:14:42,942 INFO [AssemblyLine.AssemblyLines/SSOQ2TAM.1] - [Peek_MQ] Snoozing for 60000 ms..


But it is not shown on my custom logs:

2014-07-04 02:14:42,942 INFO [com.ibm.di.log.FileRollerAppender.4c2cb50c-9550-41cc-94d1-ffd9d1abc144] - [Peek_MQ] Snoozing for 60000 ms..


As you can see, it's the same msg but somehow it does not recognize its name.Since all files are the same, I'm assuming something is not being loaded good or TDIv7 does it different.

Eddie Hartman said...

@Unknow (aka Ivan)

Yes, Ivan, logging is different in the newer version. Have you tried playing with the pattern for the logger? I found this link: https://logging.apache.org/log4j/1.2/apidocs/org/apache/log4j/PatternLayout.html

See if this helps you get the output you want. If not, then let's move our conversation to the TDI forum, since there will be more eyes on your question and more nimble brains to reflect on an answer. The link can be found here: http://www.tdi-users.org

Ashok said...

Hi Eddie,


I have a requirement to run 2 AssemblyLines with 2 different changeLog connectors(SUn LDAP and ISDS) on the same TDI Machine. Both the change log no's are in different sequence. However i can run only one AL correctly. The issue here is, my Second AL is also picking up the previous AL's Change log no and trying to pull the information accordingly.

Can you please guide me to overcome this conflict and run both the AL's individually using it's own changelog no's.

Thanks
Ashok

Eddie Hartman said...

@Ashok I don't see how the two ALs can interfere with each other - unless you are using the same Iterator State Key for both Change Detection Connectors. Note that this parameter value must be unique for each CDC.

Apart from that, each AL is a separate thread in the JVM and it sounds like you are using two different types of CDC as well. Maybe you need to share more details on your setup. And I would post questions like this to the TDI forum instead of YouTube comments. Here is the link:

https://groups.google.com/forum/#!forum/ibm.software.network.directory-integrator