General Help Series 6 -Upgrading a Livelink System the 971 Part

Recently I was tasked with  upgrading our systems from its initial version to two times jump so 971 to 10.5.I will chronicle my experiences here. Perhaps provide a structure.I also want to quell some myths about the whole process just to make sure you can do this logically and correctly.

First create a draft plan,in my draft plan I included this.I would need at least one VM that can house or Production 971 Binaries,a clone of the database and a clone of the EFS.If one had been storing items in perhaps Archive Server one should have a playing copy of that also. An OTAS based livelink upgrade can pose its challenges if not done correctly but it is no different than a EFS if you do it wrong you can write test data into your productive store ūüôā ¬†This involves talking to other people like DBA’s and Storage folks as a heavily used LL system can have tons and tons of data.You will most likely encounter the after effects of long time use,different administrators and their styles etc.So after a few failed ¬†attempts I understood why OT says a DB5 verification is very important.The Verification basically runs single threaded and will try to pinpoint anomalies in the database.Most likely it will pinpoint content loss such as bulk imports etc without actual version files et al .In my case the database had wrong pointers,duplicate dataid’s (wow),removed KUAf Id’s(somebody had fun with a oracle tool).So after going back and forth between OT and us I just thought like this and created a program(oscript) to check important structures.I list 3,your mileage will vary but essentially I grabbed all categories ,form templates and wf map¬†structures.None of these checks happen in DB5 it just looks for existence.In hilarious cases I uncovered a category data structure mysteriously would turn up as a drawing PDF.One could argue that the users would have noticed, so this was an old category that nobody noticed.You don’t have to do this but each of those anomalies would prevent a upgrade and it is back and forth with OT support so I did it just to get some time advantage.

The second thing was I had to get all the optional modules  and our system is no different from any other system a lot of optional modules some inserting schema some not.

The third thing was our systems were customized with Oscript to provide enhanced user experience  as well as awesome home written modules that takes the tedium of long running things like category upgrades,permission pushes et al a very intelligent add on to the OI module that people can use generically to upload their data like that.So our team sat down and thought of what we could retire and what we should re-code,refactor so many of them were thrown out .Many useful ones we kept.We particularly liked the Distributed Agent framework,ours is very similar to that but if time comes and there is an appetite we would code it for that.

About the re-coding effort here’s the clunker,we were to use CSIDE .Naturally all of my team are hard core Builder aficionados and boy that was a journey.I probably think we were the first org to do something with CSIDE so naturally I gave them a lot of screenshots and dumps and OT dev was very receptive.In extremely tight time constraints we developed parallel modules in 97 1 and 10 and just just sucked them up for our SRC code maintaining thing called TFS. We also learned how to work parallel on oscript but not being a code churning house our code is not that complicated so if I was working on a module somebody else was working on another as well.

So the environment prepared was

  1. 971 VM with the same Oracle client and the same DB type of clone.
  2. I removed Prod EFS and Prod Admin server info from clone.
  3. We decided to rebuild search index as nobody had any clue whether it was good and it was eons before new search technology.
  4. A base install of LL was done and the working 971 binary copy from prod was ¬†put as an overlay.Now many starting out people do not know this technique where you can get a copy of your prod binary replete with patches et al into a playing system.BTW this is what you hear as the “Parallel Upgrade” approach .Its database connection was redefined .In very olden times when there was no differentiation between 32 and 62 bits and VM’s were not popular and cheap, it was possible to use the existing boxes and run the upgrade.This was the “Update an Instance” method.This is largely vestigial at this time and very error prone not to say you would have a real hard time going back to the working version ūüôā
  5. All old 971 custom modules were removed so this database had only knowledge of OT software core and optional.
  6. Did my health check I said earlier like look at cats,form templates and wf maps,removed any that would hamper a upgrade.

 

At this point one can take this to a CS10.5 binary and start the upgrade.That is my next article.

 

BTW- I am not a novice when it comes to Upgrades I started using livelink software in 1999 and has worked in version 8.1.5 so I pretty much know the heartbeat of an upgrade and gets pleasure in challenges and how one methodically removes the challenges.Do not try an upgrade if you have not dry run the procedure without fail a couple of times.In most cases the upgrade can be done timely the planning can take weeks,if not months.

 

General Help Series 5 – Form Templates,Web Forms

Forms

Forms as the name implies is a object where you can capture input and store them in livelink  . The exact methods are only going to serve as what you design your forms to be.I wrote this because a user was asking about this in the KB

Main Attributes: Forms can be made to work only by installing the “Forms Module”,which in turn will give you a Form Template sub type(230). This serves as the straw man for your data requirements.

Installation of WEB FORMS will give you HTML form capability like wise if you install PDF Forms it will give you that capability.

Forms is used extensively in delivering a awesome user experience,however the form template code is very busy and in efficient as it will add unwanted(literally thousands of lines of javascript ) so anything you can do to optimize javascript you can do that. In the course of your work you will create a Form Template,create HTML Views so that user experience can be controlled via javascript logic,thousands and thousands of examples full forms etc in the KB.

In a article where I put earlier where we all were active in communities at one pointhttps://appukili.wordpress.com/2013/08/22/important-free-developer-links/

I find that a OT moderator putting my exact lines as I replied in a post

appu_form

Little has changed although the product from Patrick Vitali called “Answer Modules”¬†has out rightly taken of the tedium of developing in forms.

Every Which way a Form can be used“https://knowledge.opentext.com/knowledge/cs.dll?func=ll&objId=3501264&objAction=ArticleView&viewType=1”

Almost all organizations will need ¬†data in forms to be reported so use SQL as storage.If you¬†keep it at default it could go as version files which is not parsable by a SQL tool nor will it have header info(you won’t know which field is which). Users using the Transmittal product usually get bitten by the fact that most of the work in transmittals revolve around forms and if you don’t make it SQL storage you will never be able to report on it.Actually on my first project installing transmittals I bungled and then I looked under the covers to understand it.Even now there is no mention in any guide saying change forms to SQL storage.A recent help to a¬†assoc

You can very greatly experience a users’ livelink workflow experience by chnaging the process to use FORMS while maintaining the business logic of the workflow

Advanced Topics in Forms/Workflows

This I put here  and only if you know Oscript is this of any useadvanced Topics in Forms workflows

 

 

General Help Series 4- Workflow Maps/Instances

Livelink Workflows for the impatient

Livelink comes up with a extremely powerful Workflow Engine¬†.At the minimum if you have only installed Livelink OOB you get a decent workflow features.This involves a “business user” to design a logical map that may involve “players” or “roles”.The only criterion is the process needs to be repeatable or it can be plotted in visio or such like.Note Livelink WF came before the wf consortium came up with open standards but it will have almost everything one sees in the standard.

Some key terms-

  1. A WF Map.This is a dtree object identified by a dataid
  2. WF Manager-By default the person who created the original map.Best practices dictate that you assign a proper livelink group as the ‘Master Manager‘.Not to be confused as the livelink team’s manager or a organizational manager.In theory this manager can re-assign steps and repaint the map.One of the few places where the sysadmin profile or the ‘admin’ user won’t cut it so due diligence thinking that at some point a higher up user may be asked to assist.Please do not run maps with <Initiator /> as the Master Manager.It is quite possible these users know only to click and when something breaks nobody can intervene.You can make them do only the “See Details” which is more than enough in many cases.
  3. WF Instance- The process logic that is set in motion when a WF Map is set in motion a.k.a initiated .This is easily identifiable by a clickable link in the WF managers assignments and it will show in GREEN the step and useful info.
  4. Steps-The series of steps that can be assigned to processes and users.This is what apparently the business user can be trained to paint..there are a myriad of steps that OT puts on the palette.
  5. Role Based or Map Based-If you create a map OOB then livelink gives you a map where the forward steps can be done by business programming,like if WF attribute is green then my user is this group.If you change the map to “ROLE” based before initiation the Initiator will have to fill all the people in the workflow.People who use the Transmittals product of livelink can see this in action very clearly.
  6. Attachments Volume- By default the attachments package is permissioned Full Control to “Public Access”¬†very bad idea as when a casual user does search he will see unwanted or secretive results..Always put a good permission bit there.If you do not give participants Add Items up to Delete in that volume you will get crude messages from livelink . Note that the Truth Table implementation is observed by livelink on any kind of Object Creation so to get around problems in workflow OT assigns default full control to PA. Just observe what OT gives and change that to a good group and add all the people in the workflow to that group.
  7. WF Attributes– Helper attributes modelled after category metadata .It allows the business user to route the map.
  8. Forms-Optional Package see NEXT post.
  9. Loop Back– Be extremely careful about this as you can create a infinite wf instance.
  10. Item Reference– You can point to existing object subtypes in livelink like folders /documents etc so in conjunction with Item Handlers you can perform “auto magic”

It is quite possible that a good business user or a livelink user can be trained over a day to understand the process flow/swim lanes.I usually design my maps on a paper.I have a cheat sheet of sorts I maintain and many of the things in this article is based on that. Always¬†check “Verify Map Definition” to understand any problems this may have.

The Item Handler is a predictive step modeled by some placeholder logic( Design Book 3) although which it appear useful it could look frustrating as very little process automation can be done just by it.The workarounds or “auto magic” ¬†is usually a human at a prior step.If you use it with XML WF Extensions a lot of “auto magic” can be done with it.

The “XML WF EXtensions”¬† is a optional module that ¬† uses the capability of livelink objects to have a XML representation(everybody probably has heard of XML Export/XML Import)

the XML WF Ext uses XML representations of livelink objects and manipulation of those objects by a SAX(or DOM not sure I know LL has both in it ) parser. Usually people get bitten by load balancers and file system permissions.

Like wise -XML Work Flow Interchange.Can send /call Web Services of other systems.

ESign – A specialized workflow that can do electronic signatures prevalent in 21 CFR11 operations.

Perfectly suited for livelink organizations who will not invest in Oscript coding /scared by OT sales/marketing in not writing Oscript understanding of this remarkable product.It was very expensive when I started livelink programming so it is a personal bias as well since I like to look at the map and many times I can get a 1-1 representation to the process.

OT Workflow Design 1 and Design 2 is a must read if you are doing anything with workflows.it also has decent interface methods for web services  as well

General Help Series 3-Privileges (Am I allowed something)

Privileges

Privileges apply to you as the “user” versus permissions are what the Object allows a “user” to do.There are differences.For e.g for installing something in windows you need to be part of the “administrators” group similarly to install something you need to be in the “system administrators” group. A “administrator group user” has almost very high privileges to do things in a OS similarly they have a high degree of authority in livelink as well.So when a user is created we set at the minimum these privileges Login Enabled and Public Access So when a objects is accessed at run time livelink’s algorithm is does user have login,does user have PA and does Object have PA and what like S,or SC so it is a given that the user will be able to click the link. In very simple terms use PA effectively to address contents that is useful for the whole environment almost like using livelink as a glorified web server.

Now other things are self explanatory has User Creation,System administration and others leading to making Group Leaders etc.In short if you spend some time understanding the architecture you will be amazed and the simple but very effective thought being given into its devlopment.

The Add New Item Truth Table

The livelink system is replete with several subtypes some ¬†of the are very useful and harmless like folders,documents etc so nobody changes anything from default. However you would be very concerned if somebody go unauthorized access to Live Reports,workflows,form templates etc,so what I do is I will look at the personal workspace of a default user and see all the “New Items” being shown there.Then I evaluate what should a untrained user be allowed to create in my system.For all the complicated ones I will create a manual group called “r-Form Template Creators” and add ‘Admin’ to that.Now if I have trust to give a group or a user I would expand my group.So when these new person goes and he has permissions to “Add Items,Reserve ,¬†Delete” this user will see “Form Template” there. However h(s)e moves to another folder where he has only “S,SC” he does not see it there.Note different object subtypes do different things.I am not keen to explain everything like Tasks,Discussions,Projects ec they are for you to explore and set right.Before releasing a system to production give a simple user access and do some Gorilla Testing¬† that will help you more than the amount of money spent on stress testing.Obviously you should have at laest oe or two good front ends and a good beefy admin server to do your search stuff.

General Help Series 2-Permissions (Would the Object allow me to do this)

Permissions 

At the outset a Livelink system represents some vestiges of a File System like Unix or Dos.People having worked in that is probably familiar with the term U G O which stood for User,Group,Others.This is what you see in the std permission bit of an object on its ACL. Every object has one ,many times you set the container object right and let the inheritance trickle in.In many places if you set it right you can forget it.But if you install livelink Out of the Box and has no training or nobody to watch it over you are probably going to end up in rogue territory.This is because the user ‘Admin’ and ‘Default Group’ has very high permission bits set.Out of which the owner is a role,so if you start a folder where Admin did not willfully do the permissions set normalization and if you gave ‘Appu’ creation privileges¬† for folder when ‘Appu’ creates objects he becomes owner who is very powerful and so on.Now this could be argued as not a big problem until it runs amok.So what a good permission model is Owner See,See Contents or Nothing,Group See,SC or Nothing and PA,S,SC or nothing.Now you will create administrative groups something simlilar to these discussions and you will be fine.

Always rely on proper groups that you can create and maintain in a directory service (AD is very common) and have those groups synchronized in.It is quite possible that HR systems release feeds into Directory services hence when users leave your organization they will go as well.There will be no permission problems because the real administrative group has overriding permissions that the owner based approach.Simple try it and  you will be happy and 50 % of your help desk tickets will come down.

This is what one hears in OpenText engagements as “Community Modelling” well my next part will cover object truth tables as well.

Permissions bit use BitWise logic in FileSystems as well as livelink does recently a programmer told me he finds the awesome bitwise logic un understandable.I just thought wow where is the programming world heading into ūüôā

There is nothing wrong in owner having high permissions so long as the administration is willing to support it and understand it.

Security Clearance and Supplemental Markings are OT’s way of securing against inheritance rules in big hush hush organizations.

Oscript what is it what and would I use it?

Oscript is ¬†a 4GL that is the basis of the application that the OpenText¬†company adopted when they chose to market Livelink the Document Management System.It is safe to say that anything¬†¬†the Livelink product lacks and ¬†is a valid requirement can be coded in this language.The power of the the Oscript language takes root in the philosophy that was prevalent in the 60’s¬†. Oscript is a superior offering to traditional OOP languages such as Java or C++ that is what Pundits say.

To a trained Oscripter one is able to understand how livelink code is received as well as change it.Now change to any COTS product comes at a cost.That is where the strategy as well as overall grasp in the product comes into mind.If the organization has decided for itself that it will spend the big bucks on livelink, one approach is to identify the ways to use it more.Where the organization is going to make mistakes is when it sees the cost of the¬†“customization”.Many¬†decision makers wrongly decide to go after fly by night programmers/vendors to be totally left in the fray.OT has a professional vested interest to sell customers more code and services so I have been in decisions where sales and marketing ¬†will say do not customize livelink, but that is only half truth in them.If you buy these OT customizations, they are almost certain to be written in Oscript .In many cases it would have been written by senior Oscripters with thought on maintenance aspects ¬†and upgrade ability as well.The same kind of service can be bought by other Oscript houses,in many cases these are extremely talented ex OT ‘ers. Many of the Professional Services modules sometimes outgrows the intents and becomes a marketable module as well.I have no knowledge and am just speculating on that.Once you have identified and ¬†decided to use livelink the best avenue is to hire or ask¬†services of reputed vendors.Modules that contained Oscript code that I used to code in 1999 (first time was on a livelink 8 system)¬†livelink 9.1 version I can almost compile in the modern 10.5 version.My entire career was ¬†and is fruitful because I tried to interact/collaborate and share with the OT team in the KB as well as smaller websites like Greg Griffiths.I was also able to mentor myself amongst very die hard oscripters that I knew from work and just by tracing code to find bugs that needed closure before OT could provide fixes.The knowledge of OT code also allows me to become a better application architect,administrator and integrator.

If the organization is strong in Java and .Net skills then note that the user interface at this time cannot be changed using those languages,however you may use exposed API’s to make functioning applications that use Livelink as the data store

Typical things organizations can and should do.

  1. Create implementations of Nodes just like Folder or Document if there is a need for it to be there.Just because OT shows you a “Addressbook” or “Contacts” module not every livelink installation in the world needs new subtypes.It is there if you want to use it ,if there is a complex business requirement that involves a “document” like implementation or a “folder” like implementation.Do search in the KB for such terms as “invasive customizations”, overrides,orphans,subclasses, customizationsRT, weblingo customization.
  2. Add Commands in the Nodes these are basically very easy to do in a short order.These two are taught very nicely in class as well.
  3. Put a lot of functioning logic into workflows also heard as event scripting.
  4. Write code so that livelink can interact with other systemslike using the POJO object.
  5. Write agents,distributed agents,expose to columns and facets api something that OT does not give you OOB.
  6. Extending SOAP and REST can only be done using Oscript.If you are a shop who has a need to program a integration using java or C# and you see a “GAP” in existing offering it is rather easy for a Oscripter to extend that.

Opentext Programmers have black boxed thing so that many things acts as internal API’s.For e.g we are able to do work without understanding how the “Node” itself is created in livelink. Other examples are the way Oscript can talk to the search system.

Not many people know that livelink uses a api to search.

Out of the world Oscript Offerings.

Christopher Meyer a Long OT veteran his website . It is un believable stuff that I have seen.

Patrick Vitali the maker of AnswerModules

IDE’s one can use

CSIDE a eclipse jar file for versions CS10.5

Livelink Builder CS10 and Versions prior.

Common Debug Protocol

****************************************** ***START UPDATE 10/3/2016*****************

LAPI and its client installer has become very hard to find. Moreover clients written in LAPI ¬†in say Java/.NET will only work if your Livelink a.k.a Content Server is of version less than CS16.Readers who are new to LL programming is encouraged to read this to the approach and not to the exact lines of the code.What I mean is when you used to program in LAPI you were basically passing parameters to discrete calls by modelling it based on the webgui of livelink .SOAP based webservices called CWS is also the same,so if you do not try to do the task in the webgui and try to understand the business rules you will almost have no success in CWS too. OT is notorious for not putting fully functioning use cases and a walk through,so whenever possible I write code assuming the user has not worked in Livelink for X number of years and try to educate you all. Livelink,Content Server,Enterprise Server all of this has been Livelink’s marketing brand name changes over the years.CS i sused in many of the integrations like AGA, XECM, RMLINK ¬†and you know you are programming against livelink if you see a link that looks like this ¬†http(s)://somefriendlyURL/livelink.exe|cs.exe|llisapi.dll|cs.dll|livelink.In many places SAP/ SP /Exchange will be configured to talk to Archive Server and then they will use Livelink to read into archive server and turn that into LL objects for better presentment/RM and other aspects. The AGA product is moving away from LAPI(not sure totally or not) to REST API in LL.

**********************************************END UPDATE****************************

 

 

In most cases I have noticed that programmers brought into livelink a.k.a Content Server web server programming lack a general understanding of livelink and its business rules.The Vendor maintained links   is written almost to a starting novice but again people will run into problems.To mitigate this one has to have a simple protocol

  1. Ask the livelink team a representative container(folder ,project,compound documents,document work spaces all of them are shells or containers to hold additional objects) that the CWS programmer can access using the webgui.
  2. Ask how is authentication enforced.In almost 90% of places it will be SSO(NTLM,IWA,other LDAP methods)
  3. In most cases CWS application where you source the WSDL’s from will be “anonymous” because that is the level of documentation OT gives admins which means that the CWS user has to be manual user in livelink also known as a user with a password the livelink admin team gives you
  4. I have published several successful posts here  that allow me to do SSO(IWA/NTLM) that is because I know how to set it correctly.Most of it was looking at IWA webservices samples.
  5. I have installed and configured OTDS as well ,it is a re-directional(kerberos implementation) .Simply put when you access a link such as http://localhost/livelink/llisapi.dll if OTDS authentication is involved your call is redirected to the OTDS server(Tomcat in the version I tried).The kerberos token that is prevalent in your domain login is used to establish who you are against a configured LDAP appliance like AD and returned back to the livelink server.Anybody who knows enough about livelink URL’s know the web escaped URL’s, you should see the re-direction when using wireshark or fiddler as well.¬† This ideas are my interpretaion and not OT’s so take this with a piece of slat.
  6. The above OTDS authentication is overkill if Livelink was employed as a DMS or in your org people know that Livelink is there.It would make sense to tunnel xECM SAP Users your piggy backing on MYSAPSSO2   mostly integrations form other systems who want a seamless experienec.Altogether it is very easy to setup and implement save for the fact that creating SSO against AD is basically a one click operation.

Allright so what is the post about

If you are a .NET CWS/EWS programmer you will basically be sending this token with each service client call because livelink is http and state based and you never know if  the request is being handled by the same server or the same thread.

You would see .NET code like this

DocumentManagement.OTAuthentication dmOTAuth = new DocumentManagement.OTAuthentication();
ContentService.OTAuthentication     csOTAuth = new ContentService.OTAuthentication();
SearchService.OTAuthentication      ssOTAuth = new SearchService.OTAuthentication();

and things like

string token = authclient.AuthenticateUser(username, password);
if (token != null)
{
ssOTAuth.AuthenticationToken = token;
dmOTAuth.AuthenticationToken = token;
csOTAuth.AuthenticationToken = token;

}

contextID = dm.GetVersionContentsContext(ref dmOTAuth, dataID, 0);

the above is akin to a logged in user trying to download.the ref keyword is modern .net above 2.5 CLR or higher

If you are a JAVA CWS/EWS

In Java, however, you don’t need to do this and instead you set authentication information into the object representing the service. In your code you do this

setSoapHeader((WSBindingProvider) search, otAuth);

There are many more subtle differences with Java and .NET but for the most part it should work cleanly

If one were interested in debugging at the livelink server

  • Method 1: Login to the livelink URL that usually ends in /llisapi.dll or livelink.exe or livelink.Do you see an authentication screen that looks like an HTML page and not one that looks like a windows auth prompt?.If you were seamlessly taken into the application then this “instance” of livelink server has its authentication set to some authentication.assuming the simplest mechanism it could be IWA also interchangeably known as NTLM,SSO etc etc.
  • Method 3-Put Fiddler and learn how to use it when you hit a webserver.Then you can actually see your WSAPI client code making requests in Action.Many¬† times they are going to indicate something.
  • Method 4: There is a livelink sysadmin request handler that goes like <LIVELINKURL>?func=admin.testargs.Do you see “REMOTE_USER” filled and a auth mechanism something resembling this¬† AUTH_TYPE =Negotiate REMOTE_USER =DOMAIN\USERID or userid@domain REQUEST_METHOD =GET Then chances are this livelink is protected by a auth method other than “anonymous” Most webservices samples written by OT pertains to simple anonymously authenticated livelink servers.Do you have SSO samples lying in the web or here.Sure search for Guy Pomerleaux or me for a few who have ventured with it in the forums .Just by trial and error of the Web.Config and App.Config I have suceeded.Why wouldnt OT put a official sample,guesses are different web server different ways of deploying livelink,too much work to officially support it although the OTDS mechanism is kind of a middleware to address that ūüôā
  • Method 5:Search Debugging Livelink when people or programs search use the Livelink Search API to call it. While it looks integrated believe it or not the searching is a different software and it can be used for any general purpose searching.Read OT history as the first searching company in the world(Yahoo search Engine :).Sometimes it becomes difficult to create queries properly so it is very easy however I do it like this.Isolate a named server(some address that wont be like a LB URL).Find that server’s Opentext.ini and in the [options] area add wantSearchLogs=true.Then run a search from the GUI.In all likelihood for each search you will find a new txt file that will contain the query.Try to understand it and repurpose. Similar files will exist in the Admin server for prospector queries and intelligent classification queries.

Now I have given you many pointers that would lead to successful coding.However let it be known that a livelink can be configured in very many different ways such as an admin can put a expiration  of livelink cookie to a very low value,you could be bouncing off livelink servers ,your code may come from a redirected system such as SAP or SP lots of problems could occur if the architecture is not well understood by replying people like me or even you.I am not a OT employee but have worked enough to know certain things that could occur.So my simple request to you would be to as much as possible educate yourself and the forum about the same. Also try the simplest mechanisms of OT code first to iron out the difficulties.It should not be too  hard.