Vince's Dominolog  

About Lotus and other stuff

  Declan commented on David's blog post that was a response to an article on JSFCentral .
Still with me?
Anyway he claimed/joked? about the 30 servers needed for the application described on JSFCentral.
That got me thinking, we migrated a pretty complex application from Domino To JEE last year and of course the management thought that things would get a lot simpler/cheaper in the new environment.

Well, if counting servers is an indication...

Only listing servers for the SAAS website (so excluding internal software, webservices and other stuff) we started with the following Linux servers:
2 Apache
1 SQL Logging server
1 Tomcat search server
1 SQL Search server (plus 1 backup)
2 clustered Domino Webservers
1 Domino webserver for 3rd party connections
1 Domino Hub
1 Domino Index
1 Domino ‘Edit’ server (backoffice/maintenance/agents)
1 Windows server for PDF processing (OCR server)
TOTAL: 12

And we now have:
2 Apache
1 SQL Logging server
2 Tomcat Webservers
1 Tomcat Batch server (agents)
3 Elasticsearch search servers
2 clustered SQL servers
1 Windows server for PDF processing (OCR server)
TOTAL: 12

Now as you can see this environment was already partly JEE-iffied, so there were already some additional servers, which taints the comparison. So, if anything the number of servers overall probably increased rather than decreased. Plus a change in the hardware specs of course; used to be 32bit with 8Gb memory and now 64bit with 32Gb. But who's counting... ;)

From my inbox:


Vince,
 
It’s hard to believe that it has been 20 years since THE VIEW first arrived into an exciting Lotus market!  During that time we’ve had the pleasure of doing business with thousands of subscribers and attendees of our Admin and Developer conferences. Throughout the years, we worked to deliver trusted and valued information to help you do your job better. We remain humbled by the favorable response to our products over the years. 
 
It's with a sad farewell that we announce we are no longer supporting a subscription model and closing eview.com. As a subscriber, you’ll retain your access to THE VIEW archives through June 30, 2015.
 
We’re heartened that the community still has a supportive group of content contributors through blogs, forums and user groups. Our parent company, Wellesley Information Services, will continue to manage the SocialBiz User Group, and we encourage you to visit us there, find a local user group, access technical webcasts and ConnectED presentations, and read and contribute technical tips and tricks on the blogs.
 
Other technical resources we recommend include:
OpenNTF
developerWorks
Notes in 9 (XPages)
Connections101.net
Planet Lotus
VIEW virtual training
The VIEW Anthologies
We hope to see you on SocialBizUG.org and at various IBM events. Until next time...
 
Regards,
 
 
Celia Hamilton
THE VIEW


Shocking news for me, didn't see this coming.
I know the Lotus market 'has been better', but I never considered that an institution like THE VIEW, one of my main sources of information on Lotus matter for the past 18 or so years would stop so abruptly.
All the best to you guys and thanks for everything!

Now here is something I never did before.
I have a Neato vacuum cleaning 'robot' and after a year of 'hard work' it started to show errors on the display. The Neato website showed me how to 'reboot' the vacuum cleaner, but there was also a section about software updates.... Software updates for a vacuum cleaner. Quite logical if you think about it, but for some reason it never occurred to me that you could actually upgrade the software of a vacuum cleaner.
So I thought what the heck, just try it, and indeed after 3 attempts I managed to upgrade the software from 3.2.something to 3.4. something. Do I see a difference? Yes, noticed some considerable changes actually. The new software is claimed to have a better battery management system etc, but to me the most notable change is that is a lot more aggressive while vacuuming. It relies less on it's optical sensors and bumps harder and more often into obstacles.
The good news is that you can change the settings by adjusting the software yourself. You may think that as a software developer I will not be able to resist the challenge, but I decided to let the vacuum cleaner's software do it's thing unless it really starts to annoy me.

Ok, now this was a very simple problem (in hindsight) but took me quite some time to figure out:

We use Elasticsearch to search through pdf's and the Elasticsearch Attachment Mapper to index the pdf's.

However we have over 500k documents and we noticed that after one or two days Elasticsearch tended to spike the CPU use to 100% due to the attachment mapper. Because even Elasticsearch experts could not find the problem (btw try to find an expert first ;)), we decided to use Tika (the plugin used by the attachment mapper) directly.
It seemed pretty straightforward; added tika-core to the Pom file and change 2 lines of code and away you go....

At least so we thought because all our test code worked flawlessly. However running it in tomcat directly did not extract any text from the pdf's.
After lots of debugging, we noticed that the parser called from the tests were different from the ones invoked by the server. Further investigation revealed that my Java IDE IntelliJ contained the tika-core library as well as the tika-parsers library.
So after checking the documentation (yeah probably a bit late) I found a comment that the tika-core library can identify but not parse the contents of the document.
After exchanging the tika-core with the tika-parsers library in the pom.xml, I got an error that suggested library incompatibilities:

Handler processing failed; nested exception is java.lang.VerifyError: class net.sf.cglib.core.DebuggingClassWriter overrides final method visit.(IILjava/lang/String;Ljava/lang/String;Ljava/lang/String;[Ljava/lang/String;)V

This error took some more research but after a lot of coffee and even more strong words I found that there were a lot of libraries in the tika-parsers library that were already in our own pom, but that the culprit was the asm library.
So here is what worked for us, depending on your pom you may need to exclude other libraries as well.

<dependency>
<groupId>org.apache.tika</groupId>
<artifactId>tika-parsers</artifactId>
<version>1.7</version>
<exclusions>
<exclusion>
<groupId>org.ow2.asm</groupId>
<artifactId>asm-debug-all</artifactId>
</exclusion>
</exclusions>
</dependency>

Now that brought some memories back for me.
A few years after Bruce arranged to host openntf.org at his work, he bought a server and hosted it at PSC in 2002.
After a couple more years OpenNTF really started to attract attention and traffic spiked big time. We started to worry about up-time and data backup so I bought a Dell PowerEdge 1800 in 2006 to act as the European backup and failover server for the American based OpenNTF site.
After the production site was transferred to Prominic (where it still runs today, thanks to Justin Hill and the other guys at Prominic), it continued to act as development server for years, but today sadly it started to squeak and squirrel and after a couple of restart attempts died completely.
Not bad actually; 9 years is way over the average for a server, even for one with every fail safe option available.
Anyway, it got me thinking about the early days of OpenNTF and how simple 'development life' was back then (for me anyway).
I am working on major JEE projects with large multidisciplinar teams nowadays, a long way from the 'one person one project a week' Domino development environments I was involved in those days.
It feels like the death of the old server symbolises the ending of my involvement with the Lotus/IBM/Domino community as well. I am still involved in several Notes/Domino/Traveler/Sametime upgrades and such, but development work on that platform seems hard to find.
Some ISV's are still developing their commercial applications, but I don't see much development at the end-user level. Could be I am not in contact with the current Domino development scene anymore, but that is how i experience it.
Sad because the Domino community seemed so much more involved in the platform than the Java world. Yes there are all kinds of venues and conferences etcetera, but I do not see the same involvement as in the Lotus/Notes/Domino scene.
A sad thought at the start of the Lotusphere Connect ConnectED conference, but I'm sure it will work out eventually.