Searchable XML Documents with DataStax Enterprise

 Comments Off on Searchable XML Documents with DataStax Enterprise
Feb 072016
XML icon

Recently a customer asked for advice on storing a large number of XML documents in DataStax Enterprise. Though each XML document will be just a few kilobytes in size, there will be many billions of them. Multiple fields inside each XML document must be indexed and searchable in real time.

I have come across this and similar use cases a few times as a Solutions Engineer for DataStax, and indeed we are an excellent database platform for building a large, distributed, searchable document store. So in turn I created the DataStax XML Demo to show one way to store, index, and search XML documents with DataStax Enterprise. Please check out the GitHub repository and write me with your thoughts and questions.

DataStax Summit Europe 2016
To strike a related but tangential note, please come heckle me and my colleagues at DataStax Summit Europe 2016 in April. Open to all, free to register. You have no excuse, and you just might learn something.

Apple and Adobe: It’s the Censorship, Stupid

 Comments Off on Apple and Adobe: It’s the Censorship, Stupid
May 172010
Apple logo

The hearts of trendy technology fanboys the world ’round have been aflutter over the Apple versus Adobe showdown. All this agitation on the intellectual sea has produced some foam, as agitation will, but I worry when emotions run high and hyperbole runs rampant. A private technology mailing list to which I am privy began dissecting Kevin Lynch’s interview with Brady Forrest at Web 2.0 Expo SF. One gentleman, targeting Mr Lynch’s dark depiction of “walled gardens” (read: the iPad), went so far as to call Mr Lynch an “evil, evil man” for his demands that Apple open its precious platform to Adobe’s products.

Kevin Lynch a twice-evil man, is that not a tick harsh? An unabashed puller of the corporate line, certainly…though as the CTO, he helps write the corporate line. A egotistical geek, sure…again, CTO. Evil? The legal-but-immoral thoughts Steve Jobs wakes up with on a given Tuesday would make Mr Lynch curl up in a ball and beg for his mother, yet still I would hesitate to call Mr Jobs evil. (What is Steve, then? It rhymes with brass soul.)

To consumers, Apple sells delicious little bundles of hardware and software: iPhones, iPods, iPads. Apple also sells, to developers, access to that army of captive consumers, taking its famous 30% cut on App Store purchases. Apple created a platform and is her gatekeeper, censoring content and now dictating development methods. Do we as a society want Apple, or any corporation, to be the lone censor for a widespread and important consumer platform? I submit that society does not. The printing press manufacturer may not censor the daily newspaper. The descendants of Johannes Gutenberg may not challenge J. K. Rowling, no matter how derivative her imagination or childish her writing.

Our entire body of work here at Lab49 depends on tools and platforms built by others, and specifically, much of my own craft involves running Adobe tools on Apple platforms (excepting when bank clients force me onto Secure Build Windows, itself an oxymoron if ever there were one). On one hand I want Apple and Adobe to kiss, make up, and get the band back together for an overpriced reunion tour. On the other hand I enjoy watching two promoters of closed systems accuse each other of promoting closed systems. Hypocrisy may be a scoundrel’s refuge, but it is my entertainment.

Apr 292010
Flash Builder icon

Flash Builder is, speaking the King’s for a tick, pure rubbish. Adobe took its sweet time releasing this purportedly major upgrade, and I held a fool’s hope that Flex developers the world over might no longer gaze longingly at the highly functional and pleasantly integrated Silverlight tools across the aisle. Indeed, Adobe incremented the version (to 4 from 3) and switched the label (to Flash Builder from Flex Builder), but still the product is rubbish. I know it was you, Adobe. You broke my heart.

I am not alone in my sentiments. My associate Scott Selikoff focuses on Flex 4’s code verbosity and duplicate component libraries, but his verdict is the same. Rubbish. Unfortunately, Flash Builder is my primary development environment.

My present Flex frustration (Flexstration!) is resource modules, by which a developer may externalize resources into SWFs that may be loaded at run time. It sounds wonderful, but sadly the process is quite manual, requiring a human to take a number of heavy steps:

  1. Determine the resource bundles required, by running the compiler with the -resource-bundle-list option
  2. Gather the contents of the output file generated in step 1
  3. Run mxmlc from the command line — Flash Builder offers zero support here — specifying no MXML file but supplying the -locale, -source-path and -include-resource-bundles options

The poor developer must repeat this process for each resource module in each locale. As an example, let us assume the application has four resource modules and must support merely four locales: American English (en_US), British English (en_GB), Mexican Spanish (es_MX), and Spanish Spanish (es_ES). Even for these decidedly unambitious numbers, the now-heartbroken developer — or his helper monkey — must repeat the manual operations listed above 4 x 4 = 16 times. Pray for Mojo.

Of course, a clever developer could easily write a build script, perhaps leveraging Ant and Adobe’s Ant tasks for Flex. Most Flex developers, myself included, have done just that. My point is that this sort of automation of common tasks is exactly what a £400 enterprise development tool should do. Epic fail, Adobe.

[Update 21-July-2011] Corrected a typo in the “heavy steps” list.

Apr 132010
Golden Buddha

In my previous article, Achieving Nirvana on Mac OS X, I set up a Nirvana server on port 8887 and imported a realm for the TradeSpace demo. The TradeSpace demo instructions had impudently asked me to run two .BAT scripts, setup.bat and startall.bat, and when we left off, I performed manually the equivalent operations for the former. Today I will tackle the latter which, according to the instructions, will create and populate the demonstration news and rates channels in our Nirvana realm.

The script contains only two simple lines to run two other scripts, startnews.bat and startrates.bat, and thankfully these secondary scripts are short and sweet. Inspecting them, .BAT-style rem comments redacted:

set CLASSPATH=%CLASSPATH%;..\classes
java -cp "%CLASSPATH%;..\classes" -DRNAME=nhps:// -DRNAME2=nsp://localhost:9000 -DCHANNAME=/showcase/consolidatedrss com.pcbsys.demos.feedsimulators.NewsFeeder

set CLASSPATH=%CLASSPATH%;..\classes
java -cp "%CLASSPATH%;..\classes" -DRNAME=nsp://localhost:9000 -DRNAME2=nsp://localhost:9000 -DCHANNAME=/showcase/stockindices com.pcbsys.demos.feedsimulators.Stockindices

It is clear I must run two Java programs. Unfortunately, my herculean efforts to run them from the terminal end as assorted failures, from Java classpath issues to brutal Mac OS X crashes (see Figure 1).

Instead, I launch Eclipse, create a Java project, and add the Java source from the TradeSpace demo package (in my case, ~/Sites/nirvana/demo/src/java/tradespace). I also add three entries to the build path:

  • External Class Folder – /Applications/Nirvana_5.1.6710/classes
  • External JAR – {NIRVANA_HOME}/lib/nClient.jar
  • External JAR – {NIRVANA_HOME}/lib/swing-layout-1.0.jar

Now I create two launch configurations:

Main Class: com.pcbsys.demos.feedsimulators.NewsFeeder

VM Arguments:

Main Class: com.pcbsys.demos.feedsimulators.Stockindices

VM Arguments:

I launch both configurations in debug mode. Inspecting the Nirvana Enterprise Manager, I can see activity in the Event History chart and the Event Status table. I navigate to http://localhost:8887/ in a browser once again and click on the Flex link under the Web Client Demos section. The Flex client starts up and loads the news and rates data (see Figure 2). Voilà.

Mac OS X Crash Screen

Figure 1: Mac OS X Epic Fail

TradeSpace Web Client Demo in Flex

Figure 2: TradeSpace Web Client Demo in Flex

Apr 062010
"Giant Buddha" on Lantau Island, Hong Kong

Here at Lab49 we develop right on technology’s bleeding edge. Innovation requires investigation, and so in its wisdom the Lab encourages a culture of research. Our own Matt Davey championed an internal venture, now involving a small team of 49ers, to demonstrate a new risk management architecture. While the project at its core focuses on back-end technologies such as data stores, for this article I will discuss mostly middleware.

Early on (and prior to my involvement) the team planned to create a straightforward Microsoft Silverlight front-end that connected to Nirvana middleware. We have since added a twist: the team will build two front-ends in parallel, one in Silverlight and one in Adobe Flex. Given the political intesity surrounding RIA technologies, I have no doubt this will be an interesting experiment. With luck it will also be informative beyond the usual Flex versus Silverlight religiosity, as we will be able to judge how easily each platform integrates with a proven middleware product in Nirvana. Of course, to begin I must install Nirvana Enterprise on my trusty MacBook Pro.

The shop behind Nirvana, my-Channels, provides a free developer edition. Generally I work on a Mac, so I grabbed the Mac OS X package. They also provide official installation instructions, but unfortunately the curtains do not match the carpet, as it were. There are instructions for Win32, Linux/Solaris and Generic Unix, but there is not a word about OS X. A reasonable man might assume Generic Unix is the appropriate replacement, but he would be wrong. The good news is that when I download and unzip, out pops a proper OS X application I launch the installer and go with the defaults most of the way. I do choose a custom data directory, ~/Documents/nirvana_data. Installation proceeds without trouble, using the default /Applications/Nirvana_5.1.6710, henceforth called {NIRVANA_HOME}, for the Nirvana application folder.

Moving on, I download the TradeSpace demo package, which is free on the my-Channels downloads page. The demo instructions recommend unzipping to the Nirvana application folder, but instead I choose ~/Sites/nirvana/demo. Following the Environment Configuration section of the demo instructions, I update ~/Sites/nirvana/demo/cfg/demo.xml.

  • Update value of property BasePath to ~/Sites/nirvana/demo/www.
  • Update value of property port to 8887. This number is somewhat arbitrary, though I recommend avoiding the default 80.

Now I am able to launch the Nirvana server. In the Finder, I navigate to {NIRVANA_HOME}/server/nirvana and double-click the Start Nirvana Realm Server alias. A Terminal window opens and launches the Nirvana server, which in turn prints some server and environment details to the console. The server lives. The next section of the instructions, Interface, Plugin and Channel Setup, suggests I open an item from my Windows Start menu and then execute a pair of .BAT scripts. This is clearly impossible on OS X, and the instructions offer no alternative. I am on my own.

I open the first script, setup.bat, in a text editor and see a single command, nimportrealm ..\cfg\demo.xml -all. This implies the next step is to import the demo realm, and fortunately I can do this by GUI. I navigate to {NIRVANA_HOME}/Administration/nirvana/ and double-click the Nirvana Enterprise Manager. I find my default realm nirvana under the Realms node and right-click to bring up the context menu, selecting Import Realm from XML. I find the demo.xml file from earlier and click Import All. After I click OK, a directory showcase with four channels appears beneath my realm nirvana. To confirm the server is operational, I head to http://localhost:8887/ in a browser, and sure enough, a welcome page congratulates me.

For my encore, I will take care of the second .BAT script, startall.bat, and at last connect a Flex client to Nirvana.

[Update 13-April-2010] Continue with the follow up article, Achieving Nirvana with Flex.