If you're interested in functional programming, you might also want to checkout my second blog which i'm actively working on!!

Sunday, December 23, 2012

Getting Play2.0.x to work with Git Bash on Windows

I unzipped play2.0.4 and created a PLAY_HOME variable. I also made sure to add PLAY_HOME to my PATH variable.

nxp10009@NXL01366 /c/development/play-2.0.4
$ echo $PLAY_HOME
C:\development\play-2.0.4

When I tried to run play from the Git Bash shell I ran into below exception. In the past I didn't bother that much and just used the DOS shell as a workaround. But I like to work from one shell only so time to look for a solution.
nxp10009@NXL01366 /c/workspaces/scala
$ play
Error during sbt execution: Could not find configuration file 'c:/development/play-2.0.4/framework/sbt/play.boot.properties'.  Searched:
        file:/c:/workspaces/scala/
        file:/C:/Users/nxp10009/
        file:/C:/development/play-2.0.4/framework/sbt/

Luckily someone created a patch to resolve this issue. You can download the gist and unzip it to some folder. It will contain a file named play_cgywin.patch. So it's matter of applying the patch from within Git Bash. It will log an error but play will work nonetheless.
nxp10009@NXL01366 /c/development/play-2.0.4
$ patch -p1 < c:/tmp/play_cygwin.patch
patching file `framework/build'
Hunk #1 FAILED at 8.
1 out of 1 hunk FAILED -- saving rejects to framework/build.rej
patching file `play'


nxp10009@NXL01366 /c/workspaces/scala
$ play
Getting play console_2.9.1 2.0.4 ...
:: retrieving :: org.scala-sbt#boot-app
        confs: [default]
        5 artifacts copied, 0 already retrieved (3667kB/83ms)
       _            _
 _ __ | | __ _ _  _| |
| '_ \| |/ _' | || |_|
|  __/|_|\____|\__ (_)
|_|            |__/

play! 2.0.4, http://www.playframework.org

Thursday, December 20, 2012

StringJoining lines of file Unix

This week I had to generate a batch (400) of DITA files. The actual ids were handed to me in an excel sheet. As I was using a scripting language (Cocoon flowscript == Javascript) it would be convenient to transform the id column from excel to Javascript array notation. So the first thing I did was copying the content of the first column and paste it into batch2.txt.

batch2.txt
2N7000
BLF1822-10
BT150-800R
BU1506DX
BUT18A
BUX87P
BY229-600

So now I needed to find a way to basically transform each line by wrapping each id in double quotes and next do a string-join of all lines. As I was a bit rusty in shell scripting I started looking online. My colleague and I stumbled upon the same approach which got the job done for 99.9%. The only problem was that there was a comma after the last id.
for line in `cat batch2.txt`; do echo -n "\"$line\"," ; done >> output.txt

"2N7000","BLF1822-10","BT150-800R","BU1506DX","BUT18A","BUX87P","BY229-600",

When i woke up this morning I felt restless... surely doing a simple stringjoin can't be that difficult? So I started reading about a few unix commands and you can use 'sed' to easily do string replacement. So the trick I use is to first wrap all ids within double quotes. I also have to make sure to use the -n flag with the echo command so the newlines are removed. Next I just replace 2 sequential double quotes "" by "," and that's it.
nxp10009@NXL01366 /c/tmp
$ for line in `cat batch2.txt`; do echo -n "\"$line\""; done | sed 's/""/","/g' >> output.txt

nxp10009@NXL01366 /c/tmp
$ less output.txt
"2N7000","BLF1822-10","BT150-800R","BU1506DX","BUT18A","BUX87P","BY229-600"

Wednesday, December 19, 2012

Using Options in XQuery

As a developer you try to use best practices from different toolkits, frameworks, programming languages. One of the things Scala solves nicely in their core libraries is avoiding nullpointer exceptions. So I decided to hack this idea into some XQuery functions for fun.
   <!-- Taking  the same approach as Scala we return options which are either none or some -->
   <option>
     <none/>
   </option>
   
   <option>
     <some>4</some>
   </option> 

   <!-- Example of a lookup map represented in XML -->
   <map>
     <entry key="PH3330L">SOT669</entry>
     <entry key="BUK100-50GL">SOT78B</entry>
     <entry key="PSMN003-30B">SOT404</entry>
   </map>   

An example of how you could model options in XQuery and build some utility functions to make your code more safe.
declare function local:getOrElse($map as element(map), $key as xs:string, $else)  {
    let $option := local:get($map, $key)
    return (if (empty($option/some)) then $else else data($option/some))
};

declare function local:get($map as element(map), $key as xs:string) as element(option) {
    (if (empty($map/entry[@key=$key])) then <option><none/></option> else <option><some>{data($map/entry[@key=$key])}</some></option>)
};

let $map1 := 
   <map>
     <entry key="PH3330L">SOT669</entry>
     <entry key="BUK100-50GL">SOT78B</entry>
     <entry key="PSMN003-30B">SOT404</entry>
   </map> 
 
let $map2 :=    
   <map>
     <entry key="robby">1977</entry>
     <entry key="ivan">1987</entry>
   </map>  
 
return 
  <lookups>
    <package>{local:getOrElse($map1, "PH3330L", "test1")}</package>
    <package>{local:getOrElse($map1, "INVALID", "test1")}</package>
    <age>{2012 - local:getOrElse($map2, "robby", 0)}</age>
    <age>{2012 - local:getOrElse($map2, "amanda", 2012)}</age>
  </lookups>  


The result looks like this if you test it on e.g. Zorba.
<lookups>
  <package>SOT669</package>
  <package>test1</package>
  <age>35</age>
  <age>0</age>
</lookups>

Handling Nil values in XSLT / XQuery

Today I had to generate another batch of DITA files but I discovered that some failed. So the problem was that some dates were nillable and I was formatting the dates so I ran into an exception. So you can in fact pretty easily check if that element has a nil value.

<xsl:variable name="now" select="current-date()"/>
<xsl:variable name="date-format" select="'[Y0001]-[M01]-[D01]T00:00:00'"/>  

<created date="{if (not($productInfo/InitialWebPublicationDate/@xsi:nil='true')) then format-date($productInfo/InitialWebPublicationDate, $date-format) else (format-date($now, $date-format))}"/>


As Ryan Dew commented there is a shorter way like below:
<created date="{if (not(nilled($productInfo/InitialWebPublicationDate))) then format-date($productInfo/InitialWebPublicationDate, $date-format) else (format-date($now, $date-format))}"/>

Monday, December 17, 2012

Using sbt plugin for creating IntelliJ IDEA project files (update)

This article contains up-to-date information. I followed the tutorial from the Guardian but there were some issues and I managed to resolve everything. So first we will install the sbt-idea plugin. This time I will install the plugin not on individual project basis but for our SBT installation directly.

First I had to create a plugins folder in my %USER_HOME%/.sbt/ which in below snippet has already been taken care of. Next I created a build.sbt and I added the mpeltonen plugin.
$ pwd
/c/Users/nxp10009/.sbt/plugins

nxp10009@NXL01366 ~/.sbt/plugins
$ ls -la
total 3
drwxr-xr-x    5 nxp10009 Administ     4096 Dec 17 14:14 .
drwxr-xr-x    1 nxp10009 Administ        0 Dec 17 14:13 ..
-rw-r--r--    1 nxp10009 Administ       61 Dec 17 14:14 build.sbt

$ less build.sbt
addSbtPlugin("com.github.mpeltonen" % "sbt-idea" % "1.1.0")


So next I created a new project according to my previous article and tried to check if everything worked ok. But I soon ran into following issue:
[warn] Host repo.typesafe.com not found. url=http://repo.typesafe.com/typesafe/ivy-releases/com.github.mpeltonen/sbt-idea/scala_2.9.2/sbt_0.12/1.1.0-M2-TYPESAFE/ivys
/ivy.xml
[info] You probably access the destination server through a proxy server that is not well configured.

So I had to configure a proxy for SBT to use. I could have created an alias in my bashrc script but I decided to change the SBT start script and add the proxy to the JAVA_OPTS
nxp10009@NXL01366 /c/development/sbt
$ less sbt
#!/bin/sh
# sbt launcher script for Cygwin and MSYS

JAVA_CMD=java
JAVA_OPTS="-Dhttp.proxyHost=http://your_proxy -Dhttp.proxyPort=8080 -Dhttps.proxyHost=http://your_proxy -Dhttps.proxyPort=8080 -Xmx512M"

....

Sunday, December 9, 2012

Using sbt plugin for creating IntelliJ IDEA project files

Of course I'm a lazy developer -- who not -- ? So I started searching for a sbt plugin to generate the IntelliJ IDEA project files. I found two which seem to work pretty well.


So I decided to go with the mpeltonen plugin. You have two choices. You can configure the plugin for sbt globally or on a per project basis. I will now configure the plugin for the demo project only.
nxp10009@NXL01366 /c/workspaces/pelssers/demo/project
$ vi plugins.sbt

resolvers += "Sonatype snapshots" at "http://oss.sonatype.org/content/repositories/snapshots/"

addSbtPlugin("com.github.mpeltonen" % "sbt-idea" % "1.2.0-SNAPSHOT")

If you start sbt again now for the demo project you will see it will fetch the dependencies.
nxp10009@NXL01366 /c/workspaces/pelssers/demo
$ sbt
[info] Loading project definition from C:\workspaces\pelssers\demo\project
[info] Updating {file:/C:/workspaces/pelssers/demo/project/}default-a9f0a5...
[info] Resolving com.github.mpeltonen#sbt-idea;1.2.0-SNAPSHOT ...
[info] Resolving commons-io#commons-io;2.0.1 ...
[info] Resolving org.scala-sbt#sbt;0.12.0 ...

Now we can use the gen-idea command to generate our project files
> gen-idea
[info] Trying to create an Idea module demo
[info] Resolving org.scala-lang#scala-library;2.9.2 ...
[info] downloading http://repo1.maven.org/maven2/org/scala-lang/scala-library/2.9.2/scala-library-2.9.2-sources.jar ...
[info]  [SUCCESSFUL ] org.scala-lang#scala-library;2.9.2!scala-library.jar(src) (887ms)
[info] downloading http://repo1.maven.org/maven2/org/scala-lang/scala-library/2.9.2/scala-library-2.9.2-javadoc.jar ...
[info]  [SUCCESSFUL ] org.scala-lang#scala-library;2.9.2!scala-library.jar(doc) (5127ms)
[info] Excluding folder target
[info] Created C:\workspaces\pelssers\demo/.idea/IdeaProject.iml
[info] Created C:\workspaces\pelssers\demo\.idea
[info] Excluding folder C:\workspaces\pelssers\demo\target
[info] Created C:\workspaces\pelssers\demo\.idea_modules/demo.iml
[info] Created C:\workspaces\pelssers\demo\.idea_modules/demo-build.iml
>
And my colleague Ivan pointed me to this link from the guardian which covers the topic into more detail.

SBT Quick start - Part 1

Now that I have Scala and SBT working from GitBash we will setup our first demo project.
$ mkdir demo
$ cd demo
$ mkdir -p src/{main,test}/{java,scala,resources}

Create a script hw.scala and place it under demo/src/scala
/*********** hw.scala ***************/
object Hi {
  def main(args: Array[String]) = println("Demo application says HELLO!")
}

now run sbt -> Running sbt with no command line arguments starts it in interactive mode.
nxp10009@NXL01366 /c/workspaces/pelssers/demo
$ sbt
[info] Set current project to default-8388aa (in build file:/C:/workspaces/pelssers/demo/)
> exit  

We still have to create a build.sbt in the project root so let's do this first
name := "demo"

version := "1.0-SNAPSHOT"

scalaVersion := "2.9.2"

If we now rerun sbt again everything seems ok
$ sbt
[info] Set current project to demo (in build file:/C:/workspaces/pelssers/demo/)
> exit

You will now see sbt created a project folder for you. Here we will create a new build.properties file
nxp10009@NXL01366 /c/workspaces/pelssers/demo
$ ls -la
total 3
drwxr-xr-x    6 nxp10009 Administ        0 Dec  9 15:13 .
drwxr-xr-x    8 nxp10009 Administ     4096 Dec  9 15:04 ..
-rw-r--r--    1 nxp10009 Administ       70 Dec  9 15:04 build.sbt
drwxr-xr-x    3 nxp10009 Administ        0 Dec  9 15:13 project
drwxr-xr-x    4 nxp10009 Administ        0 Dec  9 15:05 src
drwxr-xr-x    3 nxp10009 Administ        0 Dec  9 15:12 target


$ cd project
$ vi build.properties


sbt.version=0.12.0
Enabling continous build and test
> ~ compile
[success] Total time: 0 s, completed Dec 9, 2012 3:32:39 PM
1. Waiting for source changes... (press enter to interrupt)

Now edit the hw.scala and make a little change:
object Hi {
  def main(args: Array[String]) = println("Demo application says HELLO again!")
}

[info] Compiling 1 Scala source to C:\workspaces\pelssers\demo\target\scala-2.9.2\classes...
[success] Total time: 1 s, completed Dec 9, 2012 3:33:02 PM
2. Waiting for source changes... (press enter to interrupt)

Getting Git Bash and Scala working on windows

I found a great blog post about fixing issues. I already avoid using spaces in directory names so I only ran into the last issue:
nxp10009@NXL01366 /c/development/scala/scala-2.9.2/bin
$ scala
Exception in thread "main" java.lang.NoClassDefFoundError: scala/tools/nsc/MainGenericRunner
Caused by: java.lang.ClassNotFoundException: scala.tools.nsc.MainGenericRunner
        at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
Could not find the main class: scala.tools.nsc.MainGenericRunner.  Program will exit.

So I only need to create a bash script and add an alias to fix the issue
nxp10009@NXL01366 ~
$ cd ~

nxp10009@NXL01366 ~
$ pwd
/c/Users/nxp10009

nxp10009@NXL01366 ~
$ vi .bashrc

alias scala='scala -nobootcp'

Now exit Git Bash and open a new shell:
nxp10009@NXL01366 ~
$ scala
Welcome to Scala version 2.9.2 (Java HotSpot(TM) 64-Bit Server VM, Java 1.6.0_32).
Type in expressions to have them evaluated.
Type :help for more information.

scala>

Thursday, December 6, 2012

Merging DITA maps and topics

This week I did a major data conversion. For about 2k products we generated DITA maps (each pointing to 3 topics). But many products had the same data so the generated topics had the same <body> tag. So we decided to first merge all topics. This also meant we had to rewrite the topicrefs in the maps. And next we could also merge the maps themselves if they had the same topicrefs.

One important lesson learned.. I first used timestamps for the merged files. It seemed like Saxon was able to merge 4 use cases in 1 millisecond so they ended up overwriting each other. So I quickly had to look for another alternative and switched to using the hashcode of the grouping-keys.

Example map:
<?xml version="1.0" encoding="utf-8"?>
<value-proposition id="vp_BC51-10PA" rev="001.001" title="Value proposition" xml:lang="en-US">
  <topicmeta translate="no">
    <subtitle translate="yes">45 V, 1 A PNP medium power transistor</subtitle>
    <prodinfo><prodname>BC51-10PA</prodname></prodinfo>
  </topicmeta>
  <technical-summary-ref href="technical-summary/ts_BC51-10PA.dita"/>
  <features-benefits-ref href="features-benefits/fb_BC51-10PA.dita"/>
  <target-applications-ref href="target-applications/ta_BC51-10PA.dita"/>
</value-proposition>

Example topic
<?xml version="1.0" encoding="utf-8"?>
<p-topic id="fb_BC51-10PA" rev="001.001" xml:lang="en-US">
  <title translate="no">Features and benefits</title>
  <prolog translate="no">...</prolog>
  <body>
    <ul>
      <li><p>High current</p></li>
      <li><p>Three current gain selections</p></li>
      <li><p>High power dissipation capability</p></li>
      <li><p>Exposed heatsink for excellent thermal and electrical conductivity</p></li>
      <li><p>Leadless very small SMD plastic package with medium power capability</p></li>
      <li><p>AEC-Q101 qualified</p></li>
    </ul>
  </body>
</p-topic>

I just am going to share the XSLT's that did the hard work to merge the topics and maps. I'm sure I can reuse the same approach in the future.
topicmerge.xslt
<?xml version="1.0" encoding="UTF-8"?>
<!--
Author: Robby Pelssers
This stylesheet will merge topics if they have the same body tag
-->

<xsl:stylesheet version="2.0"
  xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
  xmlns:xs="http://www.w3.org/2001/XMLSchema"
  xmlns:nxp="http://www.nxp.com">
  
  <xsl:output method="xml" version="1.0" encoding="UTF-8" indent="yes"/>

  <xsl:param name="exportFolder"/>
  <xsl:param name="subFolder"/>
  <xsl:variable name="folderTemplate" select="concat('file:///', $exportFolder, $subFolder, '/topic-type/?select=*.dita')"/>

  <xsl:variable name="featuresandbenefits" select="collection(replace($folderTemplate, 'topic-type', 'features-benefits'))"/>
  <xsl:variable name="technicalsummaries" select="collection(replace($folderTemplate, 'topic-type', 'technical-summary'))"/>
  <xsl:variable name="targetapplications" select="collection(replace($folderTemplate, 'topic-type', 'target-applications'))"/>

  <xsl:variable name="date-format" select="'[Y0001]-[M01]-[D01]T[h01]:[m01]:[s01]'"/>

  <xsl:function name="nxp:getHashCode">
    <xsl:param name="stringvalue" as="xs:string"/>
    <xsl:value-of select="string:hashCode($stringvalue)" xmlns:string="java:java.lang.String"/>
  </xsl:function>

  <!-- handles a logical group of documents (featuresandbenefits | technicalsummaries | targetapplications) -->
  <xsl:template name="mergeDocumentGroup">
    <xsl:param name="documents"/>
    <xsl:for-each-group select="$documents" group-by="p-topic/body">
      <xsl:call-template name="p-topic">
        <xsl:with-param name="topics" select="current-group()/p-topic"/>
        <xsl:with-param name="grouping_key"  select="current-grouping-key()"/>
      </xsl:call-template>
    </xsl:for-each-group>
  </xsl:template>

  <xsl:template match="/">
    <result>
      <xsl:call-template name="mergeDocumentGroup">
        <xsl:with-param name="documents" select="$featuresandbenefits"/>
      </xsl:call-template>
      <xsl:call-template name="mergeDocumentGroup">
        <xsl:with-param name="documents" select="$technicalsummaries"/>
      </xsl:call-template>
      <xsl:call-template name="mergeDocumentGroup">
        <xsl:with-param name="documents" select="$targetapplications"/>
      </xsl:call-template>
    </result>
  </xsl:template>


  <xsl:template name="p-topic">
    <xsl:param name="topics"/>
    <xsl:param name="grouping_key"/>
    <xsl:variable name="topic" select="$topics[1]"/>
    <p-topic>
      <xsl:choose>
        <xsl:when test="count($topics) > 1">
          <xsl:apply-templates select="$topic/@* | $topic/node()" mode="merge">
            <xsl:with-param name="grouping_key" select="$grouping_key" tunnel="yes"/>
          </xsl:apply-templates>
        </xsl:when>
        <xsl:otherwise>
          <xsl:apply-templates select="$topic/@* | $topic/node()"/>
        </xsl:otherwise>
      </xsl:choose>
      <!-- we temporarily add the original topic id's so we can easily alter the topicrefs in a subsequent transform -->
      <topics>
        <xsl:for-each select="$topics">
          <id><xsl:value-of select="./@id"/></id>
        </xsl:for-each>
      </topics>
    </p-topic>
  </xsl:template>

  <xsl:template match="p-topic/@id" mode="merge">
    <xsl:param name="grouping_key" tunnel="yes"/>
    <xsl:attribute name="id"
        select="concat(substring-before(., '_'), '_', translate(nxp:getHashCode($grouping_key), '-', ''))"/>
  </xsl:template>

    <!-- copy all nodes and attributes which are not processed by one of available templates -->
  <xsl:template match="@* | node()">
    <xsl:copy copy-namespaces="no">
      <xsl:apply-templates select="@*"/>
      <xsl:apply-templates/>
    </xsl:copy>
  </xsl:template>

  <xsl:template match="@* | node()" mode="merge">
    <xsl:copy copy-namespaces="no">
      <xsl:apply-templates select="@*" mode="merge"/>
      <xsl:apply-templates mode="merge"/>
    </xsl:copy>
  </xsl:template>


</xsl:stylesheet>

mapmerge.xslt
<?xml version="1.0" encoding="UTF-8"?>
<!--
Author: Robby Pelssers
This stylesheet will merge maps which have same topic refs and same title.
-->
<xsl:stylesheet version="2.0"
                xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
                xmlns:xs="http://www.w3.org/2001/XMLSchema"
                xmlns:nxp="http://www.nxp.com">
  
  <xsl:output method="xml" version="1.0" encoding="UTF-8" indent="yes"/>

  <xsl:variable name="date-format" select="'[Y0001]-[M01]-[D01]T[h01]:[m01]:[s01]'"/>

  <xsl:function name="nxp:getHashCode">
    <xsl:param name="stringvalue" as="xs:string"/>
    <xsl:value-of select="string:hashCode($stringvalue)" xmlns:string="java:java.lang.String"/>
  </xsl:function>

  <xsl:function name="nxp:getMapGroupingKey" as="xs:string">
    <xsl:param name="vp" as="element(value-proposition)"/>
    <xsl:sequence select="concat($vp/topicmeta/subtitle, $vp/technical-summary-ref/@href,
      $vp/features-benefits-ref/@href, $vp/target-applications-ref/@href)"/>
  </xsl:function>

  <xsl:template match="/">
    <xsl:apply-templates/>
  </xsl:template>

  <xsl:template match="result">
    <result>
      <xsl:apply-templates select="p-topic"/>
      <xsl:for-each-group select="value-proposition" group-by="nxp:getMapGroupingKey(.)">
        <xsl:call-template name="value-proposition">
          <xsl:with-param name="valuepropositions" select="current-group()"/>
          <xsl:with-param name="grouping_key"  select="current-grouping-key()"/>
        </xsl:call-template>
      </xsl:for-each-group>
    </result>
  </xsl:template>

  <xsl:template name="value-proposition">
    <xsl:param name="valuepropositions"/>
    <xsl:param name="grouping_key"/>
    <xsl:variable name="vp" select="$valuepropositions[1]"/>
    <value-proposition>
      <xsl:choose>
        <xsl:when test="count($valuepropositions) > 1">
          <xsl:apply-templates select="$vp/@* | $vp/node()" mode="merge">
            <xsl:with-param name="valuepropositions" select="$valuepropositions" tunnel="yes"/>
            <xsl:with-param name="grouping_key" select="$grouping_key" tunnel="yes"/>
          </xsl:apply-templates>
        </xsl:when>
        <xsl:otherwise>
          <xsl:apply-templates select="$vp/@* | $vp/node()"/>
        </xsl:otherwise>
      </xsl:choose>
    </value-proposition>
  </xsl:template>


  <xsl:template match="value-proposition/@id" mode="merge">
    <xsl:param name="grouping_key" tunnel="yes"/>
    <xsl:attribute name="id"
         select="concat(substring-before(., '_'), '_', translate(nxp:getHashCode($grouping_key), '-', ''))"/>
  </xsl:template>

  <!-- copy all nodes and attributes which are not processed by one of available templates -->
  <xsl:template match="@* | node()">
    <xsl:copy copy-namespaces="no">
      <xsl:apply-templates select="@*"/>
      <xsl:apply-templates/>
    </xsl:copy>
  </xsl:template>

  <xsl:template match="@* | node()" mode="merge">
    <xsl:copy copy-namespaces="no">
      <xsl:apply-templates select="@*" mode="merge"/>
      <xsl:apply-templates mode="merge"/>
    </xsl:copy>
  </xsl:template>


</xsl:stylesheet>