Moving from boot2docker to Docker Toolbox

Docker Toolbox comes from Docker, Inc as an improved way of creating your docker containers. I believe it uses boot2docker underneath. Here are quick instructions on getting set up:

1. Download Docker Toolbox from here: [Choose the Mac download].
2. Go ahead with the installation as suggested here.
3. While verifying the installation, you can also run the following:
• . '/Applications/Docker/Docker Quickstart Terminal.app/Contents/Resources/Scripts/start.sh'
• You can either run the above script or just run "Docker QuickStart Terminal" via spotlight.

Now to get started, run the following:

• docker-machine start default
• the above will start the vm
• eval "$(docker-machine env default)"
• the above will set the needed env variables
• docker images
• the image listing should appear after that.

The equivalent of "boot2docker ssh" is now "docker-machine ssh default". You'll need to do this to set up insecure registries, etc. After updating the proflile file (/var/lib/boot2docker/profile), you can restart the vm by "docker-machine restart default".

textutil in Mac OS X

Lynx had a useful option to convert html pages to text. 

lynx -dump input.html > output.txt

And I miss that on Mac OSX. I stumbled upon an alternative for this - textutil. It comes bundled with MAC OS X. Converting an html file to text can be done in this way:

textutil -convert txt ~/Downloads/your-fav-profile.html

There should be a file created: ~/Downloads/your-fav-profile.txt

RTFM. Textutil is a utility extracted from the Cocoa text system 

Pushing Docker containers to more than one registry using Gradle

In the early days, we had to maintain two docker registries, one in AWS us-east-1 and the other in eu-west-1. This required us to build our containers and then push it to both registries. Below is a build.grade using Ben’s Gradle-Docker plugin to do the following:

1. Builds a docker container based on the Dockerfile in the root directory.
2. Tag and push to both repos (us-east-1 and eu-west-1).

The root directory (project folder) should contain your Dockerfile and build.gradle.

This is how the build.grade looks like:


Note that you can also build this on Jenkins.

S3's Eventual Consistency

If you are working with AWS services, particularly S3, you might have encountered S3's eventual consistency model (or you'll eventually encounter it). This post here describes concisely the eventual consistency issue. We have similar systems, where data is published to S3 and then notified to an online system about its availability using a messaging bus. There have been occasions when the message has been consumed by an online service to later discover that the S3 data was still not accessible.

Collections vs Iterables: Java

One important note while considering Collections vs Iterables is that Collections in some sense assume that you have loaded the data into memory and have access to the size(). On the other hand, Iterables allow for lazy evaluation of the data. This becomes an important considerable when you are working with large scale data just being loaded from a database. In a sense, it allows you “stream” the data.

Check this Guava link here:
https://code.google.com/p/google-collections/wiki/Faq#Why_so_much_emphasis_on_Iterators_and_Iterables?

Here is another good discussion:
http://stackoverflow.com/questions/1159797/when-should-i-use-iterable-vs-collection-in-java

  

Getting Docker to work on OS X

Chris Jones has an excellent resource on getting started with Docker on MAC OS X. You can access it here:
http://viget.com/extend/how-to-use-docker-on-os-x-the-missing-guide

My only deviation in the above guide is to install boot2docker from here:
https://github.com/boot2docker/osx-installer/releases

Everything else should work fine for you.

Accessing insecure private repos:

I have had some issues accessing insecure docker registries. The error is usually something like this:
laptop-rmenezes:$ docker push dockerregistry.dynprod.roger-menezes.net:7001/minions_ubuntubase-x86_64-20140910-ebs

2014/11/07 16:06:17 Error: Invalid registry endpoint https://dockerregistry.dynprod.roger-menezes.net:7001/v1/: Get https://dockerregistry.dynprod.roger-menezes.net:7001/v1/_ping: tls: oversized record received with length 20527. If this private registry supports only HTTP or HTTPS with an unknown CA certificate, please add `--insecure-registry dockerregistry.dynprod.roger-menezes.net:7001` to the daemon's arguments. In the case of HTTPS, if you have access to the registry's CA certificate, no need for the flag; simply place the CA certificate at /etc/docker/certs.d/dockerregistry.dynprod.roger-menezes.net:7001/ca.crt

The fix over there is change the following file inside the boot2docker vm:
/var/lib/boot2docker/profile

boot2docker ssh
sudo vi /var/lib/boot2docker/profile

You need to add the following line to this fine:
EXTRA_ARGS="  --insecure-registry dockerregistry.eu-west-1.dyntest.roger-menezes.net:7001 --insecure-registry dockerregistry.dynprod.roger-menezes.net:7001  "

The above is an example that you can add multiple registries the above format.

Then you just need to do:
boot2docker up
boot2docker down

=============

Running a Docker command:
sudo docker run -v /mnt/data:/data -e "ENVIRONMENT=prod" -m 39g  d34ca1989ab7 /apps/python/bin/JobRunner.py  --config params.email=rmenezes@chakde.com

-v is used to mount a Machine directory onto a docker directory.
-e is used to pass environment variables.
-m tells how much memory to use
d34ca1989ab7 is the image id
the rest is the command

Scala Http Client

I have started using scalaj for writing REST client side calls in Scala. The Clever Cloud team has a convenient tool to generate Scala case classes from json.

Alternative for Fiddler on Mac OS

I miss Fiddler2 very much on Mac OS but there is a decent alternative to it. Open up Chrome and type:
chrome://net-internals in the address bar. Go to Events in the drop down box.

No Visio on Macs :-(

Started working on Macs recently and got to know that there's no Visio on Mac :-(. Visio is my favorite tool for all things drawn - UML diagrams, architecture diagrams, everything. The flexibility and the breath of what's possible in Visio is enormous. I stumbled for sometime to realize a block diagram in Skitch but then finally gave up. OmniGraffle is a good alternative. Still a learning curve but very close to Visio.

Signing PDF docs using Adobe Reader

Adobe Reader now provides a way to place your handwritten signature on PDF docs. This has saved me a ton of time (avoind the following workflow....print doc => sign => scan)

1. Create a signature using Penultimate iPad app.
2. Save it as a photo and email it to you.
3. Open PDF in Adobe Reader X
4. Click on the Signature icon...This opens up a tab on the right side
5. Click on Place Signature - this will give you the option of either scribbling a signature or opening an image
6. Import your Penultimate Signature....
7. Now place your signature wherever you want in the document.

git conflict resolution using p4merge

Documented here - Evernote Note link 

Sending books to your Kindle

Found this article that introduced me to the SendToKindle desktop app from Amazon. This is the effective way of having your non Amazon books synced to your Kindle cloud account, so that all of your devices can have them, rather than side loading them. There are versions for PC and Mac.

Additionally, you can use Calibre to convert PDF docs to .mobi before sending them to your Kindle account. 

Outputting JMX Monitoring Points

If you need to programmatically access all JMX monitoring points for your Servlet app, do the following:

1. Add the following VM arguments to the Tomcat Run Configuration (This can be done in Eclipse/STS run configurations):

-Dcom.sun.management.jmxremote
-Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.port=8234
-Dcom.sun.management.jmxremote.ssl=false

2. Then run the following Java Program:

package com.rim.platform.mdm.core.service.email;

import java.io.IOException;
import java.util.Set;

import javax.management.InstanceNotFoundException;
import javax.management.IntrospectionException;
import javax.management.MBeanAttributeInfo;
import javax.management.MBeanInfo;
import javax.management.MBeanServerConnection;
import javax.management.MalformedObjectNameException;
import javax.management.ObjectInstance;
import javax.management.ReflectionException;
import javax.management.remote.JMXConnector;
import javax.management.remote.JMXConnectorFactory;
import javax.management.remote.JMXServiceURL;

public class MonitoringPoints
{

    public MonitoringPoints()
    {
    }

    /**
     * @param args
     * @throws Exception
     */
    public static void main(String[] args) throws Exception
    {
        String host = "localhost"; 
        int port = 8234;
        String url = "service:jmx:rmi:///jndi/rmi://" + host + ":" + port + "/jmxrmi";
        printAll(url);
    }

    public static void printAll(String url) throws IOException, MalformedObjectNameException, 
            InstanceNotFoundException, IntrospectionException, ReflectionException
    {
        JMXServiceURL serviceUrl = new JMXServiceURL(url);
        JMXConnector jmxConnector = JMXConnectorFactory.connect(serviceUrl, null);
        try
        {
            MBeanServerConnection mbeanConn = jmxConnector.getMBeanServerConnection();
            // now query to get the beans or whatever
            Set beanSet = mbeanConn.queryMBeans(null, null);
            for (ObjectInstance instance : beanSet)
            {
                MBeanInfo info = mbeanConn.getMBeanInfo(instance.getObjectName());
                MBeanAttributeInfo[] mai = info.getAttributes();
                System.out.println("\n******* Monitoring points for: " + instance.getObjectName());
                for (MBeanAttributeInfo object : mai)
                {
                    System.out.println(object.getName());
                }
            }
        }
        finally
        {
            jmxConnector.close();
        }
    }
}

Content assist for static imports in Eclipse

Take a look at the following link on how you can enable auto complete in Eclipse for static imports:
http://stackoverflow.com/questions/288861/eclipse-optimize-imports-to-include-static-imports

This is very helpful when you are writing JUnit and Mockito code. Below is the relevant part of the StackOverflow answer: 

Partial implementation of Interface

Java Stuff: If you are implementing an interface and planning on not supporting all the methods, it a good practice to do the following for not implemented methods:

        throw new org.apache.commons.lang.NotImplementedException.NotImplementedException(
                  "method not implemented by This class");

More details here:
http://stackoverflow.com/questions/1062937/java-equivalent-to-nets-notsupportedexception

Git Tricks

I started maintaining a new page at Zoho docs to document my git learning:

Code highlighting in Blogger

There are helpful instructions on the link below for high-lighting code in your blogger posts:
http://lukabloga.blogspot.com/2008/10/to-test-new-highlighting.html

UPDATE: I found GitHub Gist to be a faster and elegant alternative

There are other alternatives here:
http://stackoverflow.com/q/679189/747479

Stack Overflow rocks!

Dealing with International Characters in Web Services [Java]

Recently, we had to develop an API where data (resource in RESTful terms) gets created with an emailId as a primary key and then a subsequent lookup is performed to access that resource.

There is an RFC that allows email ids to have international characters:

http://tools.ietf.org/html/rfc6530

e.g. Rδοκιμή@παράδειγμα.δοκιμή

More details: http://en.wikipedia.org/wiki/E-mail_address#Internationalization

To support international characters in our API, we had to do the following:

We develop our web services in Java Servlets using the Spring Framework and deploy them using Tomcat. The Create (resource) API had the emailId in the requestBody and the lookup API was a HTTP GET call with the emailId in the URL. 

To get the Create (resource API) working, the client needs to make the request with the charset parameter in the ContentType header:

Content-Type: application/json; charset=utf-8

This instructs the Servlet container to treat the requestBody as an UTF8 encoded string. Without the charset parameter, the encoding is assumed to be ISO-8859-1 (default for Java Servlets). If you ever have to store the string in a DB or encrypt it, remember to retrieve the string in UTF8 encoding. String.getBytes("UTF-8") like stuff.

Now for the lookup API, where the emailId is in the URI, you need to keep two things in mind:

1. Special characters in URIs are percent encoded.  For a non-ASCII character, it is typically converted to its byte sequence in UTF-8, and then each byte value is percent encoded.
2. Instruct the Servlet container about the UTF8 URI encoding. This can be done by updating the HTTP connector in server.xml:
   

   <Connector connectionTimeout="20000" port="8080" 
   protocol="HTTP/1.1" redirectPort="8443" URIEncoding="UTF-8"/>
   

For additional insight, the below StackOverflow link has great nuggets of knowledge:

http://stackoverflow.com/questions/138948/how-to-get-utf-8-working-in-java-webapps/138950#138950

I did try the CharacterEncodingFilter filter approach but didn't get it to work. I did set it up as the first filter, also did the force encoding bit wasn't doing anything.

JUnit: test if a method throws correct exception

How do you write a JUnit test to check if it throws the correct exception:

You can add the following to the annotated test:
@Test(expected=RequestValidationException.class)

1:  @Test(expected=RequestValidationException.class)  
2:          public void testValidateHeader_contentTypeInvalidValue() throws Exception  
3:          {      
4:              EasyMock.expect(servletRequest.getHeader("content-type")).  
5:                 andReturn("application/xml;charset=utf8");      
6:              EasyMock.replay(servletRequest, encryptionHelper);  
7:              Whitebox.invokeMethod(validator, "validateHeaders", servletRequest);  
8:          }  

Got to know it from here:
http://www.junit.org/node/390

Additional JUnit tip: WhiteBox comes from org.powermock.reflect.Whitebox and is used for invoking private methods

Integration tests for web services with Maven

There is a very nifty Maven plugin, called Maven Jetty plugin through which you can set up quick integration tests as part of your build. This is very valuable to catch erroneous check-ins that cause your code to compile and unit tests to pass but still can induce service start up failures.

With this plug in, you can start up a Servlet container (Jetty) and run your web service in it, as part of your build, in the pre-integration test phase. A sample configuration is pasted below. The following goes in the tag of your Maven pom.xml:

     <plugins>  
             <plugin>  
                 <groupId>org.apache.maven.plugins</groupId>  
                 <artifactId>maven-failsafe-plugin</artifactId>  
                 <version>2.9</version>  
                 <executions>  
                     <execution>  
                         <goals>  
                             <goal>integration-test</goal>  
                             <goal>verify</goal>  
                         </goals>  
                     </execution>  
                 </executions>  
             </plugin>  
             <plugin>  
                 <groupId>org.mortbay.jetty</groupId>  
                 <artifactId>jetty-maven-plugin</artifactId>  
                 <version>7.4.4.v20110707</version>  
                 <configuration>  
                     <connectors>  
                         <connector implementation="org.eclipse.jetty.server.nio.SelectChannelConnector">  
                             <port>9099</port>  
                             <maxIdleTime>60000</maxIdleTime>  
                         </connector>  
                     </connectors>  
                     <stopKey>foo</stopKey>  
                     <stopPort>9999</stopPort>  
                     <!-- webXml>src/main/resources/override-web.xml</webXml> -->  
                     <webAppConfig>  
                         <contextPath>/facebookConnectServlet</contextPath>                          
                     </webAppConfig>                      
                 </configuration>  
                 <executions>  
                     <execution>  
                         <id>start-jetty</id>  
                         <phase>pre-integration-test</phase>  
                         <goals>  
                             <goal>run</goal>  
                         </goals>  
                         <configuration>  
                             <scanIntervalSeconds>0</scanIntervalSeconds>  
                             <daemon>true</daemon>  
                         </configuration>  
                     </execution>  
                     <execution>  
                         <id>stop-jetty</id>  
                         <phase>post-integration-test</phase>  
                         <goals>  
                             <goal>stop</goal>  
                         </goals>  
                     </execution>  
                 </executions>  
             </plugin>  
         </plugins>  

The first plugin, Maven Failsafe Plugin, is designed to run integration tests. The second plug in, Maven Jetty Plugin, is configured subsequently to run Jetty in the pre-integration test phase and stop Jetty in the post integration test phase.

You can then make smoke test level requests to your web service, in the integration test phase, to check if the basic functionality works.