textutil in Mac OS X

Lynx had a useful option to convert html pages to text. 

lynx -dump input.html > output.txt

And I miss that on Mac OSX. I stumbled upon an alternative for this - textutil. It comes bundled with MAC OS X. Converting an html file to text can be done in this way:

textutil -convert txt ~/Downloads/your-fav-profile.html

There should be a file created: ~/Downloads/your-fav-profile.txt

RTFM. Textutil is a utility extracted from the Cocoa text system 

Pushing Docker containers to more than one registry using Gradle

In the early days, we had to maintain two docker registries, one in AWS us-east-1 and the other in eu-west-1. This required us to build our containers and then push it to both registries. Below is a build.grade using Ben’s Gradle-Docker plugin to do the following:

1. Builds a docker container based on the Dockerfile in the root directory.
2. Tag and push to both repos (us-east-1 and eu-west-1).

The root directory (project folder) should contain your Dockerfile and build.gradle.

This is how the build.grade looks like:


Note that you can also build this on Jenkins.

S3's Eventual Consistency

If you are working with AWS services, particularly S3, you might have encountered S3's eventual consistency model (or you'll eventually encounter it). This post here describes concisely the eventual consistency issue. We have similar systems, where data is published to S3 and then notified to an online system about its availability using a messaging bus. There have been occasions when the message has been consumed by an online service to later discover that the S3 data was still not accessible.

Collections vs Iterables: Java

One important note while considering Collections vs Iterables is that Collections in some sense assume that you have loaded the data into memory and have access to the size(). On the other hand, Iterables allow for lazy evaluation of the data. This becomes an important considerable when you are working with large scale data just being loaded from a database. In a sense, it allows you “stream” the data.

Check this Guava link here:
https://code.google.com/p/google-collections/wiki/Faq#Why_so_much_emphasis_on_Iterators_and_Iterables?

Here is another good discussion:
http://stackoverflow.com/questions/1159797/when-should-i-use-iterable-vs-collection-in-java

  

Getting Docker to work on OS X

Chris Jones has an excellent resource on getting started with Docker on MAC OS X. You can access it here:
http://viget.com/extend/how-to-use-docker-on-os-x-the-missing-guide

My only deviation in the above guide is to install boot2docker from here:
https://github.com/boot2docker/osx-installer/releases

Everything else should work fine for you.

Accessing insecure private repos:

I have had some issues accessing insecure docker registries. The error is usually something like this:
laptop-rmenezes:$ docker push dockerregistry.dynprod.roger-menezes.net:7001/minions_ubuntubase-x86_64-20140910-ebs

2014/11/07 16:06:17 Error: Invalid registry endpoint https://dockerregistry.dynprod.roger-menezes.net:7001/v1/: Get https://dockerregistry.dynprod.roger-menezes.net:7001/v1/_ping: tls: oversized record received with length 20527. If this private registry supports only HTTP or HTTPS with an unknown CA certificate, please add `--insecure-registry dockerregistry.dynprod.roger-menezes.net:7001` to the daemon's arguments. In the case of HTTPS, if you have access to the registry's CA certificate, no need for the flag; simply place the CA certificate at /etc/docker/certs.d/dockerregistry.dynprod.roger-menezes.net:7001/ca.crt

The fix over there is change the following file inside the boot2docker vm:
/var/lib/boot2docker/profile

boot2docker ssh
sudo vi /var/lib/boot2docker/profile

You need to add the following line to this fine:
EXTRA_ARGS="  --insecure-registry dockerregistry.eu-west-1.dyntest.roger-menezes.net:7001 --insecure-registry dockerregistry.dynprod.roger-menezes.net:7001  "

The above is an example that you can add multiple registries the above format.

Then you just need to do:
boot2docker up
boot2docker down

=============

Running a Docker command:
sudo docker run -v /mnt/data:/data -e "ENVIRONMENT=prod" -m 39g  d34ca1989ab7 /apps/python/bin/JobRunner.py  --config params.email=rmenezes@chakde.com

-v is used to mount a Machine directory onto a docker directory.
-e is used to pass environment variables.
-m tells how much memory to use
d34ca1989ab7 is the image id
the rest is the command