Hadoop - OutOfMemoryError: Java heap space

I was writing a hadoop job which processes many files and creates multiple files from each file. I was using "MultipleOutputs" to write them. It worked fine for a small number of files but I was getting the following error for large number of files. I tried increasing the ulimit and -Xmx but to no avail.

2013-01-15 13:44:05,154 FATAL org.apache.hadoop.mapred.Child: Error running child : java.lang.OutOfMemoryError: Java heap space
    at org.apache.hadoop.hdfs.DFSOutputStream$Packet.(DFSOutputStream.java:201)
    at org.apache.hadoop.hdfs.DFSOutputStream.writeChunk(DFSOutputStream.java:1423)
    at org.apache.hadoop.fs.FSOutputSummer.writeChecksumChunk(FSOutputSummer.java:161)
    at org.apache.hadoop.fs.FSOutputSummer.flushBuffer(FSOutputSummer.java:136)
    at org.apache.hadoop.fs.FSOutputSummer.flushBuffer(FSOutputSummer.java:125)
    at org.apache.hadoop.fs.FSOutputSummer.write1(FSOutputSummer.java:116)
    at org.apache.hadoop.fs.FSOutputSummer.write(FSOutputSummer.java:90)
    at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOutputStream.java:54)
    at java.io.DataOutputStream.write(DataOutputStream.java:90)
    at org.apache.hadoop.mapreduce.lib.output.TextOutputFormat$LineRecordWriter. writeObject( TextOutputFormat.java:78)
    at org.apache.hadoop.mapreduce.lib.output.TextOutputFormat$LineRecordWriter. write(TextOutputFormat.java:99)
    **at org.apache.hadoop.mapreduce.lib.output.MultipleOutputs.write( MultipleOutputs.java:386)
    at com.demoapp.collector.MPReducer.reduce(MPReducer.java:298)
    at com.demoapp.collector.MPReducer.reduce(MPReducer.java:28)**
    at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:164)
    at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:595)
    at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:433)
    at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:396)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1332)
    at org.apache.hadoop.mapred.Child.main(Child.java:262)

Solution:
I used the following configuration values to resolve it-
OPTS="-Dmapred.reduce.tasks=8 -Dio.sort.mb=640 -Dmapred.task.timeout=1200000"
hadoop jar ${JAR} ${OPTS} -src ${SRC} -dest ${DST}

iOS 6 - Apple maps needs to find a shorter route to the correct destination

We all have heard a lot about Apple Maps lately. The verdict is already out. A forgettable job by Apple. The last full update(Iphone 4) had the dropped signal issue and now it is the maps. I usually don't like to get involved in the Apple/Android war but this was one incident, I couldn't resist writing about.
Yes, I have an iPhone. It is an iPhone 4 and I recently updated to iOS 6. I am on AT&T and want to move to T-mobile. I am eligible for an upgrade but I will skip the upgrade this time for reasons I would like to keep out of this blog entry. So on a Sunday morning, when I am finished buying groceries at the nearby Lucky's, the first thing I do is search for a T-mobile store on my new Apple maps and this is what it shows-

So I start driving and I find another Lucky's supermarket at my destination and no T-mobile store anywhere close. That was such a shame.

So tried to search for the same route from Google maps and this is what I get-

Interestingly, the place is in an opposite direction all together.
Then I tried to move the destination pin to the location where Apple maps was locating the T-mobile store and this is what I get-

So it looks like there is not a problem with locating the address alone, but also the route taken. Google maps takes me via a simpler route- a U turn and a right turn but Apple maps goes through a series of twists and turns, getting on the highway, changing lanes etc. The route by Google maps is also shorter 3.2 miles (to the wrong destination of course)

I am not an expert, but I guess Apple needs to map the correct co-ordinates to the expected destination and also debug those algorithms for shortest route possible. It will be a while before I trust Apple maps. Meanwhile, I will use my secondary phone, Nokia N8 for my navigation needs.

On a side note, Nokia 920 does sound appealing to me.

GWT - Cookie provided by RPC doesn't match request cookie

Even after setting the security cookie on client and server-
bindConstant().annotatedWith(SecurityCookie.class).to("mycookie");

I was getting the following exception-

Sep 6, 2012 4:09:40 PM com.gwtplatform.dispatch.server.AbstractDispatchServiceImpl cookieMatch
INFO: No cookie sent by client in RPC. (Did you forget to bind the security cookie client-side? Or it could be an attack.)
Sep 6, 2012 4:09:40 PM com.gwtplatform.dispatch.server.AbstractDispatchServiceImpl execute
SEVERE: Cookie provided by RPC doesn't match request cookie, aborting action, possible XSRF attack. (Maybe you forgot to set the security cookie?) While executing action: com.company.appmodule.client.Login

Solution: Clear cache and cookies from browser.

GWT Celltable - Add multiple anchors to a cell

I am trying to insert multiple anchors in a single celltable cell. This is how it would look like-

The cars column has multiple cars, each car link can be clicked to perform different actions.
I have extended a MultipleAnchorCell class from AbstractSafeHtmlCell like this-



And then I use this cell in a celltable to create a column. My User class contains comma separated values of cars in a class variable.

      

       

I am still learning GWT and it took me a while to figure this out.

Comments/feedback is welcome.
===============================================================

GWT dygraph example using visualization datatable

Here is a quick example of how to create a dygraph using GWT and visualization api to get the datatable.

1. Add dygraph-gwt.jar to your build path and these inherits tags in your mysamplegwtproj.gwt.xml file

<inherits name="com.google.gwt.visualization.Visualization"/>
<inherits name="org.danvk.dygraphs"/>

2. Add the following script tag in mysamplegwtproject.html file

<script type="text/javascript" src="dygraph-combined.js">
<script type="text/javascript" src="http://www.google.com/jsapi">
3. Add the following code in your onModuleLoad() function-

        final VerticalPanel vp = new VerticalPanel();
        vp.setWidth("700px");
        vp.setHeight("700px");
        VisualizationUtils.loadVisualizationApi(new Runnable() {
            @Override
            public void run() {
                Query q = Query
                        .create("mysampleproj/getMyDatatable?params=2&id=1");

                q.send(new Callback() {
                    @Override
                    public void onResponse(QueryResponse response) {
                        JavaScriptObject jdygraph = createDygraph(
                                vp.getElement(), response.getDataTable(), 0,
                                100);
                    }
                });
            }
        }, LineChart.PACKAGE);

        RootPanel.get().add(vp);

4. Here is the JSNI function which actually creates the dygraph.




public static native JavaScriptObject createDygraph(Element element,
            DataTable dataTable, double minY, double maxY) /*-{
        var chart = new $wnd.Dygraph.GVizChart(element);
        chart.draw(dataTable, {
            valueRange : [ minY, maxY ],
            displayAnnotations : true,
            showRangeSelector : true,
            legend : 'always',
            labelsDivStyles : {
                'textAlign' : 'right'
            },
            title : 'TITLE',
            titleHeight : 25,            
            axes : {
                x : {
                    pixelsPerLabel : 50
                }
            }            
        });
        return chart;
    }-*/;




Hope this helps. Suggestions/improvements are welcome.

===============================================================
 

NoClassDefFoundError: com.google.common.collect.Sets

Error-
NoClassDefFoundError: "com.google.common.collect.Sets"

Solution-
Add guava-11.0.2.jar to WEB-INF/lib and add it to the build path.

GWT wrapper for visualization treemap with mouse events

A treemap is a helpful visualization of a data tree. It would be nice if the treemap had features like weighted box sizes for intermediate levels and explicitly setting up colors for non-leaf nodes. I wrote a  quick Google Web Toolkit(GWT) wrapper for Google visualization treemap-

 public class TreeMap extends Visualization<TreeMap.Options>
{
   public static class Options extends AbstractDrawOptions {
       public static Options create() {
           return JavaScriptObject.createObject().cast();
       }

       protected Options() { }
   }

   public static final String PACKAGE = "treemap";

   public TreeMap() {
       super();
   }

   public TreeMap(AbstractDataTable data, Options options) {
       super(data, options);
   }

   @Override
   protected native JavaScriptObject createJso(Element parent) /*-{
       return new $wnd.google.visualization.TreeMap(parent);                
   }-*/;
   
   public final void addOnMouseOverHandler(OnMouseOverHandler handler) {
       Handler.addHandler(this, "onmouseover", handler);
   }
   
   public final void addOnMouseOutHandler(OnMouseOutHandler handler) {
       Handler.addHandler(this, "onmouseout", handler);
   }
}