Getting multipart-form-data request parameters in java servlet-
Friday, February 1, 2013
Monday, January 21, 2013
Hadoop - OutOfMemoryError: Java heap space
I was writing a hadoop job which processes many files and creates multiple files from each file. I was using "MultipleOutputs" to write them. It worked fine for a small number of files but I was getting the following error for large number of files. I tried increasing the ulimit and -Xmx but to no avail.
2013-01-15 13:44:05,154 FATAL org.apache.hadoop.mapred.Child: Error running child : java.lang.OutOfMemoryError: Java heap space
at org.apache.hadoop.hdfs.DFSOutputStream$Packet.(DFSOutputStream.java:201)
at org.apache.hadoop.hdfs.DFSOutputStream.writeChunk(DFSOutputStream.java:1423)
at org.apache.hadoop.fs.FSOutputSummer.writeChecksumChunk(FSOutputSummer.java:161)
at org.apache.hadoop.fs.FSOutputSummer.flushBuffer(FSOutputSummer.java:136)
at org.apache.hadoop.fs.FSOutputSummer.flushBuffer(FSOutputSummer.java:125)
at org.apache.hadoop.fs.FSOutputSummer.write1(FSOutputSummer.java:116)
at org.apache.hadoop.fs.FSOutputSummer.write(FSOutputSummer.java:90)
at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOutputStream.java:54)
at java.io.DataOutputStream.write(DataOutputStream.java:90)
at org.apache.hadoop.mapreduce.lib.output.TextOutputFormat$LineRecordWriter. writeObject( TextOutputFormat.java:78)
at org.apache.hadoop.mapreduce.lib.output.TextOutputFormat$LineRecordWriter. write(TextOutputFormat.java:99)
**at org.apache.hadoop.mapreduce.lib.output.MultipleOutputs.write( MultipleOutputs.java:386)
at com.demoapp.collector.MPReducer.reduce(MPReducer.java:298)
at com.demoapp.collector.MPReducer.reduce(MPReducer.java:28)**
at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:164)
at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:595)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:433)
at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1332)
at org.apache.hadoop.mapred.Child.main(Child.java:262)
Solution:
I used the following configuration values to resolve it-
OPTS="-Dmapred.reduce.tasks=8 -Dio.sort.mb=640 -Dmapred.task.timeout=1200000"
hadoop jar ${JAR} ${OPTS} -src ${SRC} -dest ${DST}
2013-01-15 13:44:05,154 FATAL org.apache.hadoop.mapred.Child: Error running child : java.lang.OutOfMemoryError: Java heap space
at org.apache.hadoop.hdfs.DFSOutputStream$Packet.
at org.apache.hadoop.hdfs.DFSOutputStream.writeChunk(DFSOutputStream.java:1423)
at org.apache.hadoop.fs.FSOutputSummer.writeChecksumChunk(FSOutputSummer.java:161)
at org.apache.hadoop.fs.FSOutputSummer.flushBuffer(FSOutputSummer.java:136)
at org.apache.hadoop.fs.FSOutputSummer.flushBuffer(FSOutputSummer.java:125)
at org.apache.hadoop.fs.FSOutputSummer.write1(FSOutputSummer.java:116)
at org.apache.hadoop.fs.FSOutputSummer.write(FSOutputSummer.java:90)
at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOutputStream.java:54)
at java.io.DataOutputStream.write(DataOutputStream.java:90)
at org.apache.hadoop.mapreduce.lib.output.TextOutputFormat$LineRecordWriter. writeObject( TextOutputFormat.java:78)
at org.apache.hadoop.mapreduce.lib.output.TextOutputFormat$LineRecordWriter. write(TextOutputFormat.java:99)
**at org.apache.hadoop.mapreduce.lib.output.MultipleOutputs.write( MultipleOutputs.java:386)
at com.demoapp.collector.MPReducer.reduce(MPReducer.java:298)
at com.demoapp.collector.MPReducer.reduce(MPReducer.java:28)**
at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:164)
at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:595)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:433)
at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1332)
at org.apache.hadoop.mapred.Child.main(Child.java:262)
Solution:
I used the following configuration values to resolve it-
OPTS="-Dmapred.reduce.tasks=8 -Dio.sort.mb=640 -Dmapred.task.timeout=1200000"
hadoop jar ${JAR} ${OPTS} -src ${SRC} -dest ${DST}
Labels:
hadoop,
heap,
java,
multipleoutputs,
outofmemory,
space
Location:
Mountain View, CA, USA
Tuesday, September 25, 2012
iOS 6 - Apple maps needs to find a shorter route to the correct destination
We all have heard a lot about Apple Maps lately. The verdict is already out. A forgettable job by Apple. The last full update(Iphone 4) had the dropped signal issue and now it is the maps. I usually don't like to get involved in the Apple/Android war but this was one incident, I couldn't resist writing about.
Yes, I have an iPhone. It is an iPhone 4 and I recently updated to iOS 6. I am on AT&T and want to move to T-mobile. I am eligible for an upgrade but I will skip the upgrade this time for reasons I would like to keep out of this blog entry. So on a Sunday morning, when I am finished buying groceries at the nearby Lucky's, the first thing I do is search for a T-mobile store on my new Apple maps and this is what it shows-
So I start driving and I find another Lucky's supermarket at my destination and no T-mobile store anywhere close. That was such a shame.
So tried to search for the same route from Google maps and this is what I get-
Interestingly, the place is in an opposite direction all together.
Then I tried to move the destination pin to the location where Apple maps was locating the T-mobile store and this is what I get-
So it looks like there is not a problem with locating the address alone, but also the route taken. Google maps takes me via a simpler route- a U turn and a right turn but Apple maps goes through a series of twists and turns, getting on the highway, changing lanes etc. The route by Google maps is also shorter 3.2 miles (to the wrong destination of course)
I am not an expert, but I guess Apple needs to map the correct co-ordinates to the expected destination and also debug those algorithms for shortest route possible. It will be a while before I trust Apple maps. Meanwhile, I will use my secondary phone, Nokia N8 for my navigation needs.
On a side note, Nokia 920 does sound appealing to me.
Yes, I have an iPhone. It is an iPhone 4 and I recently updated to iOS 6. I am on AT&T and want to move to T-mobile. I am eligible for an upgrade but I will skip the upgrade this time for reasons I would like to keep out of this blog entry. So on a Sunday morning, when I am finished buying groceries at the nearby Lucky's, the first thing I do is search for a T-mobile store on my new Apple maps and this is what it shows-
So I start driving and I find another Lucky's supermarket at my destination and no T-mobile store anywhere close. That was such a shame.
So tried to search for the same route from Google maps and this is what I get-
Interestingly, the place is in an opposite direction all together.
Then I tried to move the destination pin to the location where Apple maps was locating the T-mobile store and this is what I get-
So it looks like there is not a problem with locating the address alone, but also the route taken. Google maps takes me via a simpler route- a U turn and a right turn but Apple maps goes through a series of twists and turns, getting on the highway, changing lanes etc. The route by Google maps is also shorter 3.2 miles (to the wrong destination of course)
I am not an expert, but I guess Apple needs to map the correct co-ordinates to the expected destination and also debug those algorithms for shortest route possible. It will be a while before I trust Apple maps. Meanwhile, I will use my secondary phone, Nokia N8 for my navigation needs.
On a side note, Nokia 920 does sound appealing to me.
Location:
Mountain View, CA, USA
Thursday, September 6, 2012
GWT - Cookie provided by RPC doesn't match request cookie
Even after setting the security cookie on client and server-
bindConstant().annotatedWith(SecurityCookie.class).to("mycookie");
I was getting the following exception-
Sep 6, 2012 4:09:40 PM com.gwtplatform.dispatch.server.AbstractDispatchServiceImpl cookieMatch
INFO: No cookie sent by client in RPC. (Did you forget to bind the security cookie client-side? Or it could be an attack.)
Sep 6, 2012 4:09:40 PM com.gwtplatform.dispatch.server.AbstractDispatchServiceImpl execute
SEVERE: Cookie provided by RPC doesn't match request cookie, aborting action, possible XSRF attack. (Maybe you forgot to set the security cookie?) While executing action: com.company.appmodule.client.Login
Solution: Clear cache and cookies from browser.
bindConstant().annotatedWith(SecurityCookie.class).to("mycookie");
I was getting the following exception-
Sep 6, 2012 4:09:40 PM com.gwtplatform.dispatch.server.AbstractDispatchServiceImpl cookieMatch
INFO: No cookie sent by client in RPC. (Did you forget to bind the security cookie client-side? Or it could be an attack.)
Sep 6, 2012 4:09:40 PM com.gwtplatform.dispatch.server.AbstractDispatchServiceImpl execute
SEVERE: Cookie provided by RPC doesn't match request cookie, aborting action, possible XSRF attack. (Maybe you forgot to set the security cookie?) While executing action: com.company.appmodule.client.Login
Solution: Clear cache and cookies from browser.
Labels:
AbstractDispatchServiceImpl,
cookie,
gwtp,
rpc
Location:
Mountain View, CA, USA
Sunday, August 26, 2012
GWT Celltable - Add multiple anchors to a cell
I am trying to insert multiple anchors in a single celltable cell. This is how it would look like-
The cars column has multiple cars, each car link can be clicked to perform different actions.
I have extended a MultipleAnchorCell class from AbstractSafeHtmlCell like this-
And then I use this cell in a celltable to create a column. My User class contains comma separated values of cars in a class variable.
I am still learning GWT and it took me a while to figure this out.
Comments/feedback is welcome.
===============================================================
Location:
Mountain View, CA, USA
Saturday, April 7, 2012
GWT dygraph example using visualization datatable
Here is a quick example of how to create a dygraph using GWT and visualization api to get the datatable.
1. Add dygraph-gwt.jar to your build path and these inherits tags in your mysamplegwtproj.gwt.xml file
<inherits name="com.google.gwt.visualization.Visualization"/>
<inherits name="org.danvk.dygraphs"/>
2. Add the following script tag in mysamplegwtproject.html file
<script type="text/javascript" src="dygraph-combined.js">
<script type="text/javascript" src="http://www.google.com/jsapi">
3. Add the following code in your onModuleLoad() function-
1. Add dygraph-gwt.jar to your build path and these inherits tags in your mysamplegwtproj.gwt.xml file
<inherits name="com.google.gwt.visualization.Visualization"/>
<inherits name="org.danvk.dygraphs"/>
2. Add the following script tag in mysamplegwtproject.html file
<script type="text/javascript" src="dygraph-combined.js">
<script type="text/javascript" src="http://www.google.com/jsapi">
3. Add the following code in your onModuleLoad() function-
final VerticalPanel vp = new VerticalPanel();
vp.setWidth("700px");
vp.setHeight("700px");
VisualizationUtils.loadVisualizationApi(new Runnable() {
@Override
public void run() {
Query q = Query
.create("mysampleproj/getMyDatatable?params=2&id=1");
q.send(new Callback() {
@Override
public void onResponse(QueryResponse response) {
JavaScriptObject jdygraph = createDygraph(
vp.getElement(), response.getDataTable(), 0,
100);
}
});
}
}, LineChart.PACKAGE);
RootPanel.get().add(vp);
4. Here is the JSNI function which actually creates the dygraph.
public static native JavaScriptObject createDygraph(Element element,
DataTable dataTable, double minY, double maxY) /*-{
var chart = new $wnd.Dygraph.GVizChart(element);
chart.draw(dataTable, {
valueRange : [ minY, maxY ],
displayAnnotations : true,
showRangeSelector : true,
legend : 'always',
labelsDivStyles : {
'textAlign' : 'right'
},
title : 'TITLE',
titleHeight : 25,
axes : {
x : {
pixelsPerLabel : 50
}
}
});
return chart;
}-*/;
Hope this helps. Suggestions/improvements are welcome.
===============================================================
Wednesday, April 4, 2012
NoClassDefFoundError: com.google.common.collect.Sets
Error-
NoClassDefFoundError: "com.google.common.collect.Sets"
Solution-
Add guava-11.0.2.jar to WEB-INF/lib and add it to the build path.
Subscribe to:
Posts (Atom)