중위값과 몇%의 위치를 알아내는 함수 public static double getMedian(Double[] value_list) { int size = value_list.length; int med = size / 2; if(size == 0) return 0; if(size == 1) return value_list[0]; if(size % 2 == 1) return value_list[med]; return (value_list[med-1] + value_list[med])/2; } public static double getPercentile(Double[] value_list, Double percent) { Arrays.sort(value_list); int N = value_list.length; double n = percent / 100d * ((double)N - 1) + 1; int k = (int)n; if(k == 0) return value_list[0]; if(k >= N) return value_list[N-1]; double diff = n - (double)k; return value_list.......
2015-01-11 13:46:14 · 산소소년[HADOOP] map/reduce legacy template (org.apache.hadoop.mapred)
이번엔 옛날 형식 template 입니다. Mapper public class first_mapper extends MapReduceBase implements Mapper { private String SEPARATOR = "\t"; private String LogDate = ""; public void configure(JobConf job) { super.configure(job); String full_path = job.get("map.input.file"); int StartPosi = full_path.indexOf("summarized/")+"summarized/".length(); LogDate = full_path.substring(StartPosi, StartPosi+6); } public void map(LongWritable k, Text v, OutputCollector o, Reporter r) throws IOException { o.collect(new Text("ERR"), new Text(v.toString.......
2015-01-05 19:34:14 · 산소소년[HADOOP] map/reduce legacy template (org.apache.hadoop.mapred)
이번엔 옛날 형식 template 입니다. Mapper public class first_mapper extends MapReduceBase implements Mapper<LongWritable, Text, Text, Text> { private String SEPARATOR = "\t"; private String LogDate = ""; public void configure(JobConf job) { super.configure(job); String full_path = job.get("map.input.file"); int StartPosi = full_path.indexOf("summarized/")+"summarized/".length(); LogDate = full_path.substring(StartPosi, StartPosi+6); } public void map(LongWritable k, Text v, OutputCollector<Text, Text> o, Reporter r) throws IOException { o.collect(new Text("ERR"), new Text(v.toString.......
2015-01-05 19:34:14 · 산소소년[HADOOP] map/reduce legacy template (org.apache.hadoop.mapred)
이번엔 옛날 형식 template 입니다. Mapper public class first_mapper extends MapReduceBase implements Mapper<LongWritable, Text, Text, Text> { private String SEPARATOR = "\t"; private String LogDate = ""; public void configure(JobConf job) { super.configure(job); String full_path = job.get("map.input.file"); int StartPosi = full_path.indexOf("summarized/")+"summarized/".length(); LogDate = full_path.substring(StartPosi, StartPosi+6); } public void map(LongWritable k, Text v, OutputCollector<Text, Text> o, Reporter r) throws IOException { o.collect(new Text("ERR"), new Text(v.toString.......
2015-01-05 10:34:14 · 산소소년[HADOOP] map/reduce legacy template (org.apache.hadoop.mapred)
이번엔 옛날 형식 template 입니다. Mapper public class first_mapper extends MapReduceBase implements Mapper<LongWritable, Text, Text, Text> { private String SEPARATOR = "\t"; private String LogDate = ""; public void configure(JobConf job) { super.configure(job); String full_path = job.get("map.input.file"); int StartPosi = full_path.indexOf("summarized/")+"summarized/".length(); LogDate = full_path.substring(StartPosi, StartPosi+6); } public void map(LongWritable k, Text v, OutputCollector<Text, Text> o, Reporter r) throws IOException { o.collect(new Text("ERR"), new Text(v.toString.......
2015-01-05 10:34:14 · 산소소년[HADOOP] map/reduce template (org.apache.hadoop.mapreduce)
legacy 방식(mapred) + streamming 조합으로만 M/R개발을 하다보니 종종 코드가 섞이는 상황이 발생하여 template을 메모해 놨습니다. (0.20.3 부터인가 지원되는 방식입니다.... 0.20.2였던가????) Mapper package com.airguy.mapreduce.template; import org.apache.hadoop.io.IntWritable; import org.apache.hadoop.io.LongWritable; import org.apache.hadoop.io.Text; import org.apache.hadoop.mapreduce.Mapper; public class TemplateMapper extends Mapper<LongWritable, Text, Text, Text> { private final static IntWritable outputValue = new IntWritable(1); public void map(LongWritable key, Text value, Context conte.......
2015-01-05 10:27:12 · 산소소년[HADOOP] map/reduce template (org.apache.hadoop.mapreduce)
legacy 방식(mapred) + streamming 조합으로만 M/R개발을 하다보니 종종 코드가 섞이는 상황이 발생하여 template을 메모해 놨습니다. (0.20.3 부터인가 지원되는 방식입니다.... 0.20.2였던가????) Mapper package com.airguy.mapreduce.template; import org.apache.hadoop.io.IntWritable; import org.apache.hadoop.io.LongWritable; import org.apache.hadoop.io.Text; import org.apache.hadoop.mapreduce.Mapper; public class TemplateMapper extends Mapper { private final static IntWritable outputValue = new IntWritable(1); public void map(LongWritable key, Text value, Context conte.......
2015-01-05 10:27:12 · 산소소년[HADOOP] map/reduce template (org.apache.hadoop.mapreduce)
legacy 방식(mapred) + streamming 조합으로만 M/R개발을 하다보니 종종 코드가 섞이는 상황이 발생하여 template을 메모해 놨습니다. (0.20.3 부터인가 지원되는 방식입니다.... 0.20.2였던가????) Mapper package com.airguy.mapreduce.template; import org.apache.hadoop.io.IntWritable; import org.apache.hadoop.io.LongWritable; import org.apache.hadoop.io.Text; import org.apache.hadoop.mapreduce.Mapper; public class TemplateMapper extends Mapper<LongWritable, Text, Text, Text> { private final static IntWritable outputValue = new IntWritable(1); public void map(LongWritable key, Text value, Context conte.......
2015-01-05 01:27:12 · 산소소년[HADOOP] map/reduce template (org.apache.hadoop.mapreduce)
legacy 방식(mapred) + streamming 조합으로만 M/R개발을 하다보니 종종 코드가 섞이는 상황이 발생하여 template을 메모해 놨습니다. (0.20.3 부터인가 지원되는 방식입니다.... 0.20.2였던가????) Mapper package com.airguy.mapreduce.template; import org.apache.hadoop.io.IntWritable; import org.apache.hadoop.io.LongWritable; import org.apache.hadoop.io.Text; import org.apache.hadoop.mapreduce.Mapper; public class TemplateMapper extends Mapper<LongWritable, Text, Text, Text> { private final static IntWritable outputValue = new IntWritable(1); public void map(LongWritable key, Text value, Context conte.......
2015-01-05 01:27:12 · 산소소년시각화 툴 GEPHI : https://gephi.github.io/ TreeMap : http://www.cs.umd.edu/hcil/treemap/ ImagePlot : http://lab.softwarestudies.com/p/imageplot.html IndieMapper : http://indiemapper.com/ ArcGis : http://www.arcgis.com/features/ Graphviz : http://www.graphviz.org/Home.php 챠트 D3Chart : http://d3js.org/ (Tutorial : http://alignedleft.com/tutorials/d3/) NVD3 : http://nvd3.org/ jqPlot : http://www.jqplot.com/ amCharts : http://www.amcharts.com/ Ext JS 4 Charts : http://dev.sencha.com/deploy/ChartsDemo/ YUI 3 Charts : http://yuilibrary.com/yui/docs/charts/ FusionCharts : http://www.fusioncharts.co.......
2014-10-13 10:39:43 · 산소소년시각화 툴 GEPHI : https://gephi.github.io/ TreeMap : http://www.cs.umd.edu/hcil/treemap/ ImagePlot : http://lab.softwarestudies.com/p/imageplot.html IndieMapper : http://indiemapper.com/ ArcGis : http://www.arcgis.com/features/ Graphviz : http://www.graphviz.org/Home.php 챠트 D3Chart : http://d3js.org/ (Tutorial : http://alignedleft.com/tutorials/d3/) NVD3 : http://nvd3.org/ jqPlot : http://www.jqplot.com/ amCharts : http://www.amcharts.com/ Ext JS 4 Charts : http://dev.sencha.com/deploy/ChartsDemo/ YUI 3 Charts : http://yuilibrary.com/yui/docs/charts/ FusionCharts : http://www.fusioncharts.co.......
2014-10-13 10:39:43 · 산소소년시각화 툴 GEPHI : https://gephi.github.io/ TreeMap : http://www.cs.umd.edu/hcil/treemap/ ImagePlot : http://lab.softwarestudies.com/p/imageplot.html IndieMapper : http://indiemapper.com/ ArcGis : http://www.arcgis.com/features/ Graphviz : http://www.graphviz.org/Home.php 챠트 D3Chart : http://d3js.org/ (Tutorial : http://alignedleft.com/tutorials/d3/) NVD3 : http://nvd3.org/ jqPlot : http://www.jqplot.com/ amCharts : http://www.amcharts.com/ Ext JS 4 Charts : http://dev.sencha.com/deploy/ChartsDemo/ YUI 3 Charts : http://yuilibrary.com/yui/docs/charts/ FusionCharts : http://www.fusioncharts.co.......
2014-10-13 01:39:43 · 산소소년시각화 툴 GEPHI : https://gephi.github.io/ TreeMap : http://www.cs.umd.edu/hcil/treemap/ ImagePlot : http://lab.softwarestudies.com/p/imageplot.html IndieMapper : http://indiemapper.com/ ArcGis : http://www.arcgis.com/features/ Graphviz : http://www.graphviz.org/Home.php 챠트 D3Chart : http://d3js.org/ (Tutorial : http://alignedleft.com/tutorials/d3/) NVD3 : http://nvd3.org/ jqPlot : http://www.jqplot.com/ amCharts : http://www.amcharts.com/ Ext JS 4 Charts : http://dev.sencha.com/deploy/ChartsDemo/ YUI 3 Charts : http://yuilibrary.com/yui/docs/charts/ FusionCharts : http://www.fusioncharts.co.......
2014-10-13 01:39:43 · 산소소년putty랑 puttygen 등을 이용하여 원격장비에 접속이 가능하다는 전재 하에 설명합니다. putty의 X11항목 설정 윈도우즈 장비에 Xming 설치 http://sourceforge.net/projects/xming/ 리눅스장비에 필요요소 설치 apt-get update apt-get install xorg openbox putty에서 x-terminal 실행 > xterm gnuplot 같은 것이 제대로 작동되나 확인해보면 됩니다.!!!!!!! 참조: http://simplefeatures.perroazul.net/?p=381
2014-10-02 10:49:26 · 산소소년JXL을 이용해서 만들었습니다. option.separator을 통하여 필드구분자를 설정을 option.sheetlinelimit을 통하여 라인을 초과하는 라인을 다음시트로 넘기도록 설정하였습니다. #!/bin/sh PWD="/home/hadoop/work/airguy/StdInToExcel" JAVA_BIN="/usr/bin/java" CLASS_PATH="$PWD/ReportUtils.jar:$PWD/jxl.jar" PROGRAM_NAME="com.airguy.utils.report.StdInToExcel" OUTPUT=$1 MAXLINE=$2 SEPARATOR=$3 if [ -z $MAXLINE ] ; then MAXLINE="5000" fi if [ -z $SEPARATOR ] ; then SEPARATOR="\\^" #SEPARATOR="\t" fi $JAVA_BIN \ -Doption.separator=$SEPARATOR \ -Doption.sheetlinelimit=$MAXLINE \ -classpath $CLASS_PATH \ $PROGRAM_NAM.......
2014-09-30 15:54:19 · 산소소년방법1 hadoop distcp hdfs:HDFS경로 s3n://access키:secret키@버킷경로 secret키에 + / 등이 들어가면 작동불가능한 단점이 있다. 방법2 time hadoop distcp -Dfs.s3n.awsAccessKeyId="access키" -Dfs.s3n.awsSecretAccessKey="secret키" HDFS경로 s3n:버킷경로 방법3 <property> <name>fs.s3n.awsAccessKeyId</name> <value>access키</value> </property> <property> <name>fs.s3n.awsSecretAccessKey</name> <value>secret키</value> </property> core-site, mapred-site, hdfs-site 등 config에 직접 추가 @추가로 파일용량이 커서 600sec timeout이 날 때가 있다. .......
2014-09-30 11:24:10 · 산소소년[무스패밀리 시즌2 17편] 엄마가 되는 순간부터 잠을 제대로 잘 수 없습니다
* 출처 : http://samsungtomorrow.com/%EB%AC%B4%EC%8A%A4%ED%8C%A8%EB%B0%80%EB%A6%AC-%EC%8B%9C%EC%A6%8C2-17%ED%8E%B8-%EC%97%84%EB%A7%88%EA%B0%80-%EB%90%98%EB%8A%94-%EC%88%9C%EA%B0%84%EB%B6%80%ED%84%B0-%EC%9E%A0%EC%9D%84-%EC%A0%9C
2014-08-29 10:44:56 · Into the Blue