Home > Software Dev > Installation Hadoop+Hive dan Porting data dari Postgresql ke Hive

Installation Hadoop+Hive dan Porting data dari Postgresql ke Hive

Judulnya aneh?

hmm,ngga sih,tergantung kebutuhan aja gan :P..okeh,saya akan sedkit berbagi mengenai kelanjutan tugas saya dikantor,jadi melanjutkan permasalahan yang diberikan pada manager saya,so akhirnya mulai-mulai deh install Hadoop dkk..

okeh,pertama-tama instlall hadoop dulu,download hadoop di sini untuk yang versi asli apache,dan di sini (include sqoop,dan hive-nya)untuk yang bundelan dari cloudera..saya pakai yang cloudera karena sqoop membutuhkan library hadoop yang sudah ditambah oleh cloudera..:D..

selanjutnya untar folder hadoop dan kawan-kawan,jangan lupa set Hadoop home,

 export HADOOP_HOME=/opt/hadoop-0.20.2-cdh3u0
 export PATH=$PATH:$HADOOP_HOME/bin
 export HIVE_HOME=/opt/hive-0.7.0-cdh3u0
 export PATH=$PATH:$HIVE_HOME/bin
 export SQOOP_HOME=/opt/sqoop-1.2.0
 export PATH=$PATH:$SQOOP_HOME/bin
 

kalo saya kira-kira path-nya seperti itu untuk hadoop,sqoop,dan Hivenya..

okeh,selanjutnya vim atau gedit file configure-sqoop yang ada pada direktori bin/ dari SQOOP_HOME..

pertanyaannya untuk apa?

sebenarnya hal ini dibutuhkan hanya jika bagi anda yang melakukan instalasi hadoop+hive+sqoop saja,karena pada dasarnya sqoop secara default harus terinclude ZOOKEEPER,HBASE

<pre>#!/bin/bash
#
# Licensed to Cloudera, Inc. under one or more
# contributor license agreements.  See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# Cloudera, Inc. licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License.  You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

# This is sourced in by bin/sqoop to set environment variables prior to
# invoking Hadoop.

bin="$1"

if [ -z "${bin}" ]; then
  bin=`dirname $0`
  bin=`cd ${bin} && pwd`
fi

if [ -z "$SQOOP_HOME" ]; then
  export SQOOP_HOME=${bin}/..
fi

# Find paths to our dependency systems. If they are unset, use CDH defaults.

if [ -z "${HADOOP_HOME}" ]; then
  HADOOP_HOME=/usr/lib/hadoop
fi
#if [ -z "${HBASE_HOME}" ]; then
#  HBASE_HOME=/usr/lib/hbase
#fi
#if [ -z "${ZOOKEEPER_HOME}" ]; then
#  ZOOKEEPER_HOME=/usr/lib/zookeeper
#fi

# Check: If we can't find our dependencies, give up here.
if [ ! -d "${HADOOP_HOME}" ]; then
  echo "Error: $HADOOP_HOME does not exist!"
  echo 'Please set $HADOOP_HOME to the root of your Hadoop installation.'
  exit 1
fi
#if [ ! -d "${HBASE_HOME}" ]; then
#  echo "Error: $HBASE_HOME does not exist!"
#  echo 'Please set $HBASE_HOME to the root of your HBase installation.'
#  exit 1
#fi
#if [ ! -d "${ZOOKEEPER_HOME}" ]; then
#  echo "Error: $ZOOKEEPER_HOME does not exist!"
#  echo 'Please set $ZOOKEEPER_HOME to the root of your ZooKeeper installation.'
#  exit 1
#fi

# Where to find the main Sqoop jar
SQOOP_JAR_DIR=$SQOOP_HOME

# If there's a "build" subdir, override with this, so we use
# the newly-compiled copy.
if [ -d "$SQOOP_JAR_DIR/build" ]; then
  SQOOP_JAR_DIR="${SQOOP_JAR_DIR}/build"
fi

function add_to_classpath() {
  dir=$1
  for f in $dir/*.jar; do
    SQOOP_CLASSPATH=${SQOOP_CLASSPATH}:$f;
  done

  export SQOOP_CLASSPATH
}

# Add sqoop dependencies to classpath.
SQOOP_CLASSPATH=""
if [ -d "$SQOOP_HOME/lib" ]; then
  add_to_classpath $SQOOP_HOME/lib
fi

# Add HBase to dependency list
#add_to_classpath $HBASE_HOME
#add_to_classpath $HBASE_HOME/lib

#HBASE_CONF_DIR=${HBASE_CONF_DIR:-${HBASE_HOME}/conf}
SQOOP_CLASSPATH=${HBASE_CONF_DIR}:${SQOOP_CLASSPATH}

#add_to_classpath $ZOOKEEPER_HOME
#add_to_classpath $ZOOKEEPER_HOME/lib

SQOOP_CONF_DIR=${SQOOP_CONF_DIR:-${SQOOP_HOME}/conf}
SQOOP_CLASSPATH=${SQOOP_CONF_DIR}:${SQOOP_CLASSPATH}

# If there's a build subdir, use Ivy-retrieved dependencies too.
if [ -d "$SQOOP_HOME/build/ivy/lib/sqoop" ]; then
  for f in $SQOOP_HOME/build/ivy/lib/sqoop/*/*.jar; do
    SQOOP_CLASSPATH=${SQOOP_CLASSPATH}:$f;
  done
fi

add_to_classpath ${SQOOP_JAR_DIR}

export SQOOP_CLASSPATH
export SQOOP_CONF_DIR
export SQOOP_JAR_DIR
export HADOOP_CLASSPATH="${SQOOP_CLASSPATH}:${HADOOP_CLASSPATH}"
export HADOOP_HOME
#export HBASE_HOME</pre>

uncomment sesuai yang saya kasih tanda #,selanjutnyaaaa,save dan kita sudahi hack kita dengan mengeksekusi sqoop(sql to hadoop)..

eits,perlu diingat,ketika kita akan mengeksekusi sqoop,jangan lupa tambahkan library jdbc sesuai database yang kita pakai,misalkan:


sqoop import-all-tables --connect jdbc:postgresql://localhost/TestDB --username postgres --password postgres --hive-import

artinya kita ingin agar data dari database lama kita,kita pindahkan ke sqoop..

jika anda menggunakan ubuntu dan gagal pada saat pertama kali,Tenang,mungkin kegagalan anda disebabkan oleh masalah privilege,anda bisa menjdai super user untuk kembali mengeksekusi sqoop..:D..

BTW jangan lupa buat foldernya dulu yah,baru dieksekusi commandnya di folder tersebut,folder ini sebagai HDFS..

oke,tinggal jalankan hive pada console dan lakukan query layaknya sql..

dan hasilnya?

😀

semoga manfaat

Advertisements
Categories: Software Dev
  1. igoy
    August 17, 2011 at 7:39 pm

    ok, ane udah berhsil install hive dan hadoopnya.
    permasalahannya koq ane sama skali gk ngerti cara masukin data or recordnya ke tabel yaa?
    sulit banget, nggak kya mysql.
    ane bingung banget dah biar udah udah baca tutorialnya juga.
    mohon petunjuk..
    terima kasih

  1. No trackbacks yet.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: