Configuring Hive metastore for spark

Hi Folk’s ,

This post is going to be about how to configure hive metastore to access your hive table’s in spark sql.

Before Configuring

Defaults is spark’s default metastore respective to the session.

1)In Spark-env.sh Add your mysql connector jar path in the class path and hive home variable as shown below.

spark-env.sh

This is for spark to access your hive metastore i’m using Mysql as metastore , hence Mysql jar.

Copy hive-site.xml to $SPARK_HOME/conf directory.

$SPARK_HOME/conf
Hive
After Configuring

Note: spark 2.X will only support hive metastore versions ranging from 0.12.0 to 2.3.3 (As give in the documentation https://spark.apache.org/docs/latest/sql-data-sources-hive-tables.html). Default one is 1.2.1 but i am going to use 1.2.0. Hive version 2.X is not supported by spark, i have tried but it didn’t work.

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Thulasitharan Govindaraj

Big data engineer - Spark Scala,Hadoop,HIVE,Impala,Kafka,AWS,HBASE,Snowflake,Deltalake,CDH,DOCKER,k8s . For the love of Formula one #SennaSempre