Introduction to Spark SQL in Python

Learn Spark SQL

If you’re familiar with SQL and have heard great things about Apache Spark, this course is for you. Apache Spark is a computing framework for processing big data, and Spark SQL is a component of Apache Spark. This four-hour course will show you how to take Spark to a new level of usefulness, using advanced SQL features, such as window functions.

Over the course of four chapters, you’ll use Spark SQL to analyze time series data, extract the most common words from a text document, create feature sets from natural language text, and use them to predict the last word in a sentence using logistic regression.

Discover the Uses of Spark SQL

You’ll start by creating and querying an SQL table in Spark, as well as learning how to use SQL window functions to perform running sums, running differences, and other operations.

Next, you’ll explore how to use the window function in Spark SQL for natural language processing, including using a moving window analysis to find common word sequences.

In chapter 3, you’ll learn how to use the SQL Spark UI to properly cache DataFrames and SQL tables before exploring the best practices for logging in Spark.

Finally, you use all of the skills learned so far to load and tokenize raw text before extracting word sequences. You’ll then use logistic regression to classify the text, using raw natural language data to train a text classifier.

Gain a Thorough Introduction to Spark SQL

By the end of the course, you’ll have a firm understanding of Spark SQL and will understand how Spark combines the power of distributed computing with the ease of use of Python and SQL.