This document summarizes the steps taken to perform linear regression on a dataset using PySpark. It loads data from a CSV file, cleans and prepares the data for modeling, trains a linear regression model in a pipeline, evaluates the model on test data to calculate the RMSE, and computes the r2 score. Key steps include splitting the data into training and test sets, fitting a linear regression model in a pipeline, calculating performance metrics like RMSE and r2 score on test data, and extracting the model coefficients and statistics.
This document summarizes the steps taken to perform linear regression on a dataset using PySpark. It loads data from a CSV file, cleans and prepares the data for modeling, trains a linear regression model in a pipeline, evaluates the model on test data to calculate the RMSE, and computes the r2 score. Key steps include splitting the data into training and test sets, fitting a linear regression model in a pipeline, calculating performance metrics like RMSE and r2 score on test data, and extracting the model coefficients and statistics.
Original Description:
regression machine learning sur spark
Universite de Djibouti
This document summarizes the steps taken to perform linear regression on a dataset using PySpark. It loads data from a CSV file, cleans and prepares the data for modeling, trains a linear regression model in a pipeline, evaluates the model on test data to calculate the RMSE, and computes the r2 score. Key steps include splitting the data into training and test sets, fitting a linear regression model in a pipeline, calculating performance metrics like RMSE and r2 score on test data, and extracting the model coefficients and statistics.
This document summarizes the steps taken to perform linear regression on a dataset using PySpark. It loads data from a CSV file, cleans and prepares the data for modeling, trains a linear regression model in a pipeline, evaluates the model on test data to calculate the RMSE, and computes the r2 score. Key steps include splitting the data into training and test sets, fitting a linear regression model in a pipeline, calculating performance metrics like RMSE and r2 score on test data, and extracting the model coefficients and statistics.
[35]: # Split the data into training and test sets (40% held out for testing) (trainingData, testData) = transformed.randomSplit([0.8, 0.2])
[36]: # Import LinearRegression class
from pyspark.ml.regression import LinearRegression
# Define LinearRegression algorithm
lr = LinearRegression()
[38]: import warnings
warnings.filterwarnings('ignore') # Chain indexer and tree in a Pipeline pipeline = Pipeline(stages=[featureIndexer, lr])
model = pipeline.fit(trainingData)
21/12/14 14:37:13 WARN Instrumentation: [8d638690] regParam is zero, which might
cause numerical instability and overfitting.
[39]: def modelsummary(model):
import numpy as np print ("Note: the last rows are the information for Intercept") print ("##","-------------------------------------------------") print ("##"," Estimate | Std.Error | t Values | P-value") coef = np.append(list(model.coefficients),model.intercept) Summary=model.summary
3 for i in range(len(Summary.pValues)): print ("##",'{:10.6f}'.format(coef[i]),\ '{:10.6f}'.format(Summary.coefficientStandardErrors[i]),\ '{:8.3f}'.format(Summary.tValues[i]),\ '{:10.6f}'.format(Summary.pValues[i]))
