DATABRICKS-MACHINE-LEARNING-ASSOCIATE Online Practice Questions and Answers

Correct Answer: C

For large datasets with many variables, Spark ML distributes the training of a linear regression model using iterative optimization methods. Specifically, Spark ML employs algorithms such as Gradient Descent or L-BFGS (Limited-memory

Broyden璅letcher璆oldfarb璖hanno) to iteratively minimize the loss function. These iterative methods are suitable for distributed computing environments and can handle large-scale data efficiently by partitioning the data across nodes in a

cluster and performing parallel updates.References:

Spark MLlib Documentation (Linear Regression with Iterative Optimization).

Correct Answer: A

Including the entire pipeline as the estimator in the cross-validation process means that all stages of the pipeline, including data preprocessing steps like string indexing and vector assembling, will be refit or retransformed for each fold of the cross-validation. This results in a longer runtime because each fold requires re-execution of these preprocessing steps, which can be computationally expensive. If only the random forest regressor (rfr) were included as the estimator, the preprocessing steps would be performed once, and only the model fitting would be repeated for each fold, significantly reducing the computational overhead. References: Databricks documentation on cross-validation: Cross Validation

Correct Answer: A

To calculate the overall cross-validation root-mean-squared error (RMSE), you average the RMSE values obtained from each validation fold. Given the RMSE values of 10.0, 12.0, and 17.0 for the three folds, the overall cross-validation

RMSE is calculated as the average of these three values:

Overall CV RMSE=10.0+12.0+17.03=39.03=13.0Overall CV RMSE=310.0+12.0+17.0 =339.0=13.0

Thus, the correct answer is 13.0, which accurately represents the average RMSE across all folds.References:

Cross-validation in Regression (Understanding Cross-Validation Metrics).

Correct Answer: A

Thesummary()function in PySpark's DataFrame API provides descriptive statistics which include count, mean, standard deviation, min, max, and quantiles for numeric columns. Here are the steps on how it can be used:

Import PySpark:Ensure PySpark is installed and correctly configured in the Databricks environment.

Load Data:Load the data into a Spark DataFrame.

Apply Summary:Usespark_df.summary()to generate summary statistics. View Results:The output from thesummary()function includes the statistics specified in the query (count, mean, standard deviation, min, max, and potentially quartiles

which approximate the interquartile range).

References:

PySpark

Documentation:https://spark.apache.org/docs/latest/api/python/reference/api/pysp ark.sql.DataFrame.summary.html

Correct Answer: B

Spark ML (part of Apache Spark's MLlib) is designed to handle machine learning tasks across multiple nodes in a cluster, effectively parallelizing tasks like hyperparameter tuning. It supports various machine learning algorithms that can be

optimized over a Spark cluster, making it suitable for parallelizing hyperparameter tuning for single-node machine learning models when they are adapted to run on Spark.

References:

Apache Spark MLlib Guide:https://spark.apache.org/docs/latest/ml-guide.html

Spark ML is a library within Apache Spark designed for scalable machine learning. It provides tools to handle large-scale machine learning tasks, including parallelizing the hyperparameter tuning process for single-node machine learning

models using a Spark cluster. Here's a detailed explanation of how Spark ML can be used:

Hyperparameter Tuning with CrossValidator: Spark ML includes theCrossValidatorandTrainValidationSplitclasses, which are used for hyperparameter tuning. These classes can evaluate multiple sets of hyperparameters in parallel using a

Spark cluster. from pyspark.ml.tuning import CrossValidator, ParamGridBuilder from pyspark.ml.evaluation import BinaryClassificationEvaluator

# Define the model

model = ...

# Create a parameter grid

paramGrid = ParamGridBuilder() \

addGrid(model.hyperparam1, [value1, value2]) \

addGrid(model.hyperparam2, [value3, value4]) \

build()

# Define the evaluator

evaluator = BinaryClassificationEvaluator()

# Define the CrossValidator

crossval = CrossValidator(estimator=model,

estimatorParamMaps=paramGrid,

evaluator=evaluator,

numFolds=3)

Parallel Execution: Spark distributes the tasks of training models with different hyperparameters across the cluster's nodes. Each node processes a subset of the parameter grid, which allows multiple models to be trained simultaneously.

Scalability: Spark ML leverages the distributed computing capabilities of Spark. This allows for efficient processing of large datasets and training of models across many nodes, which speeds up the hyperparameter tuning process significantly

compared to single-node computations.

References:

Apache Spark MLlib Documentation

Hyperparameter Tuning in Spark ML

Correct Answer: D

To determine the number of machine learning models that can be trained in parallel, we need to calculate the total number of combinations of hyperparameters. The given hyperparameter grid includes:

Hyperparameter 1: [2, 5, 10] (3 values)

Hyperparameter 2: [50, 100] (2 values)

The total number of combinations is the product of the number of values for each hyperparameter:3 (values of Hyperparameter 1)? (values of Hyperparameter 2)=63 (value s of Hyperparameter 1)? (values of Hyperparameter 2)=6 With 3-fold

cross-validation, each combination of hyperparameters will be evaluated 3 times. Thus, the total number of models trained will

be:6 (combinations)? (folds)=186 (combinations)? (folds)=18 However, the number of models that can be trained in parallel is equal to the number of hyperparameter combinations, not the total number of models considering cross-validation.

Therefore, 6 models can be trained in parallel.

References:

Databricks documentation on hyperparameter tuning: Hyperparameter Tuning

Correct Answer: B

For a data scientist using Spark SQL to import data and then performing machine learning tasks using Spark ML, the best-suited compute tool is a Standard cluster. A Standard cluster in Databricks provides the necessary resources and

scalability to handle large datasets and perform distributed computing tasks efficiently, making it ideal for running Spark SQL and Spark ML operations.

References:

Databricks documentation on clusters: Clusters in Databricks

Correct Answer: D

When using theHyperoptlibrary withfmin, the goal is to find the minimum of the objective function. Since you are usingcross_val_scoreto calculate the R2 score which is a measure of the proportion of the variance for a dependent variable

that's explained by an independent variable(s) in a regression model, higher values are better. However,fmin seeks to minimize the objective function, so to align withfmin's goal, you should return the negative of the R2 score (-r2). This way,

by minimizing the negative R2,fminis effectively maximizing the R2 score, which can lead to a more accurate model.

References:

Hyperopt Documentation: http://hyperopt.github.io/hyperopt/ Scikit-Learn documentation on model evaluation: https://scikit-learn.org/stable/modules/model_evaluation.html

Correct Answer: B

To ensure reproducible training and test sets, writing the split data sets to persistent storage is a reliable approach. This allows you to consistently load the same training and test data for each model run, regardless of cluster reconfiguration

or other changes in the environment.

Correct approach:

Split the data.

Write the split data to persistent storage (e.g., HDFS, S3). Load the data from storage for each model training session. train_df, test_df = spark_df.randomSplit([0.8,0.2], seed=42) train_df.write.parquet("path/to/train_df.parquet")

test_df.write.parquet("path/to/test_df.parquet")# Later, load the datatrain_df = spark.read.parquet("path/to/train_df.parquet") test_df = spark.read.parquet("path/to/test_df.parquet")

References:

Spark DataFrameWriter Documentation

Correct Answer: A

To use the pandas API on Spark, the data scientist can run the following code block:

importpyspark.pandasasps df = ps.DataFrame(spark_df) This code imports the pandas API on Spark and converts the Spark DataFramespark_df into a pandas-on-Spark DataFrame, allowing the data scientist to use familiar pandas functions

for further feature engineering.

References:

Databricks documentation on pandas API on Spark: pandas API on Spark