Getting Started

Download the prep-buddy jar and add it to your project library.

You can start writing spark program with prep buddy by writing the following code

Functionalities:

The TransformableRDD holds all the functionality that could be done over a RDD. It takes an RDD object as constructor parameter. The following functionalities are available on TransformableRDD.

It is the process of replacing missing data with substituted values. The following imputation algorithms have been implemented:

• Approx Mean Based Substitution
• Mode Based Substitution
• Naive Bayes Classifier Based Substitution
• Univariate Linear Regression Based Substitution
• Mean Based Substitution

It is helpful to find the number of occurrences for each value of a field.

It is the process Finding the group of different items but could be the alternative representation of the same item.The following imputation algorithms have been implemented

• Simple Fingerprint algorithm
• N-Gram Fingerprint algorithm
• Levenshtein distance algorithm

Is the process of removing duplicates from a dataset.

It provides only the records which has duplicates entry in the given dataset

Is the process of split a field into multiple field.

• By delimiter
• By length

Is the process of merging two or more fields into one field.

Mark rows by a symbol for a given condition.

Map on marked (flag) row.

Removes rows from dataset for a given condition

• Min Max Normalizer
• Z Score Normalizer
• Decimal Scaling Normalizer

• Simple Moving Average
• Weighted Moving Average

Examples

Let's take a sample CSV dataset with filename calls.csv consisting attributes :