spark-FeatureSelection (homepage)

Provides different FeatureSelection methods as Spark MLlib PipelineStages and a VectorMerger for merging different VectorColumns without duplicates.

Different Featureselection methods (3 filters/ 2 selectors based on scores from embedded methods) are provided as Spark MLlib PipelineStages. These are:

Filters:
- CorrelationSelector: calculates correlation ("spearman", "pearson"- adjustable through .setCorrelationType) between each feature and label.
- GiniSelector: measures impurity difference between before and after a feature value is known.
- InfoGainSelector: measures the information gain of a feature with respect to the class.

"Embedded":
- ImportanceSelector: takes FeatureImportances from any embedded method, e.g. Random Forest.
- LRSelector: takes feature weights from (L1) logistic regression. The weights are in a matrix W with dimensions #Labels X #Features. The absolute value is taken from all entries, summed column wise and scaled with the max value.

Util
- VectorMerger: takes several VectorColumns (e.g. the result of different feature selection methods) and merges them into one VectorColumn. Unlike the VectorAssembler, VectorMerger uses the metadata of the VectorColumn to remove duplicates. It supports two modes:
1) useFeaturesCol true and featuresCol set: the output column will contain the corresponding column from featuresCol (match by name) that have names appearing in one of the inputCols. Use this, if feature importances were calculated using (e.g.) discretized columns, but selection shall use original values.
2) useFeaturesCol false: the output column will contain the columns from the inputColumns, but dropping duplicates.


Tags (No tags yet, login to add one. )


How to

This package doesn't have any releases published in the Spark Packages repo, or with maven coordinates supplied. You may have to build this package from source, or it may simply be a script. To use this Spark Package, please follow the instructions in the README.

Releases

No releases yet.