Unlocking Spark: Machine Learning, New Word Discovery, and Smart Q&A Techniques

How to Use Spark for Machine Learning

Spark 1.5 provides a unified platform for real‑time and batch processing, SQL, and a rich algorithm library, running on Hadoop’s YARN and HDFS. The Spark‑Shell, combined with Scala, lets you write and execute code across hundreds of nodes as easily as a local script, enabling full‑dataset training with functions like

sample

for random sampling.

Algorithms such as Naive Bayes, Word2Vec, and linear regression are readily available; using Spark‑Shell dramatically speeds up experimentation compared to single‑machine workflows.

New Word Discovery with Spark

By processing over two million blog posts (≈200 GB), Spark can compute five metrics for each candidate term—cohesion, freedom, frequency, IDF, and overlapping substrings—to filter and rank new words. The pipeline includes HTML tag removal, tokenization, length filtering (max five Chinese characters), and character cleaning before applying the metric formulas.

Performance tricks such as preferring

reduceByKey

over

groupByKey

prevent memory blow‑outs, and dynamic worker scaling in Spark 1.5 helps manage resource usage.

Smart Q&A on Spark

Using Spark‑based Word2Vec embeddings, titles of blog posts and user queries are transformed into dense vectors (e.g., 50‑dimensional). Sentence vectors are obtained by element‑wise addition of word vectors, then similarity scores identify matching answers; thresholds of 0.9 and 0.7 denote direct answers and reference answers respectively.

This approach powers simple question‑answer bots that retrieve relevant content from forums, blogs, and documentation.

Conclusion

Data analysts and algorithm engineers should adopt Spark‑Shell for its interactive, full‑scale capabilities; the community now supports Python and R, and DataFrames borrow concepts from R.

For building machine‑learning platforms, see the author’s related article (http://www.jianshu.com/p/d59c3e037cb7) for architectural insights.