StartUp Interview Question for Data Engineers
- 0of 0 votes
Answerscreate a custom feature transformer in spark scala.Lets say dataframe is like below
- ashwini.padhy89 December 03, 2018 in India
+--------------------+ .
| email_list| .
+--------------------+ .
|testmail1115@gmail.com| .
|mavenmaven@mlail.com| .
|dnd.7899334622@gmail.com| .
+--------------------+ .
If i use the transformer it converts the input array of strings into an array of n-grams.like below:
+--------------------+--------------------+
| email_list| ngrams| .
+--------------------+--------------------+
|testmail1115@gmail.com|[t e, e s, s t, t...|
|mavenmaven@mlail.com|[m a, a v, v e, e...| .
|dnd.7899334622@gmail.com|[d n, n d, d...| .
+--------------------+--------------------+ .
How to get the distinct ngram present rather the pattern or array .| Report Duplicate | Flag | PURGE
StartUp Data Engineer
Interview Type: In-Person