Issues merging Dfs with Lamba function & Materialize

[guptaat]

I had a need to merge multiple data frames based on extracted info from a text column. Had to use lamda function. The merge/join works fine if the virtual column is present only on base df but join fails and throws the error if virtual column has lamda function on dataframe to be joined. (It really doesn't matter if the virtual column with lamda is not used as join column. Just the mere presence in df causes the error)

import pandas as pd
import vaex

df = pd.DataFrame(data = {'a': ['text_text2', 't1_t2', 't3_t4'], 'b': [4, 5, 6], 'c': [3, 5, 6]})
vdf=vaex.from_pandas(df)
vdf['m']=vdf.a.str.split('_').apply(lambda x: x[0])
vdf

d1=vdf.join(vdf,left_on='a',right_on='a',how='left',lsuffix='_lft', rsuffix='_rt',allow_duplication=True )

Get Error :
ValueError: 'lambda_function' is not in list

Thought, maybe materialize function will help...

`import pandas as pd
import vaex

df = pd.DataFrame(data = {'a': ['text_text2', 't1_t2', 't3_t4'], 'b': [4, 5, 6], 'c': [3, 5, 6]})
vdf=vaex.from_pandas(df)
vdf['m']=vdf.a.str.split('_').apply(lambda x: x[0])
vdf=vdf.materialize('m')
vdf`

It threw up another error as it could not find column. _

Supposedly materialize function can only handle single char column names.

_

but materializing the virtual column didn't help and join continue to throw lambda_function is not in list error.

Questions:

1. How to avoid lambda function error if virtual column is needed on the dataframe to be joined?
2. What does Materialize function really does? I read the documentation and thought it would actually store the values of lambda function and indeed data type becomes str but join still threw error..

Any insight is much helpful. BTW.. thank you very much for this wonderful library...and has started to do all analysis in vaex... where bunch of other alternatives failed or took way too long... so once again thank you very much.

[guptaat]

For the moment got around the problem by using string replace function and avoided using apply completely but still wondering why it caused issues especially when virtual column using apply is present on the df to be joined..

[maartenbreddels]

I think this is a duplicate of #1282 which is fixed in #1966 which should be released soon (4.9)