issue with left_join for a large dataset

[btojenniferborder]

I had a issues with using left_join for a very big dataset (13 million rows). I got this error message:

Joining with by = join_by(coord)
Error in vctrs::vec_locate_matches():
! Match procedure results in an allocation larger than 2^31-1 elements. Attempted allocation size was 3452978837.
ℹ In file match.c at line 2658.
ℹ This is an internal error that was detected in the vctrs package.
Please report it at https://github.com/r-lib/vctrs/issues with a reprex and the full backtrace.
Backtrace:
▆

1. ├─dplyr::left_join(all_ringing_data, int)
2. ├─dplyr:::left_join.data.frame(all_ringing_data, int)
3. │ └─dplyr:::join_mutate(...)
4. │ └─dplyr:::join_rows(...)
5. │ └─dplyr:::dplyr_locate_matches(...)
6. │ ├─base::withCallingHandlers(...)
7. │ └─vctrs::vec_locate_matches(...)
8. └─rlang:::stop_internal_c_lib(...)
9. └─rlang::abort(message, call = call, .internal = TRUE, .frame = frame)

[btojenniferborder]

sorted was an issue with my data

[bertils]

I have the same issue. How did u sort the problem?

[btojenniferborder]

HI, With my data I think the issue was in the joining column having too few unique values- I recreated the column allowing more decimal places and it worked. But that was quite a niche solution that's very specific to my dataset, I'm not sure if it'd work for you too, if not maybe best to open another issue? Good luck!

…

On Wed, 1 Nov 2023 at 16:57, BertilS ***@***.***> wrote: I have the same issue. How did u sort the problem? — Reply to this email directly, view it on GitHub <#1890 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AJPNFIK3LFZUOYDUHZERNEDYCJ5QTAVCNFSM6AAAAAA6ZRFAMGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTOOBZGMYTIOBZGQ> . You are receiving this because you modified the open/close state.Message ID: ***@***.***>

-- Dr Jennifer Border (*née** Taylor*) Senior Research Ecologist British Trust for Ornithology, The Nunnery, Thetford, Norfolk IP24 2PU Tel: +44 (0) 1842 750050 Fax: +44 (0) 1842 750030 Email: jennifer.border @bto.org ***@***.***> Registered Charity Number 216652 (England & Wales), SC039193 (Scotland). Company Limited by Guarantee Number 357284 (England and Wales).