UMAP coordinates between core and extended atlas

[LorenzoMerotto]

Description of the bug

Hello @nictru ,
I want to report what I think is a bug in the extension workflow of scDownstream.

I have a set of datasets from different sequencing technologies that I'm using to build an atlas. The core atlas looks perfect, but not the extended one. While the pipeline runs trough, the datasets from the extended atlas seem to cluster all together, separately from the rest. While the cell type differences are visible, it's almost as if they cluster in their own mini-umap, with a different coordinate system from the main one.
I can't provide you with public screenshots here but I can share them privately with you.

Keep up with the amazing work, and thanks for your help!

Command used and terminal output

Relevant files

No response

System information

No response

[nictru]

Hey, so as we discussed during the last days, you said that you are currently doing the extension outside of the pipeline with the following process:

1. Take model and anndata object from first integration run (core)
2. Combine core anndata object with new data
3. Load core model
4. Retrain core model on combined object from (2)
5. Get embeddings for entire combined object from retrained model

This differs from the implementation in the pipeline, which looks as follows:

1. Take model and anndata object from first integration run (core)
2. Load core model
3. Train on only extension data
4. Get extension embeddings
5. Concatenate extension embeddings with core embeddings

With your approach you get a well-integrated UMAP, with the pipeline implementation you get a weird result with two non-overlapping UMAP structures

---

Now I briefly investigated why I did the pipeline implementation the way it is done, and it basically implements what is described in the official scvi-tools reference mapping tutorial. There they also retrain on only the query objects, and not the combined object.

Now this makes me think, that what we have been talking about (making the pipeline following your approach) might actually not be desirable. What do you think?

[LorenzoMerotto]

Hello Nico,

Maybe I explained myself wrong, but in my script I do everything you do in the pipeline, all the five steps, and at the end I recompute neighbourhood and umap. And this gives me the properly rendered dimensionality reduction