{ "cells": [ { "cell_type": "markdown", "id": "thousand-hazard", "metadata": { "tags": [] }, "source": [ "# Lazily concatenating multiple AnnData objects" ] }, { "cell_type": "markdown", "id": "961d20db-ad53-4822-8434-46bce4d17106", "metadata": {}, "source": [ "**Author:** [Sergei Rybakov](https://github.com/koncopd)" ] }, { "cell_type": "markdown", "id": "equal-amazon", "metadata": {}, "source": [ "This tutorial shows how to use the `AnnCollection` class with one or multiple `AnnData` objects." ] }, { "cell_type": "code", "execution_count": 2, "id": "damaged-glossary", "metadata": {}, "outputs": [], "source": [ "from __future__ import annotations\n", "\n", "import warnings\n", "\n", "import gdown\n", "import numpy as np\n", "import pandas as pd\n", "import scanpy as sc\n", "from sklearn.decomposition import IncrementalPCA\n", "from sklearn.model_selection import train_test_split\n", "\n", "from anndata.experimental.multi_files import AnnCollection\n", "\n", "warnings.simplefilter(action=\"ignore\", category=pd.core.common.SettingWithCopyWarning)" ] }, { "cell_type": "code", "execution_count": null, "id": "discrete-professor", "metadata": {}, "outputs": [], "source": [ "# the data is from this scvi reproducibility notebook\n", "# https://yoseflab.github.io/scvi-tools-reproducibility/scarches_totalvi_seurat_data/\n", "gdown.download(\n", " url=\"https://drive.google.com/uc?id=1X5N9rOaIqiGxZRyr1fyZ6NpDPeATXoaC\",\n", " output=\"pbmc_seurat_v4.h5ad\",\n", " quiet=False,\n", ")\n", "gdown.download(\n", " url=\"https://drive.google.com/uc?id=1JgaXNwNeoEqX7zJL-jJD3cfXDGurMrq9\",\n", " output=\"covid_cite.h5ad\",\n", " quiet=False,\n", ")" ] }, { "cell_type": "code", "execution_count": 3, "id": "alien-collar", "metadata": {}, "outputs": [], "source": [ "covid = sc.read(\"covid_cite.h5ad\")\n", "pbmc = sc.read(\"pbmc_seurat_v4.h5ad\")" ] }, { "cell_type": "markdown", "id": "closed-purpose", "metadata": {}, "source": [ "## AnnCollection initialization\n", "AnnCollection is a wrapper class which allows to lazily concatenate and jointly subset AnnData objects along the observations axis." ] }, { "cell_type": "markdown", "id": "loving-contract", "metadata": {}, "source": [ "You can initialize AnnCollection with one AnnData object." ] }, { "cell_type": "code", "execution_count": 4, "id": "educated-douglas", "metadata": {}, "outputs": [], "source": [ "dataset = AnnCollection([covid])" ] }, { "cell_type": "code", "execution_count": 5, "id": "qualified-sierra", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "AnnCollection object with n_obs × n_vars = 57669 × 33538\n", " constructed from 1 AnnData objects\n", " view of obsm: 'pro_exp'\n", " obs: 'orig.ident', 'nCount_RNA', 'nFeature_RNA', 'RNA_snn_res.0.4', 'seurat_clusters', 'set', 'Resp', 'disease', 'subj_code', 'covidpt_orhealth', 'mito', 'ncount', 'nfeat', 'bust_21', 'og_clust', 'severmod_other', 'og_clusts', 'nCount_ADT', 'nFeature_ADT', 'UMAP1', 'UMAP2', 'final_clust', 'final_clust_v2', 'new_pt_id', 'Resp_og', 'final_clust_withnum', 'final_clust_review', 'Age', 'Gender', 'Gender_num'" ] }, "execution_count": 5, "metadata": {}, "output_type": "execute_result" } ], "source": [ "dataset" ] }, { "cell_type": "markdown", "id": "electrical-consumption", "metadata": {}, "source": [ "`view of obsm` means that the wrapper object doesn’t copy anything from .obsm of the underlying AnnData object.\n", "`obs` instead of `view of obs` means that the object copied .obs from the AnnData object.\n", "You can configure what is copied, please see the AnnCollection tutorial for details.\n", "The wrapper object (AnnCollection) never copies the full .X or .layers from the underlying AnnData object, all conversions happen when the AnnCollection object is subset." ] }, { "cell_type": "markdown", "id": "earned-jonathan", "metadata": {}, "source": [ "But most importantly you can lazily join several AnnData objects without copying them.\n", "Set `join_vars='inner'` if you have different variables in the AnnData objects to create a joint index of the intersection of the variables" ] }, { "cell_type": "code", "execution_count": 6, "id": "cooked-naples", "metadata": {}, "outputs": [], "source": [ "dataset = AnnCollection(\n", " [covid, pbmc], join_vars=\"inner\", join_obs=None, label=\"dataset\"\n", ")" ] }, { "cell_type": "markdown", "id": "treated-third", "metadata": {}, "source": [ "`label='dataset'` means the AnnCollection object will have the column 'dataset' in its own .obs." ] }, { "cell_type": "code", "execution_count": 7, "id": "destroyed-interference", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "AnnCollection object with n_obs × n_vars = 219433 × 20729\n", " constructed from 2 AnnData objects\n", " view of obs: 'orig.ident', 'nCount_RNA', 'nFeature_RNA', 'nCount_ADT', 'nFeature_ADT'\n", " own obs: 'dataset'" ] }, "execution_count": 7, "metadata": {}, "output_type": "execute_result" } ], "source": [ "dataset" ] }, { "cell_type": "markdown", "id": "changed-price", "metadata": {}, "source": [ "This object doesn’t copy data from AnnData objects, it uses joint index of observations and variables of the AnnData objects to allow joint subsetting.\n", "AnnCollection object has its own .obs and also view of inner join (you can select the type of join for .obs on initialization) of .obs in the passed AnnData objects.\n", "An AnnCollection object can also have its own .obsm if `join_obsm='inner'` is passed." ] }, { "cell_type": "markdown", "id": "female-watson", "metadata": {}, "source": [ "You can also initialize an AnnCollection object with backed AnnData objects" ] }, { "cell_type": "code", "execution_count": 8, "id": "suitable-census", "metadata": {}, "outputs": [], "source": [ "covid = sc.read(\"covid_cite.h5ad\", backed=\"r\")\n", "pbmc = sc.read(\"pbmc_seurat_v4.h5ad\", backed=\"r\")" ] }, { "cell_type": "code", "execution_count": 9, "id": "israeli-singing", "metadata": {}, "outputs": [], "source": [ "dataset = AnnCollection(\n", " [covid, pbmc], join_vars=\"inner\", join_obs=None, label=\"dataset\"\n", ")" ] }, { "cell_type": "code", "execution_count": 10, "id": "enclosed-hello", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "AnnCollection object with n_obs × n_vars = 219433 × 20729\n", " constructed from 2 AnnData objects\n", " view of obs: 'orig.ident', 'nCount_RNA', 'nFeature_RNA', 'nCount_ADT', 'nFeature_ADT'\n", " own obs: 'dataset'" ] }, "execution_count": 10, "metadata": {}, "output_type": "execute_result" } ], "source": [ "dataset" ] }, { "cell_type": "markdown", "id": "liable-walker", "metadata": {}, "source": [ "You can subset AnnCollection to get observation related attributes of the lazily concatenated AnnData objects." ] }, { "cell_type": "code", "execution_count": 11, "id": "variable-delaware", "metadata": {}, "outputs": [], "source": [ "subset = dataset[56000:58000]" ] }, { "cell_type": "code", "execution_count": 12, "id": "deadly-manchester", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "AnnCollectionView object with n_obs × n_vars = 2000 × 20729\n", " obs: 'orig.ident', 'nCount_RNA', 'nFeature_RNA', 'nCount_ADT', 'nFeature_ADT'" ] }, "execution_count": 12, "metadata": {}, "output_type": "execute_result" } ], "source": [ "subset" ] }, { "cell_type": "code", "execution_count": 13, "id": "affecting-factory", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "(2000, 20729)" ] }, "execution_count": 13, "metadata": {}, "output_type": "execute_result" } ], "source": [ "subset.X.shape" ] }, { "cell_type": "code", "execution_count": 14, "id": "seventh-discretion", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "View of obs with keys: 'orig.ident', 'nCount_RNA', 'nFeature_RNA', 'nCount_ADT', 'nFeature_ADT'" ] }, "execution_count": 14, "metadata": {}, "output_type": "execute_result" } ], "source": [ "subset.obs" ] }, { "cell_type": "code", "execution_count": 15, "id": "relative-military", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "array(['SeuratProject', 'SeuratProject'], dtype=object)" ] }, "execution_count": 15, "metadata": {}, "output_type": "execute_result" } ], "source": [ "subset.obs[\"orig.ident\"][:2]" ] }, { "cell_type": "markdown", "id": "progressive-facing", "metadata": {}, "source": [ "Subsetting of an AnnCollection object returns AnnCollectionView.\n", "Only these subset objects have views of .obs, .obsm, .layers, .X from the passed AnnData objects.\n", "When an AnnCollection object is initialized with `join_obs=None` and `join_obsm=None`,\n", "the subset objects only give you access to the views of .obs, .obsm, .layers, .X from the passed AnnData objects, and not to .obs of AnnCollection.\n", "This is why you don’t see dataset key among the .obs keys of the subset." ] }, { "cell_type": "markdown", "id": "hindu-walker", "metadata": {}, "source": [ "However, if you initialize an AnnCollection object with `join_obs='inner'` (for inner join of .obs in the passed AnnData objects) or `join_obs='outer'`,\n", "the AnnCollection with join and copy the result to its own .obs instead of creating a view.\n", "Then subset objects (AnnCollectionView) will give access to the .obs attribute of the AnnCollection object instead of the view of the lazily joined .obs of the underlying AnnData objects.\n", "When `join_obs` is not `None`, you have a copy of the concatenated .obs attributes in the AnnCollection object." ] }, { "cell_type": "code", "execution_count": 16, "id": "legal-reason", "metadata": {}, "outputs": [], "source": [ "dataset = AnnCollection(\n", " [covid, pbmc], join_vars=\"inner\", join_obs=\"inner\", label=\"dataset\"\n", ")" ] }, { "cell_type": "code", "execution_count": 17, "id": "placed-segment", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "AnnCollection object with n_obs × n_vars = 219433 × 20729\n", " constructed from 2 AnnData objects\n", " obs: 'orig.ident', 'nCount_RNA', 'nFeature_RNA', 'nCount_ADT', 'nFeature_ADT', 'dataset'" ] }, "execution_count": 17, "metadata": {}, "output_type": "execute_result" } ], "source": [ "dataset" ] }, { "cell_type": "markdown", "id": "killing-thanksgiving", "metadata": {}, "source": [ "As you see, now the concatenated .obs attributes are copied to .obs of the AnnCollection object." ] }, { "cell_type": "code", "execution_count": 18, "id": "boring-slope", "metadata": {}, "outputs": [], "source": [ "subset = dataset[56000:58000]" ] }, { "cell_type": "code", "execution_count": 19, "id": "proof-replication", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "AnnCollectionView object with n_obs × n_vars = 2000 × 20729\n", " obs: 'orig.ident', 'nCount_RNA', 'nFeature_RNA', 'nCount_ADT', 'nFeature_ADT', 'dataset'" ] }, "execution_count": 19, "metadata": {}, "output_type": "execute_result" } ], "source": [ "subset" ] }, { "cell_type": "markdown", "id": "average-graduate", "metadata": {}, "source": [ "Note that now you can have access to the 'dataset'.obs column from subsets." ] }, { "cell_type": "code", "execution_count": 20, "id": "residential-elimination", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "GTTGTCCAGCAACAAT-1 0\n", "GTTGTCCAGCGAGAAA-1 0\n", "Name: dataset, dtype: category\n", "Categories (2, object): ['0', '1']" ] }, "execution_count": 20, "metadata": {}, "output_type": "execute_result" } ], "source": [ "subset.obs[\"dataset\"][:2]" ] }, { "cell_type": "markdown", "id": "hollow-switch", "metadata": {}, "source": [ "Same can be done for .obsm, though only with inner join.\n", "`join_obsm='inner'` means do inner join and copy AnnCollection’s own obsm, `join_obsm=None` means do inner join without copying and do not create obsm in an AnnCollection object,\n", "though AnncollectionView will still have access to the lazily inner joined .obsm attributes of the underlying AnnData objects." ] }, { "cell_type": "code", "execution_count": 21, "id": "placed-banking", "metadata": {}, "outputs": [], "source": [ "# inner join of .obsm of the datasets covid and pbmc is empty\n", "dataset = AnnCollection(\n", " [covid, pbmc], join_vars=\"inner\", join_obsm=None, label=\"dataset\"\n", ")" ] }, { "cell_type": "code", "execution_count": 22, "id": "neural-stand", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "True" ] }, "execution_count": 22, "metadata": {}, "output_type": "execute_result" } ], "source": [ "dataset.obsm is None" ] }, { "cell_type": "code", "execution_count": 23, "id": "threatened-johnston", "metadata": {}, "outputs": [], "source": [ "# inner join of .obsm of the datasets covid and pbmc is empty\n", "dataset = AnnCollection(\n", " [covid, pbmc], join_vars=\"inner\", join_obsm=\"inner\", label=\"dataset\"\n", ")" ] }, { "cell_type": "code", "execution_count": 24, "id": "dense-customer", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "False" ] }, "execution_count": 24, "metadata": {}, "output_type": "execute_result" } ], "source": [ "dataset.obsm is None" ] }, { "cell_type": "code", "execution_count": 25, "id": "funny-threshold", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "AxisArrays with keys: " ] }, "execution_count": 25, "metadata": {}, "output_type": "execute_result" } ], "source": [ "dataset.obsm" ] }, { "cell_type": "code", "execution_count": 26, "id": "funded-dryer", "metadata": {}, "outputs": [], "source": [ "subset = dataset[56000:58000]" ] }, { "cell_type": "code", "execution_count": 27, "id": "isolated-juice", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "View of obs with keys: 'orig.ident', 'nCount_RNA', 'nFeature_RNA', 'nCount_ADT', 'nFeature_ADT', 'dataset'" ] }, "execution_count": 27, "metadata": {}, "output_type": "execute_result" } ], "source": [ "subset.obs" ] }, { "cell_type": "markdown", "id": "private-coach", "metadata": {}, "source": [ ".obs of AnnCollectionView objects is a MapObsView object, but it can be converted to pandas DataFrame." ] }, { "cell_type": "code", "execution_count": 28, "id": "finite-yesterday", "metadata": {}, "outputs": [ { "data": { "text/html": [ "
| \n", " | orig.ident | \n", "nCount_RNA | \n", "nFeature_RNA | \n", "nCount_ADT | \n", "nFeature_ADT | \n", "dataset | \n", "
|---|---|---|---|---|---|---|
| GTTGTCCAGCAACAAT-1 | \n", "SeuratProject | \n", "3763.0 | \n", "1353 | \n", "268.0 | \n", "35 | \n", "0 | \n", "
| GTTGTCCAGCGAGAAA-1 | \n", "SeuratProject | \n", "5883.0 | \n", "1931 | \n", "515.0 | \n", "32 | \n", "0 | \n", "
| GTTGTCCAGTGGTTCT-1 | \n", "SeuratProject | \n", "1479.0 | \n", "750 | \n", "147.0 | \n", "34 | \n", "0 | \n", "
| GTTGTCCCAAACGTGG-1 | \n", "SeuratProject | \n", "8920.0 | \n", "2588 | \n", "788.0 | \n", "37 | \n", "0 | \n", "
| GTTGTCCCAGCAGGAT-1 | \n", "SeuratProject | \n", "3016.0 | \n", "1261 | \n", "257.0 | \n", "34 | \n", "0 | \n", "
| ... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "
| L1_AACGTCACAACTCATG | \n", "P1_0 | \n", "4571.0 | \n", "1838 | \n", "4667.0 | \n", "218 | \n", "1 | \n", "
| L1_AACGTCACAATCCTAG | \n", "P4_7 | \n", "4351.0 | \n", "1234 | \n", "6718.0 | \n", "213 | \n", "1 | \n", "
| L1_AACGTCACAGCTCGGT | \n", "P1_0 | \n", "3922.0 | \n", "1602 | \n", "4267.0 | \n", "219 | \n", "1 | \n", "
| L1_AACGTCACAGGAATAT | \n", "P3_7 | \n", "6033.0 | \n", "1395 | \n", "5277.0 | \n", "213 | \n", "1 | \n", "
| L1_AACGTCAGTAGTCACT | \n", "P4_0 | \n", "9649.0 | \n", "2610 | \n", "7159.0 | \n", "221 | \n", "1 | \n", "
2000 rows × 6 columns
\n", "