{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Getting started with anndata" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Authors:** [Adam Gayoso](https://twitter.com/adamgayoso), [Alex Wolf](https://twitter.com/falexwolf)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ ":::{note}\n", "This tutorial is based on a blog posts by [Adam in 2021](https://adamgayoso.com/posts/ten_min_to_adata/) and [Alex in 2017](https://falexwolf.me/2017/introducing-anndata/). \n", ":::" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "In this tutorial, we introduce basic properties of the central object, [AnnData](http://anndata.readthedocs.io/en/latest/anndata.AnnData.html) (“Annotated Data”).\n", "\n", "`AnnData` is specifically designed for matrix-like data.\n", "By this we mean that we have $n$ observations, each of which can be represented as $d$-dimensional vectors, where each dimension corresponds to a variable or feature.\n", "Both the rows and columns of this $n \\times d$ matrix are special in the sense that they are indexed.\n", "\n", "For instance, in scRNA-seq data, each row corresponds to a cell with a barcode, and each column corresponds to a gene with a gene id.\n", "Furthermore, for each cell and each gene we might have additional metadata, like (1) donor information for each cell, or (2) alternative gene symbols for each gene.\n", "Finally, we might have other unstructured metadata like color palletes to use for plotting.\n", "Without going into every fancy Python-based data structure, we think that still today no other alternative really exists that:\n", "\n", "* Handles sparsity\n", "* Handles unstructured data\n", "* Handles observation- and feature-level metadata\n", "* Is user-friendly" ] }, { "cell_type": "code", "execution_count": 86, "metadata": { "jupyter": { "outputs_hidden": false } }, "outputs": [ { "data": { "text/plain": [ "'0.13.0.dev180+ga7ba5620b.d20260513'" ] }, "execution_count": 86, "metadata": {}, "output_type": "execute_result" } ], "source": [ "from __future__ import annotations\n", "\n", "from importlib.metadata import version\n", "\n", "import numpy as np\n", "import pandas as pd\n", "from scipy.sparse import csr_matrix\n", "\n", "import anndata as ad\n", "\n", "version(\"anndata\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Initializing AnnData" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Let’s start by building a basic AnnData object with some sparse count information, perhaps representing gene expression counts." ] }, { "cell_type": "code", "execution_count": 36, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "AnnData object with n_obs × n_vars = 100 × 2000\n", " layers: None" ] }, "execution_count": 36, "metadata": {}, "output_type": "execute_result" } ], "source": [ "counts = csr_matrix(np.random.poisson(1, size=(100, 2000)), dtype=np.float32)\n", "adata = ad.AnnData(counts)\n", "adata" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We see that AnnData provides a representation with summary stastics of the data The initial data we passed are accessible as a sparse matrix using `adata.X`." ] }, { "cell_type": "code", "execution_count": 37, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "" ] }, "execution_count": 37, "metadata": {}, "output_type": "execute_result" } ], "source": [ "adata.X" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now, we provide the index to both the `obs` and `var` axes using `.obs_names` (resp. `.var_names`)." ] }, { "cell_type": "code", "execution_count": 38, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Index(['Cell_0', 'Cell_1', 'Cell_2', 'Cell_3', 'Cell_4', 'Cell_5', 'Cell_6',\n", " 'Cell_7', 'Cell_8', 'Cell_9'],\n", " dtype='str')\n" ] } ], "source": [ "adata.obs_names = [f\"Cell_{i:d}\" for i in range(adata.n_obs)]\n", "adata.var_names = [f\"Gene_{i:d}\" for i in range(adata.n_vars)]\n", "print(adata.obs_names[:10])" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Subsetting AnnData" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "These index values can be used to subset the AnnData, which provides a view of the AnnData object.\n", "We can imagine this to be useful to subset the AnnData to particular cell types or gene modules of interest.\n", "The rules for subsetting AnnData are quite similar to that of a Pandas DataFrame.\n", "You can use values in the `obs/var_names`, boolean masks, or cell index integers." ] }, { "cell_type": "code", "execution_count": 39, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "View of AnnData object with n_obs × n_vars = 2 × 2\n", " layers: None" ] }, "execution_count": 39, "metadata": {}, "output_type": "execute_result" } ], "source": [ "adata[[\"Cell_1\", \"Cell_10\"], [\"Gene_5\", \"Gene_1900\"]]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Adding aligned metadata\n", "\n", "### Observation/Variable level" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "So we have the core of our object and now we’d like to add metadata at both the observation and variable levels.\n", "This is pretty simple with AnnData, both `adata.obs` and `adata.var` are Pandas DataFrames." ] }, { "cell_type": "code", "execution_count": 40, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
cell_type
Cell_0T
Cell_1Monocyte
Cell_2B
Cell_3Monocyte
Cell_4B
......
Cell_95Monocyte
Cell_96Monocyte
Cell_97B
Cell_98T
Cell_99B
\n", "

100 rows × 1 columns

\n", "
" ], "text/plain": [ " cell_type\n", "Cell_0 T\n", "Cell_1 Monocyte\n", "Cell_2 B\n", "Cell_3 Monocyte\n", "Cell_4 B\n", "... ...\n", "Cell_95 Monocyte\n", "Cell_96 Monocyte\n", "Cell_97 B\n", "Cell_98 T\n", "Cell_99 B\n", "\n", "[100 rows x 1 columns]" ] }, "execution_count": 40, "metadata": {}, "output_type": "execute_result" } ], "source": [ "ct = np.random.choice([\"B\", \"T\", \"Monocyte\"], size=(adata.n_obs,))\n", "adata.obs[\"cell_type\"] = pd.Categorical(ct) # Categoricals are preferred for efficiency\n", "adata.obs" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We can also see now that the AnnData representation has been updated:" ] }, { "cell_type": "code", "execution_count": 41, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "AnnData object with n_obs × n_vars = 100 × 2000\n", " obs: 'cell_type'\n", " layers: None" ] }, "execution_count": 41, "metadata": {}, "output_type": "execute_result" } ], "source": [ "adata" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Subsetting using metadata" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We can also subset the AnnData using these randomly generated cell types:" ] }, { "cell_type": "code", "execution_count": 42, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "View of AnnData object with n_obs × n_vars = 36 × 2000\n", " obs: 'cell_type'\n", " layers: None" ] }, "execution_count": 42, "metadata": {}, "output_type": "execute_result" } ], "source": [ "bdata = adata[adata.obs.cell_type == \"B\"]\n", "bdata" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Observation/variable-level matrices" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We might also have metadata at either level that has many dimensions to it, such as a UMAP embedding of the data.\n", "For this type of metadata, AnnData has the `.obsm/.varm` attributes.\n", "We use keys to identify the different matrices we insert.\n", "The restriction of `.obsm/.varm` are that `.obsm` matrices must length equal to the number of observations as `.n_obs` and `.varm` matrices must length equal to `.n_vars`.\n", "They can each independently have different number of dimensions.\n", "\n", "Let’s start with a randomly generated matrix that we can interpret as a UMAP embedding of the data we’d like to store, as well as some random gene-level metadata:" ] }, { "cell_type": "code", "execution_count": 43, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "AxisArrays with keys: 'X_umap'" ] }, "execution_count": 43, "metadata": {}, "output_type": "execute_result" } ], "source": [ "adata.obsm[\"X_umap\"] = np.random.normal(0, 1, size=(adata.n_obs, 2))\n", "adata.varm[\"gene_stuff\"] = np.random.normal(0, 1, size=(adata.n_vars, 5))\n", "adata.obsm" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Again, the AnnData representation is updated." ] }, { "cell_type": "code", "execution_count": 44, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "AnnData object with n_obs × n_vars = 100 × 2000\n", " obs: 'cell_type'\n", " obsm: 'X_umap'\n", " varm: 'gene_stuff'\n", " layers: None" ] }, "execution_count": 44, "metadata": {}, "output_type": "execute_result" } ], "source": [ "adata" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "A few more notes about `.obsm/.varm`\n", "\n", "1. The “array-like” metadata can originate from a Pandas DataFrame, scipy sparse matrix, or numpy dense array.\n", "2. When using scanpy, their values (columns) are not easily plotted, where instead items from `.obs` are easily plotted on, e.g., UMAP plots." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Unstructured metadata" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "AnnData has `.uns`, which allows for any unstructured metadata.\n", "This can be anything, like a list or a dictionary with some general information that was useful in the analysis of our data." ] }, { "cell_type": "code", "execution_count": 45, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "OrderedDict([('random', [1, 2, 3])])" ] }, "execution_count": 45, "metadata": {}, "output_type": "execute_result" } ], "source": [ "adata.uns[\"random\"] = [1, 2, 3]\n", "adata.uns" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Layers" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Finally, we may have different forms of our original core data, perhaps one that is normalized and one that is not.\n", "These can be stored in different layers in AnnData.\n", "For example, let’s log transform the original data and store it in a layer:" ] }, { "cell_type": "code", "execution_count": 46, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "AnnData object with n_obs × n_vars = 100 × 2000\n", " obs: 'cell_type'\n", " uns: 'random'\n", " obsm: 'X_umap'\n", " varm: 'gene_stuff'\n", " layers: None, 'log_transformed'" ] }, "execution_count": 46, "metadata": {}, "output_type": "execute_result" } ], "source": [ "adata.layers[\"log_transformed\"] = np.log1p(adata.X)\n", "adata" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Conversion to DataFrames" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We can also ask AnnData to return us a DataFrame from one of the layers:" ] }, { "cell_type": "code", "execution_count": 47, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
Gene_0Gene_1Gene_2Gene_3Gene_4Gene_5Gene_6Gene_7Gene_8Gene_9...Gene_1990Gene_1991Gene_1992Gene_1993Gene_1994Gene_1995Gene_1996Gene_1997Gene_1998Gene_1999
Cell_01.0986120.6931470.6931470.0000000.0000001.0986120.0000000.0000001.3862940.000000...1.3862940.6931471.0986121.0986121.0986120.6931471.3862940.6931470.0000000.000000
Cell_10.6931470.0000001.0986120.0000000.0000000.0000000.6931470.6931470.6931470.000000...0.0000000.6931471.0986120.6931470.0000000.0000001.0986120.6931470.6931470.693147
Cell_20.6931470.0000000.6931470.0000000.6931470.0000000.0000001.3862940.6931471.098612...0.6931470.6931470.6931470.0000000.6931470.6931470.0000000.0000000.0000000.000000
Cell_31.3862940.6931471.0986120.0000000.0000000.6931470.6931470.0000000.6931470.693147...0.0000001.3862940.6931471.0986120.0000000.0000000.6931470.6931470.6931470.000000
Cell_40.0000000.0000001.3862940.6931470.6931470.6931470.6931470.6931470.6931471.098612...0.6931470.0000000.0000000.6931471.0986121.0986120.6931470.6931471.0986120.000000
..................................................................
Cell_950.6931470.6931470.0000000.0000000.0000000.6931471.0986121.0986120.0000000.693147...0.0000000.0000001.0986120.6931470.6931470.6931470.6931470.0000000.0000000.000000
Cell_960.0000001.0986121.0986120.0000001.3862940.6931470.6931470.6931471.0986120.000000...0.0000000.0000001.0986120.0000000.0000000.6931470.6931470.6931471.0986121.386294
Cell_970.6931471.0986120.6931470.6931470.0000000.6931471.0986120.6931471.3862940.693147...0.6931470.6931470.0000000.0000000.0000000.0000000.0000000.6931470.6931471.386294
Cell_980.0000000.6931470.6931470.6931470.6931471.0986120.6931470.0000000.0000000.000000...0.0000000.0000000.6931471.6094381.3862940.0000000.0000000.0000000.6931470.693147
Cell_990.0000000.6931470.0000000.6931470.6931470.0000000.6931470.0000000.6931470.000000...0.0000001.0986121.0986121.0986121.0986120.0000000.0000000.0000000.0000000.000000
\n", "

100 rows × 2000 columns

\n", "
" ], "text/plain": [ " Gene_0 Gene_1 Gene_2 Gene_3 Gene_4 Gene_5 Gene_6 \\\n", "Cell_0 1.098612 0.693147 0.693147 0.000000 0.000000 1.098612 0.000000 \n", "Cell_1 0.693147 0.000000 1.098612 0.000000 0.000000 0.000000 0.693147 \n", "Cell_2 0.693147 0.000000 0.693147 0.000000 0.693147 0.000000 0.000000 \n", "Cell_3 1.386294 0.693147 1.098612 0.000000 0.000000 0.693147 0.693147 \n", "Cell_4 0.000000 0.000000 1.386294 0.693147 0.693147 0.693147 0.693147 \n", "... ... ... ... ... ... ... ... \n", "Cell_95 0.693147 0.693147 0.000000 0.000000 0.000000 0.693147 1.098612 \n", "Cell_96 0.000000 1.098612 1.098612 0.000000 1.386294 0.693147 0.693147 \n", "Cell_97 0.693147 1.098612 0.693147 0.693147 0.000000 0.693147 1.098612 \n", "Cell_98 0.000000 0.693147 0.693147 0.693147 0.693147 1.098612 0.693147 \n", "Cell_99 0.000000 0.693147 0.000000 0.693147 0.693147 0.000000 0.693147 \n", "\n", " Gene_7 Gene_8 Gene_9 ... Gene_1990 Gene_1991 Gene_1992 \\\n", "Cell_0 0.000000 1.386294 0.000000 ... 1.386294 0.693147 1.098612 \n", "Cell_1 0.693147 0.693147 0.000000 ... 0.000000 0.693147 1.098612 \n", "Cell_2 1.386294 0.693147 1.098612 ... 0.693147 0.693147 0.693147 \n", "Cell_3 0.000000 0.693147 0.693147 ... 0.000000 1.386294 0.693147 \n", "Cell_4 0.693147 0.693147 1.098612 ... 0.693147 0.000000 0.000000 \n", "... ... ... ... ... ... ... ... \n", "Cell_95 1.098612 0.000000 0.693147 ... 0.000000 0.000000 1.098612 \n", "Cell_96 0.693147 1.098612 0.000000 ... 0.000000 0.000000 1.098612 \n", "Cell_97 0.693147 1.386294 0.693147 ... 0.693147 0.693147 0.000000 \n", "Cell_98 0.000000 0.000000 0.000000 ... 0.000000 0.000000 0.693147 \n", "Cell_99 0.000000 0.693147 0.000000 ... 0.000000 1.098612 1.098612 \n", "\n", " Gene_1993 Gene_1994 Gene_1995 Gene_1996 Gene_1997 Gene_1998 \\\n", "Cell_0 1.098612 1.098612 0.693147 1.386294 0.693147 0.000000 \n", "Cell_1 0.693147 0.000000 0.000000 1.098612 0.693147 0.693147 \n", "Cell_2 0.000000 0.693147 0.693147 0.000000 0.000000 0.000000 \n", "Cell_3 1.098612 0.000000 0.000000 0.693147 0.693147 0.693147 \n", "Cell_4 0.693147 1.098612 1.098612 0.693147 0.693147 1.098612 \n", "... ... ... ... ... ... ... \n", "Cell_95 0.693147 0.693147 0.693147 0.693147 0.000000 0.000000 \n", "Cell_96 0.000000 0.000000 0.693147 0.693147 0.693147 1.098612 \n", "Cell_97 0.000000 0.000000 0.000000 0.000000 0.693147 0.693147 \n", "Cell_98 1.609438 1.386294 0.000000 0.000000 0.000000 0.693147 \n", "Cell_99 1.098612 1.098612 0.000000 0.000000 0.000000 0.000000 \n", "\n", " Gene_1999 \n", "Cell_0 0.000000 \n", "Cell_1 0.693147 \n", "Cell_2 0.000000 \n", "Cell_3 0.000000 \n", "Cell_4 0.000000 \n", "... ... \n", "Cell_95 0.000000 \n", "Cell_96 1.386294 \n", "Cell_97 1.386294 \n", "Cell_98 0.693147 \n", "Cell_99 0.000000 \n", "\n", "[100 rows x 2000 columns]" ] }, "execution_count": 47, "metadata": {}, "output_type": "execute_result" } ], "source": [ "adata.to_df(layer=\"log_transformed\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We see that the `.obs_names/.var_names` are used in the creation of this Pandas object." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Writing the results to disk" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "`AnnData` comes with its own persistent zarr-based file format in addition to an `h5` backed format, `h5ad`.\n", "We recommend `zarr` for a few reasons and offer some guidelines on how to use it well in our {doc}`/tutorials/zarr-v3`.\n", "\n", "`zarr` generally has much better performance than `h5` thanks to its directory structure/inherent-parallelism and can be further accelerated via [`zarrs`](https://zarrs-python.readthedocs.io/en/latest/) (which will be used by default if installed as of `anndata>=0.13` in {func}`anndata.AnnData.write_zarr` and {func}`anndata.io.read_zarr`).\n", "`zarr` is also a cloud-native format featuring an `async` backend (see {doc}`/tutorials/notebooks/read_lazy`).\n", "\n", "By contrast, `h5` files are neither cloud native nor parallelized for reads/writes.\n", "\n", "Zarr stores, despite being directories of small-ish files (in contrast to single-file `h5` files), are by default [sharded](https://zarr.readthedocs.io/en/latest/user-guide/performance/#sharding) as of `anndata>=0.13` and thus should not overwhelm filesystems or produce more than a few dozen files for most datasets.\n", "You can also zip zarr files up, manually or by using a [`ZipStore`](https://zarr.readthedocs.io/en/stable/api/zarr/storage/#zarr.storage.ZipStore) to achieve single-file usage." ] }, { "cell_type": "code", "execution_count": 48, "metadata": {}, "outputs": [], "source": [ "adata.write_zarr(\"my_results.zarr\")\n", "adata.write_h5ad(\"my_results.h5ad\")" ] }, { "cell_type": "code", "execution_count": 49, "metadata": { "jupyter": { "outputs_hidden": false } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\u001b[34mX\u001b[m\u001b[m \u001b[34mlayers\u001b[m\u001b[m \u001b[34mobs\u001b[m\u001b[m \u001b[34mobsm\u001b[m\u001b[m \u001b[34mobsp\u001b[m\u001b[m \u001b[34mraw\u001b[m\u001b[m \u001b[34muns\u001b[m\u001b[m \u001b[34mvar\u001b[m\u001b[m \u001b[34mvarm\u001b[m\u001b[m \u001b[34mvarp\u001b[m\u001b[m\n" ] } ], "source": [ "!ls 'my_results.zarr'" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Wrapping up the introduction" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "AnnData has become the standard for single-cell analysis in Python and for good reason – it’s straightforward to use and faciliatates more reproducible analyses with it’s key-based storage.\n", "It’s even becoming easier to convert to the popular R-based formats for single-cell analysis.\n", "\n", "Keep reading on to better understand “views”, on-disk backing, and other details." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Views and copies" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "For the fun of it, let’s look at another metadata use case.\n", "Imagine that the observations come from instruments characterizing 10 readouts in a multi-year study with samples taken from different subjects at different sites.\n", "We’d typically get that information in some format and then store it in a DataFrame:" ] }, { "cell_type": "code", "execution_count": 50, "metadata": { "jupyter": { "outputs_hidden": false } }, "outputs": [], "source": [ "obs_meta = pd.DataFrame(\n", " {\n", " \"time_yr\": np.random.choice([0, 2, 4, 8], adata.n_obs),\n", " \"subject_id\": np.random.choice(\n", " [\"subject 1\", \"subject 2\", \"subject 4\", \"subject 8\"], adata.n_obs\n", " ),\n", " \"instrument_type\": np.random.choice([\"type a\", \"type b\"], adata.n_obs),\n", " \"site\": np.random.choice([\"site x\", \"site y\"], adata.n_obs),\n", " },\n", " index=adata.obs.index, # these are the same IDs of observations as above!\n", ")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "This is how we join the readout data with the metadata.\n", "Of course, the first argument of the following call for `X` could also just be a DataFrame." ] }, { "cell_type": "code", "execution_count": 51, "metadata": { "jupyter": { "outputs_hidden": false } }, "outputs": [], "source": [ "adata = ad.AnnData(adata.X, obs=obs_meta, var=adata.var)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now we again have a single data container that keeps track of everything." ] }, { "cell_type": "code", "execution_count": 52, "metadata": { "jupyter": { "outputs_hidden": false } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "AnnData object with n_obs × n_vars = 100 × 2000\n", " obs: 'time_yr', 'subject_id', 'instrument_type', 'site'\n", " layers: None\n" ] } ], "source": [ "print(adata)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Subsetting the joint data matrix can be important to focus on subsets of variables or observations, or to define train-test splits for a machine learning model." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ ":::{note}\n", "Similar to numpy arrays, AnnData objects can either hold actual data or reference another `AnnData` object.\n", "In the later case, they are referred to as “view”.\n", "\n", "Subsetting AnnData objects always returns views, which has two advantages:\n", "\n", "- no new memory is allocated\n", "- it is possible to modify the underlying AnnData object\n", "\n", "You can get an actual AnnData object from a view by calling `.copy()` on the view.\n", "Usually, this is not necessary, as any modification of elements of a view (calling `.[]` on an attribute of the view) internally calls `.copy()` and makes the view an AnnData object that holds actual data.\n", "See the example below.\n", ":::" ] }, { "cell_type": "code", "execution_count": 53, "metadata": { "jupyter": { "outputs_hidden": false } }, "outputs": [ { "data": { "text/plain": [ "AnnData object with n_obs × n_vars = 100 × 2000\n", " obs: 'time_yr', 'subject_id', 'instrument_type', 'site'\n", " layers: None" ] }, "execution_count": 53, "metadata": {}, "output_type": "execute_result" } ], "source": [ "adata" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Get access to the first 5 rows for two variables." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ ":::{note}\n", "Indexing into AnnData will assume that integer arguments to `[]` behave like `.iloc` in pandas, whereas string arguments behave like `.loc`. `AnnData` always assumes string indices.\n", ":::" ] }, { "cell_type": "code", "execution_count": 54, "metadata": { "jupyter": { "outputs_hidden": false } }, "outputs": [ { "data": { "text/plain": [ "View of AnnData object with n_obs × n_vars = 5 × 2\n", " obs: 'time_yr', 'subject_id', 'instrument_type', 'site'\n", " layers: None" ] }, "execution_count": 54, "metadata": {}, "output_type": "execute_result" } ], "source": [ "adata[:5, [\"Gene_1\", \"Gene_3\"]]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "This is a view!\n", "If we want an `AnnData` that holds the data in memory, let’s call `.copy()`" ] }, { "cell_type": "code", "execution_count": 55, "metadata": {}, "outputs": [], "source": [ "adata_subset = adata[:5, [\"Gene_1\", \"Gene_3\"]].copy()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "If you try to access parts of a view of an AnnData, the content will be auto-copied and a data-storing object will be generated." ] }, { "cell_type": "code", "execution_count": 57, "metadata": { "jupyter": { "outputs_hidden": false } }, "outputs": [], "source": [ "adata_subset = adata[:3, [\"Gene_1\", \"Gene_2\"]]" ] }, { "cell_type": "code", "execution_count": 58, "metadata": { "jupyter": { "outputs_hidden": false } }, "outputs": [ { "data": { "text/plain": [ "View of AnnData object with n_obs × n_vars = 3 × 2\n", " obs: 'time_yr', 'subject_id', 'instrument_type', 'site'\n", " layers: None" ] }, "execution_count": 58, "metadata": {}, "output_type": "execute_result" } ], "source": [ "adata_subset" ] }, { "cell_type": "code", "execution_count": 59, "metadata": { "jupyter": { "outputs_hidden": false } }, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "/var/folders/k9/9wc7lvwj2g34_r74kn6cr0nr0000gn/T/ipykernel_12269/2955902014.py:1: ImplicitModificationWarning: Trying to modify attribute `.obs` of view, initializing view as actual.\n", " adata_subset.obs['foo'] = range(3)\n" ] } ], "source": [ "adata_subset.obs[\"foo\"] = range(3)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now `adata_subset` stores the actual data and is no longer just a reference to `adata`." ] }, { "cell_type": "code", "execution_count": 60, "metadata": { "jupyter": { "outputs_hidden": false } }, "outputs": [ { "data": { "text/plain": [ "AnnData object with n_obs × n_vars = 3 × 2\n", " obs: 'time_yr', 'subject_id', 'instrument_type', 'site', 'foo'\n", " layers: None" ] }, "execution_count": 60, "metadata": {}, "output_type": "execute_result" } ], "source": [ "adata_subset" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Evidently, you can use all of pandas to slice with sequences or boolean indices." ] }, { "cell_type": "code", "execution_count": 61, "metadata": { "jupyter": { "outputs_hidden": false } }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
time_yrsubject_idinstrument_typesite
Cell_02subject 8type asite y
Cell_22subject 1type bsite y
Cell_34subject 1type bsite y
Cell_42subject 1type asite x
Cell_72subject 8type bsite y
\n", "
" ], "text/plain": [ " time_yr subject_id instrument_type site\n", "Cell_0 2 subject 8 type a site y\n", "Cell_2 2 subject 1 type b site y\n", "Cell_3 4 subject 1 type b site y\n", "Cell_4 2 subject 1 type a site x\n", "Cell_7 2 subject 8 type b site y" ] }, "execution_count": 61, "metadata": {}, "output_type": "execute_result" } ], "source": [ "adata[adata.obs.time_yr.isin([2, 4])].obs.head()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "(read-partial)=\n", "\n", "## Partial reading of large data" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "If a single `.zarr`/`h5ad` is very large, you can partially read it into memory by using a backed mode, or with {func}`anndata.experimental.read_lazy`.\n", "`read_lazy` gives `dask` objects which are more useful in conjunction with `scanpy`, which supports `dask`:" ] }, { "cell_type": "code", "execution_count": 76, "metadata": {}, "outputs": [], "source": [ "adata_backed = ad.read_h5ad(\"my_results.h5ad\", backed=\"r\")\n", "adata_lazy = ad.experimental.read_lazy(\"my_results.zarr\")" ] }, { "cell_type": "code", "execution_count": 77, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "True" ] }, "execution_count": 77, "metadata": {}, "output_type": "execute_result" } ], "source": [ "adata_backed.isbacked" ] }, { "cell_type": "code", "execution_count": 78, "metadata": {}, "outputs": [ { "data": { "text/html": [ "\n", " \n", " \n", " \n", " \n", "
\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
Array Chunk
Shape (100, 2000) (100, 2000)
Dask graph 1 chunks in 1 graph layer
Data type float32 scipy.sparse._csr.csr_matrix
\n", " 		\n", " \n", "\n", " \n", " \n", " \n", "\n", " \n", " \n", " \n", "\n", " \n", " \n", "\n", " \n", " 2000\n", " 100\n", "\n", "
" ], "text/plain": [ "dask.array" ] }, "execution_count": 78, "metadata": {}, "output_type": "execute_result" } ], "source": [ "adata_lazy.X" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "However, `obs` and `var` are also lazy objects ({class}`~anndata.experimental.backed.Dataset2D`) that are not compatible yet with `scanpy` and need to be brought into memory:" ] }, { "cell_type": "code", "execution_count": 79, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "
<xarray.Dataset> Size: 2kB\n",
       "Dimensions:    (_index: 100)\n",
       "Coordinates:\n",
       "  * _index     (_index) object 800B 'Cell_0' 'Cell_1' ... 'Cell_98' 'Cell_99'\n",
       "Data variables:\n",
       "    cell_type  (_index) category 800B ...\n",
       "Attributes:\n",
       "    is_backed:  True
" ], "text/plain": [ " Size: 2kB\n", "Dimensions: (_index: 100)\n", "Coordinates:\n", " * _index (_index) object 800B 'Cell_0' 'Cell_1' ... 'Cell_98' 'Cell_99'\n", "Data variables:\n", " cell_type (_index) category 800B ...\n", "Attributes:\n", " is_backed: True" ] }, "execution_count": 79, "metadata": {}, "output_type": "execute_result" } ], "source": [ "adata_lazy.obs.ds" ] }, { "cell_type": "code", "execution_count": 80, "metadata": {}, "outputs": [], "source": [ "adata_lazy.obs = adata_lazy.obs.to_memory()\n", "adata_lazy.var = adata_lazy.var.to_memory()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "If you do use backed mode and `h5ad` files, you’ll need to remember that the `AnnData` object has an open connection to the file used for reading:" ] }, { "cell_type": "code", "execution_count": 81, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "PosixPath('my_results.h5ad')" ] }, "execution_count": 81, "metadata": {}, "output_type": "execute_result" } ], "source": [ "adata_backed.filename" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "As we’re using it in read-only mode, we can’t damage anything.\n", "To proceed with this tutorial, we still need to explicitly close it: " ] }, { "cell_type": "code", "execution_count": 82, "metadata": {}, "outputs": [], "source": [ "adata_backed.file.close()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "For this reason, we recommend lazy usage or constructing `AnnData` objects manually, which can be equivalent to backed mode:" ] }, { "cell_type": "code", "execution_count": 84, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "AnnData object with n_obs × n_vars = 100 × 2000\n", " obs: 'cell_type'\n", " layers: None" ] }, "execution_count": 84, "metadata": {}, "output_type": "execute_result" } ], "source": [ "import zarr\n", "\n", "z = zarr.open(\"my_results.zarr\")\n", "adata_backed_zarr = ad.AnnData(\n", " X=ad.io.sparse_dataset(z[\"X\"]),\n", " obs=ad.io.read_elem(z[\"obs\"]),\n", " var=ad.io.read_elem(z[\"var\"]),\n", ")\n", "\n", "adata_backed_zarr" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [] } ], "metadata": { "anaconda-cloud": {}, "kernelspec": { "display_name": "venv", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.12.3" } }, "nbformat": 4, "nbformat_minor": 4 }