{ "cells": [ { "cell_type": "markdown", "id": "faf20ef8-a00d-47dd-a089-74202423b9a7", "metadata": {}, "source": [ "# Awkward Arrays in AnnData\n", "\n", "**Author**: [Gregor Sturm](https://github.com/grst)" ] }, { "cell_type": "markdown", "id": "0238f94a-7b50-47cf-af0f-6cb462d5c47f", "metadata": { "raw_mimetype": "text/restructuredtext", "tags": [] }, "source": [ ":::{warning}\n", "Support for awkward arrays in AnnData is **experimental**.\n", "\n", "Behavior, in particular of {func}`~anndata.concat`, may change in the future. \n", "Please report any issues using the [issue tracker](https://github.com/scverse/anndata)\n", ":::" ] }, { "cell_type": "code", "execution_count": 1, "id": "3547f8e3-d19e-4cb7-bf86-b664ca8cb024", "metadata": {}, "outputs": [], "source": [ "from __future__ import annotations\n", "\n", "import awkward as ak\n", "import scanpy as sc\n", "from biothings_client import get_client as get_biothings_client" ] }, { "cell_type": "markdown", "id": "a5ae115b-ad94-48a5-a84d-09140b605428", "metadata": {}, "source": [ "[Awkward Array](https://awkward-array.org/doc/main/) is a library for working with **nested, variable-sized data** using **NumPy-like idioms**. \n", "It is considerably faster than working with lists-of-lists or lists-of-dicts in Python. \n", "\n", "Here are two simple examples what an awkward array could look like: \n", "\n", "**ragged array:**" ] }, { "cell_type": "code", "execution_count": 2, "id": "6a25de2c-a70a-4a79-bf8f-3a18d1ede78e", "metadata": {}, "outputs": [ { "data": { "text/html": [ "
[None,\n",
       " 2,\n",
       " 4]\n",
       "----------------\n",
       "type: 3 * ?int64
" ], "text/plain": [ "" ] }, "execution_count": 2, "metadata": {}, "output_type": "execute_result" } ], "source": [ "ragged = ak.Array([\n", " None,\n", " [1, 2, 3],\n", " [3, 4],\n", "])\n", "ragged[:, 1]" ] }, { "cell_type": "markdown", "id": "47a2427c-0fc1-4811-acc5-75ff0d0f81d8", "metadata": {}, "source": [ "**list of records:**" ] }, { "cell_type": "code", "execution_count": 3, "id": "d59f1c88-7a61-4cd5-990f-26b1642556a4", "metadata": {}, "outputs": [ { "data": { "text/html": [ "
[1,\n",
       " 3,\n",
       " None]\n",
       "----------------\n",
       "type: 3 * ?int64
" ], "text/plain": [ "" ] }, "execution_count": 3, "metadata": {}, "output_type": "execute_result" } ], "source": [ "records = ak.Array([\n", " {\"a\": 1, \"b\": 2},\n", " {\"a\": 3, \"c\": 4},\n", " {\"d\": 5},\n", "])\n", "records[\"a\"]" ] }, { "cell_type": "markdown", "id": "4bad02dd-0403-4eae-95ba-a13e62e1206a", "metadata": {}, "source": [ "Please refer to the [awkward array documentation](https://awkward-array.org) for more information. \n", "\n", "Since v0.9, awkward arrays are supported in AnnData in the `.layers`, `.obsm`, `.varm` and `.uns` slots. " ] }, { "cell_type": "markdown", "id": "75b641f6-0c8d-4ebf-a57f-def03c97fa69", "metadata": {}, "source": [ "In the following, we will explore how awkward arrays can be useful when working with single-cell data." ] }, { "cell_type": "markdown", "id": "e5599439-dc85-482f-9704-0acf797ccf91", "metadata": {}, "source": [ "## Storing transcripts in `.varm`\n", "\n", "Every gene can have one or many transcripts.\n", "Using awkward arrays, we can store a ragged list of transcripts for each gene in `adata.varm`. " ] }, { "cell_type": "code", "execution_count": 4, "id": "d71cd256-8189-42b8-8fc2-63c9dd51a312", "metadata": {}, "outputs": [], "source": [ "adata = sc.datasets.pbmc3k()" ] }, { "cell_type": "code", "execution_count": 5, "id": "a5f2a529-d4c0-4f96-aee8-37321a7b348c", "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
gene_ids
index
MIR1302-10ENSG00000243485
FAM138AENSG00000237613
OR4F5ENSG00000186092
RP11-34P13.7ENSG00000238009
RP11-34P13.8ENSG00000239945
\n", "
" ], "text/plain": [ " gene_ids\n", "index \n", "MIR1302-10 ENSG00000243485\n", "FAM138A ENSG00000237613\n", "OR4F5 ENSG00000186092\n", "RP11-34P13.7 ENSG00000238009\n", "RP11-34P13.8 ENSG00000239945" ] }, "execution_count": 5, "metadata": {}, "output_type": "execute_result" } ], "source": [ "adata.var.head()" ] }, { "cell_type": "markdown", "id": "0f0b58f8-c2a9-4164-83a4-1517b27a4380", "metadata": {}, "source": [ "Let’s retrieve a list of transcripts for each gene using the [MyGene.info API](https://docs.mygene.info/en/latest/). " ] }, { "cell_type": "code", "execution_count": 6, "id": "17614e98-5dee-486e-a870-958fbb060644", "metadata": {}, "outputs": [], "source": [ "mygene = get_biothings_client(\"gene\")" ] }, { "cell_type": "code", "execution_count": 7, "id": "bb5c7d5d-6b18-450b-a70f-dc3e34d10893", "metadata": {}, "outputs": [], "source": [ "%%capture\n", "mygene_res = mygene.querymany(\n", " adata.var[\"gene_ids\"],\n", " scopes=[\"ensembl.gene\"],\n", " fields=[\"ensembl.transcript\"],\n", " species=\"human\",\n", " as_dataframe=True,\n", ")\n", "# remove duplicated results\n", "mygene_res = mygene_res.loc[~mygene_res.index.duplicated()]\n", "assert adata.var[\"gene_ids\"].tolist() == mygene_res.index.tolist(), (\n", " \"Order of genes does not match\"\n", ")" ] }, { "cell_type": "markdown", "id": "81992a77-9285-4521-a32c-7b7caa65152d", "metadata": {}, "source": [ "The API call returns a data frame with transcripts in the `ensembl.transcript` key:" ] }, { "cell_type": "code", "execution_count": 8, "id": "ed0677b8-0c79-4198-ab90-15ee9d919ec7", "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
_id_scoreensembl.transcriptnotfoundensembl
query
ENSG00000243485ENSG0000024348525.719067[ENST00000469289, ENST00000473358]NaNNaN
ENSG0000023761364552025.719007[ENST00000417324, ENST00000461467]NaNNaN
ENSG000001860927950124.912605ENST00000641515NaNNaN
ENSG00000238009ENSG0000023800925.719582[ENST00000453576, ENST00000466430, ENST0000047...NaNNaN
ENSG00000239945ENSG0000023994525.719145ENST00000495576NaNNaN
\n", "
" ], "text/plain": [ " _id _score \\\n", "query \n", "ENSG00000243485 ENSG00000243485 25.719067 \n", "ENSG00000237613 645520 25.719007 \n", "ENSG00000186092 79501 24.912605 \n", "ENSG00000238009 ENSG00000238009 25.719582 \n", "ENSG00000239945 ENSG00000239945 25.719145 \n", "\n", " ensembl.transcript notfound \\\n", "query \n", "ENSG00000243485 [ENST00000469289, ENST00000473358] NaN \n", "ENSG00000237613 [ENST00000417324, ENST00000461467] NaN \n", "ENSG00000186092 ENST00000641515 NaN \n", "ENSG00000238009 [ENST00000453576, ENST00000466430, ENST0000047... NaN \n", "ENSG00000239945 ENST00000495576 NaN \n", "\n", " ensembl \n", "query \n", "ENSG00000243485 NaN \n", "ENSG00000237613 NaN \n", "ENSG00000186092 NaN \n", "ENSG00000238009 NaN \n", "ENSG00000239945 NaN " ] }, "execution_count": 8, "metadata": {}, "output_type": "execute_result" } ], "source": [ "mygene_res.head()" ] }, { "cell_type": "markdown", "id": "0d27efeb-7a70-4fa0-b8d9-f465de933314", "metadata": {}, "source": [ "Let’s construct an awkward Array from the list of dictionaries and \n", "assign the ragged list of transcripts to `adata.varm`" ] }, { "cell_type": "code", "execution_count": 9, "id": "b6cc25e3-44d6-4515-abe5-3b8ba4c8ab45", "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "/home/sturm/projects/2022/anndata/anndata/_core/aligned_mapping.py:54: ExperimentalFeatureWarning: Support for Awkward Arrays is currently experimental. Behavior may change in the future. Please report any issues you may encounter!\n", " warnings.warn(\n" ] } ], "source": [ "adata.varm[\"transcripts\"] = ak.Array(mygene_res[\"ensembl.transcript\"])" ] }, { "cell_type": "markdown", "id": "69f742f9-cf4d-4127-81ae-c0751acf70bd", "metadata": {}, "source": [ "We can now access transcripts of individual genes by slicing the AnnData object:" ] }, { "cell_type": "code", "execution_count": 10, "id": "1f6ff2d1-9260-44c0-87ee-febf17b1e851", "metadata": {}, "outputs": [ { "data": { "text/html": [ "
[['ENST00000283635', 'ENST00000352580', ..., 'ENST00000699439'],\n",
       " ['ENST00000286758', 'ENST00000506590', 'ENST00000682537']]\n",
       "----------------------------------------------------------------------------------------------------------------\n",
       "type: 2 * union[\n",
       "    var * string,\n",
       "    string,\n",
       "    float64, \n",
       "parameters={"_view_args": ["target-140242149270096", "varm", ["transcripts"]], "__array__": "AwkwardArrayView"}]
" ], "text/plain": [ "" ] }, "execution_count": 10, "metadata": {}, "output_type": "execute_result" } ], "source": [ "adata[:, [\"CD8A\", \"CXCL13\"]].varm[\"transcripts\"]" ] } ], "metadata": { "kernelspec": { "display_name": "Python [conda env:scirpy_dev]", "language": "python", "name": "conda-env-scirpy_dev-py" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.9.9" } }, "nbformat": 4, "nbformat_minor": 5 }