Data frames and AnnData

GAIn supports two resource types for working with tabular and matrix-based data. A data_frame resource contains a table, while an ann_data resource contains an annotated data matrix, commonly used for single-cell gene expression data.

Unlike other genomic resource types, these resources do not have dedicated GAIn Python classes. Instead, GAIn provides loader functions that return objects from the pandas and anndata libraries. The loaders read the data directly, without requiring a separate open() or load() call.

Loading a data frame

A data frame can be loaded from a resource ID and a repository using load_data_frame_from_resource_id():

import gain.grr as grr

repo = grr.build_genomic_resource_repository()
frame = grr.load_data_frame_from_resource_id(
    "DATA_FRAME_RESOURCE_ID", repo
)

If you already have a GenomicResource object, use load_data_frame_from_resource() instead:

resource = repo.get_resource("DATA_FRAME_RESOURCE_ID")
frame = grr.load_data_frame_from_resource(resource)

Both functions are available through gain.grr and return a pandas DataFrame. The resulting table can be accessed using standard pandas operations:

print(frame.shape)
print(frame.head())

The format: setting in the resource definition determines how the table is read. GAIn supports csv (the default), tsv, and excel formats. The parameters: block can pass additional arguments to the corresponding pandas reader, such as read_csv() or read_excel().

For TSV files, a tab separator is used unless another separator is specified. Files ending in .gz are read using gzip compression.

The loader must return a single pandas DataFrame. If the supplied parameters cause pandas to return another type, such as when using chunksize or sheet_name: null, the loader raises a ValueError identifying the resource.

Loading an AnnData object

An ann_data resource contains an annotated data matrix, typically representing cells and their measured molecular features. GAIn uses the anndata library to read these resources.

An AnnData object can be loaded using load_ann_data_from_resource_id():

ann_data = grr.load_ann_data_from_resource_id(
    "ANN_DATA_RESOURCE_ID", repo
)

Alternatively, an existing GenomicResource object can be passed to load_ann_data_from_resource():

resource = repo.get_resource("ANN_DATA_RESOURCE_ID")
ann_data = grr.load_ann_data_from_resource(resource)

Both functions return an AnnData object. Its main components include X, the data matrix, obs, annotations for observations such as cells, and var, annotations for variables such as genes.

For example:

print(ann_data.shape)
print(ann_data.obs.head())
print(ann_data.var.head())

For illustration, an AnnData object containing three cells and four genes might produce the following output:

(3, 4)
       cell_type  n_genes
cell
CELL_1    neuron        3
CELL_2      glia        2
CELL_3    neuron        4

        gene_name  highly_variable
gene
ENSG001      ACTB             True
ENSG002     GAPDH            False
ENSG003    MALAT1             True
ENSG004      XIST            False

The rows of obs represent cells, while the rows of var represent genes. The shape gives the number of cells and genes in the matrix.

AnnData resources must be accessible through local files. Directory-based GRRs and supported caching protocols provide this access. Other storage protocols raise a ValueError identifying the resource.

AnnData file formats

The format: setting in the resource definition determines how AnnData is loaded. If no format is specified, GAIn determines the format from the file name.

Format	File	Description
`h5ad`	`*.h5ad`	An AnnData HDF5 file. This is also the default if no recognized file suffix is found.
`10x_mtx`	`matrix.mtx` or `matrix.mtx.gz`	A 10x matrix with accompanying `barcodes` and `features` (or `genes`) files in the same directory.
`10x_h5`	`*.h5`	A 10x HDF5 matrix.

The resource configuration options are described in Genomic resources and repositories.

Matrix-free loading

For some analyses, only the cell and gene annotations are needed, without the expression matrix itself. The matrix_free argument of load_ann_data_from_resource() supports this use case for 10x_mtx and 10x_h5 resources.

When matrix_free=True, the returned AnnData object contains the annotations but its X matrix is filled with zeros rather than the original expression values. This option is therefore appropriate for inspecting obs and var, but not for analyzing the expression matrix.

For h5ad resources, matrix_free is ignored because the matrix is already accessed through a backed file.

Backed AnnData objects

An AnnData object loaded from an h5ad resource retains an open file handle. GAIn loads these files in read-only backed mode, allowing the expression matrix to remain on disk rather than being loaded entirely into memory.

The isbacked attribute indicates whether an AnnData object is backed by a file. When it is, the caller is responsible for closing the file after use:

if ann_data.isbacked:
    ann_data.file.close()

This is particularly important when loading multiple AnnData resources in a loop, since otherwise their file handles may remain open.

The 10x_mtx and 10x_h5 formats are loaded into memory and do not retain open file handles. Similarly, data frames are read directly into memory and require no explicit closing.

API

The functions below are defined in the modules gain.genomic_resources.data_frame_resource and gain.genomic_resources.ann_data_resource. gain.grr exports all four, as grr.load_data_frame_from_resource, grr.load_data_frame_from_resource_id, grr.load_ann_data_from_resource and grr.load_ann_data_from_resource_id. The page has no class: the loaders return the objects of pandas and anndata.

Data frame loaders

`load_data_frame_from_resource`(resource)	Load a pandas DataFrame from a `data_frame` genomic resource.
`load_data_frame_from_resource_id`(resource_id)	Load a pandas DataFrame from a `data_frame` resource id.

load_data_frame_from_resource(resource: GenomicResource | None) → DataFrame[source]

Load a pandas DataFrame from a data_frame genomic resource.

The format: key of the resource selects the reader: csv (the default), tsv or excel. The parameters: block passes keyword arguments to the pandas reader. The data are read at once, and the result holds no open file.

Parameters:: resource (GenomicResource or None) – The resource to load. Its type must be data_frame. None is an error, not an empty result.
Returns:: The table of the resource. A file with a header and no rows gives a DataFrame with columns and no rows. A file with no content at all raises the error of the pandas reader.
Return type:: pandas.DataFrame
Raises:: ValueError – If resource is None, is not of type data_frame, has no file: key, names an unknown format:, or has a parameters: block that makes pandas return something other than a DataFrame, such as chunksize or sheet_name: null. Each message except the None one names the resource.

load_data_frame_from_resource_id(resource_id: str, grr: GenomicResourceRepo | None = None) → DataFrame[source]

Load a pandas DataFrame from a data_frame resource id.

Parameters:

resource_id (str) – The id of the resource in the repository.
grr (GenomicResourceRepo or None, default None) – The repository to read from. None builds the default repository with build_genomic_resource_repository().

Returns:

The table of the resource, as load_data_frame_from_resource() returns it.

Return type:

pandas.DataFrame

Raises:

ValueError – As load_data_frame_from_resource() raises it.

AnnData loaders

`load_ann_data_from_resource`(resource, *[, ...])	Load an AnnData from an `ann_data` genomic resource.
`load_ann_data_from_resource_id`(resource_id)	Load an AnnData from an `ann_data` resource id.

load_ann_data_from_resource(resource: GenomicResource | None, *, matrix_free: bool = False) → AnnData[source]

Load an AnnData from an ann_data genomic resource.

The format: key of the resource selects the reader: h5ad, 10x_mtx or 10x_h5. When the key is missing, the suffix of the file: key decides, and h5ad is the default. The data files must be reachable as local files.

The caller owns the file handle. An h5ad file is opened read-only and backed by the file, so the AnnData keeps the file open. Call ann_data.file.close() when ann_data.isbacked is true. The two 10x formats are read into memory and hold no handle.

Parameters:

resource (GenomicResource or None) – The resource to load. Its type must be ann_data. None is an error, not an empty result.
matrix_free (bool, default False) – Skip the data matrix. It applies to the 10x_mtx and 10x_h5 formats only, and h5ad ignores it. The X of the result is a matrix of zeros of the right shape, not the data of the resource. obs and var are the same as in an ordinary read. Do not use this option to read values.

Returns:

The annotated matrix. The rows of obs are the cells and the rows of var are the genes.

Return type:

anndata.AnnData

Raises:

ValueError – If resource is None, is not of type ann_data, has no file: key, names an unknown format:, cannot reach a data file as a local file:// url, or has a parameters: block that makes the reader return something other than an AnnData. Each message except the None one names the resource.

load_ann_data_from_resource_id(resource_id: str, grr: GenomicResourceRepo | None = None) → AnnData[source]

Load an AnnData from an ann_data resource id.

The caller owns the file handle, as for load_ann_data_from_resource().

Parameters:

resource_id (str) – The id of the resource in the repository.
grr (GenomicResourceRepo or None, default None) – The repository to read from. None builds the default repository with build_genomic_resource_repository().

Returns:

The annotated matrix, as load_ann_data_from_resource() returns it.

Return type:

anndata.AnnData

Raises:

ValueError – As load_ann_data_from_resource() raises it.