dae.parquet package

Subpackages

• dae.parquet.schema2 package
  • Submodules
  • dae.parquet.schema2.annotate_schema2_parquet module
    • backup_schema2_study()
    • cli()
    • merge_partitions()
    • process_parquet()
    • produce_regions()
    • produce_schema2_annotation_tasks()
    • produce_schema2_merging_tasks()
    • symlink_pedigree_and_family_variants()
    • write_new_meta()
  • dae.parquet.schema2.loader module
    • MultiReader
      • MultiReader.close()
      • MultiReader.current_idx
    • ParquetLoader
      • ParquetLoader.FAMILY_COLUMNS
      • ParquetLoader.SUMMARY_COLUMNS
      • ParquetLoader.fetch_family_variants()
      • ParquetLoader.fetch_summary_variants()
      • ParquetLoader.fetch_variants()
      • ParquetLoader.get_family_pq_filepaths()
      • ParquetLoader.get_summary_pq_filepaths()
      • ParquetLoader.load_from_dir()
    • ParquetLoaderException
    • Reader
      • Reader.close()
      • Reader.current_idx
  • dae.parquet.schema2.merge_parquet module
    • merge_parquet_directory()
    • merge_parquets()
    • merge_variants_parquets()
  • dae.parquet.schema2.parquet_io module
    • ContinuousParquetFileWriter
      • ContinuousParquetFileWriter.BATCH_ROWS
      • ContinuousParquetFileWriter.DEFAULT_COMPRESSION
      • ContinuousParquetFileWriter.append_allele()
      • ContinuousParquetFileWriter.build_batch()
      • ContinuousParquetFileWriter.build_table()
      • ContinuousParquetFileWriter.close()
      • ContinuousParquetFileWriter.data_reset()
      • ContinuousParquetFileWriter.size()
    • VariantsParquetWriterDeprecated()
  • dae.parquet.schema2.processing_pipeline module
    • AnnotationPipelineVariantsBatchFilter
      • AnnotationPipelineVariantsBatchFilter.filter()
    • AnnotationPipelineVariantsFilter
      • AnnotationPipelineVariantsFilter.filter()
    • AnnotationPipelineVariantsFilterMixin
    • DeleteAttributesFromVariantFilter
      • DeleteAttributesFromVariantFilter.filter()
    • DeleteAttributesFromVariantsBatchFilter
      • DeleteAttributesFromVariantsBatchFilter.filter()
    • Schema2SummaryVariantsBatchSource
      • Schema2SummaryVariantsBatchSource.fetch()
    • Schema2SummaryVariantsSource
      • Schema2SummaryVariantsSource.fetch()
    • VariantsLoaderBatchSource
      • VariantsLoaderBatchSource.fetch()
    • VariantsLoaderSource
      • VariantsLoaderSource.fetch()
  • dae.parquet.schema2.serializers module
    • AlleleParquetSerializer
      • AlleleParquetSerializer.blob_column()
      • AlleleParquetSerializer.build_allele_record_dict()
      • AlleleParquetSerializer.schema()
    • FamilyAlleleParquetSerializer
      • FamilyAlleleParquetSerializer.blob_column()
      • FamilyAlleleParquetSerializer.build_allele_record_dict()
      • FamilyAlleleParquetSerializer.build_schema()
      • FamilyAlleleParquetSerializer.schema()
    • JsonVariantsDataSerializer
      • JsonVariantsDataSerializer.deserialize_family_record()
      • JsonVariantsDataSerializer.deserialize_summary_record()
      • JsonVariantsDataSerializer.serialize_family()
      • JsonVariantsDataSerializer.serialize_summary()
    • SummaryAlleleParquetSerializer
      • SummaryAlleleParquetSerializer.blob_column()
      • SummaryAlleleParquetSerializer.build_allele_record_dict()
      • SummaryAlleleParquetSerializer.build_blob_schema()
      • SummaryAlleleParquetSerializer.build_schema()
      • SummaryAlleleParquetSerializer.schema()
    • VariantsDataAvroSerializer
      • VariantsDataAvroSerializer.deserialize_family_record()
      • VariantsDataAvroSerializer.deserialize_summary_record()
      • VariantsDataAvroSerializer.serialize_family()
      • VariantsDataAvroSerializer.serialize_summary()
    • VariantsDataSerializer
      • VariantsDataSerializer.build_serializer()
      • VariantsDataSerializer.deserialize_family_record()
      • VariantsDataSerializer.deserialize_summary_record()
      • VariantsDataSerializer.serialize_family()
      • VariantsDataSerializer.serialize_summary()
    • build_family_schema()
    • build_summary_blob_schema()
    • build_summary_schema()
    • construct_avro_family_schema()
    • construct_avro_summary_schema()
  • dae.parquet.schema2.variants_parquet_writer module
    • ContinuousParquetFileWriter
      • ContinuousParquetFileWriter.DEFAULT_COMPRESSION
      • ContinuousParquetFileWriter.append_allele()
      • ContinuousParquetFileWriter.close()
      • ContinuousParquetFileWriter.size()
    • Schema2SummaryVariantBatchConsumer
      • Schema2SummaryVariantBatchConsumer.filter()
    • Schema2SummaryVariantConsumer
      • Schema2SummaryVariantConsumer.filter()
    • Schema2VariantBatchConsumer
      • Schema2VariantBatchConsumer.filter()
    • Schema2VariantConsumer
      • Schema2VariantConsumer.filter()
    • VariantsParquetWriter
      • VariantsParquetWriter.close()
      • VariantsParquetWriter.write()
      • VariantsParquetWriter.write_summary_variant()
  • Module contents

Submodules

dae.parquet.helpers module

dae.parquet.helpers.merge_parquets(in_files: list[str], out_file: str, *, delete_in_files: bool = True, row_group_size: int = 50000, parquet_version: str | None = None) → None

Merge in_files into one large file called out_file.

dae.parquet.helpers.url_to_pyarrow_fs(filename: str) → tuple[FileSystem | None, str]

Turn URL into pyarrow filesystem instance.

Parameters

filenamestr

The fsspec-compatible URL

Returns

filesystempyarrow.fs.FileSystem

The new filesystem discovered from filename and filesystem.

dae.parquet.parquet_writer module

dae.parquet.parquet_writer.add_missing_parquet_fields(pps: Schema, ped_df: DataFrame) → tuple[DataFrame, Schema]

Add missing parquet fields.

dae.parquet.parquet_writer.append_meta_to_parquet(meta_filename: str, key: list[str], value: list[str]) → None

Append key-value pair to meta data parquet file.

dae.parquet.parquet_writer.collect_pedigree_parquet_schema(ped_df: DataFrame) → Schema

Build the pedigree parquet schema.

dae.parquet.parquet_writer.fill_family_bins(families: FamiliesData, partition_descriptor: PartitionDescriptor | None = None) → None

Save families data into a parquet file.

dae.parquet.parquet_writer.merge_variants_parquets(partition_descriptor: PartitionDescriptor, variants_dir: str, partition: list[tuple[str, str]], row_group_size: int = 50000, parquet_version: str | None = None) → None

Merge parquet files in variants_dir.

dae.parquet.parquet_writer.pedigree_parquet_schema() → Schema

Return the schema for pedigree parquet file.

dae.parquet.parquet_writer.save_ped_df_to_parquet(ped_df: DataFrame, filename: str, parquet_version: str | None = None) → None

Save ped_df as a parquet file named filename.

dae.parquet.parquet_writer.serialize_summary_schema(annotation_attributes: list[AttributeInfo], partition_descriptor: PartitionDescriptor) → str

Serialize the summary schema.

dae.parquet.partition_descriptor module

class dae.parquet.partition_descriptor.Partition(region_bin: str | None = None, frequency_bin: str | None = None, coding_bin: str | None = None, family_bin: str | None = None)

Bases: object

Class to represent a partition of a genotype dataset.

coding_bin: str | None = None

family_bin: str | None = None

frequency_bin: str | None = None

static from_pylist(partition: list[tuple[str, str]]) → Partition

Create a partition from a list of tuples.

is_empty() → bool

Check if the partition is empty.

region_bin: str | None = None

to_pylist() → list[tuple[str, str]]

Convert the partition to a list of tuples.

class dae.parquet.partition_descriptor.PartitionDescriptor(*, chromosomes: list[str] | None = None, region_length: int = 0, integer_region_bins: bool = False, family_bin_size: int = 0, coding_effect_types: list[str] | None = None, rare_boundary: float = 0)

Bases: object

Class to represent partition of a genotype dataset.

build_family_partitions(chromosome_lengths: dict[str, int]) → list[Partition]

Build summary partitions for all variants in the dataset.

build_summary_partitions(chromosome_lengths: dict[str, int]) → list[Partition]

Build summary partitions for all variants in the dataset.

family_partition(allele: FamilyAllele, *, seen_as_denovo: bool) → list[tuple[str, str]]

Produce family partition for an allele.

The partition is returned as a list of tuples consiting of the name of the partition and the value.

Example: [

(“region_bin”, “chr1_0”), (“frequency_bin”, “0”), (“coding_bin”, “1”), (“family_bin”, “1)

]

family_partition_schema() → list[tuple[str, str]]

Build family dataset partition schema for table creation.

When creating an Impala or BigQuery table it is helpful to have the list of partitions and types used in the parquet dataset.

get_variant_partitions(chromosome_lengths: dict[str, int]) → tuple[list[list[tuple[str, str]]], list[list[tuple[str, str]]]]

Return the output summary and family variant partition names.

has_coding_bins() → bool

has_family_bins() → bool

has_family_partitions() → bool

Check if partition applicable to family allele are defined.

has_frequency_bins() → bool

has_partitions() → bool

Equivalent to has_family_partitions method.

has_region_bins() → bool

has_summary_partitions() → bool

Check if partition applicable to summary allele are defined.

make_all_region_bins(chromosome_lengths: dict[str, int]) → list[str]

Produce all region bins for all chromosomes.

make_coding_bin(effect_types: Iterable[str]) → int

Produce coding bin for given list of effect types.

make_family_bin(family_id: str) → int

Produce family bin for given family ID.

make_frequency_bin(allele_count: int, allele_freq: float, *, is_denovo: bool = False) → str

Produce frequency bin from allele count, frequency and de Novo flag.

Params are allele count, allele frequence and de Novo flag.

make_region_bin(chrom: str, pos: int) → str

Produce region bin for given chromosome and position.

make_region_bins_regions(chromosomes: list[str], chromosome_lengths: dict[str, int]) → dict[str, list[Region]]

Generate region_bin to regions based on a partition descriptor.

static parse(path_name: Path | str) → PartitionDescriptor

Parse partition description from a file.

When the file name has a .conf suffix or is without suffix the format of the file is assumed to be python config file and it is parsed using the Python ConfigParser class.

When the file name has .yaml suffix the file is parsed using the YAML parser.

static parse_dict(config_dict: dict[str, Any]) → PartitionDescriptor

Parse configuration dictionary and create a partion descriptor.

static parse_string(content: str, content_format: str = 'conf') → PartitionDescriptor

Parse partition description from a string.

The supported formats are the Python config format and YAML. Example string content should be as follows.

Example Python config format: ` [region_bin] chromosomes = chr1, chr2 region_length = 10 integer_region_bins = False [frequency_bin] rare_boundary = 5.0 [coding_bin] coding_effect_types = frame-shift,splice-site,nonsense,missense [family_bin] family_bin_size=10 `

Example YAML format: ``` region_bin:

chromosomes: chr1, chr2 region_length: 10 integer_region_bins: False

frequency_bin:

rare_boundary: 5.0

coding_bin:

coding_effect_types: frame-shift,splice-site,nonsense,missense

family_bin:

family_bin_size: 10

```

static partition_directory(dataset_dir: str, partition: Partition | list[tuple[str, str]]) → str

Construct a partition dataset directory.

Given a partition in the format returned by summary_parition or family_partition methods, this function constructs the directory name corresponding to the partition.

static partition_filename(prefix: str, partition: Partition | list[tuple[str, str]], bucket_index: int | None) → str

Construct a partition dataset base filename.

Given a partition in the format returned by summary_parition or family_partition methods, this function constructs the file name corresponding to the partition.

static path_to_partitions(raw_path: str) → list[tuple[str, str]]

Convert a path into the partitions it is composed of.

region_bins_count(chrom: str, chromosome_lengths: dict[str, int]) → int

region_to_region_bins(region: Region, chrom_lens: dict[str, int]) → list[str]

Provide a list of bins the given region intersects.

schema1_partition(allele: FamilyAllele) → list[tuple[str, str]]

Produce Schema1 family partition for an allele.

The partition is returned as a list of tuples consiting of the name of the partition and the value.

Example: [

(“region_bin”, “chr1_0”), (“frequency_bin”, “0”), (“coding_bin”, “1”), (“family_bin”, “1)

]

serialize(output_format: str = 'conf') → str

Serialize a partition descriptor into a string.

summary_partition(allele: SummaryAllele, *, seen_as_denovo: bool) → list[tuple[str, str]]

Produce summary partition for an allele.

The partition is returned as a list of tuples consiting of the name of the partition and the value.

Example: [

(“region_bin”, “chr1_0”), (“frequency_bin”, “0”), (“coding_bin”, “1”),

]

summary_partition_schema() → list[tuple[str, str]]

Build summary parquet dataset partition schema for table creation.

When creating an Impala or BigQuery table it is helpful to have the list of partitions and types used in the parquet dataset.

to_dict() → dict[str, Any]

Convert the partition descriptor to a dict.

Module contents