Data Tutorial¶

– On data handling –

One of the contributions of perturb-lib is to provide users with one-liners to manage available perturbation data. This tutorial covers:

Browsing through available data.
Loading and handling data in AnnData format suitable for exploration, filtering, and preprocessing.
Loading and handling data in built-in PlibData format suitable for machine-learning operations.

We start by listing currently available data. The basic unit of data in perturb-lib is context. We assume that each context collects the data produced within a specific experimental context and without confounders within a single context such as batch effects. Larger datasets that are given in batches across which differences are considerable (think statistically significant), are split into contexts.

[1]:

import perturb_lib as plib

plib.list_contexts()

[1]:

['DummyData',
 'DummyDataLongStrings',
 'HumanCellLine_1HAE_L1000-RNA-seq_LINCS-CMap2020_CMP',
 'HumanCellLine_A375_L1000-RNA-seq_LINCS-CMap2020_CMP',
 'HumanCellLine_A375_L1000-RNA-seq_LINCS-CMap2020_XPR',
 'HumanCellLine_A549_L1000-RNA-seq_LINCS-CMap2020_CMP',
 'HumanCellLine_A549_L1000-RNA-seq_LINCS-CMap2020_XPR',
 'HumanCellLine_AGS_L1000-RNA-seq_LINCS-CMap2020_XPR',
 'HumanCellLine_ASC_L1000-RNA-seq_LINCS-CMap2020_CMP',
 'HumanCellLine_BICR6_L1000-RNA-seq_LINCS-CMap2020_XPR',
 'HumanCellLine_BJAB_L1000-RNA-seq_LINCS-CMap2020_CMP',
 'HumanCellLine_BT20_L1000-RNA-seq_LINCS-CMap2020_CMP',
 'HumanCellLine_CD34_L1000-RNA-seq_LINCS-CMap2020_CMP',
 'HumanCellLine_DANG_L1000-RNA-seq_LINCS-CMap2020_XPR',
 'HumanCellLine_ES2_L1000-RNA-seq_LINCS-CMap2020_XPR',
 'HumanCellLine_HA1E_L1000-RNA-seq_LINCS-CMap2020_CMP',
 'HumanCellLine_HAP1_L1000-RNA-seq_LINCS-CMap2020_CMP',
 'HumanCellLine_HBL1_L1000-RNA-seq_LINCS-CMap2020_CMP',
 'HumanCellLine_HCC1806_L1000-RNA-seq_LINCS-CMap2020_XPR',
 'HumanCellLine_HCC515_L1000-RNA-seq_LINCS-CMap2020_CMP',
 'HumanCellLine_HCT116_L1000-RNA-seq_LINCS-CMap2020_CMP',
 'HumanCellLine_HEK293_L1000-RNA-seq_LINCS-CMap2020_CMP',
 'HumanCellLine_HELA_L1000-RNA-seq_LINCS-CMap2020_CMP',
 'HumanCellLine_HEPG2_L1000-RNA-seq_LINCS-CMap2020_CMP',
 'HumanCellLine_HFL1_L1000-RNA-seq_LINCS-CMap2020_CMP',
 'HumanCellLine_HME1_L1000-RNA-seq_LINCS-CMap2020_CMP',
 'HumanCellLine_HPTEC_L1000-RNA-seq_LINCS-CMap2020_CMP',
 'HumanCellLine_HS578T_L1000-RNA-seq_LINCS-CMap2020_CMP',
 'HumanCellLine_HS944T_L1000-RNA-seq_LINCS-CMap2020_XPR',
 'HumanCellLine_HT29_L1000-RNA-seq_LINCS-CMap2020_CMP',
 'HumanCellLine_HT29_L1000-RNA-seq_LINCS-CMap2020_XPR',
 'HumanCellLine_HUES3_L1000-RNA-seq_LINCS-CMap2020_CMP',
 'HumanCellLine_HUH7_L1000-RNA-seq_LINCS-CMap2020_CMP',
 'HumanCellLine_HUVEC_L1000-RNA-seq_LINCS-CMap2020_CMP',
 'HumanCellLine_IMR90_L1000-RNA-seq_LINCS-CMap2020_CMP',
 'HumanCellLine_IPC298_L1000-RNA-seq_LINCS-CMap2020_XPR',
 'HumanCellLine_JURKAT_L1000-RNA-seq_LINCS-CMap2020_CMP',
 'HumanCellLine_Jurkat_GrowthScreen_Horlbeck18',
 'HumanCellLine_K562_10xChromium3-scRNA-seq_Replogle22',
 'HumanCellLine_K562_GrowthScreen_Horlbeck18',
 'HumanCellLine_K562_L1000-RNA-seq_LINCS-CMap2020_CMP',
 'HumanCellLine_KELLY_L1000-RNA-seq_LINCS-CMap2020_XPR',
 'HumanCellLine_KMS34_L1000-RNA-seq_LINCS-CMap2020_CMP',
 'HumanCellLine_KYSE30_L1000-RNA-seq_LINCS-CMap2020_XPR',
 'HumanCellLine_LCLC103H_L1000-RNA-seq_LINCS-CMap2020_XPR',
 'HumanCellLine_LNCAP_L1000-RNA-seq_LINCS-CMap2020_CMP',
 'HumanCellLine_MCF10A_L1000-RNA-seq_LINCS-CMap2020_CMP',
 'HumanCellLine_MCF7_L1000-RNA-seq_LINCS-CMap2020_CMP',
 'HumanCellLine_MCF7_L1000-RNA-seq_LINCS-CMap2020_XPR',
 'HumanCellLine_MDAMB231_L1000-RNA-seq_LINCS-CMap2020_CMP',
 'HumanCellLine_MINO_L1000-RNA-seq_LINCS-CMap2020_CMP',
 'HumanCellLine_MNEU_L1000-RNA-seq_LINCS-CMap2020_CMP',
 'HumanCellLine_NALM6_L1000-RNA-seq_LINCS-CMap2020_CMP',
 'HumanCellLine_NKDBA_L1000-RNA-seq_LINCS-CMap2020_CMP',
 'HumanCellLine_NL20_L1000-RNA-seq_LINCS-CMap2020_CMP',
 'HumanCellLine_NPC_L1000-RNA-seq_LINCS-CMap2020_CMP',
 'HumanCellLine_OCILY10_L1000-RNA-seq_LINCS-CMap2020_CMP',
 'HumanCellLine_OCILY19_L1000-RNA-seq_LINCS-CMap2020_CMP',
 'HumanCellLine_OCILY3_L1000-RNA-seq_LINCS-CMap2020_CMP',
 'HumanCellLine_P1A82_L1000-RNA-seq_LINCS-CMap2020_CMP',
 'HumanCellLine_PC3_L1000-RNA-seq_LINCS-CMap2020_CMP',
 'HumanCellLine_PC3_L1000-RNA-seq_LINCS-CMap2020_XPR',
 'HumanCellLine_PHH_L1000-RNA-seq_LINCS-CMap2020_CMP',
 'HumanCellLine_RPE1_10xChromium3-scRNA-seq_Replogle22',
 'HumanCellLine_SHSY5Y_L1000-RNA-seq_LINCS-CMap2020_CMP',
 'HumanCellLine_SKBR3_L1000-RNA-seq_LINCS-CMap2020_CMP',
 'HumanCellLine_SKB_L1000-RNA-seq_LINCS-CMap2020_CMP',
 'HumanCellLine_SKL_L1000-RNA-seq_LINCS-CMap2020_CMP',
 'HumanCellLine_SKMEL5_L1000-RNA-seq_LINCS-CMap2020_CMP',
 'HumanCellLine_SKNSH_L1000-RNA-seq_LINCS-CMap2020_CMP',
 'HumanCellLine_SNGM_L1000-RNA-seq_LINCS-CMap2020_XPR',
 'HumanCellLine_SNU761_L1000-RNA-seq_LINCS-CMap2020_XPR',
 'HumanCellLine_THP1_L1000-RNA-seq_LINCS-CMap2020_CMP',
 'HumanCellLine_TMD8_L1000-RNA-seq_LINCS-CMap2020_CMP',
 'HumanCellLine_U251MG_L1000-RNA-seq_LINCS-CMap2020_XPR',
 'HumanCellLine_U2OS_L1000-RNA-seq_LINCS-CMap2020_CMP',
 'HumanCellLine_VCAP_L1000-RNA-seq_LINCS-CMap2020_CMP',
 'HumanCellLine_WA09_L1000-RNA-seq_LINCS-CMap2020_CMP',
 'HumanCellLine_WI38_L1000-RNA-seq_LINCS-CMap2020_CMP',
 'HumanCellLine_XC.L10_L1000-RNA-seq_LINCS-CMap2020_CMP',
 'HumanCellLine_XC.P026_L1000-RNA-seq_LINCS-CMap2020_CMP',
 'HumanCellLine_XC.P031_L1000-RNA-seq_LINCS-CMap2020_CMP',
 'HumanCellLine_XC.P033_L1000-RNA-seq_LINCS-CMap2020_CMP',
 'HumanCellLine_XC.P091_L1000-RNA-seq_LINCS-CMap2020_CMP',
 'HumanCellLine_XC.P092_L1000-RNA-seq_LINCS-CMap2020_CMP',
 'HumanCellLine_XC.P901_L1000-RNA-seq_LINCS-CMap2020_CMP',
 'HumanCellLine_XC.P904_L1000-RNA-seq_LINCS-CMap2020_CMP',
 'HumanCellLine_XC.P905_L1000-RNA-seq_LINCS-CMap2020_CMP',
 'HumanCellLine_XC.P906_L1000-RNA-seq_LINCS-CMap2020_CMP',
 'HumanCellLine_XC.P907_L1000-RNA-seq_LINCS-CMap2020_CMP',
 'HumanCellLine_XC.P908_L1000-RNA-seq_LINCS-CMap2020_CMP',
 'HumanCellLine_XC.P909_L1000-RNA-seq_LINCS-CMap2020_CMP',
 'HumanCellLine_XC.P910_L1000-RNA-seq_LINCS-CMap2020_CMP',
 'HumanCellLine_XC.P911_L1000-RNA-seq_LINCS-CMap2020_CMP',
 'HumanCellLine_XC.P912_L1000-RNA-seq_LINCS-CMap2020_CMP',
 'HumanCellLine_XC.P914_L1000-RNA-seq_LINCS-CMap2020_CMP',
 'HumanCellLine_XC.P915_L1000-RNA-seq_LINCS-CMap2020_CMP',
 'HumanCellLine_XC.P922_L1000-RNA-seq_LINCS-CMap2020_CMP',
 'HumanCellLine_XC.P930_L1000-RNA-seq_LINCS-CMap2020_CMP',
 'HumanCellLine_XC.P931_L1000-RNA-seq_LINCS-CMap2020_CMP',
 'HumanCellLine_XC.P932_L1000-RNA-seq_LINCS-CMap2020_CMP',
 'HumanCellLine_XC.P933_L1000-RNA-seq_LINCS-CMap2020_CMP',
 'HumanCellLine_XC.P934_L1000-RNA-seq_LINCS-CMap2020_CMP',
 'HumanCellLine_XC.P935_L1000-RNA-seq_LINCS-CMap2020_CMP',
 'HumanCellLine_XC.P936_L1000-RNA-seq_LINCS-CMap2020_CMP',
 'HumanCellLine_XC.R10_L1000-RNA-seq_LINCS-CMap2020_CMP',
 'HumanCellLine_YAPC_L1000-RNA-seq_LINCS-CMap2020_CMP',
 'HumanCellLine_YAPC_L1000-RNA-seq_LINCS-CMap2020_XPR']

Note the naming logic: The name of the context contains details that describe the experimental context as closely as possible. The model system is mentioned, the technology, the type of readouts, as well as the name that is specific to the dataset used. To get better understanding of individual contexts, e.g. to identify the type of outcome measured in the particular experiment (gene expression, cell viability, etc.), we can obtain the high-level description as follows:

[5]:

context = "HumanCellLine_K562_10xChromium3-scRNA-seq_Replogle22"
print(plib.describe_context(context))

Human cell line K562 prepared for perturbation analysis of essential genes as described in`2022 Replogle et al. <https://pubmed.ncbi.nlm.nih.gov/35688146>`_.

To get your hands on the particular dataset, the best is to load the dataset in AnnData format.

[6]:

adata = plib.load_anndata(context)
type(adata)

15:14:54 | INFO | Loading dataset 'replogle_K562' which is originally given in AnnData format..
15:14:54 | INFO | Downloading replogle_K562.h5ad...
/Users/dm922386/Library/Caches/pypoetry/virtualenvs/perturb-lib-h68r_ta--py3.12/lib/python3.12/site-packages/urllib3/connectionpool.py:1097: InsecureRequestWarning: Unverified HTTPS request is being made to host 'plus.figshare.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings
  warnings.warn(
/Users/dm922386/Library/Caches/pypoetry/virtualenvs/perturb-lib-h68r_ta--py3.12/lib/python3.12/site-packages/urllib3/connectionpool.py:1097: InsecureRequestWarning: Unverified HTTPS request is being made to host 's3-eu-west-1.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings
  warnings.warn(
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 10.7G/10.7G [09:23<00:00, 18.9MiB/s]
15:24:30 | INFO | Adding train/val/test splits..

[6]:

perturb_lib.data.collection.replogle22.HumanCellLine_K562_10xChromium3scRNAseq_Replogle22

AnnData object contains a data matrix that describes the value of a readout after a certain perturbation has occurred. We can explore its object to investigate the dataset:

[9]:

adata

[9]:

AnnData object with n_obs × n_vars = 310385 × 8563
    obs: 'cell_barcode', 'perturbation_type', 'perturbation_target', 'batch', 'perturbation', 'context', 'split'
    var: 'readout', 'readout_type', 'readout_target'

[10]:

adata.obs[0:5]

[10]:

	cell_barcode	perturbation_type	perturbation_target	batch	perturbation	context	split
0	AAACCCAAGAAATCCA-27	CRISPRi	NAF1	27	CRISPRi_NAF1	HumanCellLine_K562_10xChromium3-scRNA-seq_Repl...	train
1	AAACCCAAGAACTTCC-31	CRISPRi	BUB1	31	CRISPRi_BUB1	HumanCellLine_K562_10xChromium3-scRNA-seq_Repl...	train
2	AAACCCAAGAAGCCAC-34	CRISPRi	UBL5	34	CRISPRi_UBL5	HumanCellLine_K562_10xChromium3-scRNA-seq_Repl...	train
3	AAACCCAAGAATAGTC-43	CRISPRi	C9orf16	43	CRISPRi_C9orf16	HumanCellLine_K562_10xChromium3-scRNA-seq_Repl...	train
4	AAACCCAAGACAGCGT-28	CRISPRi	TIMM9	28	CRISPRi_TIMM9	HumanCellLine_K562_10xChromium3-scRNA-seq_Repl...	train

[11]:

adata.var[0:5]

[11]:

	readout	readout_type	readout_target
0	Transcriptome_LINC01409	Transcriptome	LINC01409
1	Transcriptome_LINC01128	Transcriptome	LINC01128
2	Transcriptome_NOC2L	Transcriptome	NOC2L
3	Transcriptome_KLHL17	Transcriptome	KLHL17
4	Transcriptome_HES4	Transcriptome	HES4

[12]:

adata.X.shape, type(adata.X)

[12]:

((310385, 8563), numpy.ndarray)

[13]:

adata.X[45:50, 45:51]

[13]:

array([[ 0.5457988 , -0.21936147, -0.6191385 ,  1.8765817 , -0.18855709,
        -0.70445   ],
       [-0.7178899 , -1.0262779 , -0.5378534 , -0.3611275 , -1.1285864 ,
         0.3140653 ],
       [ 0.20289904,  0.8282157 ,  0.5052937 , -0.36487058, -0.9389877 ,
        -0.7573743 ],
       [-0.67229235, -0.2952343 , -0.5420161 , -0.26392677, -0.8980309 ,
         1.6804963 ],
       [-0.7307865 , -0.2523772 , -0.548757  , -0.3520529 , -0.75549585,
         0.5462019 ]], dtype=float32)

As we see, AnnData is suitable for exploration and also for data manipulation routines. However, performing machine-learning operations such as batching is not trivial. To enable data handling optimized for training machine-learning models, we introduce PlibData.

PlibData data structure comes in two main flavours: One based on in-memory (InMemoryPlibData) and one based on-disk (OnDiskPlibData) data handling routines (built on top of polars and pyarrow for optimized performance). Using PlibData, we can load multiple contexts stacked together in the context-perturbation-readout (CPR) format.

[14]:

pdata = plib.load_plibdata(
    contexts_ids=[
        "HumanCellLine_K562_10xChromium3-scRNA-seq_Replogle22",
        "HumanCellLine_RPE1_10xChromium3-scRNA-seq_Replogle22",
    ]
)
type(pdata)

15:35:32 | INFO | Loading dataset 'replogle_K562' which is originally given in AnnData format..
15:35:32 | INFO | replogle_K562.h5ad found in cache.
15:35:44 | INFO | Adding train/val/test splits..
15:35:44 | INFO | mean-aggregating data..
15:36:19 | INFO | Casting to standardized DataFrame format..
15:36:22 | INFO | Loading dataset 'replogle_RPE1' which is originally given in AnnData format..
15:36:22 | INFO | Downloading replogle_RPE1.h5ad...
/Users/dm922386/Library/Caches/pypoetry/virtualenvs/perturb-lib-h68r_ta--py3.12/lib/python3.12/site-packages/urllib3/connectionpool.py:1097: InsecureRequestWarning: Unverified HTTPS request is being made to host 'plus.figshare.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings
  warnings.warn(
/Users/dm922386/Library/Caches/pypoetry/virtualenvs/perturb-lib-h68r_ta--py3.12/lib/python3.12/site-packages/urllib3/connectionpool.py:1097: InsecureRequestWarning: Unverified HTTPS request is being made to host 's3-eu-west-1.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings
  warnings.warn(
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 8.70G/8.70G [07:32<00:00, 19.2MiB/s]
15:44:04 | INFO | Adding train/val/test splits..
15:44:04 | INFO | mean-aggregating data..
15:44:24 | INFO | Casting to standardized DataFrame format..

[14]:

perturb_lib.data.plibdata.InMemoryPlibData

Note that after PlibData construction the instance is cached by default for fast retrieval in later attempts. In fact, each context is cached separately.

[15]:

pdata.columns

[15]:

['context', 'perturbation', 'readout', 'value']

As we can see, there are four columns in the table defined by our data structure:

context: contains symbols that this value came from.
perturbation: contains perturbation symbols.
readout: contains readout symbols.
value: the value of the readout.

[16]:

batch_of_four = pdata[1000:1004]
type(batch_of_four)

[16]:

polars.dataframe.frame.DataFrame

Notice the batches of the dataset are given in the form of pd.DataFrame to enable simple exploration.

[17]:

batch_of_four

[17]:

shape: (4, 4)

context	perturbation	readout	value
str	str	str	f32
"HumanCellLine_K562_10xChromium…	"CRISPRi_RPA2"	"Transcriptome_A1BG"	0.026806
"HumanCellLine_K562_10xChromium…	"CRISPRi_RPA3"	"Transcriptome_A1BG"	0.138879
"HumanCellLine_K562_10xChromium…	"CRISPRi_RPAP2"	"Transcriptome_A1BG"	-0.084605
"HumanCellLine_K562_10xChromium…	"CRISPRi_RPAP3"	"Transcriptome_A1BG"	0.060994

where the columns are of the following types:

[18]:

batch_of_four.dtypes

[18]:

[String, String, String, Float32]

To prepare the data for predictive modelling, we can split the dataset into train, validation, and test parts as follows:

[20]:

traindata, valdata, testdata = plib.split_plibdata_3fold(
    pdata, context_ids="HumanCellLine_K562_10xChromium3-scRNA-seq_Replogle22"
)

Splitting operation provides traindata valdata, and testdata as specified in the AnnData format for the specified context. The valdata and testdata contain only data from the target context. The traindata optionally contains data from other datasets.

Finally, we can fetch a pytorch DataLoader for the corresponding dataset as follows:

[21]:

train_loader = traindata.get_data_loader(batch_size=2, num_workers=0, pin_memory=False, shuffle=True)
next(iter(train_loader))

[21]:

shape: (2, 4)

context	perturbation	readout	value
str	str	str	f32
"HumanCellLine_K562_10xChromium…	"CRISPRi_RRP15"	"Transcriptome_USF3"	-0.066255
"HumanCellLine_K562_10xChromium…	"CRISPRi_CWC25"	"Transcriptome_ZNF276"	0.01206