3. Input data structure

Here we will describe the structure of the files required to create and run an INCA model.

import pandas as pd
import numpy as np
import pathlib
import ast
import incawrapper
from incawrapper import utils

The INCAWrapper does not directly read any files, but takes pandas dataframes as input. Thus, the user can make these dataframes using there preferred methods. In the end of this guide, we have a guide to write and read .csv or excel files to import them as correctly formatted dataframes.

The data is validated using the Pandera python package. The validation covers both columns names and data types, thus column names must be specified exactly as shown in these examples. However the dataframes are allowed to contain more columns than the once required, though these extra columns are not parsed to INCA or to the INCA results object. To view the exact validation criteria for a specific data type inspect the data schema’s found in the dataschema module.

As an example we will show how to inspect the ReactionsSchema, which further described in the section Reactions data

incawrapper.ReactionsSchema.columns

{'rxn_id': <Schema Column(name=rxn_id, type=DataType(str))>,
 'rxn_eqn': <Schema Column(name=rxn_eqn, type=DataType(str))>}

3.1. Reactions data

This is were the reactions and the atom map in the model are defined. INCA requires the reactions to be defined with arrows -> for irreversible reactions and <-> for reversible reactions. There are two different syntaxes for the atom maps. First and simples is just to use letters, e.g. abc. If one requires more fine-grained control INCA also supports a syntax which specifies the individual atoms more explicit: C1:a C2:b C3:c.

utils.present_schema_overview(incawrapper.ReactionsSchema)

	column name	dtype	required	nullable	description
0	rxn_id	str	True	False	The unique id of the reaction
1	rxn_eqn	str	True	False	The reaction equation with atom map. Allowed reaction arrows: ->, <->.

model_reactions_example = pd.DataFrame(
    {
        "rxn_id": ["R1", "R2", "R3"],
        "rxn_eqn": ["A (abc) -> B (ab) + D (c)", "B (C1:a C1:b) <-> C (C1:b C2:a)", "C -> D"],
    }
)
model_reactions_example.head()

	rxn_id	rxn_eqn
0	R1	A (abc) -> B (ab) + D (c)
1	R2	B (C1:a C1:b) <-> C (C1:b C2:a)
2	R3	C -> D

Notice that is is allowed to mix the atom mapping syntax and the model may contain reactions without an atom map.

3.2. Tracer data

The tracer data specify the labelled compounds added to the experiment. The dataframe has a row for each experiment-tracer-labelling group combination, more on this later. For most users it will be sufficient to consider each row one experiment-tracer combination. The tracer dataframe has the following required columns:

utils.present_schema_overview(incawrapper.TracerSchema)

	column name	dtype	required	nullable	description
0	experiment_id	str	True	False	ID of the experiment. Must be a valid MATLAB variable name, legal characters are a-z, A-Z, 0-9, and the underscore character.
1	tracer_id	str	True	False	The unique id of the tracer compound.
2	met_id	str	True	False	The metabolite id of the labelled compound.
3	atom_ids	object	True	False	Ids of the labelled atoms in the labelled atom group (equivalent to columns of the same name in the INCA GUI)
4	atom_mdv	object	True	False	mass/isotopomer distribution vector of the \nlabelled atom group (equivalent to columns of the same name in the INCA GUI). The simplest way to use this column is to specify the purity of \nthe labelling group. This is done supplying a list two numbers, e.g. [0.01, 0.99].
5	enrichment	float64	True	False	mass/isotopomer distribution vector of the labelled atom \ngroup (equivalent to columns of the same name in the INCA GUI). The simplest way to use this column is to specify the purity of the labelling \ngroup. This is done supplying a list two numbers, e.g. `[0.5, 0.95]` specifies 95% of the compound will be fully labelled in this labelling \ngroup. If different atom positions has different purity create a different labelling group for each position. For further description please \nrefer to the INCA manual. Currently, the incawrapper only supports `atom_mdv` of length 2 for each labelling group.

Lets look at a simple and common example.

simple_tracer = pd.DataFrame(
    {
        "experiment_id": ["exp1","exp1"],
        "met_id": ["glc", "glc"],
        "tracer_id": ["[1,2-13C]glucose","[U-13C]glucose"],
        "atom_ids" : [[1,2], [1,2,3,4,5,6]],
        "atom_mdv" : [[0.02, 0.98], [0.001, 0.999]],
        "enrichment" : [0.2, 0.8],
    }
)
simple_tracer.head()

	experiment_id	met_id	tracer_id	atom_ids	atom_mdv	enrichment
0	exp1	glc	[1,2-13C]glucose	[1, 2]	[0.02, 0.98]	0.2
1	exp1	glc	[U-13C]glucose	[1, 2, 3, 4, 5, 6]	[0.001, 0.999]	0.8

The specification above specifies a single experiment were a mixture of two types of labelled glucose was used. The enrichment specifies that the labelled medium contained 20% of [1,2-13C]glucose and 80% [U-13C]glucose. The purity of the used tracers (atom_mdv) is [1,2-13C]glucose was 98% pure, i.e. 98% of the of the 20% glucose is labelled on the carbon 1 and 2, while 2% of the 20% glucose is has no labels (i.e. not considered the naturally abundant labelling).

3.2.1. Parallel isotopomer labelling experiment

To specify the a set of parallel isotopomer labelling experiment, i.e. different experiments that which should fitted simultaneously. In the following example, specify the experiment from above in parallel with an experiment conducted with 100% [1-13C]glucose.

parallel_experiment_tracer = pd.DataFrame(
    {
        "experiment_id": ["exp1","exp1","exp2"],
        "met_id": ["glc", "glc", "glc"],
        "tracer_id": ["[1,2-13C]glucose","[U-13C]glucose","[1-13C]glucose"],
        "atom_ids" : [[1,2], [1,2,3,4,5,6],[1]],
        "atom_mdv" : [[0.02, 0.98], [0.001, 0.999],[0.05, 0.95]],
        "enrichment" : [0.2, 0.8,1],
    }
)
parallel_experiment_tracer.head()

	experiment_id	met_id	tracer_id	atom_ids	atom_mdv	enrichment
0	exp1	glc	[1,2-13C]glucose	[1, 2]	[0.02, 0.98]	0.2
1	exp1	glc	[U-13C]glucose	[1, 2, 3, 4, 5, 6]	[0.001, 0.999]	0.8
2	exp2	glc	[1-13C]glucose	[1]	[0.05, 0.95]	1.0

3.2.2. Using labelling groups

The important part for different atom labelling groups is that the tracer_id is the same. In the following, we specify that we used a [1,2-13C]glucose tracer where 98% is labelled at carbon atom 1, and the 95% is labelled carbon atom 2. In this case the enrichment has to be the same for each labelling group.

two_labelling_groups_tracer = pd.DataFrame(
    {
        "experiment_id": ["exp1","exp1"],
        "met_id": ["glc", "glc"],
        "tracer_id": ["[1,2-13C]glucose","[1,2-13C]glucose"],
        "atom_ids" : [[1], [2]],
        "atom_mdv" : [[0.02, 0.98], [0.05, 0.95]],
        "enrichment" : [1, 1],
    }
)
two_labelling_groups_tracer.head()

	experiment_id	met_id	tracer_id	atom_ids	atom_mdv	enrichment
0	exp1	glc	[1,2-13C]glucose	[1]	[0.02, 0.98]	1
1	exp1	glc	[1,2-13C]glucose	[2]	[0.05, 0.95]	1

3.3. Flux measurement data

Flux measurements are typically uptake or secretion rates which this does not require labelling. Therefore this data is also quite simple to define. The required columns are the following:

utils.present_schema_overview(incawrapper.FluxMeasurementsSchema)

	column name	dtype	required	nullable	description
0	experiment_id	str	True	False	ID of the experiment. Must be a valid MATLAB variable name, legal characters are a-z, A-Z, 0-9, and the underscore character.
1	rxn_id	str	True	False	The unique id of the reaction
2	flux	float64	True	False	Measured/estimated rate typically in mmol/gDW/h
3	flux_std_error	float64	True	False	Standard error of the measured/estimated rate

The units has to be consistent for all measurements, because INCA assumes that all rates have the same units. Notice that it is not possible supply a time point for the rate estimates. This is because INCA supports steady state and isotopically non-stationary labelling analysis. Both of these methods assumes that all rates are constant over the time duration and only isotopomer distribution vector are allowed to change over time.

flux_measurements_example = pd.DataFrame(
    {
        "experiment_id": ["exp1", "exp1", "exp1", "exp2", "exp2", "exp2"],
        "rxn_id": ["R1", "R2", "R3", "R1", "R2", "R3"],
        "flux": [1.0, 2.0, 3.0, 1.2, 1.8, 2.8],
        "flux_std_error": [0.1, 0.5, 0.2, 0.1, 0.5, 0.2],
    }
)
flux_measurements_example.head()

	experiment_id	rxn_id	flux	flux_std_error
0	exp1	R1	1.0	0.1
1	exp1	R2	2.0	0.5
2	exp1	R3	3.0	0.2
3	exp2	R1	1.2	0.1
4	exp2	R2	1.8	0.5

3.4. Mass spectrometry measurements

Mass spectrometry measurements are given as isotopomer distribution vectors and the measurement standard error. These can be corrected for natural abundance or not, but by default INCA performs natural abundance correction, thus this needs to be turned of in the options if it is not required (See XX). The required columns for ms measurements are:

utils.present_schema_overview(incawrapper.MSMeasurementsSchema)

	column name	dtype	required	nullable	description
0	experiment_id	str	True	False	ID of the experiment. Must be a valid MATLAB variable name, legal characters are a-z, A-Z, 0-9, and the underscore character.
1	met_id	str	True	False	Metabolite ID of metabolite which is directly measured or from which the fragment is derived through a derivatization method.
2	ms_id	str	True	False	ID of the measured ms fragment - often multiple fragment can be measured from the same metabolite
3	measurement_replicate	int64	True	False	Replicate number of the measurement of the same fragment in the same experiment. \n"In most cases, the data will only have one measurement per fragment per experiment.
4	labelled_atom_ids	object	True	False	List of atom ids of the labelled atoms in the metabolite.
5	unlabelled_atoms	str	False	True	The molecular formula of the all atoms that cannot be labelled through \nthe introduced labels in the tracers. This typically includes non-carbon elements of the fragment and all elements originating from derivatization agent. \nINCA uses the unlabelled atoms to correct for natural abundance.
6	mass_isotope	int64	True	False	The mass isotopomer of the fragment.\nE.g. M0, M+1, etc. Specified as an integer. It is allowed to have gaps in the isotopmer of a given fragment, e.g. 0, 2, 3. In this case the intensity and \nstd error of missing isotopomers are filled with NaN before inserted in INCA.
7	intensity	float64	True	True	The measured intensity of the fragment mass isotope.
8	intensity_std_error	float64	True	True	The standard error of the measured intensity of the fragment mass isotope.
9	time	float64	True	False	Time point of measurement only relevant for isotopically non-stationary MFA analysis

import numpy as np
n_isotopes = [3, 4, 3]
ms_measurements_example = pd.DataFrame(
    {
        "experiment_id": np.repeat(["exp1"], sum(n_isotopes)),
        "met_id": np.repeat(["tyr", "ser", "ser"], n_isotopes),
        "ms_id": np.repeat(["tyr_302", "ser_390", "ser_362"], n_isotopes),
        "measurement_replicate": np.repeat([1], sum(n_isotopes)),
        "labelled_atom_ids": np.repeat(np.array([[1,2], [1,2,3], [2,3]], dtype=object), n_isotopes),
        "unlabelled_atoms": np.repeat(["C12H32O2NSi2", "C14H40O3NSi3", "C14H40O2NSi3"], n_isotopes),
        "mass_isotope": [0, 1, 2, 0, 1, 2, 3, 0, 1, 2],
        "intensity": [0.1, 0.2, 0.4, 0.2, 0.0001, 0.003, 0.3, 0.4, 0.1, 0.5],
        "intensity_std_error": [0.01, 0.02, 0.04, 0.02, 0.0001, 0.003, 0.03, 0.04, 0.01, 0.05],
        "time": np.repeat([0], sum(n_isotopes)),
    }
)
ms_measurements_example

	experiment_id	met_id	ms_id	measurement_replicate	labelled_atom_ids	unlabelled_atoms	mass_isotope	intensity	intensity_std_error	time
0	exp1	tyr	tyr_302	1	[1, 2]	C12H32O2NSi2	0	0.1000	0.0100	0
1	exp1	tyr	tyr_302	1	[1, 2]	C12H32O2NSi2	1	0.2000	0.0200	0
2	exp1	tyr	tyr_302	1	[1, 2]	C12H32O2NSi2	2	0.4000	0.0400	0
3	exp1	ser	ser_390	1	[1, 2, 3]	C14H40O3NSi3	0	0.2000	0.0200	0
4	exp1	ser	ser_390	1	[1, 2, 3]	C14H40O3NSi3	1	0.0001	0.0001	0
5	exp1	ser	ser_390	1	[1, 2, 3]	C14H40O3NSi3	2	0.0030	0.0030	0
6	exp1	ser	ser_390	1	[1, 2, 3]	C14H40O3NSi3	3	0.3000	0.0300	0
7	exp1	ser	ser_362	1	[2, 3]	C14H40O2NSi3	0	0.4000	0.0400	0
8	exp1	ser	ser_362	1	[2, 3]	C14H40O2NSi3	1	0.1000	0.0100	0
9	exp1	ser	ser_362	1	[2, 3]	C14H40O2NSi3	2	0.5000	0.0500	0

3.5. Note about formatting when reading csv and and excel files

Some of the data inputs requires the element of a cell to be a python list. This can cause issues when reading the data from .csv or excel files. To accommodate the issue simply write a list in python syntax as a string in the csv or excel file. When the file is read using pandas you will evaluate the columns the list strings, which will convert the strings in to python lists.

3.5.1. CSV files

For .csv files we can use the convert argument in the pd.read_csv functions. Lets use the ms data as an example. The csv file would look as follows

csv_illtration_file = pathlib.Path("./examples/Literature data/simple model/ms_measurement_csv_input_example.csv")
with open(csv_illtration_file, "r") as f:
    print(f.read())

experiment_id,met_id,ms_id,measurement_replicate,labelled_atom_ids,unlabelled_atoms,mass_isotope,intensity,intensity_std_error,time
exp1,tyr,tyr_302,1,"[1, 2]",C12H32O2NSi2,0,0.1000,0.0100,0
exp1,tyr,tyr_302,1,"[1, 2]",C12H32O2NSi2,1,0.2000,0.0200,0
exp1,tyr,tyr_302,1,"[1, 2]",C12H32O2NSi2,2,0.4000,0.0400,0
exp1,ser,ser_390,1,"[1, 2, 3]",C14H40O3NSi3,0,0.2000,0.0200,0
exp1,ser,ser_390,1,"[1, 2, 3]",C14H40O3NSi3,1,0.0001,0.0001,0
exp1,ser,ser_390,1,"[1, 2, 3]",C14H40O3NSi3,2,0.0030,0.0030,0
exp1,ser,ser_390,1,"[1, 2, 3]",C14H40O3NSi3,3,0.3000,0.0300,0
exp1,ser,ser_362,1,"[2, 3]",C14H40O2NSi3,0,0.4000,0.0400,0
exp1,ser,ser_362,1,"[2, 3]",C14H40O2NSi3,1,0.1000,0.0100,0
exp1,ser,ser_362,1,"[2, 3]",C14H40O2NSi3,2,0.5000,0.0500,0

Notice that the list definitions ([]) are inclosed in double quotes, but importantly when the lists contains strings these strings should be also be inclosed in single quotes, e.g. "['C3','C4']". When this csv file is read through pd.read_csv the string lists are correctly read.

from_csv = pd.read_csv(csv_illtration_file, converters={"labelled_atom_ids": ast.literal_eval})
from_csv.head()

	experiment_id	met_id	ms_id	measurement_replicate	labelled_atom_ids	unlabelled_atoms	mass_isotope	intensity	intensity_std_error	time
0	exp1	tyr	tyr_302	1	[1, 2]	C12H32O2NSi2	0	0.1000	0.0100	0
1	exp1	tyr	tyr_302	1	[1, 2]	C12H32O2NSi2	1	0.2000	0.0200	0
2	exp1	tyr	tyr_302	1	[1, 2]	C12H32O2NSi2	2	0.4000	0.0400	0
3	exp1	ser	ser_390	1	[1, 2, 3]	C14H40O3NSi3	0	0.2000	0.0200	0
4	exp1	ser	ser_390	1	[1, 2, 3]	C14H40O3NSi3	1	0.0001	0.0001	0

We can verify that the data type of the idv is a list.

type(from_csv["labelled_atom_ids"][0])

list

And check that the dataframe passes the schema validation

incawrapper.MSMeasurementsSchema.validate(from_csv)

	experiment_id	met_id	ms_id	measurement_replicate	labelled_atom_ids	unlabelled_atoms	mass_isotope	intensity	intensity_std_error	time
0	exp1	tyr	tyr_302	1	[1, 2]	C12H32O2NSi2	0	0.1000	0.0100	0.0
1	exp1	tyr	tyr_302	1	[1, 2]	C12H32O2NSi2	1	0.2000	0.0200	0.0
2	exp1	tyr	tyr_302	1	[1, 2]	C12H32O2NSi2	2	0.4000	0.0400	0.0
3	exp1	ser	ser_390	1	[1, 2, 3]	C14H40O3NSi3	0	0.2000	0.0200	0.0
4	exp1	ser	ser_390	1	[1, 2, 3]	C14H40O3NSi3	1	0.0001	0.0001	0.0
5	exp1	ser	ser_390	1	[1, 2, 3]	C14H40O3NSi3	2	0.0030	0.0030	0.0
6	exp1	ser	ser_390	1	[1, 2, 3]	C14H40O3NSi3	3	0.3000	0.0300	0.0
7	exp1	ser	ser_362	1	[2, 3]	C14H40O2NSi3	0	0.4000	0.0400	0.0
8	exp1	ser	ser_362	1	[2, 3]	C14H40O2NSi3	1	0.1000	0.0100	0.0
9	exp1	ser	ser_362	1	[2, 3]	C14H40O2NSi3	2	0.5000	0.0500	0.0

3.5.2. Excel files

The excel file can written more straight forward without double quotes around the lists. For example a cell in labelled_atom_ids can be ['C3','C4']. Notice, that the atom ids are still wrapped in single quotes because they are defined using the C notation.

Reading from an excel file is slightly more verbose. Here we first read the file and then use ast.literal_eval() to convert the strings to python-lists.

excel_illtration_file = pathlib.Path("./examples/Literature data/simple model/ms_measurement_csv_input_example.xlsx")
from_excel = pd.read_excel(excel_illtration_file)
from_excel[['labelled_atom_ids']] = from_excel[['labelled_atom_ids']].applymap(lambda x: ast.literal_eval(x))

Again, we see that the data read from excel passes the schema validation.

incawrapper.MSMeasurementsSchema.validate(from_excel)

	experiment_id	met_id	ms_id	measurement_replicate	labelled_atom_ids	unlabelled_atoms	mass_isotope	intensity	intensity_std_error	time
0	exp1	tyr	tyr_302	1	[1, 2]	C12H32O2NSi2	0	0.1000	0.0100	0.0
1	exp1	tyr	tyr_302	1	[1, 2]	C12H32O2NSi2	1	0.2000	0.0200	0.0
2	exp1	tyr	tyr_302	1	[1, 2]	C12H32O2NSi2	2	0.4000	0.0400	0.0
3	exp1	ser	ser_390	1	[1, 2, 3]	C14H40O3NSi3	0	0.2000	0.0200	0.0
4	exp1	ser	ser_390	1	[1, 2, 3]	C14H40O3NSi3	1	0.0001	0.0001	0.0
5	exp1	ser	ser_390	1	[1, 2, 3]	C14H40O3NSi3	2	0.0030	0.0030	0.0
6	exp1	ser	ser_390	1	[1, 2, 3]	C14H40O3NSi3	3	0.3000	0.0300	0.0
7	exp1	ser	ser_362	1	[2, 3]	C14H40O2NSi3	0	0.4000	0.0400	0.0
8	exp1	ser	ser_362	1	[2, 3]	C14H40O2NSi3	1	0.1000	0.0100	0.0
9	exp1	ser	ser_362	1	[2, 3]	C14H40O2NSi3	2	0.5000	0.0500	0.0

Now we have showed the expected structure of the input data you can move on to the next section to read about the how to use the INCAWrapper.