pycarol.tools.dag

pycarol.tools.dag.find_keys(node, kv)

Find recursively all the values from a given key.

Args:

node (dict of dict): Nested dictionary. kv (str): dictionary key to find.

Yields:

dict: dictionary

pycarol.tools.dag.generate_dependency_graph(carol, connector_list=None, dm_prefix='DM_', staging_prefix='connector_name', only_mapping=False)

Generates dependency graph for a list of connectors.

Args:

carol (pycarol.Carol): INstance of Carol connector_list (list, optional): List of connectors to create the dependency graph. Defaults to None. dm_prefix: ‘str` default DM_

data model prefix to add to the data model name. e.g., if dm_name=’mydatamoldel’, the result will be “DM_mydatamoldel`. This is only applied for DataModel.

staging_prefix: ‘str` default None

Prefix for the staging name. e.g., if staging_prefix=’connector_name` the output will be:

{ “connector_name_stag3” : {“connector_name_stag1”, “connector_name_stag2”}}

Possible values are: ‘connector_name’, ‘connector_id’, None

only_mapping (bool, optional): Only generates mappings and constraints dependency graph. Defaults to False.

Returns:

default: defaultdict(set). Dependency graph.

pycarol.tools.dag.get_dm_relationship_constraints(carol, connector_name=None, connector_id=None, dm_prefix='DM_', staging_prefix=None)

Create relationship between data models based on their relationship constraints

Args:

carol: pycarol.Carol

Carol() object.

connector_name: str

Connector Name

dm_prefix: ‘str` default DM_

data model prefix to add to the data model name. e.g., if dm_name=’mydatamoldel’, the result will be “DM_mydatamoldel`

Returns: defaultdict(set)

dictionary { “dm1” : {“dm2”, “dm3”}} where “dm1” depends on “dm2”” and “dm3”

pycarol.tools.dag.get_etl_constraints(carol, connector_name=None, connector_id=None, staging_prefix=None)

Create relationship between stagings in ETLs.

Args:

carol: pycarol.Carol

Carol() object.

connector_name: str

connector name to get the relationships.

connector_id: str

connector ID to get the relationships.

staging_prefix: ‘str` default None

Prefix for the staging name. e.g., if staging_prefix=’connector_name` the output will be:

{ “connector_name_stag3” : {“connector_name_stag1”, “connector_name_stag2”}}

Possible values are: ‘connector_name’, ‘connector_id’, None

Returns: defaultdict(set)

dictionary { “stag3” : {“stag1”, “stag2”}} where “stag3” depends on “stag1”” and “stag2”

pycarol.tools.dag.get_mapping_constraints(carol, connector_name=None, connector_id=None, dm_prefix='DM_', staging_prefix=None)

Create relationship between data models and stagings in mappings.

Args:

carol: pycarol.Carol

Carol() object.

connector_name: str

Connector Name

connector_id: str

Connector ID

dm_prefix: ‘str` default DM_

data model prefix to add to the data model name. e.g., if dm_name=’mydatamoldel’, the result will be “DM_mydatamoldel`. This is only applied for DataModel.

staging_prefix: ‘str` default None

Prefix for the staging name. e.g., if staging_prefix=’connector_name` the output will be:

{ “connector_name_stag3” : {“connector_name_stag1”, “connector_name_stag2”}}

Possible values are: ‘connector_name’, ‘connector_id’, None

Returns: defaultdict(set)

dictionary { “dm1” : {“stag1”, “stag2”}} where “dm1” depends on “stag1”” and “stag2”

pycarol.tools.dag.get_staging_prefix(carol, staging_prefix, connector_name, connector_id)

Create the prefix for a given set of staging_prefix, connector_name, connector_id

Args:

carol: pycarol.Carol

Carol() object.

connector_name: str

Connector Name

prefix: ‘str` default DM_

prefix to add to the data model name. e.g., if dm_name=’mydatamoldel’, the result will be “DM_mydatamoldel`. This is only applied for DataModel.

staging_prefix: ‘str` default None

Prefix for the staging name. e.g., if staging_prefix=’connector_name` the output will be:

{ “connector_name_stag3” : {“connector_name_stag1”, “connector_name_stag2”}}

Possible values are: ‘connector_name’, ‘connector_id’, None

pycarol.tools.dag.unroll_list(l)

Unroll a list of lists to a flat list

Args:

l (list of lists): List of list to unroll

Yields:

list: unrolled list.