4.4. Using the `Network` object: Auxiliary files¶

GeNet supports reading in files in JSON and CSV formats which reference Network link or node IDs. Each file needs to reference only links or nodes, not both, in case there is overlap in IDs between nodes and links.

An AuxiliaryFile attaches itself to the Network IDs, finding where the overlap in IDs is with the set of IDs stored in the network. A map is kept within the AuxiliaryFile object during a GeNet session, if using GeNet's own modification methods. The updates are applied right before saving the Network to file or when update method on the AuxiliaryFile is called.

Some processes which will result in ID changes are:

node or link reindexing
network simplification

And those which will result in data loss:

node or link removal

Some examples of auxiliary files:

benchmarks snaped to the network
road pricing csv snapped to the network

Right now, GeNet supports two types of auxiliary files:

CSV: table with single level indexing. The IDs can be nested in lists
JSON: any level of nestedness is allowed, the IDs can live singularly or within lists

In [1]:

Copied!

import json
import os

import pandas as pd

def read_json_file(path):
    with open(path) as json_file:
        return json.load(json_file)

def read_csv_file(path):
    return pd.read_csv(path)
import json
import os

import pandas as pd

def read_json_file(path):
    with open(path) as json_file:
        return json.load(json_file)

def read_csv_file(path):
    return pd.read_csv(path)

Let's show a couple of examples.

In [2]:

Copied!

example_json = os.path.join("..", "tests", "test_data", "auxiliary_files", "links_benchmark.json")
example_csv = os.path.join("..", "tests", "test_data", "auxiliary_files", "links_benchmark.csv")
example_json = os.path.join("..", "tests", "test_data", "auxiliary_files", "links_benchmark.json")
example_csv = os.path.join("..", "tests", "test_data", "auxiliary_files", "links_benchmark.csv")

In [3]:

Copied!

read_json_file(example_json)
read_json_file(example_json)

Out[3]:

{'car': {'1': {'in': {'links': ['0'],
    'counts': {'0': 78.0,
     '1': 46.0,
     '2': 39.0,
     '3': 45.0,
     '4': 72.0,
     '5': 188.0,
     '6': 475.0,
     '7': 734.0,
     '8': 651.0,
     '9': 605.0,
     '10': 605.0,
     '11': 625.0,
     '12': 569.0,
     '13': 632.0,
     '14': 586.0,
     '15': 585.0,
     '16': 825.0,
     '17': 756.0,
     '18': 711.0,
     '19': 597.0,
     '20': 405.0,
     '21': 285.0,
     '22': 218.0,
     '23': 136.0}},
   'out': {'links': ['1'],
    'counts': {'0': 76.0,
     '1': 45.0,
     '2': 40.0,
     '3': 38.0,
     '4': 63.0,
     '5': 165.0,
     '6': 608.0,
     '7': 858.0,
     '8': 725.0,
     '9': 514.0,
     '10': 415.0,
     '11': 485.0,
     '12': 554.0,
     '13': 463.0,
     '14': 589.0,
     '15': 616.0,
     '16': 835.0,
     '17': 901.0,
     '18': 704.0,
     '19': 476.0,
     '20': 355.0,
     '21': 283.0,
     '22': 219.0,
     '23': 134.0}}},
  '2': {'in': {'links': ['2'],
    'counts': {'0': 92.0,
     '1': 57.0,
     '2': 53.0,
     '3': 55.0,
     '4': 88.0,
     '5': 222.0,
     '6': 637.0,
     '7': 1146.0,
     '8': 1017.0,
     '9': 691.0,
     '10': 578.0,
     '11': 519.0,
     '12': 540.0,
     '13': 615.0,
     '14': 619.0,
     '15': 630.0,
     '16': 828.0,
     '17': 913.0,
     '18': 890.0,
     '19': 629.0,
     '20': 326.0,
     '21': 315.0,
     '22': 252.0,
     '23': 159.0}},
   'out': {'links': ['3', '4'],
    'counts': {'0': 81.0,
     '1': 53.0,
     '2': 47.0,
     '3': 45.0,
     '4': 77.0,
     '5': 182.0,
     '6': 385.0,
     '7': 721.0,
     '8': 592.0,
     '9': 487.0,
     '10': 488.0,
     '11': 514.0,
     '12': 498.0,
     '13': 659.0,
     '14': 749.0,
     '15': 786.0,
     '16': 1009.0,
     '17': 908.0,
     '18': 845.0,
     '19': 578.0,
     '20': 370.0,
     '21': 273.0,
     '22': 230.0,
     '23': 137.0}}}},
 'bus': {'1': {'in': {'links': ['0'],
    'counts': {'0': 78.0,
     '1': 46.0,
     '2': 39.0,
     '3': 45.0,
     '4': 72.0,
     '5': 188.0,
     '6': 475.0,
     '7': 734.0,
     '8': 651.0,
     '9': 605.0,
     '10': 605.0,
     '11': 625.0,
     '12': 569.0,
     '13': 632.0,
     '14': 586.0,
     '15': 585.0,
     '16': 825.0,
     '17': 756.0,
     '18': 711.0,
     '19': 597.0,
     '20': 405.0,
     '21': 285.0,
     '22': 218.0,
     '23': 136.0}},
   'out': {'links': ['1'],
    'counts': {'0': 76.0,
     '1': 45.0,
     '2': 40.0,
     '3': 38.0,
     '4': 63.0,
     '5': 165.0,
     '6': 608.0,
     '7': 858.0,
     '8': 725.0,
     '9': 514.0,
     '10': 415.0,
     '11': 485.0,
     '12': 554.0,
     '13': 463.0,
     '14': 589.0,
     '15': 616.0,
     '16': 835.0,
     '17': 901.0,
     '18': 704.0,
     '19': 476.0,
     '20': 355.0,
     '21': 283.0,
     '22': 219.0,
     '23': 134.0}}},
  '2': {'in': {'links': ['2'],
    'counts': {'0': 92.0,
     '1': 57.0,
     '2': 53.0,
     '3': 55.0,
     '4': 88.0,
     '5': 222.0,
     '6': 637.0,
     '7': 1146.0,
     '8': 1017.0,
     '9': 691.0,
     '10': 578.0,
     '11': 519.0,
     '12': 540.0,
     '13': 615.0,
     '14': 619.0,
     '15': 630.0,
     '16': 828.0,
     '17': 913.0,
     '18': 890.0,
     '19': 629.0,
     '20': 326.0,
     '21': 315.0,
     '22': 252.0,
     '23': 159.0}},
   'out': {'links': ['3', '4'],
    'counts': {'0': 81.0,
     '1': 53.0,
     '2': 47.0,
     '3': 45.0,
     '4': 77.0,
     '5': 182.0,
     '6': 385.0,
     '7': 721.0,
     '8': 592.0,
     '9': 487.0,
     '10': 488.0,
     '11': 514.0,
     '12': 498.0,
     '13': 659.0,
     '14': 749.0,
     '15': 786.0,
     '16': 1009.0,
     '17': 908.0,
     '18': 845.0,
     '19': 578.0,
     '20': 370.0,
     '21': 273.0,
     '22': 230.0,
     '23': 137.0}}}}}

In [4]:

Copied!

read_csv_file(example_csv)
read_csv_file(example_csv)

Out[4]:

	id	direction	latitude	longitude	osm_id	links	other_links
0	L001	north	51.445503	-0.234686	19074660	['1']	'5'
1	L001	south	51.445503	-0.234686	19074660	['2']	'6'
2	L002	east	51.444698	-0.233398	142592756	['3']	'7'
3	L002	west	51.444698	-0.233398	132778373	['4']	'8'

We can attach them to a Network object using dedicated methods. Each file needs to pertain only to one set of IDs, either node or link IDs and the relevant method should be used to read the file.

In [5]:

Copied!





from genet import Network

n = Network("epsg:27700")
n.add_nodes(
    {
        "1": {"x": 1, "y": 2, "s2_id": 0},
        "2": {"x": 1, "y": 2, "s2_id": 0},
        "3": {"x": 1, "y": 2, "s2_id": 0},
        "4": {"x": 1, "y": 2, "s2_id": 0},
    }
)
n.add_links(
    {
        "1": {
            "from": "1",
            "to": "2",
            "freespeed": 1,
            "capacity": 1,
            "permlanes": 1,
            "length": 1,
            "modes": {"car"},
        },
        "2": {
            "from": "1",
            "to": "3",
            "freespeed": 1,
            "capacity": 1,
            "permlanes": 1,
            "length": 1,
            "modes": {"car"},
        },
        "3": {
            "from": "2",
            "to": "4",
            "freespeed": 1,
            "capacity": 1,
            "permlanes": 1,
            "length": 1,
            "modes": {"car"},
        },
        "4": {
            "from": "3",
            "to": "4",
            "freespeed": 1,
            "capacity": 1,
            "permlanes": 1,
            "length": 1,
            "modes": {"car"},
        },
    }
)
n.read_auxiliary_link_file(example_json)
n.read_auxiliary_node_file(example_csv)
from genet import Network

n = Network("epsg:27700")
n.add_nodes(
    {
        "1": {"x": 1, "y": 2, "s2_id": 0},
        "2": {"x": 1, "y": 2, "s2_id": 0},
        "3": {"x": 1, "y": 2, "s2_id": 0},
        "4": {"x": 1, "y": 2, "s2_id": 0},
    }
)
n.add_links(
    {
        "1": {
            "from": "1",
            "to": "2",
            "freespeed": 1,
            "capacity": 1,
            "permlanes": 1,
            "length": 1,
            "modes": {"car"},
        },
        "2": {
            "from": "1",
            "to": "3",
            "freespeed": 1,
            "capacity": 1,
            "permlanes": 1,
            "length": 1,
            "modes": {"car"},
        },
        "3": {
            "from": "2",
            "to": "4",
            "freespeed": 1,
            "capacity": 1,
            "permlanes": 1,
            "length": 1,
            "modes": {"car"},
        },
        "4": {
            "from": "3",
            "to": "4",
            "freespeed": 1,
            "capacity": 1,
            "permlanes": 1,
            "length": 1,
            "modes": {"car"},
        },
    }
)
n.read_auxiliary_link_file(example_json)
n.read_auxiliary_node_file(example_csv)

2022-07-14 15:41:18,911 - Added 4 nodes
2022-07-14 15:41:18,921 - Added 4 links

Auxiliary files will record where within the file the node or link ID data is stored and use it later for updating.

In [6]:

Copied!

n.auxiliary_files["node"]["links_benchmark.csv"].attachments
n.auxiliary_files["node"]["links_benchmark.csv"].attachments

Out[6]:

['links']

In [7]:

Copied!

n.auxiliary_files["link"]["links_benchmark.json"].attachments
n.auxiliary_files["link"]["links_benchmark.json"].attachments

Out[7]:

[{'car': {'1': {'out': 'links'}}},
 {'car': {'2': {'in': 'links'}}},
 {'car': {'2': {'out': 'links'}}},
 {'bus': {'1': {'out': 'links'}}},
 {'bus': {'2': {'in': 'links'}}},
 {'bus': {'2': {'out': 'links'}}}]

Now we can use methods to change the Network. The changes will be recorded and applied to the files when we save the network.

In [8]:

Copied!

n.simplify()
n.simplify()

2022-07-14 15:41:18,960 - Begin simplifying the graph
2022-07-14 15:41:18,962 - Generating paths to be simplified
2022-07-14 15:41:18,963 - Identified 2 edge endpoints
2022-07-14 15:41:18,964 - Identified 2 possible paths
2022-07-14 15:41:18,965 - Processing 2 paths
2022-07-14 15:41:18,966 - Found 2 paths to simplify.
2022-07-14 15:41:18,967 - Generated 2 link ids.
2022-07-14 15:41:18,967 - Processing links for all paths to be simplified
2022-07-14 15:41:18,969 - Adding new simplified links
2022-07-14 15:41:18,983 - Added 2 links
2022-07-14 15:41:18,986 - Simplified graph: 4 to 2 nodes, 4 to 2 edges

In [9]:

Copied!

n.attributes
n.attributes

Out[9]:

{'crs': 'epsg:27700', 'simplified': True}

In [10]:

Copied!

n.write_to_matsim(os.path.join("example_data", "outputs"))
n.write_to_matsim(os.path.join("example_data", "outputs"))

2022-07-14 15:41:19,006 - Writing ../genet_output/network.xml
2022-07-14 15:41:19,015 - Saving auxiliary file links_benchmark.json in ../genet_output/auxiliary_files
2022-07-14 15:41:19,019 - Saving auxiliary file links_benchmark.csv in ../genet_output/auxiliary_files

In [11]:

Copied!

read_json_file(os.path.join("example_data", "outputs", "auxiliary_files", "links_benchmark.json"))
read_json_file(os.path.join("example_data", "outputs", "auxiliary_files", "links_benchmark.json"))

Out[11]:

{'car': {'1': {'in': {'links': ['0'],
    'counts': {'0': 78.0,
     '1': 46.0,
     '2': 39.0,
     '3': 45.0,
     '4': 72.0,
     '5': 188.0,
     '6': 475.0,
     '7': 734.0,
     '8': 651.0,
     '9': 605.0,
     '10': 605.0,
     '11': 625.0,
     '12': 569.0,
     '13': 632.0,
     '14': 586.0,
     '15': 585.0,
     '16': 825.0,
     '17': 756.0,
     '18': 711.0,
     '19': 597.0,
     '20': 405.0,
     '21': 285.0,
     '22': 218.0,
     '23': 136.0}},
   'out': {'links': ['5'],
    'counts': {'0': 76.0,
     '1': 45.0,
     '2': 40.0,
     '3': 38.0,
     '4': 63.0,
     '5': 165.0,
     '6': 608.0,
     '7': 858.0,
     '8': 725.0,
     '9': 514.0,
     '10': 415.0,
     '11': 485.0,
     '12': 554.0,
     '13': 463.0,
     '14': 589.0,
     '15': 616.0,
     '16': 835.0,
     '17': 901.0,
     '18': 704.0,
     '19': 476.0,
     '20': 355.0,
     '21': 283.0,
     '22': 219.0,
     '23': 134.0}}},
  '2': {'in': {'links': ['0'],
    'counts': {'0': 92.0,
     '1': 57.0,
     '2': 53.0,
     '3': 55.0,
     '4': 88.0,
     '5': 222.0,
     '6': 637.0,
     '7': 1146.0,
     '8': 1017.0,
     '9': 691.0,
     '10': 578.0,
     '11': 519.0,
     '12': 540.0,
     '13': 615.0,
     '14': 619.0,
     '15': 630.0,
     '16': 828.0,
     '17': 913.0,
     '18': 890.0,
     '19': 629.0,
     '20': 326.0,
     '21': 315.0,
     '22': 252.0,
     '23': 159.0}},
   'out': {'links': ['5', '0'],
    'counts': {'0': 81.0,
     '1': 53.0,
     '2': 47.0,
     '3': 45.0,
     '4': 77.0,
     '5': 182.0,
     '6': 385.0,
     '7': 721.0,
     '8': 592.0,
     '9': 487.0,
     '10': 488.0,
     '11': 514.0,
     '12': 498.0,
     '13': 659.0,
     '14': 749.0,
     '15': 786.0,
     '16': 1009.0,
     '17': 908.0,
     '18': 845.0,
     '19': 578.0,
     '20': 370.0,
     '21': 273.0,
     '22': 230.0,
     '23': 137.0}}}},
 'bus': {'1': {'in': {'links': ['0'],
    'counts': {'0': 78.0,
     '1': 46.0,
     '2': 39.0,
     '3': 45.0,
     '4': 72.0,
     '5': 188.0,
     '6': 475.0,
     '7': 734.0,
     '8': 651.0,
     '9': 605.0,
     '10': 605.0,
     '11': 625.0,
     '12': 569.0,
     '13': 632.0,
     '14': 586.0,
     '15': 585.0,
     '16': 825.0,
     '17': 756.0,
     '18': 711.0,
     '19': 597.0,
     '20': 405.0,
     '21': 285.0,
     '22': 218.0,
     '23': 136.0}},
   'out': {'links': ['5'],
    'counts': {'0': 76.0,
     '1': 45.0,
     '2': 40.0,
     '3': 38.0,
     '4': 63.0,
     '5': 165.0,
     '6': 608.0,
     '7': 858.0,
     '8': 725.0,
     '9': 514.0,
     '10': 415.0,
     '11': 485.0,
     '12': 554.0,
     '13': 463.0,
     '14': 589.0,
     '15': 616.0,
     '16': 835.0,
     '17': 901.0,
     '18': 704.0,
     '19': 476.0,
     '20': 355.0,
     '21': 283.0,
     '22': 219.0,
     '23': 134.0}}},
  '2': {'in': {'links': ['0'],
    'counts': {'0': 92.0,
     '1': 57.0,
     '2': 53.0,
     '3': 55.0,
     '4': 88.0,
     '5': 222.0,
     '6': 637.0,
     '7': 1146.0,
     '8': 1017.0,
     '9': 691.0,
     '10': 578.0,
     '11': 519.0,
     '12': 540.0,
     '13': 615.0,
     '14': 619.0,
     '15': 630.0,
     '16': 828.0,
     '17': 913.0,
     '18': 890.0,
     '19': 629.0,
     '20': 326.0,
     '21': 315.0,
     '22': 252.0,
     '23': 159.0}},
   'out': {'links': ['5', '0'],
    'counts': {'0': 81.0,
     '1': 53.0,
     '2': 47.0,
     '3': 45.0,
     '4': 77.0,
     '5': 182.0,
     '6': 385.0,
     '7': 721.0,
     '8': 592.0,
     '9': 487.0,
     '10': 488.0,
     '11': 514.0,
     '12': 498.0,
     '13': 659.0,
     '14': 749.0,
     '15': 786.0,
     '16': 1009.0,
     '17': 908.0,
     '18': 845.0,
     '19': 578.0,
     '20': 370.0,
     '21': 273.0,
     '22': 230.0,
     '23': 137.0}}}}}

In [12]:

Copied!

read_csv_file(os.path.join("example_data", "outputs", "auxiliary_files", "links_benchmark.csv"))
read_csv_file(os.path.join("example_data", "outputs", "auxiliary_files", "links_benchmark.csv"))

Out[12]:

	Unnamed: 0	id	direction	latitude	longitude	osm_id	links	other_links
0	0	L001	north	51.445503	-0.234686	19074660	['1']	5
1	1	L001	south	51.445503	-0.234686	19074660	[None]	6
2	2	L002	east	51.444698	-0.233398	142592756	[None]	7
3	3	L002	west	51.444698	-0.233398	132778373	['4']	8

4.4. Using the Network object: Auxiliary files¶

4.4. Using the `Network` object: Auxiliary files¶