cngi.direct.framework

InitializeFramework(workers=2, memory='8GB', processes=True, **kwargs)[source]

Initialize the CNGI framework

This sets up the processing environment such that all future calls to Dask Dataframes, arrays, etc will automatically use the scheduler that is configured here.

Parameters

  • workers (int) – Number of processor cores to use, Default=2

  • memory (str) – Max memory allocated to each worker in string format. Default=’8GB’

  • processes (bool) – Whether to use processes (True) or threads (False), Default=True

  • threads_per_worker (int) – Only used if processes = True. Number of threads per python worker process, Default=1

Returns

Client from Dask Distributed for use by Dask objects

Return type

distributed.client.Client

GetFrameworkClient()[source]

Return the CNGI framework scheduler client

Returns

Client from Dask Distributed for use by Dask objects

Return type

distributed.client.Client