Skip to content

Latest commit

 

History

History
134 lines (93 loc) · 4.47 KB

api.md

File metadata and controls

134 lines (93 loc) · 4.47 KB

API

The API allows interacting with ftio directly, rather than using the command line interface provided in the cli folder. Below or several examples of this.

General

The file ftio_api.py provides an example how to directly call use ftio:

import numpy as np
from ftio.cli.ftio_core import core
from ftio.parse.args import parse_args
from ftio.freq._dft import display_prediction
from ftio.plot.freq_plot import convert_and_plot
from ftio.parse.bandwidth import overlap

ranks = 10
total_bytes = 100

# Set up data
## 1) overlap for rank level metrics
b_rank   = [0.0,0.0,1000.0,1000.0,0.0,0.0,1000.0,1000.0,0.0,0.0,1000.0,1000.0,0.0,0.0]
t_rank_s = [0.5,0.0,10.5,10.0,20.5,20.0,30.5,30.0,40.5,40.0,50.5,50.0,60.5,60]
t_rank_e = [5.0,4.5,15.0,14.5,25.0,24.5,35.0,34.5,45.0,44.5,55.0,54.5,65.0,64.5]
b,t = overlap(b_rank,t_rank_s, t_rank_e)

# ## 2) or directly specify the app level metrics
# t = [10.0, 20.1, 30.0, 40.2, 50.3, 60, 70, 80.0,]
# b = [10, 0, 10, 0, 10, 0, 10, 0]


# command line arguments
argv = ["-e", "no"] #["-e", "mat"]

# set up data
data = {
        "time": np.array(t),
        "bandwidth": np.array(b),
        "total_bytes": total_bytes,
        "ranks": ranks 
        }

#parse args
args = parse_args(argv,"ftio")

# perform prediction
prediction, dfs = core([data], args)


# plot and print info
convert_and_plot(data, dfs, args)
display_prediction("ftio", prediction)

⬆

Metric Proxy

This API allows interaction with the metric proxy. Once executed, the proxy outputs a JSON file which can be directly used with this API. The file proxy.py provides an example. To execute it, simply call:

python proxy.py

The following line in proxy.py can be changed to specify the needed metric and the path to the JSON file:

b, t = parse("some_location/filename.json", "metric")

To suppress the output, the function display_prediction can be commented out. Moreover, argv = ["-e", "plotly"] can be changed to ["-e", "no"] to disable the plots.

Furthermore, at the end of proxy.py, postprocessing occurs to label the phases according to the function label_phases from processing.py.

⬆

GekkoFS with Msgpack/JSON support

The file ftio_gekko.py provides an example for the adapted ftio api for gekkoFs. The file predictor_gekko.py provides an example for predictor.

For ftio_gekko.py, the path to the files needs to be specified in the code:

if __name__ == "__main__":
    # absolute path to search all text files inside a specific folder
    # path=r'/d/github/FTIO/examples/API/gekkoFs/JSON/*.json' # For JSON
    path = r"/d/github/FTIO/examples/API/gekkoFs/MSGPACK/write*.msgpack"  # For MSGPCK
    matched_files = glob.glob(path)
    run(matched_files)

Similarly, for predictor_gekko.py, the following lines can be adjusted:

def main(args: list[str] = []) -> None:

    n_buffers = 4 # number of buffers 
    args = ["-e", "plotly", "-f", "0.01"] #arguments for ftio
    # path=r'/d/github/FTIO/examples/API/gekkoFs/JSON/*.json' # For JSON
    path = r"/d/github/FTIO/examples/API/gekkoFs/MSGPACK/write*.msgpack"  # For MSGPCK
    matched_files = glob.glob(path)

GekkoFS with ZMQ

The file predictor_zmq_gekko deploys this functionality.

Download and compile the file test_mpi.cxx from the TMIO repo. Then either execute it with a single rank (interactive) or with multiple ranks (with sleep). Simply execute the Makefile in the folder:

make test_mpi

make run_mpi # or run_single_mpi

for ftio, first navigate to the script and then execute it:

cd ftio/api/gekkoFs
python3 predictor_zmq_gekko.py

⬆