Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add ability to wait for an environment #1957

Open
wants to merge 80 commits into
base: main
Choose a base branch
from
Open

Add ability to wait for an environment #1957

wants to merge 80 commits into from

Commits on Sep 16, 2024

  1. Send in app threads directly from args

    * This allows us not to read the config for the
    app threads until absolutely necessary.
    8W9aG committed Sep 16, 2024
    Configuration menu
    Copy the full SHA
    6075901 View commit details
    Browse the repository at this point in the history
  2. Load config right before necessary

    * Delay loading the cog yaml until it is read from
    in the code.
    8W9aG committed Sep 16, 2024
    Configuration menu
    Copy the full SHA
    3d348cc View commit details
    Browse the repository at this point in the history
  3. Add waiting for a wait file

    * Allow environment variables to control whether
    we wait for a file to appear before further
    processing
    * This allows the python interpreter to boot up
    while we wait for other files to become available
    8W9aG committed Sep 16, 2024
    Configuration menu
    Copy the full SHA
    1024cac View commit details
    Browse the repository at this point in the history
  4. Add wait_for_imports ability

    * Allow waiting for a general environment by
    importing select python packages while the system
    boots up around cog.
    8W9aG committed Sep 16, 2024
    Configuration menu
    Copy the full SHA
    cc7fccf View commit details
    Browse the repository at this point in the history

Commits on Sep 17, 2024

  1. Fix lint on src_path

    8W9aG committed Sep 17, 2024
    Configuration menu
    Copy the full SHA
    19cc38d View commit details
    Browse the repository at this point in the history
  2. Fix watchdog version

    8W9aG committed Sep 17, 2024
    Configuration menu
    Copy the full SHA
    e79e25d View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    6dfe3ec View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    03f7011 View commit details
    Browse the repository at this point in the history
  5. Set recursive to true

    * I shouldn’t need to do this but want to see if
    it relieves the errors on linux.
    8W9aG committed Sep 17, 2024
    Configuration menu
    Copy the full SHA
    4f75de4 View commit details
    Browse the repository at this point in the history
  6. Watch the directory instead

    8W9aG committed Sep 17, 2024
    Configuration menu
    Copy the full SHA
    949792c View commit details
    Browse the repository at this point in the history
  7. Add code_xforms test

    * A small test to make sense of what the code
    stripper is doing.
    8W9aG committed Sep 17, 2024
    Configuration menu
    Copy the full SHA
    c83f2e8 View commit details
    Browse the repository at this point in the history
  8. Configuration menu
    Copy the full SHA
    7265582 View commit details
    Browse the repository at this point in the history

Commits on Sep 18, 2024

  1. Abstract away cog config

    * Create a class for accessing cog config
    * Only access variables from the config by
    properties
    * Gate those properties with environment variable
    function decorators to allow fetching the config
    from the environment rather than a file.
    * This allows the environment to begin running
    without needing the /src
    8W9aG committed Sep 18, 2024
    Configuration menu
    Copy the full SHA
    71854b9 View commit details
    Browse the repository at this point in the history
  2. Wait for environment before executing setup

    * While waiting for the environment to boot, load
    The designated imports to speed up interpreter
    Time.
    8W9aG committed Sep 18, 2024
    Configuration menu
    Copy the full SHA
    5ff8030 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    8f3f41d View commit details
    Browse the repository at this point in the history
  4. Skip test_strip_model_source_code if < 3.9

     * In strip model source code we use an AST
    function that isn’t available prior to 3.9
    8W9aG committed Sep 18, 2024
    Configuration menu
    Copy the full SHA
    1bebbc3 View commit details
    Browse the repository at this point in the history
  5. Change COG_WAIT_IMPORTS to COG_EAGER_IMPORTS

    * We aren’t waiting for these imports, we are
    loading them eagerly while we wait.
    8W9aG committed Sep 18, 2024
    Configuration menu
    Copy the full SHA
    d61d782 View commit details
    Browse the repository at this point in the history
  6. Bump integration test timeout to 20 mins

    * We have so many that we need to increase this.
    8W9aG committed Sep 18, 2024
    Configuration menu
    Copy the full SHA
    e9464df View commit details
    Browse the repository at this point in the history
  7. Add tests for Config class

    * Fix an issue with env_property where it could
    not handle Optional or Union
    8W9aG committed Sep 18, 2024
    Configuration menu
    Copy the full SHA
    adb04a4 View commit details
    Browse the repository at this point in the history

Commits on Sep 19, 2024

  1. Fix get_args and get_origin in python 3.7

    * Create these functions because they don’t exist
    in python 3.7
    8W9aG committed Sep 19, 2024
    Configuration menu
    Copy the full SHA
    25fb547 View commit details
    Browse the repository at this point in the history
  2. Add more tests for config

    8W9aG committed Sep 19, 2024
    Configuration menu
    Copy the full SHA
    ef2a1a0 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    061ff16 View commit details
    Browse the repository at this point in the history
  4. Add watch handler tests

    8W9aG committed Sep 19, 2024
    Configuration menu
    Copy the full SHA
    33b4106 View commit details
    Browse the repository at this point in the history
  5. Configuration menu
    Copy the full SHA
    2c1f2b9 View commit details
    Browse the repository at this point in the history
  6. Fix no torch import in tests

    8W9aG committed Sep 19, 2024
    Configuration menu
    Copy the full SHA
    f6d0e45 View commit details
    Browse the repository at this point in the history
  7. Configuration menu
    Copy the full SHA
    b480479 View commit details
    Browse the repository at this point in the history

Commits on Sep 20, 2024

  1. Do naive waiting for file

    * Do not wait for a signal, just use a while loop
    with a 10 ms interval to check for presence of
    file.
    8W9aG committed Sep 20, 2024
    Configuration menu
    Copy the full SHA
    42b7306 View commit details
    Browse the repository at this point in the history

Commits on Sep 24, 2024

  1. Configuration menu
    Copy the full SHA
    f79c597 View commit details
    Browse the repository at this point in the history

Commits on Sep 25, 2024

  1. Merge branch 'main' into add-waiting-env

    Signed-off-by: Will Sackfield <[email protected]>
    8W9aG authored Sep 25, 2024
    Configuration menu
    Copy the full SHA
    f0c5b79 View commit details
    Browse the repository at this point in the history
  2. Consolidate code_xforms tests

    * test_code_xforms actually lived in the server
    directory and not in the cog directory, move these
    tests into the cog directory.
    * Add a test and code for removing function
    decorators from the target functions, these can
    remove the ability to load the slimmed predictor.
    8W9aG committed Sep 25, 2024
    Configuration menu
    Copy the full SHA
    8fc79b0 View commit details
    Browse the repository at this point in the history
  3. Add logic to keep referenced globals

    * Keep a global if it is referenced as a default
    from a functions arguments.
    8W9aG committed Sep 25, 2024
    Configuration menu
    Copy the full SHA
    6b15565 View commit details
    Browse the repository at this point in the history

Commits on Sep 26, 2024

  1. Convert set of globals to list

    8W9aG committed Sep 26, 2024
    Configuration menu
    Copy the full SHA
    e4c2675 View commit details
    Browse the repository at this point in the history
  2. Add logging for file waiting

    8W9aG committed Sep 26, 2024
    Configuration menu
    Copy the full SHA
    bcc14a4 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    c33ead0 View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    065cf19 View commit details
    Browse the repository at this point in the history
  5. Fix more List issues

    8W9aG committed Sep 26, 2024
    Configuration menu
    Copy the full SHA
    75d7430 View commit details
    Browse the repository at this point in the history
  6. Configuration menu
    Copy the full SHA
    f1921c2 View commit details
    Browse the repository at this point in the history
  7. Configuration menu
    Copy the full SHA
    f3494c8 View commit details
    Browse the repository at this point in the history
  8. Add setup logs to mirror prediction logs

    * Allow the logs to show when certain setup steps
    are happening
    8W9aG committed Sep 26, 2024
    Configuration menu
    Copy the full SHA
    9fbb7ad View commit details
    Browse the repository at this point in the history

Commits on Sep 30, 2024

  1. Add 3.12 to torch 2.3.0

    8W9aG committed Sep 30, 2024
    Configuration menu
    Copy the full SHA
    096c45b View commit details
    Browse the repository at this point in the history

Commits on Oct 2, 2024

  1. Add insertion of python path

    8W9aG committed Oct 2, 2024
    Configuration menu
    Copy the full SHA
    a522a0f View commit details
    Browse the repository at this point in the history

Commits on Oct 3, 2024

  1. Configuration menu
    Copy the full SHA
    92d0b75 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    f51c576 View commit details
    Browse the repository at this point in the history
  3. Fix listing issues

    8W9aG committed Oct 3, 2024
    Configuration menu
    Copy the full SHA
    1a55b0d View commit details
    Browse the repository at this point in the history
  4. Wrap in str

    8W9aG committed Oct 3, 2024
    Configuration menu
    Copy the full SHA
    3b7756a View commit details
    Browse the repository at this point in the history
  5. Remove extraneous print

    8W9aG committed Oct 3, 2024
    Configuration menu
    Copy the full SHA
    21c1724 View commit details
    Browse the repository at this point in the history
  6. Fix python path race condition

    8W9aG committed Oct 3, 2024
    Configuration menu
    Copy the full SHA
    4598529 View commit details
    Browse the repository at this point in the history

Commits on Oct 9, 2024

  1. Merge branch 'main' into add-waiting-env

    Signed-off-by: Will Sackfield <[email protected]>
    8W9aG authored Oct 9, 2024
    Configuration menu
    Copy the full SHA
    3dc9e9d View commit details
    Browse the repository at this point in the history
  2. Remove predictor import

    8W9aG committed Oct 9, 2024
    Configuration menu
    Copy the full SHA
    0031765 View commit details
    Browse the repository at this point in the history
  3. Merge branch 'main' into add-waiting-env

    Signed-off-by: Will Sackfield <[email protected]>
    8W9aG authored Oct 9, 2024
    Configuration menu
    Copy the full SHA
    272756f View commit details
    Browse the repository at this point in the history
  4. Fix pydantic 2 errors

    8W9aG committed Oct 9, 2024
    Configuration menu
    Copy the full SHA
    1a8a374 View commit details
    Browse the repository at this point in the history
  5. More pydantic 2 fixes

    8W9aG committed Oct 9, 2024
    Configuration menu
    Copy the full SHA
    6e7b4c7 View commit details
    Browse the repository at this point in the history

Commits on Oct 11, 2024

  1. Support multiple class names and method names

    * This allows consumers to strip source code but
    Maintain multiple classes and methods
    * This is important if they want to maintain both
    Predict and train types
    8W9aG committed Oct 11, 2024
    Configuration menu
    Copy the full SHA
    84e947d View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    dc89489 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    5f86293 View commit details
    Browse the repository at this point in the history

Commits on Oct 14, 2024

  1. Merge branch 'main' into add-waiting-env

    Signed-off-by: Will Sackfield <[email protected]>
    8W9aG authored Oct 14, 2024
    Configuration menu
    Copy the full SHA
    5e1c842 View commit details
    Browse the repository at this point in the history
  2. Fix types imports

    8W9aG committed Oct 14, 2024
    Configuration menu
    Copy the full SHA
    2f883e4 View commit details
    Browse the repository at this point in the history

Commits on Oct 15, 2024

  1. Configuration menu
    Copy the full SHA
    697adf9 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    c2d3269 View commit details
    Browse the repository at this point in the history
  3. Revert "Add 3.12 to torch 2.3.0"

    This reverts commit 096c45b.
    8W9aG committed Oct 15, 2024
    Configuration menu
    Copy the full SHA
    6b05a79 View commit details
    Browse the repository at this point in the history

Commits on Oct 16, 2024

  1. Configuration menu
    Copy the full SHA
    926f2f5 View commit details
    Browse the repository at this point in the history
  2. Merge branch 'main' into add-waiting-env

    Signed-off-by: Will Sackfield <[email protected]>
    8W9aG authored Oct 16, 2024
    Configuration menu
    Copy the full SHA
    bd9c6cf View commit details
    Browse the repository at this point in the history
  3. Fix imports after merge

    8W9aG committed Oct 16, 2024
    Configuration menu
    Copy the full SHA
    8d6c4ce View commit details
    Browse the repository at this point in the history
  4. Add COG_TRAIN_PREDICTOR

    * Ensure that train/predictor stripped code is
    kept separate in the event a user uses separate
    files to refer to these different modes.
    8W9aG committed Oct 16, 2024
    Configuration menu
    Copy the full SHA
    bf222ce View commit details
    Browse the repository at this point in the history
  5. Rename env vars to be sensical

    * Old ones used to be many overloaded concepts.
    8W9aG committed Oct 16, 2024
    Configuration menu
    Copy the full SHA
    0c3c6f1 View commit details
    Browse the repository at this point in the history

Commits on Oct 17, 2024

  1. Add PYTHONPATH environment variables

    * We can’t just update the system path, we also
    need to change the PYTHONPATH environment variable
    * This allows forked processes to see this
    environment variable and act accordingly in user
    code.
    8W9aG committed Oct 17, 2024
    Configuration menu
    Copy the full SHA
    00b98bc View commit details
    Browse the repository at this point in the history

Commits on Oct 18, 2024

  1. Configuration menu
    Copy the full SHA
    5100333 View commit details
    Browse the repository at this point in the history

Commits on Oct 21, 2024

  1. Configuration menu
    Copy the full SHA
    07de3f1 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    8ea4663 View commit details
    Browse the repository at this point in the history
  3. Add consistent debug logging in config

    * Some tests previously checked the debug logging
    * Make sure we conform to the same debug logging
    8W9aG committed Oct 21, 2024
    Configuration menu
    Copy the full SHA
    3a29229 View commit details
    Browse the repository at this point in the history
  4. Add test_strip_model_source_code_keeps_referenced_class_from_function

    * Confirm that this behaviour is consistent with
    functions.
    8W9aG committed Oct 21, 2024
    Configuration menu
    Copy the full SHA
    7657c23 View commit details
    Browse the repository at this point in the history
  5. Explicitly check return code in test train

    * Allows better debugging of stderr et al
    8W9aG committed Oct 21, 2024
    Configuration menu
    Copy the full SHA
    e3c39c3 View commit details
    Browse the repository at this point in the history

Commits on Oct 22, 2024

  1. Handle response types in _predict

    * Currently predict assumes that PredictionResponse
    is the serialisation target.
    * This isn’t the case if we are calling _predict
    from a training endpoint
    * Allow response_type to be fed into the _predict
    method to inform it of what kind of response it
    should expect.
    8W9aG committed Oct 22, 2024
    Configuration menu
    Copy the full SHA
    5dc23a4 View commit details
    Browse the repository at this point in the history
  2. Make cog train a first class CLI function

    * Use the proper endpoints to call training
    functions.
    * Log the command properly to the user.
    8W9aG committed Oct 22, 2024
    Configuration menu
    Copy the full SHA
    a0477d8 View commit details
    Browse the repository at this point in the history
  3. Merge branch 'main' into add-waiting-env

    Signed-off-by: Will Sackfield <[email protected]>
    8W9aG authored Oct 22, 2024
    Configuration menu
    Copy the full SHA
    c046ff0 View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    94d1ff4 View commit details
    Browse the repository at this point in the history

Commits on Oct 23, 2024

  1. Merge branch 'main' into add-waiting-env

    Signed-off-by: Will Sackfield <[email protected]>
    8W9aG authored Oct 23, 2024
    Configuration menu
    Copy the full SHA
    187eeea View commit details
    Browse the repository at this point in the history

Commits on Oct 28, 2024

  1. Merge branch 'main' into add-waiting-env

    Signed-off-by: Will Sackfield <[email protected]>
    8W9aG authored Oct 28, 2024
    Configuration menu
    Copy the full SHA
    8ca482f View commit details
    Browse the repository at this point in the history
  2. Remove connection import

    8W9aG committed Oct 28, 2024
    Configuration menu
    Copy the full SHA
    2840fc7 View commit details
    Browse the repository at this point in the history

Commits on Nov 1, 2024

  1. Merge branch 'main' into add-waiting-env

    Signed-off-by: Will Sackfield <[email protected]>
    8W9aG authored Nov 1, 2024
    Configuration menu
    Copy the full SHA
    200173c View commit details
    Browse the repository at this point in the history