How to correctly access private policy repo? #139

hongbo-miao · 2021-09-09T18:45:08Z

hongbo-miao
Sep 9, 2021

Hi team, I am trying to run
https://github.com/authorizon/opal-fetcher-postgres

I generated my SSH key by

ssh-keygen -t ed25519 -C "[email protected]"

~/.ssh/id_ed25519
~/.ssh/id_ed25519.pub

I added the content of id_ed25519.pub in the GitHub:

Then I am using these two lines for opal server in docker-compose.yml file:

  opal_server:
    environment:
      - OPAL_POLICY_REPO_SSH_KEY="ssh-ed25519 xxx/xxx/xxx [email protected]"
      - OPAL_POLICY_REPO_URL=https://github.com/Hongbo-Miao/test-opa-policy
      - ...

The https://github.com/Hongbo-Miao/test-opa-policy repo is a private repo which is exactly same with https://github.com/authorizon/opal-example-policy-repo

However, I still got error

opal_server_1 | git.exc.GitCommandError: Cmd('git') failed due to: exit code(128)
opal_server_1 | cmdline: git clone -v https://github.com/Hongbo-Miao/test-opa-policy /regoclone
opal_server_1 | stderr: 'Cloning into '/regoclone'...
opal_server_1 | fatal: could not read Username for 'https://github.com': No such device or address

Here is the full log:

Click to expand

➜ docker compose up --force-recreate

[+] Running 4/4
⠿ Container opal-fetcher-postgres_example_db_1         Recreated                                                                                               0.2s
⠿ Container opal-fetcher-postgres_broadcast_channel_1  Recreated                                                                                               0.2s
⠿ Container opal-fetcher-postgres_opal_server_1        Recreated                                                                                               0.2s
⠿ Container opal-fetcher-postgres_opal_client_1        Recreated                                                                                               0.2s
Attaching to broadcast_channel_1, example_db_1, opal_client_1, opal_server_1
example_db_1         |
example_db_1         | PostgreSQL Database directory appears to contain a database; Skipping initialization
example_db_1         |
example_db_1         | 2021-09-09 18:33:52.567 UTC [1] LOG:  starting PostgreSQL 13.4 (Debian 13.4-1.pgdg100+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 8.3.0-6) 8.3.0, 64-bit
example_db_1         | 2021-09-09 18:33:52.567 UTC [1] LOG:  listening on IPv4 address "0.0.0.0", port 5432
example_db_1         | 2021-09-09 18:33:52.567 UTC [1] LOG:  listening on IPv6 address "::", port 5432
broadcast_channel_1  |
broadcast_channel_1  | PostgreSQL Database directory appears to contain a database; Skipping initialization
broadcast_channel_1  |
example_db_1         | 2021-09-09 18:33:52.578 UTC [1] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
example_db_1         | 2021-09-09 18:33:52.587 UTC [26] LOG:  database system was shut down at 2021-09-09 18:33:43 UTC
example_db_1         | 2021-09-09 18:33:52.597 UTC [1] LOG:  database system is ready to accept connections
broadcast_channel_1  | 2021-09-09 18:33:52.600 UTC [1] LOG:  starting PostgreSQL 13.4 on x86_64-pc-linux-musl, compiled by gcc (Alpine 10.3.1_git20210424) 10.3.1 20210424, 64-bit
broadcast_channel_1  | 2021-09-09 18:33:52.600 UTC [1] LOG:  listening on IPv4 address "0.0.0.0", port 5432
broadcast_channel_1  | 2021-09-09 18:33:52.600 UTC [1] LOG:  listening on IPv6 address "::", port 5432
broadcast_channel_1  | 2021-09-09 18:33:52.603 UTC [1] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
broadcast_channel_1  | 2021-09-09 18:33:52.607 UTC [22] LOG:  database system was shut down at 2021-09-09 18:33:44 UTC
broadcast_channel_1  | 2021-09-09 18:33:52.612 UTC [1] LOG:  database system is ready to accept connections
opal_server_1        | [2021-09-09 18:33:54 +0000] [1] [INFO] Starting gunicorn 20.1.0
opal_server_1        | [2021-09-09 18:33:54 +0000] [1] [INFO] Listening at: http://0.0.0.0:7002 (1)
opal_server_1        | [2021-09-09 18:33:54 +0000] [1] [INFO] Using worker: uvicorn.workers.UvicornWorker
opal_server_1        | [2021-09-09 18:33:54 +0000] [9] [INFO] Booting worker with pid: 9
opal_server_1        | [2021-09-09 18:33:54 +0000] [10] [INFO] Booting worker with pid: 10
opal_server_1        | [2021-09-09 18:33:54 +0000] [11] [INFO] Booting worker with pid: 11
opal_server_1        | [2021-09-09 18:33:54 +0000] [12] [INFO] Booting worker with pid: 12
opal_server_1        | 2021-09-09T18:33:55.799144+0000 |opal_common.authentication.signer       | INFO  | OPAL was not provided with JWT encryption keys, cannot verify api requests!
opal_server_1        | 2021-09-09T18:33:55.803584+0000 |opal_common.authentication.signer       | INFO  | OPAL was not provided with JWT encryption keys, cannot verify api requests!
opal_server_1        | 2021-09-09T18:33:55.834990+0000 |opal_common.authentication.signer       | INFO  | OPAL was not provided with JWT encryption keys, cannot verify api requests!
opal_server_1        | 2021-09-09T18:33:55.899002+0000 |opal_common.authentication.signer       | INFO  | OPAL was not provided with JWT encryption keys, cannot verify api requests!
opal_server_1        | 2021-09-09T18:33:55.900078+0000 |opal_server.server                      | INFO  | triggered startup event
opal_server_1        | 2021-09-09T18:33:55.900787+0000 |opal_common.topics.publisher            | INFO  | started topic publisher
opal_server_1        | 2021-09-09T18:33:55.901493+0000 |opal_server.server                      | INFO  | leadership lock acquired, leader pid: 11
opal_server_1        | 2021-09-09T18:33:55.901790+0000 |opal_server.server                      | INFO  | listening on webhook topic: 'webhook'
opal_server_1        | 2021-09-09T18:33:55.903716+0000 |fastapi_websocket_pubsub.event_notifier | INFO  | New subscription {'id': 'c8469df502dc48eabdfda3accc2afd16', 'subscriber_id': 'a45563df99b1447bbf793bb6d8c631ff', 'topic': 'webhook', 'callback': functools.partial(<function trigger_repo_watcher_pull at 0x7fed88ff31f0>, <opal_server.policy.watcher.task.RepoWatcherTask object at 0x7fed88ff4e50>), 'notifier_id': None}
opal_server_1        | 2021-09-09T18:33:55.904452+0000 |opal_server.policy.watcher.task         | INFO  | Launching repo watcher
opal_server_1        | 2021-09-09T18:33:55.905744+0000 |opal_common.git.repo_cloner             | INFO  | Cloning repo from 'https://github.com/Hongbo-Miao/test-opa-policy' to '/regoclone'
opal_server_1        | 2021-09-09T18:33:55.939090+0000 |opal_server.server                      | INFO  | triggered startup event
opal_server_1        | 2021-09-09T18:33:55.939461+0000 |opal_common.topics.publisher            | INFO  | started topic publisher
opal_server_1        | 2021-09-09T18:33:56.041325+0000 |opal_server.server                      | INFO  | triggered startup event
opal_server_1        | 2021-09-09T18:33:56.043827+0000 |opal_common.topics.publisher            | INFO  | started topic publisher
opal_server_1        | 2021-09-09T18:33:56.057541+0000 |opal_server.server                      | INFO  | triggered startup event
opal_server_1        | 2021-09-09T18:33:56.058145+0000 |opal_common.topics.publisher            | INFO  | started topic publisher
opal_client_1        | [2021-09-09 18:33:56 +0000] [9] [INFO] Starting gunicorn 20.1.0
opal_client_1        | [2021-09-09 18:33:56 +0000] [9] [INFO] Listening at: http://0.0.0.0:7000 (9)
opal_client_1        | [2021-09-09 18:33:56 +0000] [9] [INFO] Using worker: uvicorn.workers.UvicornWorker
opal_client_1        | [2021-09-09 18:33:56 +0000] [11] [INFO] Booting worker with pid: 11
opal_client_1        | 2021-09-09T18:33:57.237384+0000 |opal_common.fetcher.fetcher_register    | INFO  | Loading FetcherProvider 'FastApiRpcFetchProvider' found at: <class 'opal_common.fetcher.providers.fastapi_rpc_fetch_provider.FastApiRpcFetchProvider'>
opal_client_1        | 2021-09-09T18:33:57.237713+0000 |opal_common.fetcher.fetcher_register    | INFO  | Loading FetcherProvider 'HttpFetchProvider' found at: <class 'opal_common.fetcher.providers.http_fetch_provider.HttpFetchProvider'>
opal_client_1        | 2021-09-09T18:33:57.260430+0000 |opal_common.fetcher.fetcher_register    | INFO  | Loading FetcherProvider 'PostgresFetchProvider' found at: <class 'opal_fetcher_postgres.provider.PostgresFetchProvider'>
opal_client_1        | 2021-09-09T18:33:57.260668+0000 |opal_common.fetcher.fetcher_register    | INFO  | Fetcher Register loaded
opal_client_1        | 2021-09-09T18:33:57.284150+0000 |opal_client.opa.runner                  | INFO  | Launching opa runner
opal_client_1        | 2021-09-09T18:33:57.286187+0000 |opal_client.opa.runner                  | INFO  | Running OPA inline: opa run --server --addr=:8181 --authentication=off --authorization=off --log-level=info
opal_client_1        | 2021-09-09T18:33:57.302907+0000 |opal_client.opa.logger                  | INFO  | Initializing server. {'addrs': [':8181'], 'diagnostic-addrs': [], 'time': '2021-09-09T18:33:57Z'}
opal_client_1        | 2021-09-09T18:33:58.293121+0000 |opal_client.opa.runner                  | INFO  | Running OPA initial start callbacks
opal_client_1        | 2021-09-09T18:33:58.293735+0000 |opal_client.data.updater                | INFO  | Launching data updater
opal_client_1        | 2021-09-09T18:33:58.294058+0000 |opal_client.policy.updater              | INFO  | Launching policy updater
opal_client_1        | 2021-09-09T18:33:58.294430+0000 |opal_client.data.updater                | INFO  | Subscribing to topics: ['policy_data']
opal_client_1        | 2021-09-09T18:33:58.294970+0000 |opal_client.policy.updater              | INFO  | Subscribing to topics: ['policy:.']
opal_client_1        | 2021-09-09T18:33:58.295651+0000 |fastapi_websocket_pubsub.pub_sub_client | INFO  | Trying to connect to Pub/Sub server - ws://opal_server:7002/ws
opal_client_1        | 2021-09-09T18:33:58.296445+0000 |fastapi_websocket_rpc.websocket_rpc_c...| INFO  | Trying server - ws://opal_server:7002/ws
opal_client_1        | 2021-09-09T18:33:58.300505+0000 |fastapi_websocket_pubsub.pub_sub_client | INFO  | Trying to connect to Pub/Sub server - ws://opal_server:7002/ws
opal_client_1        | 2021-09-09T18:33:58.300802+0000 |fastapi_websocket_rpc.websocket_rpc_c...| INFO  | Trying server - ws://opal_server:7002/ws
opal_server_1        | 2021-09-09T18:33:58.305235+0000 |fastapi_websocket_pubsub.event_broadc...| INFO  | Listening for incoming events from broadcast channel (first listener started)
opal_server_1        | 2021-09-09T18:33:58.305474+0000 |fastapi_websocket_pubsub.event_broadc...| INFO  | Spawning broadcast listen task
opal_server_1        | 2021-09-09T18:33:58.306015+0000 |fastapi_websocket_pubsub.event_broadc...| INFO  | Listening for incoming events from broadcast channel (first listener started)
opal_server_1        | 2021-09-09T18:33:58.306217+0000 |fastapi_websocket_pubsub.event_broadc...| INFO  | Spawning broadcast listen task
opal_server_1        | 2021-09-09T18:33:58.333964+0000 |fastapi_websocket_pubsub.event_broadc...| INFO  | Subscribing to ALL TOPICS, and sharing messages with broadcast channel
opal_server_1        | 2021-09-09T18:33:58.333975+0000 |fastapi_websocket_pubsub.event_broadc...| INFO  | Subscribing to ALL TOPICS, and sharing messages with broadcast channel
opal_server_1        | 2021-09-09T18:33:58.334795+0000 |fastapi_websocket_pubsub.event_notifier | INFO  | New subscription {'id': 'aa5e79acfc8f4ce0a7d3f64c73ab9585', 'subscriber_id': '4206461d26f54039818a219629b06a36', 'topic': '__EventNotifier_ALL_TOPICS__', 'callback': <bound method EventBroadcaster.__broadcast_notifications__ of <fastapi_websocket_pubsub.event_broadcaster.EventBroadcaster object at 0x7fed88fc85e0>>, 'notifier_id': None}
opal_server_1        | 2021-09-09T18:33:58.334805+0000 |fastapi_websocket_pubsub.event_notifier | INFO  | New subscription {'id': 'c399dafbdf2947e18e73bed692edfa1b', 'subscriber_id': 'b5921c323dbd4bf5878c0433ba2b4a10', 'topic': '__EventNotifier_ALL_TOPICS__', 'callback': <bound method EventBroadcaster.__broadcast_notifications__ of <fastapi_websocket_pubsub.event_broadcaster.EventBroadcaster object at 0x7fed89008370>>, 'notifier_id': None}
opal_server_1        | 2021-09-09T18:33:58.335354+0000 |fastapi_websocket_rpc.websocket_rpc_e...| INFO  | Client connected
opal_server_1        | 2021-09-09T18:33:58.335561+0000 |fastapi_websocket_rpc.websocket_rpc_e...| INFO  | Client connected
opal_server_1        | 2021-09-09T18:33:58.335819+0000 |fastapi_websocket_pubsub.event_broadc...| INFO  | Starting broadcaster listener
opal_server_1        | 2021-09-09T18:33:58.336127+0000 |fastapi_websocket_pubsub.event_broadc...| INFO  | Starting broadcaster listener
opal_server_1        | 2021-09-09T18:33:58.361695+0000 |fastapi_websocket_pubsub.event_notifier | INFO  | New subscription {'id': 'a1565e82fa5e4868a244ef6876e55f9f', 'subscriber_id': '7ba89096967749ebb9ac00efcbef640d', 'topic': 'policy_data', 'callback': <function RpcEventServerMethods.subscribe.<locals>.callback at 0x7fed88a4d940>, 'notifier_id': None}
opal_server_1        | 2021-09-09T18:33:58.363103+0000 |fastapi_websocket_pubsub.event_notifier | INFO  | New subscription {'id': '285682ead8ea4885b0041791831d4ae5', 'subscriber_id': 'c0cf6efbc3e8484bb0220be6ae487b04', 'topic': 'policy:.', 'callback': <function RpcEventServerMethods.subscribe.<locals>.callback at 0x7fed88a56700>, 'notifier_id': None}
opal_client_1        | 2021-09-09T18:33:58.365544+0000 |opal_client.data.updater                | INFO  | Connected to server
opal_client_1        | 2021-09-09T18:33:58.365695+0000 |opal_client.data.updater                | INFO  | Performing data configuration, reason: Initial load
opal_client_1        | 2021-09-09T18:33:58.365772+0000 |opal_client.data.updater                | INFO  | Getting data-sources configuration from 'http://opal_server:7002/data/config'
opal_server_1        | 2021-09-09T18:33:58.368902+0000 |opal_server.data.api                    | INFO  | Serving source configuration
opal_client_1        | 2021-09-09T18:33:58.369145+0000 |opal_client.policy.updater              | INFO  | Connected to server
opal_client_1        | 2021-09-09T18:33:58.369432+0000 |opal_client.policy.updater              | INFO  | Refetching policy code (full bundle)
opal_server_1        | 2021-09-09T18:33:58.369707+0000 |uvicorn.protocols.http.httptools_impl   | INFO  | 172.25.0.5:33086 - "GET /data/config HTTP/1.1" 200
opal_client_1        | 2021-09-09T18:33:58.371705+0000 |opal_client.data.updater                | INFO  | Triggering data update with id: c53d90b401104890b63c190b1d63f33a
opal_client_1        | 2021-09-09T18:33:58.371939+0000 |opal_client.data.updater                | INFO  | Fetching policy data
opal_client_1        | 2021-09-09T18:33:58.372157+0000 |opal_client.data.fetcher                | INFO  | Fetching data from url: postgresql://postgres@example_db:5432/postgres
opal_client_1        | 2021-09-09T18:33:58.387815+0000 |fastapi_websocket_pubsub.pub_sub_client | INFO  | Connected to PubSub server ws://opal_server:7002/ws
opal_server_1        | 2021-09-09T18:33:58.390771+0000 |uvicorn.protocols.http.httptools_impl   | INFO  | 172.25.0.5:33088 - "GET /policy?path=. HTTP/1.1" 503
opal_client_1        | 2021-09-09T18:33:58.391670+0000 |opal_client.policy.fetcher              |WARNING | Unexpected response code 503: {'detail': 'policy repo was not found'}
opal_client_1        | 2021-09-09T18:33:58.402206+0000 |opal_client.data.updater                | INFO  | Saving fetched data to policy-store: source url='postgresql://postgres@example_db:5432/postgres', destination path='/cities'
opal_client_1        | 2021-09-09T18:33:58.404223+0000 |opal_client.opa.logger                  | INFO  | Received request.    PUT /v1/data/cities
opal_client_1        | 2021-09-09T18:33:58.406014+0000 |opal_client.opa.logger                  | INFO  | Sent response.       PUT /v1/data/cities -> 204
opal_server_1        | 2021-09-09T18:33:58.922604+0000 |uvicorn.protocols.http.httptools_impl   | INFO  | 172.25.0.5:33094 - "GET /policy?path=. HTTP/1.1" 503
opal_client_1        | 2021-09-09T18:33:58.923231+0000 |opal_client.policy.fetcher              |WARNING | Unexpected response code 503: {'detail': 'policy repo was not found'}
opal_server_1        | 2021-09-09T18:34:00.124712+0000 |uvicorn.protocols.http.httptools_impl   | INFO  | 172.25.0.5:33096 - "GET /policy?path=. HTTP/1.1" 503
opal_client_1        | 2021-09-09T18:34:00.125457+0000 |opal_client.policy.fetcher              |WARNING | Unexpected response code 503: {'detail': 'policy repo was not found'}
opal_server_1        | 2021-09-09T18:34:02.569649+0000 |uvicorn.protocols.http.httptools_impl   | INFO  | 172.25.0.5:33098 - "GET /policy?path=. HTTP/1.1" 503
opal_client_1        | 2021-09-09T18:34:02.570409+0000 |opal_client.policy.fetcher              |WARNING | Unexpected response code 503: {'detail': 'policy repo was not found'}
opal_server_1        | 2021-09-09T18:34:09.112089+0000 |opal_common.git.repo_cloner             |ERROR  | cannot clone policy repo: Cmd('git') failed due to: exit code(128)
opal_server_1        |   cmdline: git clone -v https://github.com/Hongbo-Miao/test-opa-policy /regoclone
opal_server_1        |   stderr: 'Cloning into '/regoclone'...
opal_server_1        | fatal: could not read Username for 'https://github.com': No such device or address
opal_server_1        | '
opal_server_1        | Traceback (most recent call last):
opal_server_1        |
opal_server_1        |   File "/root/.local/bin/gunicorn", line 8, in <module>
opal_server_1        |     sys.exit(run())
opal_server_1        |     │   │    └ <function run at 0x7fed8b231430>
opal_server_1        |     │   └ <built-in function exit>
opal_server_1        |     └ <module 'sys' (built-in)>
opal_server_1        |   File "/root/.local/lib/python3.8/site-packages/gunicorn/app/wsgiapp.py", line 67, in run
opal_server_1        |     WSGIApplication("%(prog)s [OPTIONS] [APP_MODULE]").run()
opal_server_1        |     └ <class 'gunicorn.app.wsgiapp.WSGIApplication'>
opal_server_1        |   File "/root/.local/lib/python3.8/site-packages/gunicorn/app/base.py", line 231, in run
opal_server_1        |     super().run()
opal_server_1        |   File "/root/.local/lib/python3.8/site-packages/gunicorn/app/base.py", line 72, in run
opal_server_1        |     Arbiter(self).run()
opal_server_1        |     │       └ <gunicorn.app.wsgiapp.WSGIApplication object at 0x7fed8b23e850>
opal_server_1        |     └ <class 'gunicorn.arbiter.Arbiter'>
opal_server_1        |   File "/root/.local/lib/python3.8/site-packages/gunicorn/arbiter.py", line 202, in run
opal_server_1        |     self.manage_workers()
opal_server_1        |     │    └ <function Arbiter.manage_workers at 0x7fed8ab7bf70>
opal_server_1        |     └ <gunicorn.arbiter.Arbiter object at 0x7fed8b23e130>
opal_server_1        |   File "/root/.local/lib/python3.8/site-packages/gunicorn/arbiter.py", line 551, in manage_workers
opal_server_1        |     self.spawn_workers()
opal_server_1        |     │    └ <function Arbiter.spawn_workers at 0x7fed8ab7e0d0>
opal_server_1        |     └ <gunicorn.arbiter.Arbiter object at 0x7fed8b23e130>
opal_server_1        |   File "/root/.local/lib/python3.8/site-packages/gunicorn/arbiter.py", line 622, in spawn_workers
opal_server_1        |     self.spawn_worker()
opal_server_1        |     │    └ <function Arbiter.spawn_worker at 0x7fed8ab7e040>
opal_server_1        |     └ <gunicorn.arbiter.Arbiter object at 0x7fed8b23e130>
opal_server_1        |   File "/root/.local/lib/python3.8/site-packages/gunicorn/arbiter.py", line 589, in spawn_worker
opal_server_1        |     worker.init_process()
opal_server_1        |     │      └ <function UvicornWorker.init_process at 0x7fed8a4959d0>
opal_server_1        |     └ <uvicorn.workers.UvicornWorker object at 0x7fed8a77e400>
opal_server_1        |   File "/root/.local/lib/python3.8/site-packages/uvicorn/workers.py", line 64, in init_process
opal_server_1        |     super(UvicornWorker, self).init_process()
opal_server_1        |           │              └ <uvicorn.workers.UvicornWorker object at 0x7fed8a77e400>
opal_server_1        |           └ <class 'uvicorn.workers.UvicornWorker'>
opal_server_1        |   File "/root/.local/lib/python3.8/site-packages/gunicorn/workers/base.py", line 142, in init_process
opal_server_1        |     self.run()
opal_server_1        |     │    └ <function UvicornWorker.run at 0x7fed8a495af0>
opal_server_1        |     └ <uvicorn.workers.UvicornWorker object at 0x7fed8a77e400>
opal_server_1        |   File "/root/.local/lib/python3.8/site-packages/uvicorn/workers.py", line 77, in run
opal_server_1        |     loop.run_until_complete(server.serve(sockets=self.sockets))
opal_server_1        |     │    │                  │      │             │    └ [<gunicorn.sock.TCPSocket object at 0x7fed8a23bfd0>]
opal_server_1        |     │    │                  │      │             └ <uvicorn.workers.UvicornWorker object at 0x7fed8a77e400>
opal_server_1        |     │    │                  │      └ <function Server.serve at 0x7fed8a6aa9d0>
opal_server_1        |     │    │                  └ <uvicorn.server.Server object at 0x7fed8a4fd8b0>
opal_server_1        |     │    └ <method 'run_until_complete' of 'uvloop.loop.Loop' objects>
opal_server_1        |     └ <uvloop.Loop running=True closed=False debug=False>
opal_server_1        |
opal_server_1        |   File "/opal_common/git/repo_watcher.py", line 63, in run
opal_server_1        |     result = self._cloner.clone()
opal_server_1        |              │    │       └ <function RepoCloner.clone at 0x7fed88fdaee0>
opal_server_1        |              │    └ <opal_common.git.repo_cloner.RepoCloner object at 0x7fed88ff4940>
opal_server_1        |              └ <opal_common.git.repo_watcher.RepoWatcher object at 0x7fed88ff4b50>
opal_server_1        |
opal_server_1        |   File "/opal_common/git/repo_cloner.py", line 113, in clone
opal_server_1        |     return self._attempt_clone_from_url()
opal_server_1        |            │    └ <function RepoCloner._attempt_clone_from_url at 0x7fed88fed040>
opal_server_1        |            └ <opal_common.git.repo_cloner.RepoCloner object at 0x7fed88ff4940>
opal_server_1        |
opal_server_1        | > File "/opal_common/git/repo_cloner.py", line 136, in _attempt_clone_from_url
opal_server_1        |     repo = _clone_with_retries()
opal_server_1        |            └ <function BaseRetrying.wraps.<locals>.wrapped_f at 0x7fed88dc99d0>
opal_server_1        |
opal_server_1        |   File "/root/.local/lib/python3.8/site-packages/tenacity/__init__.py", line 333, in wrapped_f
opal_server_1        |     return self(f, *args, **kw)
opal_server_1        |            │    │   │       └ {}
opal_server_1        |            │    │   └ ()
opal_server_1        |            │    └ functools.partial(<bound method Repo.clone_from of <class 'git.repo.base.Repo'>>, url='https://github.com/Hongbo-Miao/test-op...
opal_server_1        |            └ <Retrying object at 0x7fed88dd8280 (stop=<tenacity.stop.stop_after_attempt object at 0x7fed88fe69a0>, wait=<tenacity.wait.wai...
opal_server_1        |   File "/root/.local/lib/python3.8/site-packages/tenacity/__init__.py", line 423, in __call__
opal_server_1        |     do = self.iter(retry_state=retry_state)
opal_server_1        |          │    │                └ <tenacity.RetryCallState object at 0x7fed88dbcdc0>
opal_server_1        |          │    └ <function BaseRetrying.iter at 0x7fed897aaca0>
opal_server_1        |          └ <Retrying object at 0x7fed88dd8280 (stop=<tenacity.stop.stop_after_attempt object at 0x7fed88fe69a0>, wait=<tenacity.wait.wai...
opal_server_1        |   File "/root/.local/lib/python3.8/site-packages/tenacity/__init__.py", line 372, in iter
opal_server_1        |     raise retry_exc.reraise()
opal_server_1        |           │         └ <function RetryError.reraise at 0x7fed897aa1f0>
opal_server_1        |           └ RetryError(<Future at 0x7fed88dd8340 state=finished raised GitCommandError>)
opal_server_1        |   File "/root/.local/lib/python3.8/site-packages/tenacity/__init__.py", line 189, in reraise
opal_server_1        |     raise self.last_attempt.result()
opal_server_1        |           │    │            └ <function Future.result at 0x7fed8a52f280>
opal_server_1        |           │    └ <Future at 0x7fed88dd8340 state=finished raised GitCommandError>
opal_server_1        |           └ RetryError(<Future at 0x7fed88dd8340 state=finished raised GitCommandError>)
opal_server_1        |   File "/usr/local/lib/python3.8/concurrent/futures/_base.py", line 432, in result
opal_server_1        |     return self.__get_result()
opal_server_1        |            └ <Future at 0x7fed88dd8340 state=finished raised GitCommandError>
opal_server_1        |   File "/usr/local/lib/python3.8/concurrent/futures/_base.py", line 388, in __get_result
opal_server_1        |     raise self._exception
opal_server_1        |           │    └ GitCommandError(['git', 'clone', '-v', 'https://github.com/Hongbo-Miao/test-opa-policy', '/regoclone'], 128, b"Cloning into '...
opal_server_1        |           └ <Future at 0x7fed88dd8340 state=finished raised GitCommandError>
opal_server_1        |   File "/root/.local/lib/python3.8/site-packages/tenacity/__init__.py", line 426, in __call__
opal_server_1        |     result = fn(*args, **kwargs)
opal_server_1        |              │   │       └ {}
opal_server_1        |              │   └ ()
opal_server_1        |              └ functools.partial(<bound method Repo.clone_from of <class 'git.repo.base.Repo'>>, url='https://github.com/Hongbo-Miao/test-op...
opal_server_1        |   File "/root/.local/lib/python3.8/site-packages/git/repo/base.py", line 1111, in clone_from
opal_server_1        |     return cls._clone(git, url, to_path, GitCmdObjectDB, progress, multi_options, **kwargs)
opal_server_1        |            │   │      │    │    │        │               │         │                └ {}
opal_server_1        |            │   │      │    │    │        │               │         └ None
opal_server_1        |            │   │      │    │    │        │               └ None
opal_server_1        |            │   │      │    │    │        └ <class 'git.db.GitCmdObjectDB'>
opal_server_1        |            │   │      │    │    └ '/regoclone'
opal_server_1        |            │   │      │    └ 'https://github.com/Hongbo-Miao/test-opa-policy'
opal_server_1        |            │   │      └ <git.cmd.Git object at 0x7fed88dd9400>
opal_server_1        |            │   └ <classmethod object at 0x7fed8905e970>
opal_server_1        |            └ <class 'git.repo.base.Repo'>
opal_server_1        |   File "/root/.local/lib/python3.8/site-packages/git/repo/base.py", line 1049, in _clone
opal_server_1        |     finalize_process(proc, stderr=stderr)
opal_server_1        |     │                │            └ "Cloning into '/regoclone'...\nfatal: could not read Username for 'https://github.com': No such device or address\n"
opal_server_1        |     │                └ <git.cmd.Git.AutoInterrupt object at 0x7fed88dd8610>
opal_server_1        |     └ <function finalize_process at 0x7fed88e7dd30>
opal_server_1        |   File "/root/.local/lib/python3.8/site-packages/git/util.py", line 370, in finalize_process
opal_server_1        |     proc.wait(**kwargs)
opal_server_1        |     │    │      └ {'stderr': "Cloning into '/regoclone'...\nfatal: could not read Username for 'https://github.com': No such device or address\n"}
opal_server_1        |     │    └ <function Git.AutoInterrupt.wait at 0x7fed890820d0>
opal_server_1        |     └ <git.cmd.Git.AutoInterrupt object at 0x7fed88dd8610>
opal_server_1        |   File "/root/.local/lib/python3.8/site-packages/git/cmd.py", line 447, in wait
opal_server_1        |     raise GitCommandError(remove_password_if_present(self.args), status, errstr)
opal_server_1        |           │               │                          │    │      │       └ b"Cloning into '/regoclone'...\nfatal: could not read Username for 'https://github.com': No such device or address\n"
opal_server_1        |           │               │                          │    │      └ 128
opal_server_1        |           │               │                          │    └ <member 'args' of 'AutoInterrupt' objects>
opal_server_1        |           │               │                          └ <git.cmd.Git.AutoInterrupt object at 0x7fed88dd8610>
opal_server_1        |           │               └ <function remove_password_if_present at 0x7fed88e7de50>
opal_server_1        |           └ <class 'git.exc.GitCommandError'>
opal_server_1        |
opal_server_1        | git.exc.GitCommandError: Cmd('git') failed due to: exit code(128)
opal_server_1        |   cmdline: git clone -v https://github.com/Hongbo-Miao/test-opa-policy /regoclone
opal_server_1        |   stderr: 'Cloning into '/regoclone'...
opal_server_1        | fatal: could not read Username for 'https://github.com': No such device or address
opal_server_1        | '
opal_server_1        | 2021-09-09T18:34:09.148975+0000 |opal_server.policy.watcher.task         |ERROR  | watcher failed with exception: GitFailed()
opal_server_1        | 2021-09-09T18:34:09.149744+0000 |opal_server.policy.watcher.task         | INFO  | Stopping repo watcher
opal_server_1        | 2021-09-09T18:34:10.518144+0000 |uvicorn.protocols.http.httptools_impl   | INFO  | 172.25.0.5:33102 - "GET /policy?path=. HTTP/1.1" 503
opal_client_1        | 2021-09-09T18:34:10.519006+0000 |opal_client.policy.fetcher              |WARNING | Unexpected response code 503: {'detail': 'policy repo was not found'}
opal_client_1        | 2021-09-09T18:34:10.519664+0000 |opal_client.policy.fetcher              |WARNING | Failed all attempts to fetch bundle, got error: ValueError('unexpected response code while fetching bundle: 503')
opal_client_1        | 2021-09-09T18:34:10.520155+0000 |fastapi_websocket_pubsub.pub_sub_client | INFO  | Connected to PubSub server ws://opal_server:7002/ws

Any idea? Thanks!

Answered by asafc

Sep 9, 2021

Hi @hongbo-miao,

TL;DR

You accidentally used a repo url that uses the https protocol:

OPAL_POLICY_REPO_URL=https://github.com/Hongbo-Miao/test-opa-policy

The https:// prefix can only be used with public repos.

For private repo you must use the ssh git protocol.

I am guessing for your repo, the ssh url is something like:

[email protected]/Hongbo-Miao/test-opa-policy.git

The complete doc about this is found here.

Here is a complete guide how to work with private repos.

Step 1: generate a private-public key pair

Run this command

ssh-keygen -t ed25519 -C "[email protected]"

I actually recommend to select a custom filename, so you will not override other keys by mistake.
I p…

View full answer

asafc · 2021-09-09T21:38:16Z

asafc
Sep 9, 2021

Hi @hongbo-miao,

TL;DR

You accidentally used a repo url that uses the https protocol:

OPAL_POLICY_REPO_URL=https://github.com/Hongbo-Miao/test-opa-policy

The https:// prefix can only be used with public repos.

For private repo you must use the ssh git protocol.

I am guessing for your repo, the ssh url is something like:

[email protected]/Hongbo-Miao/test-opa-policy.git

The complete doc about this is found here.

Here is a complete guide how to work with private repos.

Step 1: generate a private-public key pair

Run this command

ssh-keygen -t ed25519 -C "[email protected]"

I actually recommend to select a custom filename, so you will not override other keys by mistake.
I picked this filename for this example:

Enter file in which to save the key (/Users/asafc/.ssh/id_ed25519): /Users/asafc/.ssh/examplekey_ed25519

I got the following public key file:

❯ cat ~/.ssh/examplekey_ed25519.pub
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIE0IUOAmHhF3jJAaYSSmq0lWeuIOka8+/laSPr8dE6oj [email protected]

and a private key file that looks like this:

❯ cat ~/.ssh/examplekey_ed25519
-----BEGIN OPENSSH PRIVATE KEY-----
b3B
<REDACTED>
UGBw==
-----END OPENSSH PRIVATE KEY-----

Step 2: on your private github repo, add the public key as a deploy key

Deploy keys are read only - and they are recommended for use with OPAL.
OPAL does not need write access to your repo - limit credentials where you can :)

Fill the form as shown in this image and click on Add Key:

This is what you will see after:

Step 3 - set `OPAL_POLICY_REPO_URL` value to the the repo ssh url

You will need to pass the repo ssh url to opal via OPAL_POLICY_REPO_URL.

On the github repo page click on the green Code button and select the ssh url, like shown in the image:

Click on the side button to copy the url to the clipboard.

example input:

[email protected]:authorizon/opal-example-private-policy-repo.git

Step 4 - set `OPAL_POLICY_REPO_SSH_KEY` value with the private ssh key

encode the key to one line first:

❯ cat ~/.ssh/examplekey_ed25519 | tr '\n' '_'

you will get an output looking something like this:

-----BEGIN OPENSSH PRIVATE KEY-----_b3B<REDACTED>GBw==_-----END OPENSSH PRIVATE KEY-----_

set the env var:

OPAL_POLICY_REPO_SSH_KEY=-----BEGIN OPENSSH PRIVATE KEY-----_b3B<REDACTED>GBw==_-----END OPENSSH PRIVATE KEY-----_

You should be able to clone the repo now, i used this configuration with the exception of changing these two env vars for OPAL server:

opal_server:
    image: authorizon/opal-server:latest
    environment:
      ...
      - [email protected]:authorizon/opal-example-private-policy-repo.git
      - OPAL_POLICY_REPO_SSH_KEY=-----BEGIN OPENSSH PRIVATE KEY-----_b3B<REDACTED>GBw==_-----END OPENSSH PRIVATE KEY-----_

BTW the opal-example-private-policy-repo repo is a real repo (exact clone of the example repo, but defined as a private repo).

0 replies

avinashs433 · 2023-09-13T14:43:13Z

avinashs433
Sep 13, 2023

Hi,

Is there a way to access a private gitlab repo through https?

Regards,
Avinash

4 replies

orweis Sep 13, 2023
Maintainer

Currently only SSH key based access is supportedbfor private repositories.

avinashs433 Sep 14, 2023

Is this on your roadmap for the near future ?

avinashs433 Sep 14, 2023

Is there an alternative to using a git repo as the policy repo ? Can I not mount a volume and makes changes in there to propagate ?

I see there is the bundle option
https://docs.opal.ac/tutorials/track_an_api_bundle_server

orweis Sep 14, 2023
Maintainer

Hi, it's not something that is currently on the roadmap - it's pretty are that one can't use SSH for Git (can you share on your use case ?).
Maybe consider contributing this change on your own? I can provide guidance on where to start.

And yes you can of course use the API-bundle-server option

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

How to correctly access private policy repo? #139

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 2 comments 4 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

Select a reply

How to correctly access private policy repo? #139

hongbo-miao Sep 9, 2021

TL;DR

Step 1: generate a private-public key pair

Replies: 2 comments · 4 replies

asafc Sep 9, 2021

TL;DR

Step 1: generate a private-public key pair

Step 2: on your private github repo, add the public key as a deploy key

Step 3 - set OPAL_POLICY_REPO_URL value to the the repo ssh url

Step 4 - set OPAL_POLICY_REPO_SSH_KEY value with the private ssh key

avinashs433 Sep 13, 2023

orweis Sep 13, 2023 Maintainer

avinashs433 Sep 14, 2023

avinashs433 Sep 14, 2023

orweis Sep 14, 2023 Maintainer

hongbo-miao
Sep 9, 2021

Replies: 2 comments 4 replies

asafc
Sep 9, 2021

Step 3 - set `OPAL_POLICY_REPO_URL` value to the the repo ssh url

Step 4 - set `OPAL_POLICY_REPO_SSH_KEY` value with the private ssh key

avinashs433
Sep 13, 2023

orweis Sep 13, 2023
Maintainer

orweis Sep 14, 2023
Maintainer