feat: batch_mode config option to unlock 3x multiple problem solving speed by reusing openebs and observe namespace (1mins speed up), plus helm pull policy (1mins speed up)

[zhangddjs]

No description provided.

[zhangddjs]

@microsoft-github-policy-service agree

[HacksonClark]

I'm running into this issue when running a problem with cli.py:

== Fault Injection ==
Traceback (most recent call last):
  File "/Users/jackson/Library/Caches/pypoetry/virtualenvs/aiopslab-85Fme-83-py3.12/lib/python3.12/site-packages/urllib3/connectionpool.py", line 787, in urlopen
    response = self._make_request(
               ^^^^^^^^^^^^^^^^^^^
  File "/Users/jackson/Library/Caches/pypoetry/virtualenvs/aiopslab-85Fme-83-py3.12/lib/python3.12/site-packages/urllib3/connectionpool.py", line 493, in _make_request
    conn.request(
  File "/Users/jackson/Library/Caches/pypoetry/virtualenvs/aiopslab-85Fme-83-py3.12/lib/python3.12/site-packages/urllib3/connection.py", line 494, in request
    self.endheaders()
  File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/http/client.py", line 1314, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/http/client.py", line 1074, in _send_output
    self.send(msg)
  File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/http/client.py", line 1018, in send
    self.connect()
  File "/Users/jackson/Library/Caches/pypoetry/virtualenvs/aiopslab-85Fme-83-py3.12/lib/python3.12/site-packages/docker/transport/unixconn.py", line 26, in connect
    sock.connect(self.unix_socket)
FileNotFoundError: [Errno 2] No such file or directory

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/jackson/Library/Caches/pypoetry/virtualenvs/aiopslab-85Fme-83-py3.12/lib/python3.12/site-packages/requests/adapters.py", line 667, in send
    resp = conn.urlopen(
           ^^^^^^^^^^^^^
  File "/Users/jackson/Library/Caches/pypoetry/virtualenvs/aiopslab-85Fme-83-py3.12/lib/python3.12/site-packages/urllib3/connectionpool.py", line 841, in urlopen
    retries = retries.increment(
              ^^^^^^^^^^^^^^^^^^
  File "/Users/jackson/Library/Caches/pypoetry/virtualenvs/aiopslab-85Fme-83-py3.12/lib/python3.12/site-packages/urllib3/util/retry.py", line 474, in increment
    raise reraise(type(error), error, _stacktrace)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jackson/Library/Caches/pypoetry/virtualenvs/aiopslab-85Fme-83-py3.12/lib/python3.12/site-packages/urllib3/util/util.py", line 38, in reraise
    raise value.with_traceback(tb)
  File "/Users/jackson/Library/Caches/pypoetry/virtualenvs/aiopslab-85Fme-83-py3.12/lib/python3.12/site-packages/urllib3/connectionpool.py", line 787, in urlopen
    response = self._make_request(
               ^^^^^^^^^^^^^^^^^^^
  File "/Users/jackson/Library/Caches/pypoetry/virtualenvs/aiopslab-85Fme-83-py3.12/lib/python3.12/site-packages/urllib3/connectionpool.py", line 493, in _make_request
    conn.request(
  File "/Users/jackson/Library/Caches/pypoetry/virtualenvs/aiopslab-85Fme-83-py3.12/lib/python3.12/site-packages/urllib3/connection.py", line 494, in request
    self.endheaders()
  File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/http/client.py", line 1314, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/http/client.py", line 1074, in _send_output
    self.send(msg)
  File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/http/client.py", line 1018, in send
    self.connect()
  File "/Users/jackson/Library/Caches/pypoetry/virtualenvs/aiopslab-85Fme-83-py3.12/lib/python3.12/site-packages/docker/transport/unixconn.py", line 26, in connect
    sock.connect(self.unix_socket)
urllib3.exceptions.ProtocolError: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/jackson/Library/Caches/pypoetry/virtualenvs/aiopslab-85Fme-83-py3.12/lib/python3.12/site-packages/docker/api/client.py", line 223, in _retrieve_server_version
    return self.version(api_version=False)["ApiVersion"]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jackson/Library/Caches/pypoetry/virtualenvs/aiopslab-85Fme-83-py3.12/lib/python3.12/site-packages/docker/api/daemon.py", line 181, in version
    return self._result(self._get(url), json=True)
                        ^^^^^^^^^^^^^^
  File "/Users/jackson/Library/Caches/pypoetry/virtualenvs/aiopslab-85Fme-83-py3.12/lib/python3.12/site-packages/docker/utils/decorators.py", line 44, in inner
    return f(self, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jackson/Library/Caches/pypoetry/virtualenvs/aiopslab-85Fme-83-py3.12/lib/python3.12/site-packages/docker/api/client.py", line 246, in _get
    return self.get(url, **self._set_request_timeout(kwargs))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jackson/Library/Caches/pypoetry/virtualenvs/aiopslab-85Fme-83-py3.12/lib/python3.12/site-packages/requests/sessions.py", line 602, in get
    return self.request("GET", url, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jackson/Library/Caches/pypoetry/virtualenvs/aiopslab-85Fme-83-py3.12/lib/python3.12/site-packages/requests/sessions.py", line 589, in request
    resp = self.send(prep, **send_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jackson/Library/Caches/pypoetry/virtualenvs/aiopslab-85Fme-83-py3.12/lib/python3.12/site-packages/requests/sessions.py", line 703, in send
    r = adapter.send(request, **kwargs)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jackson/Library/Caches/pypoetry/virtualenvs/aiopslab-85Fme-83-py3.12/lib/python3.12/site-packages/requests/adapters.py", line 682, in send
    raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/jackson/Desktop/MSR-AIOpsLab/cli.py", line 141, in <module>
    asyncio.run(main())
  File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/asyncio/runners.py", line 194, in run
    return runner.run(main)
           ^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/asyncio/runners.py", line 118, in run
    return self._loop.run_until_complete(task)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/asyncio/base_events.py", line 664, in run_until_complete
    return future.result()
           ^^^^^^^^^^^^^^^
  File "/Users/jackson/Desktop/MSR-AIOpsLab/cli.py", line 135, in main
    await agent.set_problem()
  File "/Users/jackson/Desktop/MSR-AIOpsLab/cli.py", line 89, in set_problem
    self.init_problem(problem_id.strip())
  File "/Users/jackson/Desktop/MSR-AIOpsLab/cli.py", line 101, in init_problem
    problem_desc, _, apis = self.orchestrator.init_problem(problem_id)
                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jackson/Desktop/MSR-AIOpsLab/aiopslab/orchestrator/orchestrator.py", line 77, in init_problem
    prob.inject_fault()
  File "/Users/jackson/Desktop/MSR-AIOpsLab/aiopslab/orchestrator/problems/k8s_target_port_misconfig/target_port.py", line 43, in inject_fault
    injector = VirtualizationFaultInjector(namespace=self.namespace)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jackson/Desktop/MSR-AIOpsLab/aiopslab/generators/fault/inject_virtual.py", line 22, in __init__
    self.docker = Docker()
                  ^^^^^^^^
  File "/Users/jackson/Desktop/MSR-AIOpsLab/aiopslab/service/dock.py", line 12, in __init__
    self.client = docker.from_env()
                  ^^^^^^^^^^^^^^^^^
  File "/Users/jackson/Library/Caches/pypoetry/virtualenvs/aiopslab-85Fme-83-py3.12/lib/python3.12/site-packages/docker/client.py", line 94, in from_env
    return cls(
           ^^^^
  File "/Users/jackson/Library/Caches/pypoetry/virtualenvs/aiopslab-85Fme-83-py3.12/lib/python3.12/site-packages/docker/client.py", line 45, in __init__
    self.api = APIClient(*args, **kwargs)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jackson/Library/Caches/pypoetry/virtualenvs/aiopslab-85Fme-83-py3.12/lib/python3.12/site-packages/docker/api/client.py", line 207, in __init__
    self._version = self._retrieve_server_version()
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jackson/Library/Caches/pypoetry/virtualenvs/aiopslab-85Fme-83-py3.12/lib/python3.12/site-packages/docker/api/client.py", line 230, in _retrieve_server_version
    raise DockerException(
docker.errors.DockerException: Error while fetching server API version: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))

Thought it seems potentially unrelated. Let me know if you see this as well!

[HacksonClark]

See #64 (comment)

[zhangddjs]

See #64 (comment)

Thank you for sharing, may I confirm about the reproduce steps, is such like this :

# Step1: clone repo and checkout
git clone git@github.com:zhangddjs/AIOpsLab.git --recurse-submodules
cd AIOpsLab; git checkout zhangdd/feat/batch_mode
# Step2: settle config.yml file
cp -a aiopslab/config.example.yml aiopslab/config.yml; vim aiopslab/config.yml
# Step3: start cli.py
poetry run python cli.py
# Step4: start problem
aiopslab> start k8s_target_port-misconfig-detection-1

And may I know does main branch also have the mentioned issues in your environment?

[HacksonClark]

I don't believe the issue is in main.

[zhangddjs]

ok, let me try to reproduce, from my side currently looks ok

[HacksonClark]

@zhangddjs what environment are you using? I'm using a kubernetes cluster deployed on cloudlab. Are you using a real Kubernetes cluster or an emulated one like KinD?