Continuous error loop after test completed

[jhughesoti]

Just started seeing a new issue show up. The tests run fine and the report is generated but then DAQ gets into this weird loop and never finishes.

Nov 02 19:16:17 runner   INFO    Remaining target sets: []
Nov 02 19:16:17 runner   INFO    Port 1 dpid 737894404660 learned dc:a6:32:8c:a7:3c
Nov 02 19:16:17 runner   INFO    Creating new device dc:a6:32:8c:a7:3c on None, port info True
Nov 02 19:16:17 runner   ERROR   Target device dca6328ca73c running False exception: type object 'datetime.datetime' has no                                 attribute 'fromisoformat'
Nov 02 19:16:17 runner   ERROR   type object 'datetime.datetime' has no attribute 'fromisoformat'
Traceback (most recent call last):
  File "/home/ubuntu/faucetsdn/daq/daq/runner.py", line 559, in _handle_system_idle_raw
    self.target_set_complete(device, 'target set not active')
  File "/home/ubuntu/faucetsdn/daq/daq/runner.py", line 1049, in target_set_complete
    self._target_set_finalize(device, device.host.results, reason)
  File "/home/ubuntu/faucetsdn/daq/daq/runner.py", line 1075, in _target_set_finalize
    self._handle_faucet_events()
  File "/home/ubuntu/faucetsdn/daq/daq/runner.py", line 425, in _handle_faucet_events
    self._process_faucet_event(event)
  File "/home/ubuntu/faucetsdn/daq/daq/runner.py", line 440, in _process_faucet_event
    self._handle_port_learn(dpid, port, target_mac)
  File "/home/ubuntu/faucetsdn/daq/daq/runner.py", line 497, in _handle_port_learn
    self._target_set_trigger(device)
  File "/home/ubuntu/faucetsdn/daq/daq/runner.py", line 680, in _target_set_trigger
    if not self._should_trigger_device(device, remote_trigger):
  File "/home/ubuntu/faucetsdn/daq/daq/runner.py", line 663, in _should_trigger_device
    if device.should_block():
  File "/home/ubuntu/faucetsdn/daq/daq/runner.py", line 85, in should_block
    endtime = datetime.fromisoformat(stream.read().strip())

[grafnu]

Interesting... really should be defined as per https://docs.python.org/3/library/datetime.html#datetime.date.fromisoformat -- it says since Python3.7 At the beginning of the run log or something it should indicate what version of Python it's running... but I'm having a hard time imagining how it would be <3.7...

…

On Tue, Nov 2, 2021 at 12:30 PM jhughesbiot ***@***.***> wrote: Just started seeing a new issue show up. The tests run fine and the report is generated but then DAQ gets into this weird loop and never finishes. Nov 02 19:16:17 runner INFO Remaining target sets: [] Nov 02 19:16:17 runner INFO Port 1 dpid 737894404660 learned dc:a6:32:8c:a7:3c Nov 02 19:16:17 runner INFO Creating new device dc:a6:32:8c:a7:3c on None, port info True Nov 02 19:16:17 runner ERROR Target device dca6328ca73c running False exception: type object 'datetime.datetime' has no attribute 'fromisoformat' Nov 02 19:16:17 runner ERROR type object 'datetime.datetime' has no attribute 'fromisoformat' Traceback (most recent call last): File "/home/ubuntu/faucetsdn/daq/daq/runner.py", line 559, in _handle_system_idle_raw self.target_set_complete(device, 'target set not active') File "/home/ubuntu/faucetsdn/daq/daq/runner.py", line 1049, in target_set_complete self._target_set_finalize(device, device.host.results, reason) File "/home/ubuntu/faucetsdn/daq/daq/runner.py", line 1075, in _target_set_finalize self._handle_faucet_events() File "/home/ubuntu/faucetsdn/daq/daq/runner.py", line 425, in _handle_faucet_events self._process_faucet_event(event) File "/home/ubuntu/faucetsdn/daq/daq/runner.py", line 440, in _process_faucet_event self._handle_port_learn(dpid, port, target_mac) File "/home/ubuntu/faucetsdn/daq/daq/runner.py", line 497, in _handle_port_learn self._target_set_trigger(device) File "/home/ubuntu/faucetsdn/daq/daq/runner.py", line 680, in _target_set_trigger if not self._should_trigger_device(device, remote_trigger): File "/home/ubuntu/faucetsdn/daq/daq/runner.py", line 663, in _should_trigger_device if device.should_block(): File "/home/ubuntu/faucetsdn/daq/daq/runner.py", line 85, in should_block endtime = datetime.fromisoformat(stream.read().strip()) — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#931>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAIEPD3WXHW3BL3SRD7WOPLUKBC6DANCNFSM5HHH2SFA> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

[jhughesoti]

Python 3.6 is default of Ubuntu 18.04, which I was running. When you run the cmd/build function, it throws an error and prompts an override command to build at this version, which I did.

I upgraded python to 3.8 and rebuilt the environment and haven't seen the issue come back yet but it wasn't 100% of the time with 3.6 either so it's hard to say if it's fully resolved. Upgrading to python 3.8 on ubuntu 18.04 does break some other core packages (netifaces, netplan, pyyaml) so it's a bit of a headache to get it stable after the upgrade.

[grafnu]

Ah, I think what we might need to do is more specifically stipulate the base Ubuntu/Python version. Some systems we're working with are up to Ubuntu 20 and Python 3.9, and maintaining the system across all versions can be problematic. Is there something pinning you to 18.04, or is that just "what you had" (because at one point it was the latest).

…

On Mon, Nov 15, 2021 at 2:02 PM jhughesbiot ***@***.***> wrote: Python 3.6 is default of Ubuntu 18.04, which I was running. When you run the cmd/build function, it throws an error and prompts an override command to build at this version, which I did. I upgraded python to 3.8 and rebuilt the environment and haven't seen the issue come back yet but it wasn't 100% of the time with 3.6 either so it's hard to say if it's fully resolved. Upgrading to python 3.8 on ubuntu 18.04 does break some other core packages (apt, netplan) so it's a bit of a headache to get it stable after the upgrade. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#931 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAIEPDZMMI2NDK4BQIT6OG3UMF7OTANCNFSM5HHH2SFA> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

[jhughesoti]

Base image I built when it was the latest to rebuild the OS and VM whenever I had to due to corruptions or any other catastrophic loss. Nothing specifically pinning me there. Sounds like an upgrade might be worth it.