[Bug]: New experimenal MSSentinelSearch data provider doesn't correctly use the AzureSearchDriver

[JPvRiel]

Describe the bug

The MSSentinelSearch query provider / data environment seems to get confused between using the MSSentinel vs MSSentinelSearch data environments and fails to correctly connect the AzureSearchDriver.

To Reproduce

Steps to reproduce the behavior:

1. Clone from this git repo and checkout main in order to test PR
   Ianhelle/az monitor search driver 2025 02 05 #825 included in main.
2. Create an editable venv from the source and activate.
3. Configure msticpyconfig.yaml with a 'Sentinel' workspace that inlcudes a basic table.
4. Run a test script with the AzureSearchDriver on a table with the 'basic' plan.

import datetime

# Set debug logging
import logging
logging.basicConfig(level=logging.DEBUG)

# Inherit log level
import msticpy
print(f'msticpy version: {msticpy.__version__}')

# Config
msticpy.init_notebook()

# ws_config = msticpy.common.wsconfig.WorkspaceConfig(workspace="MyWorkspace")
# print(f'Workspace config: {ws_config}')
#qry_prov_basic_search = msticpy.QueryProvider(data_environment='MSSentinelSearch', ws_config=ws_config, workspace='BasicLogs')
qry_prov_basic_search = msticpy.QueryProvider('MSSentinelSearch')
qry_prov_basic_search.connect(workspace='BasicLogs')
print(f'Query provider driver: {qry_prov_basic_search.driver_class}')
print(f'Query provider environment: {qry_prov_basic_search.environment}')
print(f'Query provider connections: {qry_prov_basic_search.list_connections()}')

# Prep a small time range to limit basic logs query costs
lookback_period = datetime.timedelta(hours=1)
ingest_grace_period = datetime.timedelta(minutes=15)
end = datetime.datetime.now(datetime.timezone.utc) - ingest_grace_period
start = end - lookback_period
print(f'Start: {start}, End: {end}')

# Test query
df = qry_prov_basic_search.exec_query('SyslogBasic_CL | take 1', start=start, end=end)
print(df)

Expected behavior

AzureSearchDriver is connected and used with the corresponding MSSentinelSearch data environment.

Screenshots and/or Traceback

INFO:msticpy.data.drivers.azure_monitor_driver:AzureMonitorDriver loaded. connect_str  None, kwargs: {'data_environment': <DataEnvironment.MSSentinelSearch: 25>}
INFO:msticpy.data.core.data_providers:Using data environment MSSentinel
INFO:msticpy.data.core.data_providers:Driver class: AzureSearchDriver
...
INFO:msticpy.data.core.data_providers:Calling connect on driver
INFO:msticpy.data.drivers.azure_monitor_driver:WorkspaceConfig created from workspace name BasicLogs
...
INFO:msticpy.data.drivers.azure_monitor_driver:WorkspaceConfig created from workspace name BasicLogs
INFO:msticpy.data.drivers.azure_search_driver:Created HTTP-based query client using /search endpoint.
connected
INFO:msticpy.data.core.data_providers:Adding query pivot functions
Query provider driver: <class 'msticpy.data.drivers.azure_search_driver.AzureSearchDriver'>
Query provider environment: MSSentinel
Query provider connections: ['Default: BasicLogs']
Start: 2025-02-16 19:20:23.193644+00:00, End: 2025-02-16 20:20:23.193644+00:00
INFO:msticpy.data.core.query_provider_connections_mixin:Executing query 'SyslogBasic_CL | take 1...'
DEBUG:msticpy.data.core.query_provider_connections_mixin:Full query: SyslogBasic_CL | take 1
DEBUG:msticpy.data.core.query_provider_connections_mixin:Query options: {'start': datetime.datetime(2025, 2, 16, 19, 20, 23, 193644, tzinfo=datetime.timezone.utc), 'end': datetime.datetime(2025, 2, 16, 20, 20, 23, 193644, tzinfo=datetime.timezone.utc)}
Traceback (most recent call last):
  File "/home/***/SecurityNotebooks/demo/_bugs/msticpy/msticpy_cannot_query_azure_monitor_basic_logs/test_data_prov_ms_sentinel_search.py", line 39, in <module>
    df = qry_prov_basic_search.exec_query('SyslogBasic_CL | take 1', start=start, end=end)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/***/SecurityNotebooks/demo/_bugs/msticpy/msticpy_cannot_query_azure_monitor_basic_logs/src/msticpy/msticpy/data/core/query_provider_connections_mixin.py", line 99, in exec_query
    return self._query_provider.query(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/***/SecurityNotebooks/demo/_bugs/msticpy/msticpy_cannot_query_azure_monitor_basic_logs/src/msticpy/msticpy/data/drivers/azure_monitor_driver.py", line 294, in query
    raise MsticpyNotConnectedError(
msticpy.common.exceptions.MsticpyNotConnectedError: ('Workspace not connected.', 'Please run connect() to connect to the workspace', 'before running a query.', 'https://msticpy.readthedocs.io/en/latest/data_acquisition/DataProv-MSSentinel.html')

Environment (please complete the following information):

• Python Version: 3.111
• OS: Ubuntu
• Python environment: editable venv from cloned msticpy source with main branch checked out
• MSTICPy Version: 2.16 (pre-release, main branch, commit a4b0b72)

Additional context

PR #825 was merged to main and closed #819.

This is related to another issue with the PR: #830.

[ianhelle]

I'm away for a week but will try to look at this when I get back.

[ianhelle]

Will try to look at this later in the week - this was a very quick and dirty attempt and I don't have any data source to test against, so fumbling around in the dark a little