RTX-KG2 is returning invalid timestamps in the logs in PROD

[maximusunc]

When I send this (or any) query to RTX-KG2 in PROD (this looks localized to PROD and not any of the other environments), I get back invalid timestamps in the logs according to the TRAPI spec: https://github.com/NCATSTranslator/ReasonerAPI/blob/master/TranslatorReasonerAPI.yaml#L565

curl -X POST https://kg2cploverdb.transltr.io/query \
  -H 'accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '{
        "message": {
          "query_graph": {
            "nodes": {
              "n0": {
                "ids": ["MONDO:0005148"],
                "categories": ["biolink:Disease"],
                "set_interpretation": "BATCH",
                "constraints": [],
                "member_ids": []
              },
              "n1": {
                "categories": ["biolink:PhenotypicFeature"],
                "set_interpretation": "BATCH",
                "constraints": [],
                "member_ids": []
              }
            },
            "edges": {
              "e01": {
                "subject": "n0",
                "object": "n1",
                "predicates": ["biolink:has_phenotype"],
                "attribute_constraints": [],
                "qualifier_constraints": []
              }
            }
          }
        },
        "bypass_cache": true,
        "submitter": "infores:aragorn"
      }' > rtx_kg2_prod_response.json

[saramsey]

@maximusunc thank you for bringing this to our attention. Also, your reprex was very helpful. I ran it and confirmed that in ITRB CI, the format of the timestamp string returned by KG2/PloverDB is:

    {
      "level": "INFO",
      "message": "kg2c: Done with query, returning TRAPI response (0 results)",
      "timestamp": "Tue, 16 Dec 2025 19:01:21 GMT"
    }

whereas the TRAPI spec requires ISO 8601:

      properties:
        timestamp:
          type: string
          format: date-time
          description: >-
            Timestamp in ISO 8601 format, providing the LogEntry time
            either in univeral coordinated time (UTC) using the 'Z' tag
            (e.g 2020-09-03T18:13:49Z), or, if local time is provided,
            the timezone offset must be provided
            (e.g. 2020-09-03T18:13:49-04:00).
          example: '2020-09-03T18:13:49+00:00'
          nullable: false

(I have transferred this issue from the RTX-KG2 project area to the PloverDB project area).

[saramsey]

Note, this is, evidently, already fixed in ITRB TEST:

    {
      "level": "INFO",
      "message": "kg2c: Done with query, returning TRAPI response (0 results)",
      "timestamp": "2025-12-16T19:08:59Z"
    }

so this fix will roll out during the next push to PROD. I don't know when that will happen, though (maybe Sarah Stemann will know).

[saramsey]

@maximusunc OK to close this issue?

[saramsey]

Following up on information I've received by DM:

Because this bug in KG2/PloverDB in PROD is causing all KG2 results to be discarded by Aragorn currently, @maximusunc is suggesting an out-of-cycle (as in, real soon) deployment of updated (patched) for KG2/PloverDB in ITRB PROD, to fix this issue. ARAX & PloverDB team, what do you think of the feasibility of this?

[edeutsch]

Have we confirmed that there is an actual code difference between TEST and PROD? I'm think the code should be identical at this time.

Could this be due to some weird system configuration like system default datetime format or something?

[saramsey]

@sundareswarpullela What branch of RTXteam/PloverDB is deployed to ITRB TEST?

[saramsey]

I that recall the setup of RTXteam/PloverDB is different than RTXteam/RTX (in terms of which branches are deployed to which ITRB systems), but I don't recall the details of how they are different.

[sundareswarpullela]

The main branch is deployed to ITRB TEST environment @saramsey

[saramsey]

What about ITRB PROD? Which branch is deployed to that?

[sundareswarpullela]

The main branch tag from ITRB TEST is then deployed into ITRB PROD @saramsey

[saramsey]

Like these tags?

[saramsey]

Thank you, @sundareswarpullela

@sundareswarpullela can you please check if the code fix for this issue is, in fact, in the v2.1.0 release? I don't see any newer release tag than that (and it was from March 1, 2025).