API Downtime/Slow Queries at ~23:20 UTC Daily | API Developmen…

API Downtime/Slow Queries at ~23:20 UTC Daily

    • Glasnost [1844049]
    • Role: Committee
    • Level: 68
    • Posts: 428
    • Karma: 2,175
    • Last Action: 9 minutes
      • 0
    • Reason:
      Are you sure you want to report this post to staff?
      Cancel
    Thread created on 00:16:39 - 05/09/24 (3 months ago)
    |
    Last replied 07:48:52 - 26/09/24 (2 months ago)

    So I noticed the API has been up and down while developing my tool, and decided to setup a dashboard in my monitoring to keep an eye on it.

     

    Anyway, from just 3 days of monitoring, seems a pattern is developing that around 23:30 UTC each day, it hits slow queries (I run with a timeout of 10s and only request basic user info, not a heavy request).

     

    2024-09-05-CSkneNkZrW4E.png

     

    I have been using rDacted's public key too, and the monitoring reflects that on his key too, ruling out it just being my key. Furthermore, I setup additional monitoring via UptimeRobot just to be sure it was my server, and UptimeRobot provides a similar story (albeit with 60 second checks rather than my own 30 second checks).

     

     

    Precise Timestamps

     

    01/09/24 - 23:20:40 - 23:46:20

    02/09/24 - 23:20:20 - 23:47:40

    03/09/24 - 23:20:45 - 23:49:15

     

    Of course I recognise the API is done on a "best effort" basis for availability and will always be a lower priority than the main Torn site if resources are tight, but I was unsure whether the dev team were aware of this.

    Co-owner and Developer of TornPal

    TornPal API - Extract market & bazaar data with your own tools/sheets

    • Glasnost [1844049]
    • Role: Committee
    • Level: 68
    • Posts: 428
    • Karma: 2,175
    • Last Action: 9 minutes
      • 0
    • Reason:
      Are you sure you want to report this post to staff?
      Cancel
    Posted on 00:29:02 - 05/09/24 (3 months ago)
    Post link copied to clipboard Copy post link

    2024-09-05-2F2HCJNLCvFw.png

     

    Slightly better view of the patchiness tonight. Latency is either high or drops to -1ms (indicating offline - see SLA graph for down indicator dropping to 0%).

    Co-owner and Developer of TornPal

    TornPal API - Extract market & bazaar data with your own tools/sheets

    • tiksan [2383326]
    • Role: Civilian
    • Level: 100
    • Posts: 947
    • Karma: 1,257
    • Last Action: 36 minutes
      • 0
    • Reason:
      Are you sure you want to report this post to staff?
      Cancel
    Posted on 14:07:59 - 05/09/24 (3 months ago)
    Post link copied to clipboard Copy post link

    Glasnost [1844049]

    So I noticed the API has been up and down while developing my tool, and decided to setup a dashboard in my monitoring to keep an eye on it.

     

    Anyway, from just 3 days of monitoring, seems a pattern is developing that around 23:30 UTC each day, it hits slow queries (I run with a timeout of 10s and only request basic user info, not a heavy request).

     

    2024-09-05-CSkneNkZrW4E.png

     

    I have been using rDacted's public key too, and the monitoring reflects that on his key too, ruling out it just being my key. Furthermore, I setup additional monitoring via UptimeRobot just to be sure it was my server, and UptimeRobot provides a similar story (albeit with 60 second checks rather than my own 30 second checks).

     

     

    Precise Timestamps

     

    01/09/24 - 23:20:40 - 23:46:20

    02/09/24 - 23:20:20 - 23:47:40

    03/09/24 - 23:20:45 - 23:49:15

     

    Of course I recognise the API is done on a "best effort" basis for availability and will always be a lower priority than the main Torn site if resources are tight, but I was unsure whether the dev team were aware of this.

    It started on the 29th by the way. Although it doesn't look as bad for me with only ~6% of requests time out (at a 5 second timeout).

    LnNwLrL.gif

    • Glasnost [1844049]
    • Role: Committee
    • Level: 68
    • Posts: 428
    • Karma: 2,175
    • Last Action: 9 minutes
      • 0
    • Reason:
      Are you sure you want to report this post to staff?
      Cancel
    Posted on 15:34:06 - 05/09/24 (3 months ago)
    Post link copied to clipboard Copy post link

    Glasnost [1844049]

    So I noticed the API has been up and down while developing my tool, and decided to setup a dashboard in my monitoring to keep an eye on it.

     

    Anyway, from just 3 days of monitoring, seems a pattern is developing that around 23:30 UTC each day, it hits slow queries (I run with a timeout of 10s and only request basic user info, not a heavy request).

     

    2024-09-05-CSkneNkZrW4E.png

     

    I have been using rDacted's public key too, and the monitoring reflects that on his key too, ruling out it just being my key. Furthermore, I setup additional monitoring via UptimeRobot just to be sure it was my server, and UptimeRobot provides a similar story (albeit with 60 second checks rather than my own 30 second checks).

     

     

    Precise Timestamps

     

    01/09/24 - 23:20:40 - 23:46:20

    02/09/24 - 23:20:20 - 23:47:40

    03/09/24 - 23:20:45 - 23:49:15

     

    Of course I recognise the API is done on a "best effort" basis for availability and will always be a lower priority than the main Torn site if resources are tight, but I was unsure whether the dev team were aware of this.

    tiksan [2383326]

    It started on the 29th by the way. Although it doesn't look as bad for me with only ~6% of requests time out (at a 5 second timeout).

    I suppose there is the chance I've been unlucky, but my monitoring isn't hitting it hard for obvious reasons so can't tell where I fit on that particular bell curve.

     

    Going to keep an eye on it either way, figured best to open the floor on the issue.

    Co-owner and Developer of TornPal

    TornPal API - Extract market & bazaar data with your own tools/sheets

    • Titanic_ [2968477]
    • Role: Civilian
    • Level: 72
    • Posts: 314
    • Karma: 316
    • Last Action: 3 minutes
      • 0
    • Reason:
      Are you sure you want to report this post to staff?
      Cancel
    Posted on 21:12:02 - 05/09/24 (3 months ago)
    Post link copied to clipboard Copy post link

    Noticed the same, since last week.

    • Glasnost [1844049]
    • Role: Committee
    • Level: 68
    • Posts: 428
    • Karma: 2,175
    • Last Action: 9 minutes
      • 0
    • Reason:
      Are you sure you want to report this post to staff?
      Cancel
    Posted on 23:51:07 - 05/09/24 (3 months ago)
    Post link copied to clipboard Copy post link

    Same again tonight: 23:19:30 - 23:51:15

     

    2024-09-06-sifARFAR4esW.png

    Last edited by Glasnost on 00:08:04 - 06/09/24 (3 months ago)

    Co-owner and Developer of TornPal

    TornPal API - Extract market & bazaar data with your own tools/sheets

    • Reborn121 [1864577]
    • Role: Civilian
    • Level: 69
    • Posts: 141
    • Karma: 91
    • Last Action: 40 minutes
      • 0
    • Reason:
      Are you sure you want to report this post to staff?
      Cancel
    Posted on 13:55:31 - 06/09/24 (3 months ago)
    Post link copied to clipboard Copy post link

    What are you using to setup this dashboard? I'd like to have access to something like this

    Last edited by Reborn121 on 13:55:44 - 06/09/24 (3 months ago)
    • Hemicopter [2780600]
    • Role: Civilian
    • Level: 74
    • Posts: 1,488
    • Karma: 3,567
    • Last Action: 6 hours
      • 0
    • Reason:
      Are you sure you want to report this post to staff?
      Cancel
    Posted on 14:12:43 - 06/09/24 (3 months ago)
    Post link copied to clipboard Copy post link

    Reborn121 [1864577]

    What are you using to setup this dashboard? I'd like to have access to something like this

    99% sure the dashboard is grafana

    • Glasnost [1844049]
    • Role: Committee
    • Level: 68
    • Posts: 428
    • Karma: 2,175
    • Last Action: 9 minutes
      • 0
    • Reason:
      Are you sure you want to report this post to staff?
      Cancel
    Posted on 15:59:09 - 06/09/24 (3 months ago)
    Post link copied to clipboard Copy post link

    Reborn121 [1864577]

    What are you using to setup this dashboard? I'd like to have access to something like this

    As Hemi mentioned, it's Grafana. Specifically a Grafana dashboard with Uptime Kuma for monitoring and Prometheus as the time series DB.

     

    Would you like me to make a public version so everyone can access?

    Last edited by Glasnost on 16:02:02 - 06/09/24 (3 months ago)

    Co-owner and Developer of TornPal

    TornPal API - Extract market & bazaar data with your own tools/sheets

    • Glasnost [1844049]
    • Role: Committee
    • Level: 68
    • Posts: 428
    • Karma: 2,175
    • Last Action: 9 minutes
      • 0
    • Reason:
      Are you sure you want to report this post to staff?
      Cancel
    Posted on 13:31:33 - 09/09/24 (3 months ago)
    Post link copied to clipboard Copy post link

    FYI, this seems to have been resolved for the most part. Still occasional issues, but nothing worth writing home about.

     

    2024-09-09-YuutBvByZU9k.png

    Co-owner and Developer of TornPal

    TornPal API - Extract market & bazaar data with your own tools/sheets

    • DeathFoundry [2097532]
    • Role: Civilian
    • Level: 100
    • Posts: 728
    • Karma: 1,735
    • Last Action: 2 hours
      • 0
    • Reason:
      Are you sure you want to report this post to staff?
      Cancel
    Posted on 07:48:52 - 26/09/24 (2 months ago)
    Post link copied to clipboard Copy post link

    I was also having issue for this timeframe....still had them until i just wiped all my api keys and started over....dont know if it torn and something external...but fresh start with keys has restored performance for me

    6035a411-a41e-46a0-99c6-7cefdb7b09d0-2097532.png

Reply
Thread Title: