Pulling Historical Faction Data | API Development | TORN
Pulling Historical Faction Data
  • AVAL SirEdge [2609907]SirEdge [2609907]
    • SirEdge [2609907]
    • Role: Civilian
    • Level: 50
    • Posts: 2061
    • Karma: 1569
    • Last Action: 10 minutes
    • Quote
    • Report
      • 0
    • Reason:
      Are you sure you want to report this post to staff?
      Cancel
    Thread created on 17:48:28 - 11/06/21 (4 months ago)
    |
    Last replied 15:33:22 - 17/06/21 (4 months ago)
    So I understand that faction data is limited to 100 items at once.  But it's hard to know what 100 items is to request when you haven't pulled it before.  So I am trying to ask for 1 day at a time, starting with the day our faction was formed back last September.  But the data it gives me is for the last 24 hours.  Here is the request string I am using (minus key and comment):

    https://api.torn.com/faction/?selections=attacks&from=&1600819200&to=1600905600&key=

    Any idea why the response is recent instead of the historical information I'm requesting?

    I've verified this information is available through the chain reports, but it would be nice to obtain things the chain reports don't include.

    Thanks in advance.
    Last edited by SirEdge on 19:41:49 - 11/06/21

    88325a84-98c2-374d-2609907.gif

  • PT Kelsie [340367]Kelsie [340367]
    • Kelsie [340367]
    • Role: Moderator
    • Level: 100
    • Posts: 6529
    • Karma: 11183
    • Last Action: 2 hours
    • Quote
    • Report
      • 0
    • Reason:
      Are you sure you want to report this post to staff?
      Cancel
    Posted on 21:56:46 - 11/06/21 (4 months ago)
    Post link copied to clipboard Copy post link
    Moved to API Development.

  • IA Lugburz [2386297]Lugburz [2386297]
    • Lugburz [2386297]
    • Role: Civilian
    • Level: 58
    • Posts: 720
    • Karma: 1035
    • Last Action: 30 minutes
    • Quote
    • Report
      • 0
    • Reason:
      Are you sure you want to report this post to staff?
      Cancel
    Posted on 00:03:37 - 12/06/21 (4 months ago)
    Post link copied to clipboard Copy post link
    Looks like you have an ampersand before the first timestamp? It shouldn't be there.
  • AVAL SirEdge [2609907]SirEdge [2609907]
    • SirEdge [2609907]
    • Role: Civilian
    • Level: 50
    • Posts: 2061
    • Karma: 1569
    • Last Action: 10 minutes
    • Quote
    • Report
      • 0
    • Reason:
      Are you sure you want to report this post to staff?
      Cancel
    Posted on 04:45:44 - 13/06/21 (4 months ago)
    Post link copied to clipboard Copy post link

    Lugburz [2386297]

    Looks like you have an ampersand before the first timestamp? It shouldn't be there.
    Okay, finally got a chance to fix that.  It is behaving better, but it's still a real hassle to figure out the timestamps- if there is an hour between events, were there no events, or did I just hit a 100 event boundary and now it is time to request again?

    If anyone has an elegant solution for this, I'd appreciate hearing it. Thanks.

    88325a84-98c2-374d-2609907.gif

  • AVAL SirEdge [2609907]SirEdge [2609907]
    • SirEdge [2609907]
    • Role: Civilian
    • Level: 50
    • Posts: 2061
    • Karma: 1569
    • Last Action: 10 minutes
    • Quote
    • Report
      • 0
    • Reason:
      Are you sure you want to report this post to staff?
      Cancel
    Posted on 14:11:28 - 16/06/21 (4 months ago)
    Post link copied to clipboard Copy post link
    So I had something of an epiphany last night- if I reverse the order and focus on loading oldest first, then I can just use a max+1 function to set the lower bound of the new pull and then run the upper bound as 24 hours later.  Seems to be working much better...

    And it worked better until it didn't. I got ~6000 good rows (85 or so days), then I started getting tons of duplicates. I've gotten up to 97 days of history now, but I'm getting more duplicates than new records at this point. I am noticing that Google is choking on the spreadsheet (taking a very long time to update the formulas) so it may not even be on the Torn side of the equation, so I may need to move it out into a separate file or something, but if anyone has any suggestions, I would appreciate them.

    EDIT- Nevermind. My sheet had grown and my check to pull the last date wasn't getting all the way to the end.
    Last edited by SirEdge on 15:24:55 - 16/06/21

    88325a84-98c2-374d-2609907.gif

  • AVAL SirEdge [2609907]SirEdge [2609907]
    • SirEdge [2609907]
    • Role: Civilian
    • Level: 50
    • Posts: 2061
    • Karma: 1569
    • Last Action: 10 minutes
    • Quote
    • Report
      • 0
    • Reason:
      Are you sure you want to report this post to staff?
      Cancel
    Posted on 15:33:22 - 17/06/21 (4 months ago)
    Post link copied to clipboard Copy post link
    By way of update, I finally have all of the faction data pulled (thank goodness it is less than a year- I can only imagine pulling everything for something with 100 players and many years of history).  Once the job was reliable, I set it to pull another 100 rows every 5 minutes.  Now that it is finished, I set it to once per hour.

    So now to setup a functional dashboard and charts off of the data.

    88325a84-98c2-374d-2609907.gif

Reply
Thread Title: