Side-game data project / FPL data collection
- Tacalabala
- FISO Knight
- Posts: 19010
- Joined: 07 Sep 2008, 01:03
- FPL:
Re: Side-game data project / FPL data collection
I swear you could at least access the player history page, certainly not the squad for obvious reasons. Maybe I'm going senile, it is a possibility given the way work has gone this week
- Mav3rick
- FISO Jedi Knight
- Posts: 20858
- Joined: 20 Jul 2009, 20:35
- FS Record: FPL: 1082, 1201, 1800, 10203
The stats are dark and full of errors.
Re: Side-game data project / FPL data collection
My hazy memory is the same as Stem, in that the links only appear for entrants, not pending entrants.Tacalabala wrote:Will let you know what the response is - they really have to bring this back in, crazy to stop this.To: support@mail.fantasy.premierleague.com
Subject: Other player's history pages
Date: Wed, 20 Jul 2016 22:13:03 +0100
Good evening,
Firstly, may I take the opportunity to congratulate you on a visually stunning update of the game.
Please can you resolve a problem that I appear to be having with the new set-up? I have previously been able to access other player's history pages by clicking on squad/usernames on league pages, but this is not possible at this time. Is this only a temporary restriction which will be reinstated in the coming days?
Kind regards
The thing I have been working through initially with MoSe has links to the pending entrants teams if it's helpful to you though.
- Tacalabala
- FISO Knight
- Posts: 19010
- Joined: 07 Sep 2008, 01:03
- FPL:
Re: Side-game data project / FPL data collection
Mav3rick wrote:My hazy memory is the same as Stem, in that the links only appear for entrants, not pending entrants.Tacalabala wrote:Will let you know what the response is - they really have to bring this back in, crazy to stop this.To: support@mail.fantasy.premierleague.com
Subject: Other player's history pages
Date: Wed, 20 Jul 2016 22:13:03 +0100
Good evening,
Firstly, may I take the opportunity to congratulate you on a visually stunning update of the game.
Please can you resolve a problem that I appear to be having with the new set-up? I have previously been able to access other player's history pages by clicking on squad/usernames on league pages, but this is not possible at this time. Is this only a temporary restriction which will be reinstated in the coming days?
Kind regards
The thing I have been working through initially with MoSe has links to the pending entrants teams if it's helpful to you though.
- SuperGrover
- Grumpy Old Man
- Posts: 1540
- Joined: 12 Sep 2012, 15:38
- Location: Chicago, USA
- FPL:
Re: Side-game data project / FPL data collection
Used to be my job, now I tell people who do this for a living what to dothesilkworm wrote:I'm just curious, what do the people who are contributing to this thread do for a living? Do you use these skills in your work? Obviously feel free to ignore this question
My skills are antiquated but still there to some extent.
- MoSe
- Dumbledore
- Posts: 9550
- Joined: 10 Sep 2014, 12:25
- Location: next door S.Siro stadium
- FS Record: FISODAS CUP Winner Season 25
FISO H2H Winner: 15/16 Div2 - 16/17 Div1
FISO Mirror: 16/17 PL Winner
Re: Side-game data project / FPL data collection
IIRC
1) you could click on Team names from a League Standings, which don't exist yet at this stage, leading to their team page for the current GW
2) and you could click on Manager names for the teams listed under Pending Entrants (to be added after the next points update) leading to their GW History
1) while I agree that it would be moot in the first case anyway, as it there'd be nothing to show yet before the 1st deadline,
2) I might instead like to check the Season History for players who'd gonna be playing in my League,
and that page IS definitely accessible even at this stage
1) but it would be outright ABSURD if we couldn't check other teams from league standings once the season started
I can't indeed recall how it used to be tho before the 1st deadline, hopefully BOTH features will resume working as I recalled once the season properly starts
______
1) you could click on Team names from a League Standings, which don't exist yet at this stage, leading to their team page for the current GW
2) and you could click on Manager names for the teams listed under Pending Entrants (to be added after the next points update) leading to their GW History
1) while I agree that it would be moot in the first case anyway, as it there'd be nothing to show yet before the 1st deadline,
2) I might instead like to check the Season History for players who'd gonna be playing in my League,
and that page IS definitely accessible even at this stage
- for instance, here's SG GW History page https://fantasy.premierleague.com/a/ent ... 18/history
(he hasn't updated his button yet :goob: , but posted his ID viewtopic.php?p=2911412#p2911412 )
and we CAN see there he got in the Top 1k two seasons ago
1) but it would be outright ABSURD if we couldn't check other teams from league standings once the season started
I can't indeed recall how it used to be tho before the 1st deadline, hopefully BOTH features will resume working as I recalled once the season properly starts
______
- PS:
the :goob: wasn't directed at SG, but it's a mock code conveying the trademark gooberman's "update your FPL button"
- MoSe
- Dumbledore
- Posts: 9550
- Joined: 10 Sep 2014, 12:25
- Location: next door S.Siro stadium
- FS Record: FISODAS CUP Winner Season 25
FISO H2H Winner: 15/16 Div2 - 16/17 Div1
FISO Mirror: 16/17 PL Winner
Re: Side-game data project / FPL data collection
BTW, if we get a League ID now, but we don't know the League owner FPL ID, we can't "hack" anymore the League CODE to join it....
which we know it was simply "OwnerID-LeagueID"
for instance for this season Fiso Forum League
viewtopic.php?f=82&t=121329
it's Spencer4 ID 95692 dash LeagueID 31821
which we know it was simply "OwnerID-LeagueID"
for instance for this season Fiso Forum League
viewtopic.php?f=82&t=121329
it's Spencer4 ID 95692 dash LeagueID 31821
- Mav3rick
- FISO Jedi Knight
- Posts: 20858
- Joined: 20 Jul 2009, 20:35
- FS Record: FPL: 1082, 1201, 1800, 10203
The stats are dark and full of errors.
Re: Side-game data project / FPL data collection
Just to be clear, that works right now for any league id, not just the FISO example I gave.MoSe wrote: 2) it's indeed regrettable that we can't reach that info now, unless a team owner discloses his new ID, or we get the link to Mav3rick's goldmine
- kidzio
- Wideboy
- Posts: 53
- Joined: 06 Sep 2015, 14:42
Re: Side-game data project / FPL data collection
I just want to say you guys are amazing and doing a great service! Especially Mav3rick! Sorry I can't contribute as I don't know these stuff (yet). Good luck for the season!
Sent from my iPhone using Tapatalk
Sent from my iPhone using Tapatalk
-
- Kevin and Perry
- Posts: 12
- Joined: 04 Jun 2015, 20:41
Re: Side-game data project / FPL data collection
Hey- does anyone have a link that can help one scrape the fixtures?
- Mav3rick
- FISO Jedi Knight
- Posts: 20858
- Joined: 20 Jul 2009, 20:35
- FS Record: FPL: 1082, 1201, 1800, 10203
The stats are dark and full of errors.
Re: Side-game data project / FPL data collection
The fixture data is all contained here (online parsed version).ChrisA wrote:Hey- does anyone have a link that can help one scrape the fixtures?
You'll need the lookup data here (online parsed version) (or a manual note of the team IDs) to work the data into a meaningful structure.
If you need an HTML scrape, I can't see a list of all fixtures, but you can step through all 38 GWs like so
https://fantasy.premierleague.com/a/fixtures/1
https://fantasy.premierleague.com/a/fixtures/2
-
- Kevin and Perry
- Posts: 12
- Joined: 04 Jun 2015, 20:41
Re: Side-game data project / FPL data collection
The fixture data is all contained here (online parsed version).
That's exactly what I wanted- thank you very much!
- RomynPG
- Dumbledore
- Posts: 9639
- Joined: 13 Oct 2005, 18:32
- FS Record: FPL 134 overall 06/07 ... never close since
- FPL:
Re: Side-game data project / FPL data collection
Awesome - that's made 5AS setup alot easier. (Captains not being able to get the squadIDs would have presented cat-herding possibilities)Mav3rick wrote:Just to be clear, that works right now for any league id, not just the FISO example I gave.MoSe wrote: 2) it's indeed regrettable that we can't reach that info now, unless a team owner discloses his new ID, or we get the link to Mav3rick's goldmine
Thanks dude
...not happy about my predicted score for GW1 though - https://ratemyteam.uk/Fiso/SideGameData/17443/1
- Surprised
- FISO Jedi Knight
- Posts: 26528
- Joined: 13 Oct 2005, 18:32
- Location: Home
- FS Record: TFFOSM MotW in 2008 and MotM in 2003. 78th overall in TFFO for 2002/3 and 2003/4
- FPL:
Re: Side-game data project / FPL data collection
I think this is right. The leagues are not populated until the 1st update is completedStemania wrote:Are you sure it isn't always like this before the first update since it isn't practical to see what team anyone has picked yet? The list of teams under each league aren't strictly speaking actually in he league yet, just to be added after the first update I think.
- RomynPG
- Dumbledore
- Posts: 9639
- Joined: 13 Oct 2005, 18:32
- FS Record: FPL 134 overall 06/07 ... never close since
- FPL:
Re: Side-game data project / FPL data collection
You used to be able to go to the gw history page for squads listed as pending-join in leagues.Surprised wrote:I think this is right. The leagues are not populated until the 1st update is completedStemania wrote:Are you sure it isn't always like this before the first update since it isn't practical to see what team anyone has picked yet? The list of teams under each league aren't strictly speaking actually in he league yet, just to be added after the first update I think.
- Surprised
- FISO Jedi Knight
- Posts: 26528
- Joined: 13 Oct 2005, 18:32
- Location: Home
- FS Record: TFFOSM MotW in 2008 and MotM in 2003. 78th overall in TFFO for 2002/3 and 2003/4
- FPL:
Re: Side-game data project / FPL data collection
sorry yes...pending members you could see but if you tried to download the league details it wouldn't workRomynPG wrote:You used to be able to go to the gw history page for squads listed as pending-join in leagues.Surprised wrote:I think this is right. The leagues are not populated until the 1st update is completedStemania wrote:Are you sure it isn't always like this before the first update since it isn't practical to see what team anyone has picked yet? The list of teams under each league aren't strictly speaking actually in he league yet, just to be added after the first update I think.
- FPLCasual
- Kevin and Perry
- Posts: 19
- Joined: 21 Feb 2016, 01:37
Re: Side-game data project / FPL data collection
Would there be any chance to include hits if they aren't already included as the GW score isn't the H2H score when hits are taken? Hits and Net Score would be great additions.Mav3rick wrote:Yep, it's my day job (which rather irritatingly eats into my FPL time).thesilkworm wrote:I'm just curious, what do the people who are contributing to this thread do for a living? Do you use these skills in your work? Obviously feel free to ignore this question
Ok MoSe, I had a couple of hours spare after lunch, so I put this together as a starting point, the URL contains the FPL League Id and the gameweek number, I get all the participants in the league, get their GW scores and return everything in a nice simple table:MoSe wrote:webqueries from excel were a handy and readymade built-in tool to use for us non-coders
Here's an example using the FISO league.
The gameweek data is randomised for now obviously, I'll need to return real data for the listed fields for my own site, but I could add other columns with relative ease I think. It's all static and simple HTML so Excel should be fine, if anybody wanted the actual JSON then I can expose that too. The data (which is all public info) is cached for 10 minutes but not persisted anywhere other than the FPL website.
Format of the URL is simply
https://ratemyteam.uk/Fiso/SideGameData ... /<GWNumber>
Is that useful to side game admins?
I wouldn't mind the actual JSON used if possible?
Cheers and keep up the great work.
- Mav3rick
- FISO Jedi Knight
- Posts: 20858
- Joined: 20 Jul 2009, 20:35
- FS Record: FPL: 1082, 1201, 1800, 10203
The stats are dark and full of errors.
Re: Side-game data project / FPL data collection
Once we get an actual set of FPL gameweek data I'll add those things. The score is inclusive of hits, so it would be easy to add a gross and net GW score with the hit amount shown.
- Stemania
- FISO Jedi Knight
- Posts: 20448
- Joined: 27 Aug 2006, 11:54
- Location: On the Iron Throne of xG, the seat of The Crown Prince of the Stat Perverts. Or if not, in the STC!
- FS Record: Best: TFF 321st. FPL 129th. FFS Career HoF Peak 2nd (Live 1st). Ability since lost.
- FPL:
Re: Side-game data project / FPL data collection
If FISO could do knighthoods...
Ok, you already have one, but a real one.
Ok, you already have one, but a real one.
- FPLCasual
- Kevin and Perry
- Posts: 19
- Joined: 21 Feb 2016, 01:37
Re: Side-game data project / FPL data collection
Cheers, had a play around with the ones you posted earlier. A JSON for a league in a similar format to the RMT/leagueID/GW would be great for any league runners.Mav3rick wrote:Once we get an actual set of FPL gameweek data I'll add those things. The score is inclusive of hits, so it would be easy to add a gross and net GW score with the hit amount shown.
Just over 2.5 weeks to go
- Mav3rick
- FISO Jedi Knight
- Posts: 20858
- Joined: 20 Jul 2009, 20:35
- FS Record: FPL: 1082, 1201, 1800, 10203
The stats are dark and full of errors.
Re: Side-game data project / FPL data collection
Try doing a GET from the same URL format as above, but modify your client to send an "Accept" header with the value "application/json". That should trigger content negotiation and send you back a JSON result rather than HMTL.FPLCasual wrote:Cheers, had a play around with the ones you posted earlier. A JSON for a league in a similar format to the RMT/leagueID/GW would be great for any league runners.Mav3rick wrote:Once we get an actual set of FPL gameweek data I'll add those things. The score is inclusive of hits, so it would be easy to add a gross and net GW score with the hit amount shown.
Just over 2.5 weeks to go
- FPLCasual
- Kevin and Perry
- Posts: 19
- Joined: 21 Feb 2016, 01:37
Re: Side-game data project / FPL data collection
Seems that your table got the table order correct but has the wrong scores.Mav3rick wrote:Once we get an actual set of FPL gameweek data I'll add those things. The score is inclusive of hits, so it would be easy to add a gross and net GW score with the hit amount shown.
- RomynPG
- Dumbledore
- Posts: 9639
- Joined: 13 Oct 2005, 18:32
- FS Record: FPL 134 overall 06/07 ... never close since
- FPL:
Re: Side-game data project / FPL data collection
Now that the first update has been done to the league pages I find my download code - which has been good (with tweaks) for 10 years now - no longer works
Not sure how best to sort it as there's no useful data in the raw html - it looks like it's all enhanced after init which doesn't work for Excel.
How are others getting on? ....
...going to have to look at Excel 2000's json capabilities
Not sure how best to sort it as there's no useful data in the raw html - it looks like it's all enhanced after init which doesn't work for Excel.
How are others getting on? ....
...going to have to look at Excel 2000's json capabilities
- Mav3rick
- FISO Jedi Knight
- Posts: 20858
- Joined: 20 Jul 2009, 20:35
- FS Record: FPL: 1082, 1201, 1800, 10203
The stats are dark and full of errors.
Re: Side-game data project / FPL data collection
The scores are deliberately random rather than wrong. I've not updated the code to pull the real scores yet (the data was only available today so give me a chance ). I should be able to do it before the end of the weekend though as I've already been looking at the data available.FPLCasual wrote:Seems that your table got the table order correct but has the wrong scores.Mav3rick wrote:Once we get an actual set of FPL gameweek data I'll add those things. The score is inclusive of hits, so it would be easy to add a gross and net GW score with the hit amount shown.
- FPLCasual
- Kevin and Perry
- Posts: 19
- Joined: 21 Feb 2016, 01:37
Re: Side-game data project / FPL data collection
Looks like you have already mastered itMav3rick wrote:The scores are deliberately random rather than wrong. I've not updated the code to pull the real scores yet (the data was only available today so give me a chance ). I should be able to do it before the end of the weekend though as I've already been looking at the data available.FPLCasual wrote:Seems that your table got the table order correct but has the wrong scores.Mav3rick wrote:Once we get an actual set of FPL gameweek data I'll add those things. The score is inclusive of hits, so it would be easy to add a gross and net GW score with the hit amount shown.
- Mav3rick
- FISO Jedi Knight
- Posts: 20858
- Joined: 20 Jul 2009, 20:35
- FS Record: FPL: 1082, 1201, 1800, 10203
The stats are dark and full of errors.
Re: Side-game data project / FPL data collection
I've updated the earlier post with details of accessing the gameweek data.RomynPG wrote:How are others getting on? ....
The sidegame data link has been updated too, to include the GW scores. I haven't completely checked all the data (to ensure it's correct) so let me know if you spot anything that's wrong. Here's an example for one of the FISO 5AS leagues:
https://ratemyteam.uk/Fiso/SideGameData/462148/1
(As I said in an earlier post, if you can modify your client to send an "Accept" header with the value "application/json" then you can use the same url to return the table data in JSON rather than HTML).
- MoSe
- Dumbledore
- Posts: 9550
- Joined: 10 Sep 2014, 12:25
- Location: next door S.Siro stadium
- FS Record: FISODAS CUP Winner Season 25
FISO H2H Winner: 15/16 Div2 - 16/17 Div1
FISO Mirror: 16/17 PL Winner
Re: Side-game data project / FPL data collection
At first glance, it works perfectly
I assume Hits are TC from the GW history, of course,
then Score should be GP from history, that is the GW whole pts without deduction
While Points should be Score - Hits (GP-TC aka the Net Score)
I say should as of course on GW1 there are no TC to check which is which
I also see that your GS or minutes or BP totals for a team, only counts the men on the pitch, not the benchers, which is correct
GS and GC in particular are used that way as tiebreakers fou H2H matches
the other summary data are indeed very interesting when comparing team totals in a league, although I don't know of any sidegame using them
_______
showing GP and TC separately was indeed a general requirement for Sidegames, as that's one of the usually customised options: I mean, several games use the GP without TC, several other sidegames use the net score, deducting TC
It was important to present both data, to allow any sidegame runner to use them in the appropriate way for their game format.
For instance, in Divisionals I follow the FPL classic scoring for Phases (i.e. Months)
TC get normally deducted, except on the 1st GW of a phase (GW4 scores won't be deducted from your September FPL score)
At this point, I figure it would be easy to add more fields in the table, from the gw history page
TM
(that's not the same as hits, as TC depends whether you had 1FT or 2FT)
FPL use TM as tiebreaker
Most sidegames don't
Mine does: I just reinstated TM as 1st tiebreaker, after it had been dropped for a few seasons.
If I can't access that data in the table too, I'd have to change the tiebreaker rules once again
PB / £
I don't know of any sidegame using that data, but as they're available, someone might eventually create a new sidegame using them as a scoring/ranking/tiebreaking element...
I assume Hits are TC from the GW history, of course,
then Score should be GP from history, that is the GW whole pts without deduction
While Points should be Score - Hits (GP-TC aka the Net Score)
I say should as of course on GW1 there are no TC to check which is which
I also see that your GS or minutes or BP totals for a team, only counts the men on the pitch, not the benchers, which is correct
GS and GC in particular are used that way as tiebreakers fou H2H matches
the other summary data are indeed very interesting when comparing team totals in a league, although I don't know of any sidegame using them
_______
showing GP and TC separately was indeed a general requirement for Sidegames, as that's one of the usually customised options: I mean, several games use the GP without TC, several other sidegames use the net score, deducting TC
It was important to present both data, to allow any sidegame runner to use them in the appropriate way for their game format.
For instance, in Divisionals I follow the FPL classic scoring for Phases (i.e. Months)
TC get normally deducted, except on the 1st GW of a phase (GW4 scores won't be deducted from your September FPL score)
At this point, I figure it would be easy to add more fields in the table, from the gw history page
TM
(that's not the same as hits, as TC depends whether you had 1FT or 2FT)
FPL use TM as tiebreaker
Most sidegames don't
Mine does: I just reinstated TM as 1st tiebreaker, after it had been dropped for a few seasons.
If I can't access that data in the table too, I'd have to change the tiebreaker rules once again
PB / £
I don't know of any sidegame using that data, but as they're available, someone might eventually create a new sidegame using them as a scoring/ranking/tiebreaking element...
- Tacalabala
- FISO Knight
- Posts: 19010
- Joined: 07 Sep 2008, 01:03
- FPL:
Re: Side-game data project / FPL data collection
My web queries don't work!
I'm going to need help, otherwise I really don't know what I'm going to do. Spreadsheet and iqy files attached.
I'm going to need help, otherwise I really don't know what I'm going to do. Spreadsheet and iqy files attached.
You do not have the required permissions to view the files attached to this post.
- Mav3rick
- FISO Jedi Knight
- Posts: 20858
- Joined: 20 Jul 2009, 20:35
- FS Record: FPL: 1082, 1201, 1800, 10203
The stats are dark and full of errors.
Re: Side-game data project / FPL data collection
Thanks MoSe. I've added in the properties you suggested and also an indicator for any active chips. I've also made the table a lot wider, but beware that it really is only suitable for desktop browsers as you need quite a lot of horizontal space. I think that's fine though as I can't imagine many people try to run their side games on mobile devices!MoSe wrote: ... Lots of good work...
- Mav3rick
- FISO Jedi Knight
- Posts: 20858
- Joined: 20 Jul 2009, 20:35
- FS Record: FPL: 1082, 1201, 1800, 10203
The stats are dark and full of errors.
Re: Side-game data project / FPL data collection
I've had a look at your spreadsheet, it looks like most of the data you need on the "Data-By-GW" tab would be available on the link above. So for example, the FISO 5AS Bad Boys league has things like the GW score, points benched, transfer hits, team value etc.Tacalabala wrote: My web queries don't work!
I'm going to need help, otherwise I really don't know what I'm going to do. Spreadsheet and iqy files attached.
If you can link up your FISO manager name to the FPL Team Id, then you should be able to import all the GW score data you need? I'd think that one web query to pull in all the GW scores from the FISO league into your Excel workbook would be your best bet, and then you can work out any massaging of the data within Excel if you need to keep any existing sheet formats?
This is the FISO league (31821) (it can take 20-30 seconds to load as there are 1,000+ entrants to get the GW scores for). Can you update your web query to pull in the information from that table?
If there is data that you're missing, let me know what you need and I'll try to retrieve it from the FPL JSON and display it as an extra column in the table for you to pull down. SG offered earlier to help out creating Excel files from the data, I suspect that some sort of standard FISO format downloadable Excel file would be useful, but if I'm honest I don't want to take on maintenance of that on top of the raw data and JSON parsing. I don't mind collaborating though and I can extract and present the relevant FPL data, if someone else can mould it into a useful Excel workbook.
- Tacalabala
- FISO Knight
- Posts: 19010
- Joined: 07 Sep 2008, 01:03
- FPL:
Re: Side-game data project / FPL data collection
Ignore Data by GW, its superfluous - GRID is the vital data. I'll attached last season's and then you'll see what I mean
View Latest: 1 Day View Your posts