Is Special Pages -> Export broken?

[L3laze 0]

I've been trying to use it to get all 24,000+ game entries and all I'm getting is the first 5000 entries every time. The only way that it seems it would be possible to do this would be to get every page title and add them one by one...and that's definitely not happening. It's also apparently not possible to add multiple categories at once.


So, is it possible to get the data without scraping every page using a "bot"? I've already scraped all the links from the All Games pages (first + next couple), but I know doing it that way is hard on the servers and I don't want to add any stress on them..

[Garrett 214]

5000 pages is MediaWiki's limit for the Special:Export category feature (details). This limit does not apply to page names entered manually. If you don't want to use the API a quick way to grab a full list is to click and drag over the names on Category:Games and copy-paste that into the Special:Export box, repeating for each additional page of results in the category, then start the export once you have the full list.

 

Do you have a particular use in mind? If your interest is in the settings information you can query this through Special:Ask (or the API) using one or more properties to find and display the information you're wanting to see (Semantic MediaWiki has extensive documentation for this). You could also use this to make a reduced list of relevant pages for Special:Export etc.

 

If you have any further questions feel free to ask. You can also usually find some of us in the Discord server.

 

(For reference, PCGamingWiki content is available under the terms of the Creative Commons Attribution Non-Commercial Share Alike license.)

[L3laze 0]

Dang, you're good -- I did start looking at the Semantic MediaWiki API & drill down form, but it did not seem like what I wanted so I gave up on it. Lol. And good trick with the copying all of the names from Category:Games; that worked perfectly. I tried to try copy all of the sub-category names one-by-one earlier, which was part of my issue along with them being the subs =\ It's been a day full of stuff like this both on and off the computer. I'm sorry.


Yeah, I'm not trying to make another Wiki from it, or post the data on my own site without credit/anything along those lines. I'm not a fan of Discord, but it would be good to chat with at least one of y'all privately about this idea and that would be easiest. It's about 5:30am right now though and I need sleep, so I will try to start tomorrow.


Thank you very much!!