Forums > Suggestions > Proposition: stop adding Android and iOS. Let me explain why.
twitek (13973) on 8/28/2017 6:49 AM · Permalink · Report
On one side we have the PlayStore and Apple App store. They are automated computer systems. Thousands of publishers add games to the database themselves and very little human interaction is required.
On the other side we have MobyGames. Every entry has to be done manually. The new wizard helps, but it is still a very labour intensive work.
Which leads me to this conclusion: we will never have enough manpower to add even a fraction of the Android and iOS games out there.
The only way to rectify this, is to use a automated computer system as well.
Now there are two ways to do this.
1) Put time and effort into expanding the available wizard to run as a spider and leech the information from the App stores.
2) Partner with another site that already has a working spider like appanniecom or appshopper.com. Which means to put time and effort into writing a synchronization tool for their database.
What is your opinion?
Best regards, Thomas
Harmony♡ (21848) on 8/28/2017 4:21 PM · Permalink · Report
I agree that there are absolutely improvements that can be made to the scraping process. In particular, automatic tech specs would be a very welcome addition. However, no automated system is flawless, and I can see things like miscategorized apps, duplicate games, etc. tripping us up and being more of a pain to deal with than the original issue. Plus, we do have description guidelines that not all ad blurbs adhere to, and it's been brought up several times in the approvers forum that people don't really bother adding handwritten descriptions to extant game entries.
I suppose approvers could screen these automated entries, and maybe ones that need a better description could be added to some sort of special bounty area of the site, but I have no idea if something like that is feasible on the coding side of things. Regardless, there needs to be some form of manual work done in order to maintain the quality of the database.
twitek (13973) on 8/28/2017 4:37 PM · Permalink · Report
I agree that automated entries will not be perfect. But as you said: some kind of approving mechanism could be created for these entries in particular.
Concerning the description we might have a problem with thousands of these entries. Many games are not available anymore and even from the screenshots and ad blurb you cannot always tell what the game is like. This results in a lot of guess work, or games that will never be in the MobyGames library.
Considering the coding side of things: no matter how much work it will be, it will not be nearly as much work as adding 755000 entries to the database. Considering there are currently around 113000 entries in the database TOTAL (according to MobyProgress).
Harmony♡ (21848) on 8/28/2017 7:46 PM · Permalink · Report
I see your point about the descriptions, and I personally agree that a descriptionless entry is better than none at all, but I think I'm in the minority of approvers who feels that way. Though let's not forget the value of resources like the Wayback Machine when researching games that no longer exist. It can take a while and may not turn up much, but I've usually been able to scrape together enough info to write an adequate description. Personally, even if descriptionless entries ever become a thing, I'd like them to be an absolute last resort once searches and archive trawling have already been tried.
I guess my other main concern is making sure we don't accidentally duplicate entries in the process. Since our standards call for titles to be displayed in Latin script, and for them to be in English whenever possible, some countries' app stores could be problematic when we're trying to weed out duplicates. That's something we'd need to be mindful of.
twitek (13973) on 8/29/2017 12:07 PM · Permalink · Report
Duplicates with localized versions should not be a problem. They have the same ID at least in the Apple App Store. And the approving step should be like adding a new game manually. The game title will be searched in the database and if found you get the question to rename it etc.
Non-latin entries are numerous in asian App stores. You would need an approver who can speak the local language. So we should start with the US App Store first.
Simon Carless (1834) on 8/30/2017 4:33 AM · Permalink · Report
We are in the process of improving the entry helper so it's much smoother still to use - it should all fit on one page and not be multi-step.
At that point it should be a much easier job to add individual items. You still need to add basic genres and also match the correct developer/publisher, so I think we have a preference that this be done manually. We would like it to take less time, though.
I don't think we will go to full auto-scraping because we are concerned about clones/shovelware overwhelming the DB. Although we are too selective right now, becoming incredibly nonselective may - for example - make search fairly unusable due to the amount of games in the database.
So I think it's likely we will be using entry helper for the next couple of years at least as we get Phoenix up and running. But like I said, easier still to add game & videos/screens for it...
twitek (13973) on 8/30/2017 4:40 PM · Permalink · Report
So soon™, it will be easier to add something, okay.
"You/we" have a preference to add several thing manually, which is totally feasible for most systems. But remember this is about a combined 1.5 Million(!!) entries. Manually you will only be able to add some random or popular ones. It will need more than a century to get to even 10%.
So what does Mobygames want? Have a complete database of all Android/iOS games? Or a very tiny selection of random games, that doesn't overwhelm the DB and is carefully curated?
Back to my proposal. We should stop adding Android and iOS games. It's useless to have (currently) 0.2% of those games in the database, because then nobody is using it because there are better alternatives like AppAnnie and others.
P.S. What's Phoenix?
Cavalary (11445) on 8/30/2017 6:11 PM · Permalink · Report
Phoenix is the new contribution interface, used for scraping and videos, and which all contributions will eventually be switched to I guess. Never touched it myself.
Other than that, hey, always all for not accepting anything bar PC, Mac and "dead" platforms (other than for the homebrew scene, of course) myself, and that goes even more for staying out of mobile games altogether. But guess that doesn't go with the mission statement of being the ultimate games database instead of, say, ultimate gaming history + PC gaming database. I'd much prefer the latter, but hey... Past that though, also keep saying the games submitted should be those that contributors care for enough to do proper submissions for, so you'll get no complaints from me for 0.2% carefully curated entries. The complaints are about those that aren't carefully submitted and verified.
Simon Carless (1834) on 8/31/2017 12:30 PM · Permalink · Report
We're OK with having a curated selection of iOS/Android (and PC) games, yes, as opposed to everything. We'd prefer to have absolutely everything but we don't currently have the technology to do that and not mess up the database more than we would like. (There would be no genres and likely inaccurate developer/publisher matching.)
So we'll continue the way we are. But thank you for your suggestion.
Michael Cassidy (21289) on 9/2/2017 5:36 AM · Permalink · Report
There are already games that are long delisted which would make a scraper impossible to use for those, I think. For example, there's an iPhone game called "Monkey Butt" (tee hee) from 2010. There's plenty of evidence that it exists (entries on other game documentation sites, app icons, descriptions, screenshots and even gameplay videos floating around) but I've never seen its iTunes page because it hasn't been available for a long time.