Forums > News > Moby v2023.03.08 Release
MobyReed (325) on 3/9/2023 2:12 AM · Permalink · Report
- Monthly contributor leaderboards now display top 100
- Contributor now displayed with description revision history
- Allow indexing of people without existing credits
- Only factor in one review per source for overall critic score
- Better image caching
- Banner messages now rotate
- Direct messages fix
- Don’t rank games with too few ratings
- Don’t require groups for WIP credit submissions
- Fix Frequent Collaborator values
- Fix forum paging
- Fix adding new product codes
- Fix YouTube URL handling in credit submission tool
- Fix ordering of credit submission images
- Fix incorrect developer exact matching
- Fix adding/editing attributes
- Fix forum message redirect
- Various fixes, improvements for approval tools
- Misc. other fixes, tweaks, optimization
Vereina (785) on 3/9/2023 6:24 AM · Permalink · Report
I guess I'll repeat my question here.
So I've waited for 48 hours since my last contribution approval, and my yearly contribution points finally went past 50+, effectively removing the pagination limit. But when I went to check it out, I've noticed that no matter how you set filters in the browser, you won't be able to see past 10000 (ten thousand entries), or 200 pages (if you have it set for 50 items per page). The counter down below says "out of 10.000+ entries". For example, the Adventure games list (no other filters) ends when the letter S begins, which is clearly not exhaustive list.
So, here's my question. Is it the engine limitation, a bug, or something in-between?
Foxhack (32100) on 3/9/2023 5:15 PM · Permalink · Report
Well manually trying to go to page 201 throws an error, so I'm pretty sure it's an intentional limit, designed to prevent crawlers from scraping the site.
You really should be selecting a platform or adding other filters, though. Selecting Windows, for example, shows 178 pages of games.
Vereina (785) on 3/10/2023 2:27 AM · edited · Permalink · Report
So you are saying that after getting through one barrier being approved contributions, which was clearly explained to be the scrapper protection, we are having here another one, in case a scrapper is clever enough to mimick an active user. That rather not to be the case, because it would be overly excessive, plus it hampers our abilities as a user to utilize the site. What I posted up was just one example, but there is a ton of situstions where set filters would produce more than ten thousand entries. I would have thought that half of the idea would be to let us research what falls under those criteria we set, not to tell us "sorry, the list is too big to be displayed fully"
It is the same like if WoS or Scopus or Google Scholar would not let me see past the fifty most recent articles published on fibre lasers, even if I have to make a reference to an article from 1974 like I did this January. If I am trying to access that database as a complete stranger, I'll be hampered and my search options would be severely limited. But if I am recognized as a registered user, and my affiliation is a pass, then after setting filter for the "laser" keyword I'll see nearly half a million publications since the early 1930es when the idea that a solid state body could actually amplify the radiation instead of absorbing it was voiced for the first time. There won't be a "the list is too long" situation, if you're following my drift.
MobyReed (325) on 3/10/2023 2:15 PM · Permalink · Report
Crawlers are getting blocked before that, unless they are using a contributor account. :)
10,000 is the default limit in ElasticSearch (that's the underlying search engine). There are ways to get around that, but we haven't gotten to it yet and will have to evaluate the tradeoffs in doing so.
But yeah, regardless, you'll want to use other filters and not scroll hundreds of pages.
Vereina (785) on 3/10/2023 7:11 PM · edited · Permalink · Report
I know that in the end I'll set more filters eventually, but what if there's still a lot to look at, or I plain don't know how else to narrow my search and will just say "let's go over to the letter Z and see there" and I'm unable to do so, or what if someone does, actually, want to scroll through hundreds of pages?
Look, I did say before that I might be more understanding than most why it was necessary to move onto a new backend for the site, but after poking around thoroughly just now, I was unable to see any kind of a detailed information breakdown in the [url=https://www.mobygames.com/stats/]stats section[/url], just the grand total amounts. The old engine did allow to see the complete set, however limping it was. New engine might be good and better than old in many ways, but right now it looks like a situation when an inspection comes over to a bank, and they say that they're not even sure how many client accounts they actually have, but it's pretty certain that it is more than ten thousand entries, and the first 10000 will be presented when the inspection decides whether they want to see accounts of the common people, or megacorporations.
Okay, that might have been not the most accurate comparison, but you got the big idea. I know I am nagging right now, but as a researcher and a programmer I was trained to treat as many problems as I can using the "if you only had to use a sword only once in your entire life, it was still worth wearing it every morning since you learnt to walk." approach.
My original questions and inquieres were born from the fact that one of my students couldn't aggregate the statistic he wanted to because of the site migration... And Moby Games did turn into number one database on the gaming, I daresay. They also say that if you are bullshitting something, you gotta offer a solution, and I do have a few ideas how I'd handle that specific problem, but I'm unsure that you want to hear that, so I'll keep it to myself unless asked otherwise. :P
P.S. I ain't a crawler LOL
MobyReed (325) on 3/11/2023 4:28 PM · Permalink · Report
I was unable to see any kind of a detailed information breakdown in the stats section, just the grand total amounts. The old engine did allow to see the complete set, however limping it was.
That's a different thing. The search engine is just one component of the system, and it's different from the underlying database, etc.
The stats page will get filled out, that's still on the TODO list. Soon. :)
My original questions and inquieres were born from the fact that one of my students couldn't aggregate the statistic he wanted
What stat?
Vereina (785) on 3/12/2023 9:08 AM · Permalink · Report
"That's a different thing. The search engine is just one component of the system, and it's different from the underlying database, etc.
The stats page will get filled out, that's still on the TODO list. Soon. :)"
I know that it's a different component, yeah. I just worded it lame. I meant that when you would set a filtre in the old version, you'd see the total numbers without any 10k cutoff, even if the search engine was, as you say, barely breathing.
"What stat?"
Urgh... Been a few weeks LOL
I think he wanted to give a report on the evolution of the 3D-modelling in the video games over the years, trying to figure out how the trends were changing, over the time and across the platforms, and what they could possibly be in a few years from now, and he decided to take the action games as the foundation, for obvious reasons. So what he wanted to start with was the total amount of the action games ever released, no matter where, and that's where he bumped into the pagination, and I did too after I went to check his words out. It worked only with DOS, with its 2500 something entries, and smaller platforms like Win 3.x, Amiga etc. Just setting your garden variety Windows as a platform yields also 10k cut off.
In the end, he had to look up somewhere an article about 3D evolution eras, and used strict year and platform filtering here on MG to get calculable results, but complained in the end that the list seems to be not entirely accurate, and that most of the platforms (beside the most popular like DOS, Windows, PS, XBOX), listed on MG hardly have anything but a cursory mention in the non-English sources, making this place "seemingly the most reliable database". His English is good enough to be able to figure out how to operate MG, but bad enough to read proper articles.
shrugs
Gareth Williams on 3/9/2023 10:50 PM · edited · Permalink · Report
Bring screenshot contribution dates back and the screenshots and users you deleted
Infernos (44329) on 3/10/2023 12:15 PM · Permalink · Report
1) Is GIF format no longer supported? For example, this image originally looked like this. For promos I think it's important to keep GIF image type support, as images in that format do appear in the occasional Fansite/Web/Press Kit, especially animated GIFs.
2) For future you also need to start considering allowing image formats that support HDR like AVIF for example.
3) Browse->Games->then sort them alphabetical; it doesn't ignore articles: "A", "An" and "The". So for example, "The Lawnmower Man" appears at T section and not at L; "A Dinosaur's Tale" at A and not at D. Then there's a bunch of games that start with lower case letters and appear after Z currently. For example, "if", "pac32K", "ranTrainer", "tEMPtAtION".
4) So Spellings now appear in the old place where Alt titles used to be and AKAs under the main title. Good, I like that. However you need to be on the look out where the description is different, so for example instead of saying "Japanese spelling" it says "Japanese Kanji title" like here - https://www.mobygames.com/game/19522/magical-tree/ as those still appear as AKAs no Spellings.
5) What's going with this one? https://www.mobygames.com/game/182692/valna/ On Overview it says "Released: May 19th, 1988" but in Releases it says "1989 Release". BTW the correct release date is May 19, 1989 - https://www.gamepres.org/pc88/library/1989/1989_5.htm#valna
joicrawu (80) on 3/10/2023 12:30 PM · Permalink · Report
Yes, true, the alphabetical listings happen to include most games (and possibly companies?) sorted including articles such as A, An, The. However some aren't sorted that way so it's a bit messy. I did mention it and I think they said they'd work on it. I've also noticed that lower caps come after Z, which I guess is intentional as there are very few starting with lower caps. Not sure if they were ordered that way on the old site (but the A, An, The certainly weren't, they were ordered based on the main "word" let's say. To be honest I wouldn't mind one or the other, but there's a lack of consistency.
MobyReed (325) on 3/10/2023 2:32 PM · edited · Permalink · Report
1) Thanks, good catch.
2) We just upgraded to .webp! :) But yeah, that would be good, but it needs to get to 90%+ coverage https://caniuse.com/avif
3) On the TODO list to look at that. Definitely lowercase should not be getting sorted like that.
4) Yeah, we're going to rework how AKAs are handled and have pre-defined categories instead of custom decriptions. That's way too fragile.
5) Good question. We'll look into that and fix the date.
Thanks.
Vereina (785) on 3/10/2023 7:16 PM · edited · Permalink · Report
*.webp IS finally accepted for submissions, my last few promos from the Google Play passed with the flying colours. Gotta say that before the backend upgrade, I would notice that the file is in the webp format only when trying to submit it, and I had no problems opening and viewing it whatsoever both on my PC and on my phone, so I guess it's quite widespread by now for the default viewers and editors to handle it.
MobyReed (325) on 3/11/2023 3:59 PM · edited · Permalink · Report
Yeah, not forcing it, you can still use other formats. But I get why Google is pushing it, in the browser context at least, it's objectively better than JPG/PNG/GIF.
Although as you note in your other post, it looks like JPEG XL would've been better still. :/