At the suggestions of a few of the approvers, I am starting this topic for thought and discussion on what Moby should consider to be valid sources for a "press" site and a MobyRanking. I was not aware of this, but this apparently comes in the middle of a re-defining of what sites are approved and accepted here. I suppose some brief backstory and an example are in order.
Moby's current policies are provided in the MobyRank FAQ. As I see it, the applicable sections are:
"MobyRank is the combination of reviews and rankings from professional critics and reviewers from a number of online, television and print media outlets."
"We make a determination each time a media outlet is contributed for consideration to be included in the MobyRank computation. We are always looking for new sources of well written, thoughtful, accurate reviews and rankings. At this time, the website you contributed did not meet our standards."
Based on this information, which sounds to me like a request for many sources in an attempt to be comprehensive, I submitted the site Just Games Retro. This is a site that I am involved in. The site is currently active, and reviewing "retro" games with a historical perspective. I believe MobyGames to be deficient in the areas this site covers, and many of the reviews I submitted from JGR were the only reviews of any kind for games in these systems (NES, Genesis, etc).
The site was approved by an admin, and 78 reviews from it were approved by varying Moby approvers over the course of a week.
Today, after a private conversation and vote among the approvers, the site was un-approved and all the submitted reviews are in the process of being removed. The few reasons given are that it was deemed that the site was unprofessional, not reputable, and the reviews are lacking in quality. I do not feel that the site somehow "slipped through" the approval process, or than any part of it was unfairly represented - which would justify such a drastic action instead of simply assigning the site a lower "weight." I feel that the site has been judged on criteria not yet published on the Moby, and that the definition of what makes a valid and professional "rank-worthy" site needs to be discussed, agreed upon, and quickly codified.
I am not seeking or interested in having JGR relisted on Moby. I use this site as an example of why this does not need to happen again, and to avoid pointing fingers at other, currently approved sources that are of at least the same quality as JGR. I respectfully ask that you do not offer your opinion for or against the approval/de-approval of the site, unless you want to use it as an example for the topic here. I do not see what happened as the issue itself, merely a symptom. If you have any specific comments, please send them to my mail or PM.
I first strongly suggest that any new sites submitted must be approved by more than one, preferably a cadre, of approvers. If a site can be voted out, then sites need to be voted in to prevent that necessity.
Second, I suggest clearly-defined requirements such as those on GameRankings, seen here. Even if you do not agree with their specifics, the format and requirements are quite clear, and in my opinion, fair. This is the kind of policy I would like to see explicitly laid out.
This topic is to discuss what should constitute an approved site, and what should be the criteria for judging them. Some things to consider - objectivity, backing by a corporation or media publisher, availability of contact information, profanity, spelling, grammar, length of time the site has been online, obvious care and planning in the reviews, etc.
I personally believe that having your own website does not make you a professional, this site should not be a repository for online resumes thinly disguised as independent sites, this site should not be publicity to give people ad banner hits, and that blogs and reviews of the standard GameFAQs or user reviews should not be considered as press sites. However, I also believe that all web reviews should not be discounted. Their benefits are unique (ability to read the entire article instead of only an abstract, linkability, speed of new content). The question is how to judge these sites on concrete requirements, or at least, requirements that move away from the subjective rules currently in the FAQ.
Please discuss. Everyone has an opinion, almost everyone will voice it, and it is up to this site to be the gatekeeper from the start in assigning value to those opinions. Also bear in mind that there are well-written and well-researched voices outside the mainstream media, but whether they should be counted in a MobyRanking alongside paid journalists is in question.
Just for clarification, only admins can approve a source.
As for sources being approved that are a bit sub-par, keep in mind that this is a new part of the site. As such, many sources were added at the beginning that probably shouldn't have been. This happens when adding anything new. Just look at the descriptions on the games added to the site when MobyGames first got started. We are slowly trying to go through and fix those, but it takes time. In the case of sources, it doesn't take that much time to fix... an admin can do it fairly easily.
Anything new to the site is likely to see revisions in how it is handled once we see how things work. Unfortunately, that can mean that people lose points, contributions are removed or changed, etc.
The site was approved by an admin, last week. One week later it was unapproved. Please re-read my message.
The real issue is discussing what these "new" requirements should be and how to write them down.
I did read it and I already know what happened to your site... even before it was removed. You commented on approvers approving sources, so I was clarifying that only admins approve sources.
As far as requirements, approvers and admins include all of that in the wiki and when it's all ironed out, it ends up being put on the site for everyone to read. That takes time, however.
So you're saying that all of this takes place behind the curtain, is an admin issue, and users and contributors should not be a part of that discussion?
he's not saying that at all... he just commented that admins approve sources and that approvers don't
you can discuss all you want about what is required here
(Edited by BurningStickMan (17754), Sep 13, 2006)Re: Defining What Is An Approved "Press" Source
BurningStickMan (17754), Sep 13, 2006
You guys are really splitting hairs here. I said that once, and I got it right the other times.
Also, in the context it was written, it is intended. Approvers voted this site out, an admin should table a future hypothetical site and approvers should vote it in. If you have enough admins to cover the job, then more than one admin should vote on such an approval.
Unless the issue is one of bad timing - that the site was literally okay one week and not the next. I can understand that. But saying *essentially* "we'll figure it out and post it later" seems shady.
It isn't shady to improve something when you find out that one method was not good enough or that the quality used originally wasn't up to the standards for the site. We could make a general statement of what is accepted when we release a new feature, but once we start receiving contributions, it is very probable that we'll find out that it needs to be revised. As such, a final standard isn't usually going to be provided until we know what really should be in it and how it should be worded to take care of all possible situations and contributions.
As far as splitting hairs, read just your post and then my first reply. It was nothing more than a clarification so that you and anyone reading this thread knows how source approval works. Even if you knew that and it was a typo to say that approvers approve sources, others may not know that and it's best to make sure everyone has the correct information when discussing topics such as this.
As far as bringing sites in or removing them, it really does fall under the first paragraph that I wrote in this post... the site was probably fine when we started accepting sources, but as we started seeing what was submitted as sources, we started seeing that the quality of sources needed to be taken into account as MobyRank is for professional reviews and not for blogs or personal websites or anything like that. As such, your site happened to be one that was determined to no longer fit what we were looking for as a professional source. Other sources may also be removed for the same reason. It does not happen immediately, however. We have a lot of sources and they would each need to be examined for quality as well as any other requirements.
As a final note, once MobyRanks have been around for a decent amount of time, rather than only about a month, you are unlikely to see a source added and then removed later. MobyRanks is very new and anything new is likely to get revisions in what is accepted. If you were around for any other new user-contributable features in the past, you would have seen the same thing happen.
Even with the best of intentions and even if you research what you can about possibilities of what people may submit, someone is going to submit something that wasn't considered and that is when a policy change ends up needing to be made. In the past, even point revisions across the board were made and most contributors (including me) lost a lot of points. Sometimes, we just have to do what is best for the site, even if the contributors lose something based on the decision. Long-time contributors generally accept this as they approve of the high quality standards of the site. The newer contributors are often the ones who are upset about it because they haven't been around long enough to really see why we are so concerned about quality here.
Keep in mind that any policy can be discussed and debated if you feel the need. And, it has happened that such discussions have led to changes in policy. It doesn't always happen, but it can. So, if you feel that a policy is not good, feel free to discuss it and it is *possible* that it may get the policy changed. It all depends what is really best for the site.
Ah... Through the magic of archive.org, I think I see the confusion on my part.
Like the vast majority of Moby users, I've lurked here for a long, long time, but never "signed up." I was using it as the information database that you, thankfully, don't have to register for.
But I didn't have any particular loyalty to the site until I signed up to contribute (and not just with JGR). As a result, I think my previous use of the site kind of melted together with GameRankings, Wikipedia, GameFAQs and whatever else I used in the past to answer questions that popped up about games. If one didn't have it, then the other did, and that's about as far as thinking about their differences went.
...soooooo, I thought you've had the press review section for years. I never even questioned that when I looked at the giant list of press sites you have, and the vast number of games that already have press sites. But it turns out that press listings AND MobyRankings have only been around for a few months?
If so, I have a hat to eat.
Yes, it is brand new. We have so many rankings already because it's something quick and easy to add and offers easy points for people. And, because it is new, it's not hard to find rankings that aren't entered yet. We have a backlog of rankings waiting to be approved for that very reason. :)
*sound of hat being consumed* (gulp? crunch? schhloorp?)
I agree with what WildKard said, though I also understand the intent to have some kind of reliability behind the calculation of a MobyRank. You don't want the dumb kid throwing off the grading curve (only a metaphor with no particular sites in mind).
But, as an example, I'd fight from hell to breakfast to keep Just Adventure on, though they are technically an independent site. Course if they really have 1 million hits a month, they're probably doing okay. Having corporations throw money at you is not a guarantee of quality and reliability (though it's often a start).
Yeah, Just Adventure is widely considered to be a very good site and it's not going to be removed for being "independent". Independent is fine. :)
This should probably be better defined as to (some of) the criteria of "quality" required before a source can be approved. I'm mentioned at least once "behind the curtain" that I think the rankings should be open to more sources, not less, and that smaller sites and "personal blogs" shouldn't necessarily be excluded just because they don't happen to be major media and have a marketing department. On the other hand, I do understand the desire to keep informative reviews at the forefront and not fill the rankings with a bunch of ratings that people won't read or consider reliable. So the first step I think is defining some of the basic requirements for a source to be considered for approval.
MobyRank has only been around since August 4, 2006. It's still very much a new feature. I know that people have taken to submissions enthusiastically and as such, some of the queues for rankings are full of hundreds of ratings for all kinds of magazines and websites that we have to look over. It's somewhat overwhelming. Disclaimer: I don't work on the MobyRank submissions. As of yet, I'm not even sure what steps go into approving a Ranking... it's still very confusing to me and luckily, there's enough other contributions being sent to MG that I don't have to worry about it. Maybe in the future, another approver/admin will "train" me :)
In response to this thread I have been "thinking" about revising the policy. Some considerations are
1) number of reviews the source should have
2) how they review. i am thinking we may not consider sources that review after the
fact. ie go back and review games well after the general concensus has been made about them. 3) how they look.
4) length of review. have some minimum requirement of review length.
5) other factors like do they publish contact infromation, phone number or try to hide their identity
not sure how this will pan out but thinking about it
(Edited by BurningStickMan (17754), Sep 14, 2006)Re: Defining What Is An Approved "Press" Source
BurningStickMan (17754), Sep 14, 2006
I like these, though why 2)? Nothing new to add? Does the world really need another Super Mario Brothers review? The fact that it's not a review in the general understood sense - i.e. an investigation into the merits of a product to assist a reader in deciding to buy the product?
#3) Needs more clarification by outlining some factors considered. Suggestions - sectioning, ease of navigation, "production value"
What about considering how many reviewers or contributors the site has?
A structure to their staff with defined contributors/editors/site designers so it is not the work of one or two people. With contact information for all.
I don't think we should limit it based on how old the game is that they are reviewing. Why should we not accept review sites for old games and platforms? And what is wrong with seeing how a game is rated 5 years after it was released? Personally, I think it's interesting to see how the ratings change over time... sometimes for the better, sometimes for the worse. Usually it's better thanks to patches, but it could be a variety of things and I don't know why we'd not want sites that do that.
(Edited by BurningStickMan (17754), Sep 14, 2006)Re: Defining What Is An Approved "Press" Source
BurningStickMan (17754), Sep 14, 2006
To add to what Riamus said, your only source of historical reviews for the systems we're talking about (16-bit era and back) are going to be magazine reviews, for obvious reasons. Now practically, those magazines are going to be sitting in a box rotting, and either the person who has them doesn't know about (or is not willing to contribute to) MG, or someone else who IS willing to take those magazines and enter them in the database is rightfully not going to want to pay collector's prices.
For me personally, I hate abstracts and I love the ability to read full reviews online. Seeing a magazine title and a numerical score, or a paragraph from the beginning of the review and a score, doesn't help me with any game, current or past. I skip over any such entries and go right to the web sources. Being able to read the reasons behind that number is a tremendous help. With that in mind, the ONLY way to get a web review for a game from that period is through a classic-reviewing site.
I also like the historical perspective, and like a review that tells me not to bother with Mega Man when Mega Man 2 does the same things better. I like the nostalgia factor, and seeing what someone else though of, or what their experiences were with, a game that I've played too.
You also can't assume that every current gamer is familiar with past titles or the "consensus." Though people playing games in the 80s probably know about major titles from that time, that won't be true of a new gamer looking back. There are also past titles that have been overlooked. I've had to Wiki a few games I've heard reference to, but am unfamiliar with myself, and have found some very interesting titles that way.
Also, from a personal perspective, I remember going into game stores, looking at the shelves, and wanting to play EVERYTHING. Now I have the money and some time to make that possible, and there are times I often feel like I'm still in that store, looking at a near infinite catalogue, and wondering whether to spend time or money on a certain title. Even if I wasn't involved in a *certain* site, I'd still be looking at older titles I didn't get a chance to play when I was younger (my Amazon and Ebay expenses can speak to that), and I'd still want some kind of review source to help me.
(edit) But I would recommend that some kind of consideration be given to how the site handles the older games. If they are rating games without consideration to capabilities at the time, other games from that period, or just generally not doing their homework, then they shouldn't be considered a valid source. Saying Super Mario Bros doesn't look as good as Super Mario Sunshine is obvious. Saying Grand Theft Auto 3 is a much better realization of what Death Race 2000 was going for is, well, silly.
They should also not talk about how the game plays on an emulator. THAT is best left for user reviews or comments on other sites.
Agreed. We have that problem with MobyScore (user ratings) here. If a game was made in the 80s and has 4 color or 16 color graphics, then the graphics will definitely look bad from today's perspective. However, at the time, the graphics may have been really great. But too many people don't look at it that way and they rate it based on today's games.
Thanks for quietly updating the MobyRank FAQ based based on the discussions. However, could you at least explain your insistence to go with "Reviews should have been published at or close to the release date of the game being reviewed." As I expressed concern about earlier, this would exclude Just Adventure and most reviews for older consoles. Not a concern of this site?
I can see the logic in this as reviews of a game ten years after release are goign to be quite different than a contemporary review.
I have argued about that point already. I hope that will be removed as it really isn't a good idea to exclude later reviews. Professional review sites aren't going to reduce the rating of an old game based on graphics being worse that today's games, so it's not an issue like it is when users here rate an old game with 0 or 1 for graphics just because it's 4/16/256 colors, when it could have been great graphics at the time.
Jeanne informed me she is escalating all my MobyRank submissions because there is some issue with Pelit magazine as a source. Could someone shed some light into this?
As far as Pelit is concerned it is because it requires a log in.