Mad Harlequin wrote:As a concept, this sounds like it could be a good idea, but what would we do to combat unwarranted ratings of either extreme? How would we determine whether a rating is fair? (For example, rating someone's speed based on a specific deadline set by two parties is fair, but I'm not so sure about expectations, which could potentially be unreasonable.)
My first thought is that we should have a very clear set of rules, stating *when* you should give low, medium or high ratings. For example, if you didn't discuss a deadline, you're not allowed to give a low rating if they take longer than you expected, but I think you can rate low if they take 6+ months on whatever you commissioned, deadline or not (I should know, I take longer than that sometimes because of my poor time management and I'm okay being rated as slow because of it! )
I'd be interested in having one place where I could see available artists/writers/programmers who have proven they can handle the workload that a VN requires, it'd make things a lot easier when it comes to hiring.
I don't think we could have something like that over here, unless we made an external site for that... and that would be too much of a hassle :/
Ideally, every person's "hire me" thread would have their reviews and the list of their completed and released games very visible in their post, so you can see them right away.
Sailerius wrote:Rating systems have the problem that their reliability is a function of the sample size. When a service only receives a small number of reviews (as is likely to happen in as small a community as this), then the reviews are going to necessarily be unreliable. For instance, if you're browsing Yelp and you see a business has only one review and it's 5/5, you don't know if that business is really good or if an employee or friend of the owner made that review. On the other hand, if it has 20 reviews with a 5/5 average, you can be reasonably sure it's a pretty good place (or at least has enough friends to keep giving it good reviews).
You have an even darker form of the problem, too: people encouraging friends to leave poor reviews of competition. In some places where products or services are reviewed, you often see consumers holding a rating hostage to extort favorable treatment. While I don't think that's likely to happen here, it's another indication of how unreliable a system like this is and how fragile it is with a small sample size.
Ultimately, I don't think it's a good idea because I don't think it's actually going to provide anyone with useful information.
Well, we are a small community, but we still have people complaining about people they have commissioned within the community abandoning them or taking forever, etc. This is a proposed solution for this community, at least!
Now, since this would be based off the clients leaving post in the commissioned person's thread and the commissioned person keeping their score/whatever updated instead of an automated system where you just fill a little survey and that's it, the reviews would require the person to have an account old enough *(ymmv) and the project the assets (music, art, code, whatever) were commissioned for to exist prior the review date.
annpan wrote:This system seems like an nice add-on, but of course it has it's pro and cons. If we do such a system, it must be moderator accordingly. I'm very cynical when it comes to these things on these sites. We don't know if both sides will even up kept this idea. Faking ratings or not posting them up their thread (bias or not). Honestly, I feel a little uneasy about this rating system as I experienced both sides of loss and completion to commissioning someone. So I can say, if a customer is happy/angry, feedback will be posted in the commissioner's thread. Such reviews like these will definitely add more anxiety to the commissioners with the time, quality, communication. Seems very disciplinary to me. I like the current LSF recruitment as it is, being built on trust instead of ratings.
Yes, I agree we should have a moderator able to keep things civil and clean. As I mentioned above, there should ideally be a monitoring of account age and projects the assets being commissioned for to exist before the review, this way we avoid people creating accounts just to post fake reviews.
I think an artist (coder, musician, writer, whatever) should always be okay with public review if they offer their services online. Now, they can choose not to display a "score", but every client should be able to recommend somebody they had a pleasant experience with, and to warn other potential hirers of issues.
moogsparfait wrote:My favorite reviewing sites use a 1-5 system that requires an actual review but also asks people to rate the review as helpful. The helpful rating on the review really cuts down on shoddy reviews. Amazon and Newegg do this. I don't think stats are particularly needed.
I actually like that feature, though I don't think how we could work something like that with this proposed system, but hey, we're here to discuss ideas
SeventhStranger wrote:Would simply having a "recommend this artist" type feature be enough? There would be no negative reviews per say, only people willing to recommend someone that did good work.
Then you could avoid people using a rating system to vent or having one low review ruining an artist's chances of working again. It would just be a sort of thing that says "3 LSF users recommend this artist".
I suppose you could also add in a "not recommended" if you really wanted to, (then it would be "3 of 4 LSF clients recommend this artist") but then you're likely to run into the same problem.
Yes! I like this idea!
People could leave a comment like they do sometimes on a 'hire me' thread and at the end a "reccomended", so the person could put it up on their initial post (along with the list of completed games they have participated in! )