Commission review system?

A place to discuss things that aren't specific to any one creator or game.
Forum rules
Ren'Py specific questions should be posted in the Ren'Py Questions and Annoucements forum, not here.
Message
Author
User avatar
Deji
Cheer Idol; Not Great at Secret Identities
Posts: 1592
Joined: Sat Oct 20, 2007 7:38 pm
Projects: http://bit.ly/2lieZsA
Organization: Sakevisual, Apple Cider, Mystery Parfait
Tumblr: DejiNyucu
Deviantart: DejiNyucu
Location: Chile
Contact:

Commission review system?

#1 Post by Deji »

Hey everybody!
Long time no see ;)

I've been lurking around lately and noticed a lot of people bummed about the reliability of the people they hire and the need of trusting other users reviews to see if a commissioned person (artist or otherwise) is safe to commission.

So... how about we discuss a review system that's easy for everybody? :)

I personally have some ideas, myself :P

On-thread reviews and star system.

This system would need 2 things:
  • The person offering commissions to have a 'for hire' thread on the recruitment section.
  • The client having a LSF account and be willing to leave a public short review.
So each time a client gets a commissioned piece (art, code, writing, GUI, music, etc), they comment on the thread giving a 1 to 5 star score to 3 categories: Quality, Speed and Communication, along with the kind of thing they commissioned and a short commentary if they feel like it.
  • Quality:
    Rate low if the quality is bad and lower than what they advertised, rate medium if it's acceptable and within what was advertised, and rate high if it's good or very good and within or beyond advertised.
  • Speed:
    Rate low if they take way longer than what you asked for or expected, rate medium if they work at an acceptable speed or are late within reason compared to what you discussed, rate high if they're fast and/or deliver in time or before time.
  • Communication:
    Rate low if they disappear on you, rate medium if they keep an acceptable level of communication during the process, rate high if they are friendly and communicate often (regardless of speed or quality)
So an example of a review would be...
Commissioned: Character sprites for GenericMoeGame
Quality: ****
Speed: **
Communication: *****

The artist did a really god job on my sprites and I'm very happy, they took their sweet time (and would have liked if they did things faster...), but they were always letting me know if they were busy or late, so I never had to worry about them ditching me! Highly recommended, as long as you don't have a tight deadline!
And the artist, on their top post would have to display their average rating, for example:
Hi! I'm open to commissions for a short period of time!

LSF trust: ★★★★ (3 reviews)

I'm quite fast and I'm super friendly and available whenever you have inquiries about your work getting done!
----
(the rest of their thread with examples, etc)
And to add to that, if the artist/coder/musician/etc is willing to, they could link to an external file (like in google docs) wit all their ratings for easy access, for example a google docs spreadsheet like this:
Screen Shot 2015-07-15 at 11.31.25 AM.PNG
Now, the obvious downside of a system like this is that newcomers would have no rating (score, trust, whatever) and some people may not be up to hire them, but I think that still happens anyway when somebody comes saying they don't have experience in VNs but would love to gather some?

So... let's discuss this! :)
Image
Tumblr | Twitter
Forever busy :')
When drawing something, anything, USE REFERENCES!! Use your Google-fu!
Don't trust your memory, and don't blindly trust what others teach you either.
Research, observation, analysis, experimentation and practice are the key! (:

User avatar
Mad Harlequin
Eileen-Class Veteran
Posts: 1068
Joined: Sun Jul 14, 2013 12:55 am
Projects: Emma: A Lady's Maid (editor)
IRC Nick: MadHarlequin
Location: Gotham City
Contact:

Re: Commission review system?

#2 Post by Mad Harlequin »

As a concept, this sounds like it could be a good idea, but what would we do to combat unwarranted ratings of either extreme? How would we determine whether a rating is fair? (For example, rating someone's speed based on a specific deadline set by two parties is fair, but I'm not so sure about expectations, which could potentially be unreasonable.)
Last edited by Mad Harlequin on Wed Jul 15, 2015 11:29 am, edited 2 times in total.
I'm an aspiring writer and voice talent with a passion for literature and an unhealthy attachment to video games. I am also a seasoned typo-sniper. Inquiries are encouraged. Friendly chats are welcome.
"Always do what is right. It will gratify half of mankind and astound the other."
— Mark Twain

User avatar
netravelr
Miko-Class Veteran
Posts: 504
Joined: Thu Jan 28, 2010 2:31 am
Completed: Culina: Hands in the Kitchen, Culina: The Spirit of Cooking, Saving Zoey
Projects: Love at the Laundromat
Organization: Lakeview Interactive
Deviantart: netravelr
Location: USA
Contact:

Re: Commission review system?

#3 Post by netravelr »

Well, I know there are people that do post on people's hire me page, but it's mostly friends of the person or good experiences. I haven't really seen any bad reviews in the past. I think there was a post about doing something like this on a separate site awhile back, but I forget what happened to it.

I'd be interested in having one place where I could see available artists/writers/programmers who have proven they can handle the workload that a VN requires, it'd make things a lot easier when it comes to hiring.
Image
Technical Designer/Programmer
Game Design Portfolio - Project updates on my Twitter
Experienced in: C/C++/C#, Python, Unreal, Unity, and Flash
_________________
"Space can be very lonely. The greatest adventure is having someone share it with you."

Sailerius
Regular
Posts: 72
Joined: Wed Jul 15, 2009 4:28 pm
Completed: Vacant Sky, The Vestibule, Ars Harmonia
Organization: Project BC
Contact:

Re: Commission review system?

#4 Post by Sailerius »

Rating systems have the problem that their reliability is a function of the sample size. When a service only receives a small number of reviews (as is likely to happen in as small a community as this), then the reviews are going to necessarily be unreliable. For instance, if you're browsing Yelp and you see a business has only one review and it's 5/5, you don't know if that business is really good or if an employee or friend of the owner made that review. On the other hand, if it has 20 reviews with a 5/5 average, you can be reasonably sure it's a pretty good place (or at least has enough friends to keep giving it good reviews).

You have an even darker form of the problem, too: people encouraging friends to leave poor reviews of competition. In some places where products or services are reviewed, you often see consumers holding a rating hostage to extort favorable treatment. While I don't think that's likely to happen here, it's another indication of how unreliable a system like this is and how fragile it is with a small sample size.

Ultimately, I don't think it's a good idea because I don't think it's actually going to provide anyone with useful information.

User avatar
annpan
Regular
Posts: 96
Joined: Wed Feb 04, 2015 5:14 am
Projects: Death Pledge -Wish of Blood-
Organization: Broken Wisteria or Qian.
Tumblr: waterlilyvn
Contact:

Re: Commission review system?

#5 Post by annpan »

This system seems like an nice add-on, but of course it has it's pro and cons. If we do such a system, it must be moderator accordingly. I'm very cynical when it comes to these things on these sites. We don't know if both sides will even up kept this idea. Faking ratings or not posting them up their thread (bias or not). Honestly, I feel a little uneasy about this rating system as I experienced both sides of loss and completion to commissioning someone. So I can say, if a customer is happy/angry, feedback will be posted in the commissioner's thread. Such reviews like these will definitely add more anxiety to the commissioners with the time, quality, communication. Seems very disciplinary to me. I like the current LSF recruitment as it is, being built on trust instead of ratings.

User avatar
Mad Harlequin
Eileen-Class Veteran
Posts: 1068
Joined: Sun Jul 14, 2013 12:55 am
Projects: Emma: A Lady's Maid (editor)
IRC Nick: MadHarlequin
Location: Gotham City
Contact:

Re: Commission review system?

#6 Post by Mad Harlequin »

Sailerius wrote:Rating systems have the problem that their reliability is a function of the sample size. When a service only receives a small number of reviews (as is likely to happen in as small a community as this), then the reviews are going to necessarily be unreliable. For instance, if you're browsing Yelp and you see a business has only one review and it's 5/5, you don't know if that business is really good or if an employee or friend of the owner made that review. On the other hand, if it has 20 reviews with a 5/5 average, you can be reasonably sure it's a pretty good place (or at least has enough friends to keep giving it good reviews).
Yes, exactly. If the active community were many, many times larger, we would be able to observe real trends, and an individual user could rebound from an unfavorable review. Since the number of truly active, productive users is probably in the low hundreds at most, that doesn't make for a reliable data pool.

As things are now, the effect of a single poor rating on somebody's future business could be disastrous.
I'm an aspiring writer and voice talent with a passion for literature and an unhealthy attachment to video games. I am also a seasoned typo-sniper. Inquiries are encouraged. Friendly chats are welcome.
"Always do what is right. It will gratify half of mankind and astound the other."
— Mark Twain

User avatar
moogsparfait
Regular
Posts: 53
Joined: Mon Apr 28, 2014 9:26 pm
Completed: Danmaku!! (Card Game)
Projects: The Parasol Festival
Organization: Mystery Parfait
Contact:

Re: Commission review system?

#7 Post by moogsparfait »

My favorite reviewing sites use a 1-5 system that requires an actual review but also asks people to rate the review as helpful. The helpful rating on the review really cuts down on shoddy reviews. Amazon and Newegg do this. I don't think stats are particularly needed.

Maybe it just gets talked about a lot, but abandonment seems like a big issue. I can see any review system being used as a venting place or worse, a battle ground for when things go sour.

User avatar
SeventhStranger
Regular
Posts: 43
Joined: Sun Feb 22, 2015 3:38 pm
Location: GA
Contact:

Re: Commission review system?

#8 Post by SeventhStranger »

Would simply having a "recommend this artist" type feature be enough? There would be no negative reviews per say, only people willing to recommend someone that did good work.
Then you could avoid people using a rating system to vent or having one low review ruining an artist's chances of working again. It would just be a sort of thing that says "3 LSF users recommend this artist".
I suppose you could also add in a "not recommended" if you really wanted to, (then it would be "3 of 4 LSF clients recommend this artist") but then you're likely to run into the same problem.

User avatar
Deji
Cheer Idol; Not Great at Secret Identities
Posts: 1592
Joined: Sat Oct 20, 2007 7:38 pm
Projects: http://bit.ly/2lieZsA
Organization: Sakevisual, Apple Cider, Mystery Parfait
Tumblr: DejiNyucu
Deviantart: DejiNyucu
Location: Chile
Contact:

Re: Commission review system?

#9 Post by Deji »

Mad Harlequin wrote:As a concept, this sounds like it could be a good idea, but what would we do to combat unwarranted ratings of either extreme? How would we determine whether a rating is fair? (For example, rating someone's speed based on a specific deadline set by two parties is fair, but I'm not so sure about expectations, which could potentially be unreasonable.)
My first thought is that we should have a very clear set of rules, stating *when* you should give low, medium or high ratings. For example, if you didn't discuss a deadline, you're not allowed to give a low rating if they take longer than you expected, but I think you can rate low if they take 6+ months on whatever you commissioned, deadline or not (I should know, I take longer than that sometimes because of my poor time management and I'm okay being rated as slow because of it! )
netravelr wrote: I'd be interested in having one place where I could see available artists/writers/programmers who have proven they can handle the workload that a VN requires, it'd make things a lot easier when it comes to hiring.
I don't think we could have something like that over here, unless we made an external site for that... and that would be too much of a hassle :/
Ideally, every person's "hire me" thread would have their reviews and the list of their completed and released games very visible in their post, so you can see them right away.
Sailerius wrote:Rating systems have the problem that their reliability is a function of the sample size. When a service only receives a small number of reviews (as is likely to happen in as small a community as this), then the reviews are going to necessarily be unreliable. For instance, if you're browsing Yelp and you see a business has only one review and it's 5/5, you don't know if that business is really good or if an employee or friend of the owner made that review. On the other hand, if it has 20 reviews with a 5/5 average, you can be reasonably sure it's a pretty good place (or at least has enough friends to keep giving it good reviews).

You have an even darker form of the problem, too: people encouraging friends to leave poor reviews of competition. In some places where products or services are reviewed, you often see consumers holding a rating hostage to extort favorable treatment. While I don't think that's likely to happen here, it's another indication of how unreliable a system like this is and how fragile it is with a small sample size.

Ultimately, I don't think it's a good idea because I don't think it's actually going to provide anyone with useful information.
Well, we are a small community, but we still have people complaining about people they have commissioned within the community abandoning them or taking forever, etc. This is a proposed solution for this community, at least!
Now, since this would be based off the clients leaving post in the commissioned person's thread and the commissioned person keeping their score/whatever updated instead of an automated system where you just fill a little survey and that's it, the reviews would require the person to have an account old enough *(ymmv) and the project the assets (music, art, code, whatever) were commissioned for to exist prior the review date.
annpan wrote:This system seems like an nice add-on, but of course it has it's pro and cons. If we do such a system, it must be moderator accordingly. I'm very cynical when it comes to these things on these sites. We don't know if both sides will even up kept this idea. Faking ratings or not posting them up their thread (bias or not). Honestly, I feel a little uneasy about this rating system as I experienced both sides of loss and completion to commissioning someone. So I can say, if a customer is happy/angry, feedback will be posted in the commissioner's thread. Such reviews like these will definitely add more anxiety to the commissioners with the time, quality, communication. Seems very disciplinary to me. I like the current LSF recruitment as it is, being built on trust instead of ratings.
Yes, I agree we should have a moderator able to keep things civil and clean. As I mentioned above, there should ideally be a monitoring of account age and projects the assets being commissioned for to exist before the review, this way we avoid people creating accounts just to post fake reviews.
I think an artist (coder, musician, writer, whatever) should always be okay with public review if they offer their services online. Now, they can choose not to display a "score", but every client should be able to recommend somebody they had a pleasant experience with, and to warn other potential hirers of issues.
moogsparfait wrote:My favorite reviewing sites use a 1-5 system that requires an actual review but also asks people to rate the review as helpful. The helpful rating on the review really cuts down on shoddy reviews. Amazon and Newegg do this. I don't think stats are particularly needed.
I actually like that feature, though I don't think how we could work something like that with this proposed system, but hey, we're here to discuss ideas :)
SeventhStranger wrote:Would simply having a "recommend this artist" type feature be enough? There would be no negative reviews per say, only people willing to recommend someone that did good work.
Then you could avoid people using a rating system to vent or having one low review ruining an artist's chances of working again. It would just be a sort of thing that says "3 LSF users recommend this artist".
I suppose you could also add in a "not recommended" if you really wanted to, (then it would be "3 of 4 LSF clients recommend this artist") but then you're likely to run into the same problem.
Yes! I like this idea! :)
People could leave a comment like they do sometimes on a 'hire me' thread and at the end a "reccomended", so the person could put it up on their initial post (along with the list of completed games they have participated in! )
Image
Tumblr | Twitter
Forever busy :')
When drawing something, anything, USE REFERENCES!! Use your Google-fu!
Don't trust your memory, and don't blindly trust what others teach you either.
Research, observation, analysis, experimentation and practice are the key! (:

User avatar
Deji
Cheer Idol; Not Great at Secret Identities
Posts: 1592
Joined: Sat Oct 20, 2007 7:38 pm
Projects: http://bit.ly/2lieZsA
Organization: Sakevisual, Apple Cider, Mystery Parfait
Tumblr: DejiNyucu
Deviantart: DejiNyucu
Location: Chile
Contact:

Re: Commission review system?

#10 Post by Deji »

I was randomly brainstorming about the "recommended" thing and the easiness to find well reviewed artists, musicians, etc.

I know it would be a hassle to maintain an external website for this (though it could be possible, categorising commissionable people by type, number of reviews, number of completed projects, and also viewing the commissioned people per project, or how many recommendations a certain person has made...), but we could have a little "recommended by the community" "badge" for people that decide to use it and has had a number* of positive reviews? So people can display it proudly on their thread, along with the list of the completed projects they've worked with.
I had fun doing this >>
I had fun doing this >>
Screen Shot 2015-07-15 at 1.52.18 PM.PNG (50.22 KiB) Viewed 2487 times
We'd still encourage people to leave reviews on the person's page, and I guess a mod could "revoke" the badge in case they no longer participate or get a number of bad experiences with the community, and we would not have a "competition" kind of sorts with who has the highest score, etc, and it'd remove some of the anxiety that was mentioned earlier?

How about that instead? :)
Image
Tumblr | Twitter
Forever busy :')
When drawing something, anything, USE REFERENCES!! Use your Google-fu!
Don't trust your memory, and don't blindly trust what others teach you either.
Research, observation, analysis, experimentation and practice are the key! (:

User avatar
firecat
Miko-Class Veteran
Posts: 540
Joined: Sat Oct 25, 2014 6:20 pm
Completed: The Unknowns Saga series
Projects: The Unknown Saga series
Tumblr: bigattck
Deviantart: bigattck
Skype: bigattck firecat
Soundcloud: bigattck-firecat
Contact:

Re: Commission review system?

#11 Post by firecat »

Deji wrote:I was randomly brainstorming about the "recommended" thing and the easiness to find well reviewed artists, musicians, etc.

I know it would be a hassle to maintain an external website for this (though it could be possible, categorising commissionable people by type, number of reviews, number of completed projects, and also viewing the commissioned people per project, or how many recommendations a certain person has made...), but we could have a little "recommended by the community" "badge" for people that decide to use it and has had a number* of positive reviews? So people can display it proudly on their thread, along with the list of the completed projects they've worked with.
Screen Shot 2015-07-15 at 1.52.18 PM.PNG
We'd still encourage people to leave reviews on the person's page, and I guess a mod could "revoke" the badge in case they no longer participate or get a number of bad experiences with the community, and we would not have a "competition" kind of sorts with who has the highest score, etc, and it'd remove some of the anxiety that was mentioned earlier?

How about that instead? :)
the only downside i see is that someone could make a fake account and get away with it, while the real account stays fine. there needs to be a way to not cheat and let people do it naturally like the review system, the review system has no cheats and people do it naturally.
Image


Image


special thanks to nantoka.main.jp and iichan_lolbot

User avatar
Anne
Veteran
Posts: 321
Joined: Mon Apr 01, 2013 8:19 pm
Completed: Fae
Contact:

Re: Commission review system?

#12 Post by Anne »

I think it'd be nice if people specified the amount of work done in their reviews (one thing is being fast with 3 character sketches and quite another with 40+ CGs)

User avatar
Laiska
Veteran
Posts: 323
Joined: Sat Jan 11, 2014 1:14 am
Completed: Queen At Arms, Cerulean, The Shadows That Run Alongside Our Car, Vicarwissen
Projects: Caramel Mokaccino
Tumblr: minesweeperaddict
Deviantart: koyoba
Contact:

Re: Commission review system?

#13 Post by Laiska »

Unfortunately I can't really see this being a sustainable feature of the forum, especially a recommended badge or something that would have to be moderated. Also given that this is a discussion platform in addition to being a business platform, a feature like that could prove confusing to someone only here for the former, especially considering the number of people who are lurkers and not regular members. That is, if I went to a discussion forum and saw someone with a recommended badge on them, I'm not sure I would understand what that meant—does that mean I should trust them as someone who has good opinions?

I guess I'm saying that I don't think it would work out as a forum feature necessarily. However, we should still strongly encourage that clients leave reviews on service provider's threads. That would typically weed out abuse of the system to purposely lower someone's reputation, because in an intimate setting like this, most people aren't going to go out of their way to post in someone's thread just to harass them. Though of course, that does leave the question of where to leave negative reviews.

It would be a hassle, but I sort of think an external platform would be best for this.

User avatar
noeinan
Eileen-Class Veteran
Posts: 1153
Joined: Sun Apr 04, 2010 10:10 pm
Projects: Ren'Py QuickStart, Crimson Rue
Organization: Statistically Unlikely Games
Deviantart: noeinan
Github: noeinan
Location: Washington State, USA
Contact:

Re: Commission review system?

#14 Post by noeinan »

I think this is a great idea! I'm glad someone followed up, since it was discussed in this thread: http://lemmasoft.renai.us/forums/viewto ... =4&t=33105

I really like the "recommended" comments and the badge-- I think a lot of people may not post the star thing because 1. it seems like too much work or 2. they're bad at number estimates or 3. they have anxiety over giving the "wrong" number. But recommendations, with comments, work well.

The badge in the signature seems really cool, but I am curious as to how it would be enforced. (Otherwise anyone could put it there...) Can mods edit member signatures? Would mods even be willing to do that kind of work? I really want to use badges, they are so cool! But I don't know what mods have to say.

Alternatively, maybe there could be something similar to the "Honest Critique" button, where artist/musicians/etc. could put it in their signature to let others know that they encourage reviews. People with the button could ask for a review from their clients directly, and clients may say yes or no depending. (And clients may feel less nervous about posting, since the button is basically an invitation.)
Anne wrote:I think it'd be nice if people specified the amount of work done in their reviews (one thing is being fast with 3 character sketches and quite another with 40+ CGs)

^Also this. Maybe if this gains traction, and the mods see that it is a system here to stay, we could get a sticky explaining reviewer/reviewee etiquette.
Image

Image
Image

User avatar
Deji
Cheer Idol; Not Great at Secret Identities
Posts: 1592
Joined: Sat Oct 20, 2007 7:38 pm
Projects: http://bit.ly/2lieZsA
Organization: Sakevisual, Apple Cider, Mystery Parfait
Tumblr: DejiNyucu
Deviantart: DejiNyucu
Location: Chile
Contact:

Re: Commission review system?

#15 Post by Deji »

I was thinking of having the badge on the "hire me" thread and not on the signature in any case (and mods can edit that without problems)
Image
Tumblr | Twitter
Forever busy :')
When drawing something, anything, USE REFERENCES!! Use your Google-fu!
Don't trust your memory, and don't blindly trust what others teach you either.
Research, observation, analysis, experimentation and practice are the key! (:

Post Reply

Who is online

Users browsing this forum: piinkpuddiin