Commission review system?

[trooper6]

I agree with Laiska that reviews would be the best thing.

The only problem is that this community is really, really loathe to say anything critical of anyone. ... Hence why the "Honest Critique" buttons were made...to try and illicit more honest feedback other than "this is so great."

We have positive reviews on this site...but I've yet to see a negative review on a recruitment thread. Which might lead you to think that everybody here is trustworthy. But I'm now hearing that there are artists who take people's money and don't produce any work. Artists who still have recruitment threads...but no one says anything, but in vague terms with no names because people don't want to be mean or cause drama.

I think if someone is great you should say. If someone takes your money and never produces work...that should also be on their recruitment thread. Otherwise, we are enabling that person to do the same thing to someone else in our community. On a less dramatic note. If an artist is very good, but slow...then we should know that too. A commissioner who really wants music from the slow composer will know to hire them way in advance, or for a project that will take a long time...but not for NaNoReNo.

Knowing that people don't actually write honest reviews unless they are totally positive makes be feel unsafe making commissions from the artists here--because I can't know if the artist is actually a problem but no one will say because the culture here is to be very polite. I think the community would be better if we all committed to more honest reviewing.

[Deji]

@trooper6: I completely agree, though I'm not sure how that would work in a community like LSF where everybody wants to be polite and people want to avoid feeling stressed/criticised for their work ):
While I'm in favor of being positive and welcoming to people creating games, I think it's essential to be honest and 100% transparent when you're doing business.

Like I mentioned early, if you're willing to do business on a public place online, you should be able to handle any kind of reviews that come your way. The reason I came up with Quality (relative to hat was advertised), Speed (again sort of relative) and Communication was to encourage rating by reliability and discourage rating people by skill only.
Hell, I'm ridiculously slow and some people may argue that my quality is worth it, while I know some people that are insanely fast, very good with their end product and terrible at communicating.
I think a heads up about that kind of thing would be great for people interested in HIRING others (we're talking about money transactions here, you want your money to go towards the successful completion of your project! For free stuff is a different matter, imo, you can be as guarded as you want with that.)

[infel]

As someone whose going through this now, I will say I wished I had a recommendation about if the person I was about to hire was reliable. Now I'm at a split mind about a review system. It can be a good and bad thing. Maybe what could work is something halfway like the commissioner leaving a recommendation or non recommendation, and giving details about what they thought on the artist, coder, writer etc.

I think having something like this would help commissioners know what to expect or at least give them a heads up whether the person would be good to work with. As someone who might've lost 200 dollars I think this tactic could be helpful if done right.

[Laiska]

The only problem is that this community is really, really loathe to say anything critical of anyone. ... Hence why the "Honest Critique" buttons were made...to try and illicit more honest feedback other than "this is so great."

Creating an "Honest Reviews" button as a companion to this might not be a bad idea. Service providers could link this either in their thread or in their signature, so that commissioners know it's safe to post any feedback they have, even if it's negative feedback. It obviously not something that can be enforced universally, but it would give commissioners extra confidence in hiring someone who chooses to use the button, regardless of the amount of feedback they have so far. (Unless they happen to be foolhardy, it would theoretically mean they are someone who would attempt to do a good job for honest feedback.)

Without an intensive vetting system that would be too hard to enforce in a setting like this, we sort of have to work on an honour system, but encouraging service providers to openly welcome feedback, including criticism, would be an important step in making the process safer.

[trooper6]

The only problem is that this community is really, really loathe to say anything critical of anyone. ... Hence why the "Honest Critique" buttons were made...to try and illicit more honest feedback other than "this is so great."

Creating an "Honest Reviews" button as a companion to this might not be a bad idea. Service providers could link this either in their thread or in their signature, so that commissioners know it's safe to post any feedback they have, even if it's negative feedback. It obviously not something that can be enforced universally, but it would give commissioners extra confidence in hiring someone who chooses to use the button, regardless of the amount of feedback they have so far. (Unless they happen to be foolhardy, it would theoretically mean they are someone who would attempt to do a good job for honest feedback.)

Without an intensive vetting system that would be too hard to enforce in a setting like this, we sort of have to work on an honour system, but encouraging service providers to openly welcome feedback, including criticism, would be an important step in making the process safer.

This difference between "Honest Critique" button and using something like this for commisioners is this: If someone doesn't want honest feedback on their VN...no one is hurt. They don't get the feedback they don't want and no one wastes their time giving it. On the other other hand, if I'm advertising my services, but I'm a serious flake and I say I don't want "Honest Reviews" posted--who is hurt? Those people who never find out I'm a flake until after I take their money and flake on them. I think flakey people will opt out of the "Honest Reviews" system hoping they then won't get any. I think if a person is lacking in Quality/Speed/Communication, people should just say...whether or not the person being commissioned wants that review or not.

I mean, $200 is a lot of money and I'm heartbroken that Infel got taken. I'm also really concerned that I don't know who it was that took Infel's $200 and gave nothing back. What if it is someone I'm thinking about commissioning and I don't know they are the one who did this to Infel? It is really disturbing to me that this information is not being shared in the community.

[noeinan]

I was thinking of having the badge on the "hire me" thread and not on the signature in any case (and mods can edit that without problems)

That makes more sense.

Knowing that people don't actually write honest reviews unless they are totally positive makes be feel unsafe making commissions from the artists here--because I can't know if the artist is actually a problem but no one will say because the culture here is to be very polite. I think the community would be better if we all committed to more honest reviewing.

^This. The current system is a total black box, and you don't know whether it's safe to work with people-- especially giving them your money. You don't know if you're going to be robbed. Not having reviews creates a bad atmosphere for both the client *and* the hired creator. I think it also creates unnecessary barriers to getting work done, because it's really difficult and risky to work with others.

While I'm in favor of being positive and welcoming to people creating games, I think it's essential to be honest and 100% transparent when you're doing business.

Also very yes.

Creating an "Honest Reviews" button as a companion to this might not be a bad idea. Service providers could link this either in their thread or in their signature, so that commissioners know it's safe to post any feedback they have, even if it's negative feedback. It obviously not something that can be enforced universally, but it would give commissioners extra confidence in hiring someone who chooses to use the button, regardless of the amount of feedback they have so far. (Unless they happen to be foolhardy, it would theoretically mean they are someone who would attempt to do a good job for honest feedback.)

Without an intensive vetting system that would be too hard to enforce in a setting like this, we sort of have to work on an honour system, but encouraging service providers to openly welcome feedback, including criticism, would be an important step in making the process safer.

This is exactly what I was trying to say in my first comment, though I may have worded it weird. It allows artists/musicians/whoever to take the first step in letting people know it's okay to post a review. Right now people are worried about being rude, so with an invitation it may help encourage folks to use it. And if the button gets more recognition, more artists/musicians/etc. will start using it and it can help chance the review discouraging culture.

This difference between "Honest Critique" button and using something like this for commisioners is this: If someone doesn't want honest feedback on their VN...no one is hurt.

I agree with this, I don't think the button should mean a commissioner is able to refuse reviews. If you're offering a service for money, you need to be able to take reviews on your work and communication. On the other hand, there doesn't seem to be a way to enforce a review system, so we need to start with encouragement, and cultural change. General members can't delete comments in their own threads, so I don't think we're at risk of commissioners censoring bad reviews. So making reviewers feel more comfortable seems like a good first step.

[trooper6]

I agree with this, I don't think the button should mean a commissioner is able to refuse reviews. If you're offering a service for money, you need to be able to take reviews on your work and communication. On the other hand, there doesn't seem to be a way to enforce a review system, so we need to start with encouragement, and cultural change. General members can't delete comments in their own threads, so I don't think we're at risk of commissioners censoring bad reviews. So making reviewers feel more comfortable seems like a good first step.

That is a good point. Anything that will help move the culture here towards more honesty and transparency is good. It isn't good that our culture of silence results in people losing $200 or having experiences that could be avoided.

[Kuiper]

I'm generally in agreement with many of the sentiments expressed in this thread regarding accountability, but I disagree with the "ratings system" proposed in the OP, or a "badge system" or any sort of attempt to make things look official or formal. In fact, I believe that reviews provide the most value when they are informal and don't have any numbers attached. There are several reasons for this.

The first is that trying to aggregate numerical ratings algorithmically sucks, because to me, the most important thing about a review is the value of the source, which is nigh impossible to quantify. This applies even when you strip away the issue of reviewer reputation. For example, consider the following reviews:

Six months ago, I hired John to design the UI for my first project, and since then, he has been my go-to graphics guy. Be it UI elements, backgrounds, website design, and even designing my team's banner and logo, this guy always comes up with beautiful-looking stuff (just look at his portfolio), and he's always been nothing but courteous and patient, which is a blessing for someone like me who is perpetually indecisive and always asking for last-minute changes. He's usually quick to turn stuff around (bar a couple times when my commission overlapped with his college exam schedule), but he's always quick to respond to emails and up-front about how much work he has on his plate.
Quality: ****
Speed: ****
Communication: *****

great job thank you for the gorgous backgrounds!
Quality: *****
Speed: *****
Communication: *****

All other factors held equal, I think that most of us would look at the first review and mentally give it higher weight, and would accept it as higher praise than the second one despite the fact that numerically it gives a lower score. But if you're looking at things algorithmically, the two reviews have equal weight, and the second one actually looks more glowing as an endorsement when you only look at the stats, since a 5/5/5 is higher than a 4/4/5. If I'm a real human being expecting these reviews to be read by real human beings, then of course I'd rather have more reviews of the former type. But if I have stats being tracked, then I'm more incentivized to seek out the latter type of review. When we measure something, we tend to optimize for it.

One of the fundamental problems of 5 star rating systems is that it always seems that the people who's feedback you value least tend to vote 1 or 5, and the people who provide more nuanced and valuable and use the full ratings scale have their contributions muted as a result. The fundamental issue with a 5-star rating system is that you're actually putting two different kinds of rating systems on the same scale, because some people treat it like a 5-star rating system, but other treat it like a binary system where they vote 1 for things they don't like and 5 for things they do like. You also wind up in awkward positions where you liked someone, had a positive experience, and think that they deserve a 4/5 rating, but you don't want to give them a 4-star rating because their current average is 4.5/5, and your 4-star review, despite being positive, would actually hurt their stats.

Aggregate percentages can also mean different things. Someone with an average 4.2/5 based on 5 ratings could be someone whose ratings were 4, 4, 4, 4, 5, or it could be someone whose ratings were 5, 5, 5, 5, 1. Person A received 100% "positive" reviews, person B received 80% "positive" reviews, but they're both a 4.2/5.

Incidentally, this is why many ratings systems have increasingly gone binary. Back before 2010, Youtube used to have a 5-star rating system, but due to a combination of the factors described above, they switched to a binary "like/dislike" review system, which had the positive effect of putting everyone on the same ratings scale. It means I don't have to make a labored decision over whether something is a 4/5 or a 5/5, and I don't have to worry about messing up someone's stats because I gave a positive (but not perfect) review score. It makes things less like Metacritic and more like Rotten Tomatoes.

Of course, things become even more non-numeric when you factor in things like the reputability of a reviewer. For example, if someone has an endorsement from someone like Papillon (Hanako) or Jack Norton (Winter Wolves), that matters a heck of a lot more to me than an endorsement from someone saying "thanks for volunteering for my NaNoRenO project!" That said, there are people who disagree with this mindset, and that disagreement is the problem. The value of the reviewer's reputation is impossible to quantify, because "reputation" is not a objective score. Some people look at a review from John Smith and say, "John Smith is a pretty trustworthy guy, anyone who has his endorsement is A-OK in my book," while other people will say, "Who the heck is this 'John Smith' guy?"

I actually would be rather okay with a completely informal system, where people post comments about their experience with a person on their commissions thread. In fact, I'm surprised this doesn't happen more often. In my brief experience in team-based competitive gaming/eSports, this was the only thing that people would post on recruitment threads. Someone would post a thread on a public TF2 recruitment thread with a message like, "I'm a scout player with 2 seasons of TWL division 3 experience under my belt, looking to join a divison 2 team that scrims regularly on week nights," followed by a dozen posts with people saying things like, "Solid guy, had him as a ringer for my CEVO-A team last season, someone pick him up" and "played a few pugs with this guy, decent player and a friendly person." Sometimes the feedback would be more negative or ambivalent, like "solid scout, but don't ever ask this guy to offclass" or "you're not a terrible player but you're not div 2 material, maybe spend another season in div 3 until you have some actual playoff experience." The actual queries (where people would respond to the recruitment post with offers) always happened via private message. I liked this system a lot because there was never any attempt to quantify anything.



As an alternative to having public reviews, perhaps we should move to a system where people can provide references who provide assessment privately. In the work I do, references are pretty important, and it's very common for me to provide references for people. These references are actually much more valuable than a public review, because a private phone call or email I can speak a lot more frankly than I would when posting a review that had my name attached to it. References are also more valuable because they tend to be a dialog, with two-way communication, where the inquiring party can ask specific targeted questions so that they're getting information that is valuable specifically to them.

In certain ways, the web has made it easier to assess someone's technical qualifications. It's easy to look at someone's portfolio and see how skilled they are as an artist. But that doesn't tell you anything about their reliability and communication skills. I think that it may be best to take a page from the traditional employment playbook. The purpose of a resume or CV is not only to say, "I am techically proficient and qualified for this position," it's there also to say "I have professional experience and refrences who can verify both my technical and non-technical skills." You don't even include the line on your resume "references availiable upon request," because it's implied.

Maybe commission posts should start looking more like CVs than portfolios. It's good to say "here's what I'm capable of doing," (portfolio), but it's often much more powerful to say, "here's what I've done" (list of projects). In generally, people seem to be really poor about broadcasting information apart from "here is how skilled I am, and how much money I will charge you." If you're an editor or writer, say how many words per week you can handle. (A range is fine.) If you're an artist, describe what your turnaround times have historically been. Seeing someone list this information immediately gives me more confidence in their ability to perform, because even if they don't stick rigidly to that, at least it shows that they've thought about it.

People seem to be really shy about asking for references. Many people don't want to appear distrusting, but when you're too credulous you get into all sorts of trouble. Maybe it's time to start including the line "references available upon request" when they open up for commissions. Additionally, perhaps we need to do more to encourage better hiring practices, so that people do more to protect themselves—negative reviews can stop repeat offenders, but people shouldn't take a lack of negative reviews as evidence that someone is trustworthy.

For example, I think that people who are hiring should do more "trial basis" hiring. Hire someone for a $100 job before you hire them for a $500 job. Don't pay someone for 100% of a job when they haven't even shown you 20% of the results. Nothing will teach you more about someone's work habits (good and bad) like experience. While I agree that it would be beneficial for the community to help protect buyers, it's impossible for the community to assume all of that responsibility, because if you only rely on past experience to predict future behavior, you'll never be prepared for someone's first offense. Buyers need to know how to protect themselves. If you're looking for a way that a community can facilitate that, maybe a moderated escrow service run by several trusted community members would be useful.

[Anne]

It allows artists/musicians/whoever to take the first step in letting people know it's okay to post a review

It's obviously not the artists who are benefiting from the review system so it's a bit weird to leave things up to them (especially in the button-form that'll require further explanations) vs generally encouraging writing honest reviews (by having a sticky in Recruitment section with some guidelines as was originally suggested in this thread)

For example, consider the following reviews:

Still, having reviews of the second type is better than none.

[noeinan]

It's obviously not the artists who are benefiting from the review system so it's a bit weird to leave things up to them (especially in the button-form that'll require further explanations) vs generally encouraging writing honest reviews (by having a sticky in Recruitment section with some guidelines as was originally suggested in this thread)

Well, I'm an artist and I think that the fear and uncertainty surrounding commissioning here, specifically, means that it will be harder for me to get commissions. In addition, if more well reputed artists start using the "review button" then it encourages other artists to start using it as well-- "that person uses it, they are getting commissions, the button and reviews makes them seem more trustworthy, you know I'd really like to get reviews on my services too!"

However, I would fully support adding a sticky in the Recruitment section as well, I recommended that earlier, too. We don't have to pick only one thing, but I think it's important to find out what works best, what people are going to be likely to do (as opposed to feeling not motivated to do it), etc.

[Anne]

We don't have to pick only one thing

But having a button would actually mean that it's not encouraged to write reviews for those who don't have it (that's how things are with Honest critique button) which is not really a good thing. As a commisioner I need reviews on exactly the people who won't put up a button like this (because they are unreliable or because they are not part of LSF community to know / care about buttons).

[TheKiwi]

I'm having mixed feelings about this whole thing.

Mainly, my issue is that all of the work is going to the person who has the commission thread. Giving them buttons, giving them badges, etc etc, and it really seems... Weird, to me. New creators won't have that same opportunity to get those buttons, and since people are scared of giving honest reviews, how can we really be certain that the person who has a badge or something really earned it?

I think I agree with some sentiments made about just encouraging honest reviews. An honest critique is one thing, but having a button that says "I want you to honestly leave feedback in my commission thread" really sounds like a recipe for failure.

The main idea that I've been getting is that we need the people who are spending money and are at risk at the forefront of all of this. We should work on making sure that people leave reviews or comments in threads, ESPECIALLY if they have lost a large sum of money to someone. It makes me so sad to hear stories of people running out on their commissioner, and even sadder when there seems to be a huge community centered around friendliness over professionalism. In the end, LF is supposed to be a forum for creating things, and that includes working with other people. It sucks, but we're not supposed to be enforcing a lax attitude towards people who are untrustworthy just because we don't want to cause drama. Drama shouldn't even be part of the equation: It costs a lot more trouble and money for people if they don't know who they can reliably work with.

Sorry if I'm rambling, but I guess in the end I want to say that leaving it in the hands of the commissionee wouldn't work well. A lot of our problems could just be solved if we had a more professional attitude towards commissioning in general.

None of that really applies to free artists or people either, just when its a case of exchanging money and a lot of people could get hurt by take-the-money-and-run types. Of all the threads I've seen, no one has dared to name names, but that really just creates a bigger feeling of distrust in the community.

[truefaiterman]

I'm reading a lot of great ideas here, and as an artist who is about to open his "hire me" thread (Let me say I'm pretty nervous about that, lol), I'm VERY interested on this system.

I personally think Deji's idea is good enough as a starting point. The 5-stars system (or any numeric way to show these things) have some issues, as stated by Kuiper, but it also has an important advantage: it's simple and convenient for any client. This is like a client looking at portfolios, a lot of people will tell you (and I saw that with my own eyes) that a lot of people navigate though portfolios with a VERY limited number of clicks (2 or three are a limit). And I remember that time I literally heard someone shouting "WTF? A FLASH INTRO? GET THE F*** OUTTA HERE!"

If someone goes and sees the rating, it will work as a simple way to find an average number, and THEN the client is free to take a look at the actual reviews and make a choice (it's the client's money, it's up to him/her how to spend it).

What I'd say is to put some limits to the reviewer, like the following:

-The game must have a thread on LSF.
-The reviewer must have a minimum number of posts on the community (a VERY symbolic quantity, like 5 or so [saying hi on the guestbook, the game's thread and the review would be 3 already]).
-The game must have more than a certain percentage of completition, at least on the reviewed job.
-The reviewer must say an (at least) estimate of the work comissioned (it's not the same to comission 2 sprites than 20).

These are the ideas I'm having right now. Some of them may seem a little too much, but this would be a system that could encourage someone to spend money on someone, and some creators may need this kind of feedback to have more work. When money is involved, it's better to leave nothing to randomness, and these suggestions should definetely deal with most ways to exploit the system.

About the issue on people not leaving honest reviews... well, if there IS an organized review system, and people is spending their money, I think that would already give a lot of reasons for people to just be honest. But otherwise, this would be about educating the community, a matter I can't really comment since I'm not that veteran here, nor I've never had to deal with these kind of topics on any forum I moderated.

[Quelcezot]

I think that encouraging reviews is a very good idea, however like others I think that using a number rating system would be flawed. In fact, possibly more strongly than others I think this would be a bad idea.

"This person took my money and never contacted me again." is about as damning as "1/10", but while one is immediately understandable and is damning because it is justified - inexplicable high or low ratings often have equivalent impact without needing evidence to back them up.

Accurately quantifying the quality of service that an artist, composer, or writer provides is basically impossible. I think encouraging fair and concise feedback would be most conducive to a more open and less tense atmosphere. Short reviews that fairly summarize the experience had.

As for how to "encourage", I do not know. Threads like this raise awareness that there is an issue and hopefully people reading through this thread will be able to come to their own conclusions and consider their feedback (and whether to give feedback) more carefully.

[noeinan]

But having a button would actually mean that it's not encouraged to write reviews for those who don't have it (that's how things are with Honest critique button) which is not really a good thing. As a commisioner I need reviews on exactly the people who won't put up a button like this (because they are unreliable or because they are not part of LSF community to know / care about buttons).

Eh, I don't really think having the button means "don't write reviews unless they have a button", but then I don't think it's like that for the Honest Critique button either. (I think this back and forth, though, has gotten fleshed out enough on the Honest Critique thread, and boils down to a difference in opinion.)

However, I do think that having more creators use the button will put pressure on others to also use the button. (Both carrot and stick pressure.) Even if it won't mean unreliable artists use the button, it may reassure someone if the artist they're looking at has one. (And has reviews.)

Confined to these forums, the problem right now is that clients *aren't* writing feedback, especially negative feedback, so anything that encourages people to do so will at least be an improvement over the current situation. A third party rating website may work, but before that is considered, I think we'd need someone to offer to build and maintain it.

Mainly, my issue is that all of the work is going to the person who has the commission thread.

I definitely think it favors creators who make a thread on the lemmasoft forums, instead of linking to, say, a commission page on deviantart. But even if there are only reviews, that would still be the case.

Sorry if I'm rambling, but I guess in the end I want to say that leaving it in the hands of the commissionee wouldn't work well. A lot of our problems could just be solved if we had a more professional attitude towards commissioning in general.

Agreed, this isn't a problem that can be solved exclusively by creators-- clients need to feel empowered to write honest reviews, especially in cases where someone has acted unethically. But creators can, I think, encourage this behavior, make it feel safe/easy, depending on how they make their threads.