Potential tech which could generate 2d illustration from 3d assets
Posted: Sat Dec 16, 2017 7:15 am
Edit: Gathered enough interest in other places
Supporting creators of visual novels and story-based games since 2003.
https://lemmasoft.renai.us/forums/
Well, it may look indistinguishable from the work of a human artist to you, but those of us that have studied illustration, that work in it, it is obviously the work of a computer filter on a 3D model. Similar things have been done for a long time, taking photos or 3D models and running them through filters to get the end result. This process may eliminate the human element of having to tweak the results, but it still isn't some giant leap forward in technology.conclave wrote: ↑Sat Dec 16, 2017 7:15 am This deep learning expert created an algorithm which takes 3d images as inputs and generates a 2d illustration indistinguishable from the work of human artist.
Main selling point of this tech is that there is NO human intervention during this generation process. It is a simple click and finish process instead of having to be an illustration expert and needing to fix imperfections here and there throughout the process.
---
My question is, how useful does this tech seem to you? If so, I would like to discuss particular features which would make it useful to you.
I feel this could potentially save a lot of art budget for developers by buying pre-made assets on a hobbyist budget which can be customised. For example, poser software like Daz3d allows you to change pose and facial expression of a character by simplifying adjusting sliders. There's no need to pay an artist for multiple drawings of sad/happy faces in different poses.
This is a big one too. Because the algorithm cannot understand framing, posing, focal points, and lights, you have to set up all that stuff beforehand, yourself, to get a good result.SundownKid wrote: ↑Sat Dec 16, 2017 9:37 am It will probably be a hard sell for most developers. The biggest stumbling block is not the technology itself, but the fact that pre-made assets are generic and lack originality. When you consider that, it actually would cost far more to custom make a 3D model to pose then to commission the art.
I disagree about them looking 'generic'. There are hundreds of customizable parts for a 3d character using Daz3d. And you don't need to be super technical since it's just adjusting sliders for eye size,etc here and there. You can customize a character to become many different variations of hulk if that's what you want.SundownKid wrote: ↑Sat Dec 16, 2017 9:37 am It will probably be a hard sell for most developers. The biggest stumbling block is not the technology itself, but the fact that pre-made assets are generic and lack originality. When you consider that, it actually would cost far more to custom make a 3D model to pose then to commission the art.
The screenshots I showed were meant to be a proof of concept. Just cause there were some artifacts here and there doesn't mean it's impossible to get rid of them.LateWhiteRabbit wrote: ↑Sat Dec 16, 2017 11:53 am The algorithm makes the classic computer mistake of not knowing when to omit detail, use different line weights, or break lines to do things like avoid tangents. The hyperactive cross-hatching is a DEAD GIVEAWAY that it is a computer filtered drawing and not one done by a person. The result always looks busy and indecisive. Not to mention that you lose any voice or soul an individual or human artist would have given the drawing with their unique style.
Not sure if you meant that you don't have to do this work with traditional approach because you just stick a 2d figure on a background. The algorithm will allow you to do the same.LateWhiteRabbit wrote: ↑Sat Dec 16, 2017 11:53 am This is a big one too. Because the algorithm cannot understand framing, posing, focal points, and lights, you have to set up all that stuff beforehand, yourself, to get a good result.
Even customized characters from Daz3D often end up looking 'generic' because they are in the same style. I've used Daz3D extensively in the past as well as made 3D models from scratch. (I have a bachelor's degree in Graphic Art and Design, with a focus on interactive mediums.)conclave wrote: ↑Sat Dec 16, 2017 8:38 pm I disagree about them looking 'generic'. There are hundreds of customizable parts for a 3d character using Daz3d. And you don't need to be super technical since it's just adjusting sliders for eye size,etc here and there. You can customize a character to become many different variations of hulk if that's what you want.
It isn't about 'artifacts' it's that the algorithm isn't making the same decisions a professional human artist would. Maybe it could be trained to make those decisions. And I'm not saying it isn't getting some nice results. It totally is. My whole issue is the hyperbole of claiming it can't be distinguished from a human artist.conclave wrote: ↑Sat Dec 16, 2017 8:38 pm The screenshots I showed were meant to be a proof of concept. Just cause there were some artifacts here and there doesn't mean it's impossible to get rid of them.
Also, here's a colored illustration example generated from the algorithm which I challenge you to point flaws at:
https://www.daz3d.com/forums/uploads/Fi ... 13acb9.png
I mention those things, because every algorithm is only as good as it's inputs. If you don't feed the algorithm a good model, with good lighting, I doubt you are going to get any kind of good results.conclave wrote: ↑Sat Dec 16, 2017 8:38 pm Not sure if you meant that you don't have to do this work with traditional approach because you just stick a 2d figure on a background. The algorithm will allow you to do the same.
If you want to place 2d characters in background with correct perspective and such, you still have to think about them even if you don't use this algorithm. Also, one purpose of the algorithm is to reduce amount of attention required for details compared to 3d renderings so that focal points, lights and such don't have to be setup perfectly.
The details mentioned are good to keep in mind but not relevant to the usefulness of the algorithm since it isn't supposed to be a director, just an illustrator. As long as an illustrator conveys the lightings, etc that it was told to draw, it's doing its job.
When I reverse Google searched that it gave me The House of Fata Morgana, and the game has quite an erratic art styles. Can you submit links to a workflow or an article discussing the workflow to create this piece in particular? Because it's the only one I've seen thus far that can convince me it's an illustration, if it's indeed a render.
I guess I need to make a distinction here.LateWhiteRabbit wrote: ↑Sun Dec 17, 2017 6:39 pm What if someone wants an moe anime style character? That is about a thousand times harder to model in 3D for good render, because a good cartoon render of such a model requires ignoring edges and knowing when to flatten and omit lighting detail, which ironically often involves artists hand-painting normal maps for the cartoon renders to look good.
What if you want something like Skottie Young's illustrations?
It wasn't my intention to not include the source, there were other important details I wanted to convey and this wasn't my top priority. Also no one asked about it until now. He posts new renders sometimes but they are not archived in a nice organised way for people to just skim through.Why haven't you mentioned the name or linked to information on this algorithm? Is it only being run on a university mainframe or research super computer? Why are all the examples almost 2 years old? This discussion is all purely rhetorical without that information.
As I mentioned before, while 2d assets are easier to create from scratch, there are hardly any pre-made 2d assets which are reusable. For 3d, the reverse is true. They are more difficult to create from scratch, but there are LOTS of 3d pre-made assets which are reusable AND customisable (also mentioned before).Kinjo wrote: ↑Sun Dec 17, 2017 9:02 pm However, the bottom line is this: you still need an artist to do the 3D modelling. The reason those renders came out so well is because the models were so intricately designed and composed. They probably have thousands, if not hundreds of thousands of vertices and edges, and rigging the models to animate properly is no easy feat either. So you could definitely get a variety of poses for characters out of it, and even a variety of CGs taking place in the same scene. But if you're going to the effort of creating 3D assets for a 2D game, why not just make a 3D game instead?
It's not a render. I was trying to say deep learning technology is quite capable and the possibilities aren't limited like some people like to think.Zylinder wrote: ↑Sun Dec 17, 2017 11:56 pm When I reverse Google searched that it gave me The House of Fata Morgana, and the game has quite an erratic art styles. Can you submit links to a workflow or an article discussing the workflow to create this piece in particular? Because it's the only one I've seen thus far that can convince me it's an illustration, if it's indeed a render.
Ah, yeah I'd forgotten you did say that, whoops. Though in that case, I still think that your greatest obstacle would be trying to find good assets. Yes, there's a ton of pre-made 3D stuff available, but is it going to look good? And is it going to look unique? I've used the 3D Warehouse for free models, and although there is indeed a large selection, there's also a large number of models you'll never need or wouldn't want to ever use. Maybe DAZ has a better collection, but I'm just warning you that if you really want your models to stand out, you might want to consider making your own. Otherwise, whether this tool saves money really just depends on how much the models cost and what kind of quality you're willing to settle for.conclave wrote: ↑Mon Dec 18, 2017 7:51 am As I mentioned before, while 2d assets are easier to create from scratch, there are hardly any pre-made 2d assets which are reusable. For 3d, the reverse is true. They are more difficult to create from scratch, but there are LOTS of 3d pre-made assets which are reusable AND customisable (also mentioned before).
The 3d store you linked seems to be on par with Turbosquid. Daz store assets blow both out of water in my opinion. If you check https://www.daz3d.com/explore-sam-kennedy, some professionals already are using Daz assets as a base for their illustrations. So that again speaks volumes for the capabilities of its assets.Kinjo wrote: ↑Tue Dec 19, 2017 5:06 amI still think that your greatest obstacle would be trying to find good assets. Yes, there's a ton of pre-made 3D stuff available, but is it going to look good? And is it going to look unique? I've used the 3D Warehouse for free models, and although there is indeed a large selection, there's also a large number of models you'll never need or wouldn't want to ever use. Maybe DAZ has a better collection, but I'm just warning you that if you really want your models to stand out, you might want to consider making your own. Otherwise, whether this tool saves money really just depends on how much the models cost and what kind of quality you're willing to settle for.
Thanks for acknowledging this. I talked to other deep learning experts as well. The current era of deep learning is mainly 'first generation' techniques, where you specify training methods for your algorithm. The algorithm doesn't learn new skills besides improving on optimal values and such. The next-gen deep learning is to do with advanced human traits as you described, learning new skills on its own even if we don't specify the training methods.Also, about the technology itself... Assuming it really is a deep learning algorithm, then you can just train it on data sets closer to the style you want it to create. It learns just like a person would learn how to draw, so even if a professional artist can tell it's "machine-made" there could very well come a point in the future where that's no longer true. Plus, as someone who isn't a professional artist, it certainly fooled me, and really the average player (who can't tell the difference) won't care how your assets are made -- just as long as they look good.