Q: text and ... still text

[oyster]

1. Can we save the game script without utf8 encoding? Or use something in the init block at the very beginning

Code: Select all

$ style.default.encoding = 'gb2312'     

to tell the sccipt.rpy is gb2312 encoded.

I have searched the forum and found that utf8 problem is an FAQ. I am using win98, if I want to convert my script to utf8 encoding one, I open it in IE- yes, internet explorer- then use the 'save as'. Since notepad in win98 does not support encoding, if I found something wrong with my script, I have to edit the original rpy file and launch IE again. It is tedious. Further more, my native tongue is Chinese. One Chinese characer takes up 2 chars in a no-encoding TXT, but 4 chars if it is in utf8, then the whole Ren'Py script needs more chars.

I can run this python statement in IDLE

Code: Select all

>>> sys.stdin.encoding
'cp936'
>>> codecs.utf_8_encode(u'汉字')
('\xc2\xba\xc2\xba\xc3\x97\xc3\x96', 4)
>>> hz='汉字'
>>> codecs.utf_8_encode(unicode(hz,sys.stdin.encoding))
('\xe6\xb1\x89\xe5\xad\x97', 2)

(Sorry, I donno why the latest 2 are different)

I try to change the current implement in Ren'Py by editing function list_logical_lines(filename) in parser.py
add

Code: Select all

import sys
    import locale

change

Code: Select all

f = codecs.open(filename, "rU", "utf-8")

to

Code: Select all

f = codecs.open(filename, "rU")

change

Code: Select all

rv.append((filename, start_number, line))

to

Code: Select all

rv.append((filename, start_number, codecs.utf_8_encode(unicode(line,sys.stdin.encoding))[0]))

or

Code: Select all

rv.append((filename, start_number, codecs.utf_8_encode(unicode(line,locale.getdefaultlocale()[1]))[0]))

compile parser.py to parser.pyo and replace that renpy\parser.pyo in file lib\renpy.code
copy C:\Python23\Lib\encodings\*.pyo or overwrite encodings\*.pyo in file lib\renpy.code

however, when I run run_game.exe, it pops up with

I'm sorry, but an exception occured while executing your Ren'Py
script.

LookupError: unknown encoding: cp936

While parsing game/script.rpy.

-- Full Traceback ------------------------------------------------------------

File "run_game.py", line 70, in main
File "renpy\main.pyo", line 161, in main
File "renpy\script.pyo", line 195, in load_script
File "renpy\script.pyo", line 87, in __init__
File "renpy\script.pyo", line 113, in load_file
File "C:\\renpy-5.0.1\lib\renpy.code\renpy\parser.py", line 1241, in parse
File "\renpy-5.0.1\lib\renpy.code\renpy\parser.py", line 107, in list_logical_lines
LookupError: unknown encoding: cp936

While parsing game/script.rpy.

Ren'Py Version: Ren'Py 5.0.1

Any idea?

2. How to use the default font(s) coming with OS without copying it to COMMON folder?
A Chinese font is big, normally 3~20M. Some fonts exist in all of the Chinese windows, for example, simfang.ttf(~4M). Can I set the following command to search the system font fold first? If there is no such a font file in {windows fold}\fonts, it then points to common\simfang.ttf. This is helpful in saving the space.

Code: Select all

$ style.default.font = "simfang.ttf"

3. Can I set the font of characters like this:

Code: Select all

{font simfang}Hello{/font} {font symbol}world{/font}

4. Can I translate the moonlight.rpy into Chinese if I get the time?

Thank you.

[PyTom]

1. Can we save the game script without utf8 encoding?

The problem with allowing this is that many of the character sets (especially the east-asian ones) require a substantial amount of support to process. On my system the library to support mainland-chinese is 127kb in size, and the entire cjkcodecs is 1.6M,
a noticable fraction of the entire Ren'Py download size.

I'd prefer not to play favorites with language support, so unicode is the way to go. (Admittedly, there is still some stuff that favors English, but I am trying to slowly eliminate that.) Unicode is also what Python and its various libraries use internally.

I have searched the forum and found that utf8 problem is an FAQ. I am using win98, if I want to convert my script to utf8 encoding one, I open it in IE- yes, internet explorer- then use the 'save as'. Since notepad in win98 does not support encoding, if I found something wrong with my script, I have to edit the original rpy file and launch IE again. It is tedious. Further more, my native tongue is Chinese. One Chinese characer takes up 2 chars in a no-encoding TXT, but 4 chars if it is in utf8, then the whole Ren'Py script needs more chars.

Have you looked at the SciTE for Ren'Py editor I made available? It supports saving as UTF-8 with a switch on the file menu, and I believe it should support just about any input language you want to throw at it.

As for the 4 versus 2 bytes per character thing, let me point out that the .rpyc files compress the script, so I would think that you'll probably be reduced to less than two bytes per character, regardless. Besides, one hanzi is probably worth 4 English letters and a space anyway.

2. How to use the default font(s) coming with OS without copying it to COMMON folder?
A Chinese font is big, normally 3~20M. Some fonts exist in all of the Chinese windows, for example, simfang.ttf(~4M). Can I set the following command to search the system font fold first? If there is no such a font file in {windows fold}\fonts, it then points to common\simfang.ttf. This is helpful in saving the space.

Code: Select all

$ style.default.font = "simfang.ttf"

If you do not include the .ttf in the font name, Ren'Py will search the default font paths for a matching font. So something like:

Code: Select all

$ style.default.font = "simfang"

Will probably work. (You may need to capitalize or otherwise modify the word "simfang" to match what your system calls it.)

One issue with Chinese in particular, as I understand it, is that there aren't any spaces in the text. This will probably confuse the current word-wrap algorithm in Ren'Py to no end. Thankfully, I made the word-wrap customizable, so it will be possible to write an appropriate function when I understand what "appropriate" is.

So the question is how does one wrap lines of Chinese text? Is it acceptible to wrap between any two characters, or is the rule more subtle than that?

3. Can I set the font of characters like this:

Code: Select all

{font simfang}Hello{/font} {font symbol}world{/font}

Not yet. I'll add that into 5.1.

4. Can I translate the moonlight.rpy into Chinese if I get the time?

Sure. If you send me the translation, I'll post it up on the main MW site, so that everyone can see it. I do ask you run any major changes by me... I wouldn't want a major change in setting or in character names, but localizing some of the idiom could be fine.

[oyster]

thanx for the reply

Have you looked at the SciTE for Ren'Py editor I made available? It supports saving as UTF-8 with a switch on the file menu, and I believe it should support just about any input language you want to throw at it.

How to use this SciTE? I found that no proper chinese was shown when I loaded a utf-8-encoded rpy file no matter what I choose from "file->encoing" menu. Does that mean I can only save a file with utf-8 encoding in SciTE, but when I want to edit it again, I have to open the file WITHOUT utf-8 encoding?

btw, there is no rpy filetype listed in 'file->open' dialog so we have to choose 'show all files' everytime.

If you do not include the .ttf in the font name, Ren'Py will search the default font paths for a matching font. So something like:

Code: Select all

$ style.default.font = "simfang"

Will probably work. (You may need to capitalize or otherwise modify the word "simfang" to match what your system calls it.)

I have read the doc of pygame and found that can tell the name of a font using this code:?

Code: Select all

import pygame
for i in pygame.font.get_fonts():
	print i,' ',codecs.utf_8_encode(i)[0]

however, that seems something wrong with the utf-8-encoding in script.rpy(plz have a look at the attatched pic)

Code: Select all

import codecs

print '仿宋_gb2312',
print codecs.utf_8_encode('仿宋_gb2312'),codecs.utf_8_encode('仿宋_gb2312')[0]
print '\u4eff\u5b8b_gb2312'
print '\xe4\xbb\xbf\xe5\xae\x8b_gb2312'

One issue with Chinese in particular, as I understand it, is that there aren't any spaces in the text. This will probably confuse the current word-wrap algorithm in Ren'Py to no end. Thankfully, I made the word-wrap customizable, so it will be possible to write an appropriate function when I understand what "appropriate" is.

So the question is how does one wrap lines of Chinese text? Is it acceptible to wrap between any two characters, or is the rule more subtle than that?

sorry, I cannot grasp your meaning well, what does 'confuse the current word-wrap algorithm in Ren'Py to no end' mean?

[PyTom]

How to use this SciTE? I found that no proper chinese was shown when I loaded a utf-8-encoded rpy file no matter what I choose from "file->encoing" menu. Does that mean I can only save a file with utf-8 encoding in SciTE, but when I want to edit it again, I have to open the file WITHOUT utf-8 encoding?

To be honest, I've never used SciTE in a language but English, so I'm not totally sure how to do it. (Maybe someone else can help.)

I believe that if you start with an empty file, set the encoding to utf-8, and then type, things should probably work out in terms of the file encoding.

Anyway, it looks like you have been able to get going with utf-8, at least from the picture.

If you do not include the .ttf in the font name, Ren'Py will search the default font paths for a matching font. So something like:

Code: Select all

$ style.default.font = "simfang"

Will probably work. (You may need to capitalize or otherwise modify the word "simfang" to match what your system calls it.)

I assume that you've determined that the name that windows gives the simfang font is: '仿宋_gb2312', but that when you go to try to display with it, you're getting an error.

A rule is that, in python code, unicode strings must be prefixed with a u. So if you write a line like:

Code: Select all

$ style.default.font = u'仿宋_gb2312'

It might work. (Please note that the first character renders as a ? on my system... bad fonts?)

sorry, I cannot grasp your meaning well, what does 'confuse the current word-wrap algorithm in Ren'Py to no end' mean?

Okay if you have a long line of text, Ren'Py should automatically word-wrap it. Since right now the algorithm is tuned to English and western languages in general, the rule is that we only break lines at spaces.

If I understand Chinese correctly (and I probably don't), there aren't spaces between Hanzi. (Except at the end of sentences?) So if you can tell me what the rule is for breaking a Chinese sentence, I can implement that in Ren'Py.

(Is the rule that it's okay to break between any 2 characters? That's an easy rule.)

[Icekiss]

(Please note that the first character renders as a ? on my system... bad fonts?)

Renders correctly for me, and I've got nothing to do with chinese...

[oyster]

First of all, for all who do not understand Chinese:
if your code can render the 2nd case in my former attatched pic, then it works well. But if you only find 2 little rectanlges, it does not.

I still cannot get it worked
This is the string I used in script.rpy

Code: Select all

>>>import codecs
>>>fn= '你好'
>>>print '"'+fn+'"','\t','"'+codecs.utf_8_encode(fn)[0]+'"'
"你好" 	"浣犲ソ"

and I found a bug in pygame(I think so)

Code: Select all

import pygame
for i in pygame.font.get_fonts():
	print '"'+i+'"','\t','"'+codecs.utf_8_encode(i)[0]+'"','\t',pygame.font.match_font(i)

part of the output is

Code: Select all

font name       font name (utf_8)   font file
"仿宋_gb2312" 	"浠垮畫_gb2312" 	None      <---strange, in fact, it sould be C:\WINDOWS\FONTS\SIMFANG.TTF 
"宋体" 			"瀹嬩綋" 			C:\WINDOWS\Fonts\SIMSUN.TTF

so I've tried this one other than that 'sth_gb2312' in script.rpy

Code: Select all

$ style.default.font = "瀹嬩綋"
or
$ style.default.font = u"瀹嬩綋"

but only get the situation as the 1st one in my latest attached pic

Does anyone need the simsun.ttf? It is ~10M, and I donno if it is legal to release this font, but I think it is ok if we use this font only for test

As for word-wrap in Chinese, the short answer is: it is okay to break between any 2 characters, but plz avoid placing the following Chinese punctuation at the beginning of any line. I list one punctuation per line, and they are not utf_8 encoded)

Code: Select all

；
：
，
。
、
？
！

some other punctuations are not listed here, for the case are some complex, and we have to judge by the context.

And I attached a pure text file this time if you want to copy the Chinese characters

[PyTom]

Okay, reading through the python code that implements the system font matching, it seems clear to me that it does not support unicode. So that, and only that, needs to be in the native character set. (Ugly!)

Basically, run the following code as a python program. (You seem to be able to do that.)

Code: Select all

import pygame
pygame.font.init()

for i in pygame.font.get_fonts():
    print i, repr(i.encode("string_escape"))

This should print out a list of fonts, with the second element of the list being the a quoted string that, if assigned to style.default.font will get that font.

Hopefully. Maybe. I'm just guessing, here.

[oyster]

how to judge a renpy system variant is set sucessful or not? i.e. something like this:

Code: Select all

python:
    try:
    #on win9x, simsun.ttf exists
        style.default.font = "simsun.ttf"
    except:
        #but on win2k, this font file is recompiled and renamed to simsun.ttc
        style.default.font = "simsun.ttc"

thanx

[PyTom]

If you make a comma-separated list of font names, then the system font code will pick the first one that matches.

But I don't think that will work if you give it an extension. I think simply giving whatever simsun is listed as in the call to get_fonts() should work on any windows, as the lookup goes through a registry key that maps font name to font file.

BTW, is it working? If so, can you give me what you have now?

[oyster]

how to let your game use the system font on windows OS
'宋体'( AKA, simsun) exists on both simplified chinese win9x ,win2k and above, so your game takes less space if you use this font and do not bundle it. But there is the difference

Code: Select all

                    | font name( not in utf_8) | font file name
--------------------+--------------------------+--------------
win9x               |         '宋体'           |   simsun.ttf
win2k and above     |      '宋体＆新宋体'      |   simsun.ttc
--------------------+--------------------------+--------------

1.If you want to let pygame search the system font, you have to use the font name but not the font file name. For Chinese font, use this code to get the name of your font:

Code: Select all

print u'宋体'.encode('utf_8')
plz note the [u][i]u[/i][/u] before your text

your Ren'py seems like:

Code: Select all

$ style.default.font = '脣脦脤氓,脣脦脤氓拢娄脨脗脣脦脤氓'

the 1st is the font name on win98, 2nd for win2k. plz do not add SPACE/TAB in this code

Well, it works, but there is some incompatible utf-8 issue in Ren'Py:
if you want to use Chinese as TEXT in Ren'Py script, you must encode the text in utf_8, you can get it by:

Code: Select all

print '宋体测试'.encode('utf_8')
plz note there is [u][i]no[/i][/u] u before your text, if you add the u, the displayed TEXT is in a mess. and 
"宋体".encode('utf_8')=='瀹嬩綋'
but
u"宋体".encode('utf_8')=='脣脦脤氓'

I do not like to deal with the utf_8 problem, but I think perhaps PyTom want to correct this incompatibility

Hence there is another solution:
1.For user who does not want to modify Ren'Py itself, plz use the following code in your script.rpy:

Code: Select all

    python:
        import os
        if os.sys.platform=='win32':
            config.searchpath.append(os.environ['WINDIR']+'\\fonts')
        if renpy.exists ('simfang.ttf'): 
            style.default.font = 'simfang.ttf'  #here is the font file name on win9x
        else:
            style.default.font = 'simfang.ttc'  #the font file name on win2k and above    
        

2.For user who do not willing to type in long code. Plz open renpy\main.py and add the following in line 180

Code: Select all

    if os.sys.platform=='win32':
        renpy.config.searchpath.append(os.environ['WINDIR']+'\\fonts')
        

which is in the function 'main' and just below the code

Code: Select all

    renpy.config.searchpath = [ "common", basepath ]

recompile your Ren'Py, then it is always acceptable to use

Code: Select all

        python:
            for font in ('simfang.ttf','simfang.ttc'): 
                if renpy.exists (font): 
                    style.default.font = font

in your script.rpy. Can this patch appear in the next Ren'Py release?

This method is easier than assigning font name in scipt.rpy and rescures me from the encoding/decoding trouble

Another suggestion, if we add

Code: Select all

sys.path.append("lib")

in line 50 in run_game.pyw(line 27 in run_game.py)
no prompt

The _renpy module was not found. Please read module/README.txt for
more information.

comes out when we debug the game.

[PyTom]

I'm a little confused by what you're trying to say here. Are you saying this yourself, or is this the synopsis of the conversation on some other board?

1.If you want to let pygame search the system font, you have to use the font name but not the font file name. For Chinese font, use this code to get the name of your font:

Code: Select all

print u'宋体'.encode('utf_8')
plz note the [u][i]u[/i][/u] before your text

Could this code be written directly in Ren'Py as:

Code: Select all

init:
    $ style.default.font = u'宋体'.encode('utf_8')

Is that the right way of doing it?

Well, it works, but there is some incompatible utf-8 issue in Ren'Py:
if you want to use Chinese as TEXT in Ren'Py script, you must encode the text in utf_8, you can get it by:

Code: Select all

print '宋体测试'.encode('utf_8')
plz note there is [u][i]no[/i][/u] u before your text, if you add the u, the displayed TEXT is in a mess. and 
"宋体".encode('utf_8')=='瀹嬩綋'
but
u"宋体".encode('utf_8')=='脣脦脤氓'

I do not like to deal with the utf_8 problem, but I think perhaps PyTom want to correct this incompatibility

I'm still confused by what you consider the UTF-8 problem to be. Is it that you're unable to type utf-8 in a text editor? Or is it that the font name has to be in utf-8. Or is it something else?

2.For user who do not willing to type in long code. Plz open renpy\main.py and add the following in line 180

Code: Select all

    if os.sys.platform=='win32':
        renpy.config.searchpath.append(os.environ['WINDIR']+'\\fonts')
        

which is in the function 'main' and just below the code

Code: Select all

    renpy.config.searchpath = [ "common", basepath ]

recompile your Ren'Py, then it is always acceptable to use

Code: Select all

        python:
            for font in ('simfang.ttf','simfang.ttc'): 
                if renpy.exists (font): 
                    style.default.font = font

in your script.rpy. Can this patch appear in the next Ren'Py release?

I'd prefer not to duplicate the sysfont code in Ren'Py if I don't have to... I'd rather try to solve the sysfont encoding issues directly. This may include code to properly translate unicode to utf-8, or the native character set... whatever the registry encodes fonts as.

You could really help with that by sending me the result of running the code:

Code: Select all

import pygame
pygame.font.init()

for i in pygame.font.get_fonts():
    print i, repr(i), repr(i.encode("string_escape")) 

So I can figure out if the font names are being encoded in utf-8, gb_whatever, or something else. I can then try to undo the translation, which would mean that the thing would work.

Another suggestion, if we add

Code: Select all

sys.path.append("lib")

in line 50 in run_game.pyw(line 27 in run_game.py)
no prompt

The _renpy module was not found. Please read module/README.txt for
more information.

comes out when we debug the game.

The right way to fix this is to install the ren'py module for windows, which is one of the downloads you can get from the ren'py home page.

[oyster]

Code:

Code: Select all

init: 
    $ style.default.font = u'宋体'.encode('utf_8') 

Is that the right way of doing it?

No, it raise the UnicodeDecodeError

I'm sorry, but an exception occured while executing your Ren'Py
script.

UnicodeDecodeError: 'utf8' codec can't decode bytes in position 721-722: invalid data

While parsing game/script.rpy.

-- Full Traceback ------------------------------------------------------------

File "run_game.py", line 70, in main
File "k:\renpy-5.0.1\lib\renpy.code\renpy\main.py", line 186, in main
File "renpy\script.pyo", line 195, in load_script
File "renpy\script.pyo", line 87, in __init__
File "renpy\script.pyo", line 113, in load_file
File "renpy\parser.pyo", line 1213, in parse
File "renpy\parser.pyo", line 38, in list_logical_lines
File "codecs.pyo", line 380, in read
File "codecs.pyo", line 253, in read
UnicodeDecodeError: 'utf8' codec can't decode bytes in position 721-722: invalid data

While parsing game/script.rpy.

Ren'Py Version: Ren'Py 5.0.1

I'm still confused by what you consider the UTF-8 problem to be. Is it that you're unable to type utf-8 in a text editor?

No

Or is it that the font name has to be in utf-8. Or is it something else?

Plz have a look at the attached Ren'Py game script. In this game, I use '宋体,仿宋体'( simsun,simfang) font and only print the character '宋体,仿宋体'( simsun,simfang). But you may find, the 2 Chinese representation are and have to be different. The one for appointing font is gotten by

Code: Select all

print u'宋体,仿宋体'.encode('utf_8') 

And that one for screen display is by

Code: Select all

print unicode('宋体,仿宋体','mbcs').encode('utf_8')

I have to use 2 different encoding for the same thing in one script, that is what I have said "incompatibility".

You could really help with that by sending me the result of running the code:

Code:

import pygame
pygame.font.init()

for i in pygame.font.get_fonts():
print i, repr(i), repr(i.encode("string_escape"))

in the attached file

I'd rather try to solve the sysfont encoding issues directly. This may include code to properly translate unicode to utf-8, or the native character set...

Ok. but as I have mentioned, pygame cannot find all of the font file according to the font name, so there is some limitation.

[PyTom]

Okay, two things.

First of all, you're still saving your file as a multibyte file, rather than a utf-8 file. I can see this in the .rpy file itself. So you'll want to be doing that.

You can do this in scite by starting off with a new/ascii file (say, the script.rpy from the Ren'Py distribution), and then going to SciTE's file menu, and changing the encoding to "UTF-8 Cookie".

That being said, it looks like the font registry has some sort of weird encoding on it, and I don't totally know what it is. It doesn't seem to match the encoding used in script.rpy.

What I'm thinking of doing is changing the font matching code so that it tries to match .ttf files in any system font directory, so you will be able to do:

Code: Select all

$ style.default.font = 'simsun.ttf,simsun.ttc'

And expect it to work, even if the font isn't in a game directory.

Update

Okay, a modified text.py is up at:

http://www.bishoujo.us/tmp/text.py

Drop it into the renpy/display directory, and then run run_game.py. If your font is set to "simsun.ttf,simsun.ttc", it should pick it up.

You can also do "{font=simsun.ttf,simsun.ttc}This is simsun{/font" and expect that to work.

[oyster]

First of all, you're still saving your file as a multibyte file, rather than a utf-8 file. I can see this in the .rpy file itself. So you'll want to be doing that.

You can do this in scite by starting off with a new/ascii file (say, the script.rpy from the Ren'Py distribution), and then going to SciTE's file menu, and changing the encoding to "UTF-8 Cookie".

Plz have a look at the attached pic( sorry, it is a big one). In this pic, the left part is saved in SciTE, and SciTe saves file as what this code does

Code: Select all

fin=open('ansi_script.rpy')
fout=open('wrong_utf_8_script.rpy','w')
data=fin.read()
data=unicode(data,'mbcs').encode('utf_8') 
print >>fout,data
fout.close()
fin.close()

but the file name is totally wrong, which results in 'Runtime Error!'

In fact I have to save my file as a utf-8 one manually as the part on the right showes in order to get the right font name.

Okay, a modified text.py is up at:
http://www.bishoujo.us/tmp/text.py

Code: Select all

$ style.default.font = 'font1.ttf,font2.ttf'
and
"{font=font1.ttf,font2.ttf}This is simsun{/font}"

work as expected. Only one thing to note, the font flie name's extension has to be TTF, but TTC( eg. simsun.ttc) or FON is not a working one even though they are legal in pygame.

[oyster]

Since we can use the Chinese font in/not in system font fold now, I suggest not to hang on the utf-8 issue( both PyTom and me) unless there is someone else raise this problem again.
And thank you, PyTom, for the quick and kindly reply