Unicode is hard

7th October, 2007 . 51 comments

Full unicode support is steadily being fleshed out for rst2a's rendering API.

Since the unicode branch has been merged, unicode support in Django seems to be very solid.

However, when files are posted as a file upload, Django doesn't make their contents available as a unicode string. I'm not certain how / if to handle this at the framework level - as you don't want binary files getting encoded. There's a ticket open with regards to content transfer encoding. Perhaps something could be worked in with that.

Its easy to handle though with:

from django.utils.encoding import smart_unicode
content = smart_unicode(file['content'])

This got the unit tests passing, but Duncan could still make PDF rendering blow up by mashing alt+some keys on his macbook :(

Using rst2latex with --input-encoding=UTF-8 and --output-encoding=UTF-8 we were still getting this error:

! Package inputenc Error: Unicode char \u8:? not set up for use with LaTeX.

The fix for this was:

sudo apt-get install latex-ucs

rst2a should now handle most documents you throw at it, with regards to unicode. There are some issues with PDF when rendering odd characters. For example I'm trying to work out how to render documents with the ยข character. LaTeX gives this error:

! LaTeX Error: Command \textcent unavailable in encoding T1.

Thanks Duncan and Gabriel for your help.

51 Comments

Subscribe to comments

 #1

Forest Bond - 4th January, 2008 at 2:43 p.m.

xetex is latex that handles Unicode input (and TrueType fonts). Have you tried that?

 #2

simon - 23rd February, 2011 at 6:52 a.m.

Great tips, I would like to join your blog anyway,
regards,
http://www.japanrx.com

 #3

ebuy - 15th May, 2011 at 12:06 a.m.

Supply nail beauty, nail drill, uv lamp,nail foil http://www.ebuybeauty.com/

 #4

prom dresses - 3rd July, 2011 at 11:51 p.m.

sss

 #7

dafi - 26th July, 2011 at 1:38 p.m.

Do You ever heard about BEST HDTV ANTENNA, Twitt News, Twitt Trend, Events Barcelona, Golf Shop, Blitz Time, BEST TEETH WHITENING, Antennas Direct

 #8

darasa - 5th August, 2011 at 11:09 p.m.

wow... amzing info mobiltabletipaymufaceblog

 #9

supreme11 - 9th August, 2011 at 1:58 a.m.

The details of any detected motion are sent to the client device which copies the pixels from their original location to their new location.

Regards,
Dissertation Writing | Buy Dissertation

 #10

star09 - 9th August, 2011 at 3:08 a.m.

Cue reStructuredText - an easy-to-read, plain text document format designed to keep content and the style/formatting of content separate. Author the content once, then present it in a variety of formats - like html and pdf - with infinite possibilities for styles and templates.

Regards,
Write My Essay | Essay Writing Help

 #13

Satriyo Niti Atmojo - 12th August, 2011 at 12:29 p.m.

Yes, i know it's hard | Peluang Bisnis Online Tanpa Ribet, Top 1 Oli Sintetik Mobil-Motor Indonesia, dan Mobil Keluarga Terbaik Di Indonesia, Satriyo Niti Atmojo

 #14

TOP 1 OLI SINTETIK MOBIL-MOTOR INDONESIA - 16th August, 2011 at 12:52 a.m.

It's a good article, thanks for the information. Please visit my website @ TOP 1 OLI SINTETIK MOBIL-MOTOR INDONESIA and Iklan baris gratis tanpa daftar

 #15

Pulsa Elektrik Termurah - 17th August, 2011 at 5:20 a.m.

It's good to meet your blog. Pliz visit my blog @ Pulsa Elektrik Murah

 #16

Great tips - 18th August, 2011 at 7:50 p.m.

Cara Membuat Blog
goood.. goood..

 #17

Star09 - 26th August, 2011 at 4:22 a.m.

Almost everything, I mean dealing with literary works are influenced by this work. Because this work is a work of maestro

Regards,
Al Fakher

 #18

Cell Phone Review - 27th August, 2011 at 12:18 a.m.

Howdy, thank you, your tips very helpful to us.
visit my lates post, here you are :
Cell Phone Review | Cara Menghilangkan Bekas Jerawat | Adolescent | Zodia Tanaman Pengusir Nyamuk | Ramalan Bintang | iPhone 5 and iPhone 4gs | Blackberry Bold 9900 Release on August | HiPhone 5 | Berita Unik | Ramalan Bintang

 #19

sikherwar - 30th August, 2011 at 2:34 a.m.

Useful information ..I am very happy to read this article..thanks for giving us this useful information. Fantastic walk-through.
Milestone IT
Milestone IT Contractors
Web Design London
SEO Company Las Vegas

 #20

african mango - 30th August, 2011 at 11:38 p.m.

fantastic site, thank you for the post, As always a pleasure whenever I visit your site.

african mango

 #21

Steve of Wonders - 1st September, 2011 at 5:50 p.m.

I'm not sure I like the style. Anyway, don't get disagreeable about the details of unicode, or you may just need the services of a Miami personal injury lawyer before you're through.

 #22

online dating - 13th September, 2011 at 7:36 a.m.

It was a great thing to have come across this post. Great job done.
online dating

 #23

Brian - 16th September, 2011 at 11:27 a.m.

really good post
free dating

 #25

courier service seattle - 20th September, 2011 at 2:26 p.m.

I better give sometime to learn it. courier service seattle

 #26

davis1 - 23rd September, 2011 at 7:11 a.m.

So how do you cope with this?
credit cards

 #27

ada - 25th September, 2011 at 9:29 p.m.

BLOGGER | SMADAV | PCMAV | MIVO TV | RCTI | Cerita Lucu Banget |

 #29

clistern - 29th September, 2011 at 4:04 p.m.

It is extremely helpful and beneficial to your readers. Many thanks for the exciting blog posting! I really enjoyed reading it, you are a brilliant writer. I actually added your blog to my favorites.buy triactol

 #36

kedai obat kuat - 9th October, 2011 at 3:18 a.m.


Ternyata jasa SEO murah adalah pekerjaan mulia. Dengan kerja copy paste online dan suka minum XAMthone serta meliput daftar kontes SEO terbaru membuat saya bisa merancang Souvenir Pernikahan yang pasti Souvenir Pernikahan Murah dan Undangan Pernikahan dengan mengikut sertakan situs bursa iklan, bursa promosi , serta bursa jual beli sebagai ajang promosi SEO murah membuat saya semakin semangat dalam belajar bisnis online dan melayani pesanan jasa SEO dari pelanggan saya untuk biaya kuliah. Semoga saya bisa menjadi orang yang sukses di dunia dan akhirat serta menjadi raja di dunia maya, raja virtual tentunya.

 #39

kuman - 14th October, 2011 at 10:13 p.m.

good Funny Quotes - Kata Mutiara

 #40

matamu soak - 23rd October, 2011 at 4:28 a.m.

Considerably jasa SEO, this post jasa SEO is really the sweetest on jasa SEO this notable topic.
I harmonise with jasa SEO your conclusions and will thirstily look jasa SEO forward to your incoming updates.
Saying thanks jasa SEO will not just be sufficient,
for the phenomenal clarity jasa SEO in your writing.

Try to click jasa SEO guys !

 #43

ferty - 3rd November, 2011 at 5:50 p.m.

How to Lose Belly Fat The Weight loss plan Theory. The Atkins diet plans to lose weight may perhaps be the single most well-known trend diets.

 #44

pulsa elektrik murah - 6th November, 2011 at 3:02 a.m.

join to my businees
pulsa elektrik
Harga pulsa elektrik murah

 #45

Rusdi - 9th November, 2011 at 11:54 a.m.

Rumah Dijual
Gamis
Perlengkapan Bayi
Sprei
Gamis Muslim

 #46

hoodiesta - 10th November, 2011 at 1:25 a.m.

Nice Sharing. keep posting such good articles.
T Shirt Printing

 #47

jeffrey sanders - 11th November, 2011 at 6:37 a.m.

I think this stuff sounds very useful for research paper writing on Unicode.

 #48

Assignment Writing Service - 14th November, 2011 at 4:18 a.m.

I like your working on this web, thanks for update posting i get too many useful links and information from this website.
Essay Writing Services | Write My Essay | Buy Essay | Assignment Help | Custom Paper Writing

 #49

Anonymous - 16th November, 2011 at 1:12 a.m.

Great work. Very informative to me. Essay Writing | Coursework Writing | Assignment Writing

 #50

Anonymous - 16th November, 2011 at 1:12 a.m.

I appreciate your comment and also thanks to the shearing. Thesis Writing | Dissertation Writing

 #51

andrew - 16th November, 2011 at 10:58 a.m.

I can't agree with this, please get us some more arguments!
email search

Post a comment