Diacritics and header encoding

[rsdoiel]

While the body sent from the browser can be UTF-8 the header has to be Latin1 to preserve the diacritics, RDM ran into this issue. I need to investigate and confirm this is why I cannot save the clpid records that have diacritics in the string.

[rsdoiel]

This will need to be integrated into the dataset upgrade fix since the internal API use run directly from datasetd.

[rsdoiel]

This is related to issue 80 too.

[rsdoiel]

This might be more complicated than the URL or header being ASCII and not UTF-8. Browsers send the UTF-8 correctly. That is because host names in IPv6 if not IPv4 can be UTF-8 (I get RSS feeds from host names that are in Japanese characters). When I was doing RegExp to create a validator function for clpid and clgid I ran into some really odd behaviors with some of our clpid and they might be the same ones that present a problem on saving the data. Needs much more investigation.

[rsdoiel]

Once we have a spec for clpid and clgid I can normalize the problematic identifiers and this will solve our problem. It's not the header ASCII encoding issue I thought it was.