There’s functions for % escape and % unescape, and there’s a function for entity encoding.
Is there anything for entity decoding? The string-level search-and-replace functionality is a little caveman.
mig
You can try unencodedString = roUrlTransfer.unescape(encodedString)
–Kevin
I’m already sending all my strings through the unescape, but thanks.
Can you provide an example of the kind of string you’re wanting to decode?
Entities are the funny things that start with & and end with ; and can be custom defined within a DTD.
Normally, your XML parser should just transparently decode it and I should never have to look at the values.
However, if you check this MRSS feed:
http://feeds.theonion.com/OnionNewsNetwork?format=xml
You can see them doing dumb things like putting encoded entities into a CDATA block:
<itunes:summary><![CDATA[Jim and Tracy welcome Chris Morgan, the kindergartener who wrote the latest action-packed "Fast And The Furious" sequel.]]></itunes:summary>
<itunes:summary><![CDATA[The rest of this year's pop culture to be "pretty rough,” The Economist lets readers catch up, and a Wal-Mart greeter knows exactly how many blacks are in the store. It's the week of April 18th, 2011.]]></itunes:summary>
This means I need to decode those bits by hand.
So, so far, I’ve done this:
Function ReplaceString(str As String, search as String, replace as String) As String
count = 0 ' Be terrified of infinite loops
idx = instr( 0, str, search )
while ( idx > 0 AND count < 20 )
print "before: " str
str = left( str, idx - 1 ) + replace + right( str, len( str ) - len( search ) - idx + 1 )
idx = instr( 0, str, search )
count = count + 1
end while
return str
End Function
Function decode(http As Object, s As Dynamic) As String
' Let's manually decode some XML entities since they come in from our Onion feed
if ( type(s) = "String" AND s <> invalid AND s <> "" )
s = ReplaceString( s, """, chr(34) )
s = ReplaceString( s, "”", chr(34) )
endif
return http.Unescape( validstr( s ) )
End Function
But as anyone will likely tell you, implementing things like string replacement in script rather than in native code is going to be slow slow angry slow.
Realistically, not only should you add a “decode” function, but you should also probably add a similar “replace” function that is also running in native code for performance.
Thanks for your interest!
mig
“migmigmig” wrote:
Entities are the funny things that start with & and end with ; and can be custom defined within a DTD.Normally, your XML parser should just transparently decode it and I should never have to look at the values.
However, if you check this MRSS feed:
http://feeds.theonion.com/OnionNewsNetwork?format=xmlYou can see them doing dumb things like putting encoded entities into a CDATA block:
That’s what I thought you meant, actually, but wanted to be sure. HtmlEncode and HtmlDecode functions would definitely be worthwhile additions, but in their absence, I would think a RegEx based decoder would be much more efficient and performant than that ReplaceString method.
Or maybe something a little more hacky:
Function XmlDecode(encoded As String) As String
xml = CreateObject("roXmlElement")
If xml.Parse("<encoded>" + encoded + "</encoded>") Then
Return xml.GetText()
End If
Return encoded
End Function
Well, if we were going to pull out our wishlists, I’d say we want:
- The full functionality that js strings have (including array-joins, regex replace, splice replace, etc)
- JSON parsing into associative arrays (since XML is so unwieldy in HTML-land, people have been migrating to JSON… which is itself unwieldy in systems like this)
I thought about pushing XML into and then pulling it back from the XMLElement object, but since maybe only 1 in 50 strings actually have an entity that I need to replace, I figured the first walk to find any entities at all would be much cheaper than building and destroying an XML object.
I will say, I really don’t have any good sense for performance whatsoever in this language.
Huge Bump. HTML entity encoding/decoding would be a big help.
Is HTML entity decoding available by now?
“philotas” wrote:
Is HTML entity decoding available by now?
Can you be more specific what you are trying to do?
Most HTML you can run through the roXmlElement parser and fetch the decoded element text - incl. through the ingenious hack by @TheEndless above.
The strings already have getEntityEncode() method. The opposite is a one-liner:
PRINT (myStringExpression).replace(""", """").replace("'", "'").replace("<", "<").replace(">", ">').replace("&", "&")
“EnTerr” wrote:
The strings already have getEntityEncode() method. The opposite is a one-liner:PRINT (myStringExpression).replace(""", """").replace("'", "'").replace("<", "<").replace(">", ">').replace("&", "&")
Correct. This is what I want to do, but without having to create this one-liner by hand for all entities out there, since there are more than the one in your example.
Update: I found that the above mentioned XmlDecode Function indeed work but it a quick test I did showed, that it does not work with entity names:
For example € works but € does not
“philotas” wrote:
Update: I found that the above mentioned XmlDecode Function indeed work but it a quick test I did showed, that it does not work with entity names: For example € works but € does not
That must be because XML has only 5 predefined char entites - where HTML has couple of hundreds of them.
Again - can you explain your use case?
How and why are you getting text encoded as HTML?
Why not receiving data in JSON or XML (or plain-old text in UTF8 for that matter).
I could sketch you a function to tackle that, were i persuaded it’s necessary.
I receive (simple) HTML formatted text from a server via JSON and want to display it in Label.
I strip out some tags, but there could be some HTML Entities which I want to convert.
“philotas” wrote:
I receive (simple) HTML formatted text from a server via JSON and want to display it in Label.
I strip out some tags, but there could be some HTML Entities which I want to convert.
Do you have the leeway to change that, as in simply not sending HTML?
That will make your life notably easier.
The reason being, you don’t need HTML on Roku’s side (not rendered) - nor do you need it for the transport, since JSON can transport all Unicode chars already, just make sure encoding for http is utf8 (should be the default already). The only concern remaining is that of having a font with the exotic glyphs, which has to be tackled either way.
“EnTerr” wrote:
“philotas” wrote:
I receive (simple) HTML formatted text from a server via JSON and want to display it in Label.
I strip out some tags, but there could be some HTML Entities which I want to convert.
Do you have the leeway to change that, as in simply not sending HTML?
Unfortunately not for the time being.
I have to say, the part where parsing unknown character-entity-references fails miserably - i have no respect for that. Even if that’s mandated by a spec, the pragmatic thing would have been to leave unrecognized items alone:
Brightscript Debugger> xml = CreateObject("roXmlElement")
Brightscript Debugger> ? xml.parse("<x>€ and €</x>"), xml.getText()
false
Brightscript Debugger> ? xml.parse("<x>just €</x>"), xml.getText()
true just €
Here is a $64k question: does roXmlElement.parse() support DTDs? Because if it does… piece of cake, we can just do something like this and the entities will get defined and handled:
<?xml version="1.0" encoding="utf-8"?>
<!DOCTYPE x PUBLIC "-//W3C//ENTITIES Special for XHTML//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml-special.ent">
<x> ...
Affirmative on DTD support!
Brightscript Debugger> dtd2 = "<!DOCTYPE x [ <!ENTITY euro ""€""> ]>"
Brightscript Debugger> ? xml.parse(dtd2 + "<x>mwa€hahaha</x>"), xml.getText()
true mwa€hahaha
[spoiler=victory is mine:sibykd31][youtube:sibykd31]x3rWRLOZX6U[/youtube:sibykd31][/spoiler:sibykd31]
“philotas” wrote:
I receive (simple) HTML formatted text from a server via JSON and want to display it in Label.
I strip out some tags, but there could be some HTML Entities which I want to convert.
Even if you are able to convert html-formatted JSON text to display in a Label, bear in mind that you may still need to convert some characters to characters that your Roku font can handle. This applies regardless of whether the character is represented as an entity reference or even as a raw Unicode codepoint.
For instance, while the Euro character (8364) may render correctly, a character such as a hyphen (8208) will not, so you’d have to do something like:
text = text.Replace(Chr(8208), "-")
