Encoding errors on ASCII-8BIT strings (eg: any string from the mysql adapter)

[kindjar]

The sanitizer seems to have issues when its input is a string in ASCII-8BIT encoding:

irb(main):006:0* Rails::Html::WhiteListSanitizer.new.sanitize("tooth".encode('ASCII-8BIT'))
output error : unknown encoding ASCII-8BIT
=> ""
irb(main):007:0>

While ASCII-8BIT isn't the default encoding these days, it seems that strings coming from the mysql adapter (but not the mysql2 adapter) are always in ASCII-8BIT encoding, even when the table is using charset utf8:

irb(main):004:0> Day.connection.charset
=> "utf8"
irb(main):005:0> Day.last.notes.encoding
=> #<Encoding:ASCII-8BIT>

This means that using the sanitizer on any string from the database when using the mysql adapter will result in errors. I chased the error down to Nokogiri's NodeSet#to_s method, but wasn't sure what the right approach was for addressing the issue.

Switching to the mysql2 adapter makes the issue go away, since it produces all strings in UTF-8. However, folks who've been using the mysql gem (for legacy reasons or whatever) could run into headaches trying to upgrade to Rails 4.2 because of this (it hit me by way of the highlight method in ActionView::Helpers::TextHelper).

[ncri]

+1 Our app broke completely after updating to Rails 4.2 ;-)

[ncri]

Was able to hack this adding .force_encoding('utf-8') to strings passed into the sanitizer.

[ncri]

This patch fixes it globally:

module ActionView
  module Helpers
    module SanitizeHelper
      def sanitize(html, options = {})
        self.class.white_list_sanitizer.sanitize(html.force_encoding('utf-8'), options).try(:html_safe)
      end
    end
  end
end

But there might be unintended side effects. ;-)

[kaspth]

@kindjar Thanks for reporting this and sorry it took so long to look into it ❤️

Can either you and or @ncri confirm that #33 fixes your issues?

[kindjar]

@kaspth Yes #33 does fix the issue I was seeing, and since my data was actually UTF-8 already (it's just that the mysql database adapter tags it wrong), everything is cool.

Forcing ASCII-8BIT (which is just a label for a collection of different character encodings) to be interpreted as UTF-8 could cause problems for people whose data is in some non-UTF-8-compatible encoding (like, say, Windows-1252). It would be nice to get a note into the release notes to warn people about the mysql adapter, though.

[kaspth]

Thanks for trying it out. I'll look into writing something about the mysql gem.

[kaspth]

By the way, when you say release notes are you talking about Rails' 4.2 ones?

[kindjar]

I think most people will hit the issue when updating Rails, rather than specifically updating rails-html-sanitizer, so a mention in the Rails release notes would get the attention of the most affected people, I think.

[kaspth]

👍, just saying release notes was ambiguous :)

[kaspth]

Forcing ASCII-8BIT (which is just a label for a collection of different character encodings) to be interpreted as UTF-8 could cause problems for people whose data is in some non-UTF-8-compatible encoding (like, say, Windows-1252).

@kindjar do you have a link to verify?

As far as I can tell ASCII-8BIT is Ruby-speak for binary:

irb(main):008:0> Encoding::ASCII_8BIT.names
=> ["ASCII-8BIT", "BINARY"]

[kindjar]

In Windows 1252, the Euro sign is x80 (Windows 1252 in Wikipedia). If you had a database using that encoding (such as latin1 in MySQL), and you were using the mysql gem, then it would come out of the database with the generic encoding ASCII-8BIT. If you then force UTF-8 encoding on that string, the byte sequence is invalid UTF-8 (in UTF-8, code points larger than 127 require multiple bytes to encode).

For example:

Rails::Html::Sanitizer.white_list_sanitizer.new.sanitize("\x80".force_encoding("UTF-8"))

Results in:

ArgumentError: invalid byte sequence in UTF-8
  from .rbenv/versions/2.1.5/lib/ruby/gems/2.1.0/gems/nokogiri-1.6.6.2/lib/nokogiri/html/document_fragment.rb:25:in `strip'
  from .rbenv/versions/2.1.5/lib/ruby/gems/2.1.0/gems/nokogiri-1.6.6.2/lib/nokogiri/html/document_fragment.rb:25:in `initialize'
  from .rbenv/versions/2.1.5/lib/ruby/gems/2.1.0/gems/loofah-2.0.1/lib/loofah/html/document_fragment.rb:23:in `new'
  from .rbenv/versions/2.1.5/lib/ruby/gems/2.1.0/gems/loofah-2.0.1/lib/loofah/html/document_fragment.rb:23:in `parse'
  from .rbenv/versions/2.1.5/lib/ruby/gems/2.1.0/gems/loofah-2.0.1/lib/loofah.rb:42:in `fragment'
  from .rbenv/versions/2.1.5/lib/ruby/gems/2.1.0/gems/rails-html-sanitizer-1.0.1/lib/rails/html/sanitizer.rb:109:in `sanitize'

The root of the problem lies in the old mysql library, which is failing to specify the correct encoding on the data as it is read from the database. Maybe people still using the old mysql library just need to be told to switch to the mysql2 gem....

[ncri]

#33 fixes our issue too

[kaspth]

@kindjar thanks for the detailed explanation. I think you're right that just warning about the mysql gem is the best approach.