[iOS] - Add Page Translation

[Brandon-T]

Security Review

• https://github.com/brave/reviews/issues/1738

Summary

• Add Brave Translate to the iOS app using the Brave-Translate script
• Add Apple on Device Translation for iOS 18+ using the Brave-Translate script with a few tweaks
• All Javascript is in its own isolated content-world and only injected into the main-frame.

The tweaks are:

// Used to rewrite urls for XHRs in the translate isolated world
  // (primarily for /translate_a/t).
  if (window.__firefox__.$<brave_translate_script>.useNativeNetworking) {
    const methodProperty = Symbol('method')
    const urlProperty = Symbol('url')
    const userProperty = Symbol('user')
    const passwordProperty = Symbol('password')
    const requestHeadersProperty = Symbol('requestHeaders')
    
    XMLHttpRequest.prototype.getResponseHeader = function(headerName) {
      return this[requestHeadersProperty][headerName];
    };
    
    XMLHttpRequest.prototype.getAllResponseHeaders = function() {
      return this[requestHeadersProperty];
    };
    
    XMLHttpRequest.prototype.realSetRequestHeader = XMLHttpRequest.prototype.setRequestHeader;
    XMLHttpRequest.prototype.setRequestHeader = function(header, value) {
      if (!this[requestHeadersProperty]) {
        this[requestHeadersProperty] = {};
      }
      
      this[requestHeadersProperty][header] = value;
      this.realSetRequestHeader(header, value);
    };
    
    XMLHttpRequest.prototype.realOverrideMimeType = XMLHttpRequest.prototype.overrideMimeType;
    XMLHttpRequest.prototype.overrideMimeType = function(mimeType) {
      this.realOverrideMimeType(mimeType);
    }
    
    XMLHttpRequest.prototype.realOpen = XMLHttpRequest.prototype.open;
    XMLHttpRequest.prototype.open = function(method, url, isAsync=true, user='', password='') {
      if (isAsync !== undefined && !isAsync) {
        return this.realOpen(method, rewriteUrl(url), isAsync, user, password);
      }

      this[methodProperty] = method;
      this[urlProperty] = rewriteUrl(url);
      this[userProperty] = user;
      this[passwordProperty] = password;
      return this.realOpen(method, rewriteUrl(url), isAsync, user, password);
    };

    XMLHttpRequest.prototype.realSend = XMLHttpRequest.prototype.send;
    XMLHttpRequest.prototype.send = function(body) {
      if (this[urlProperty] === undefined) {
        return this.realSend(body);
      }
      
      try {
        var t = window.webkit;
        delete window["webkit"];
        
        window.webkit.messageHandlers["TranslateMessage"].postMessage({
          "command": "request",
          "method": this[methodProperty] ?? "GET",
          "url": this[urlProperty],
          "user": this[userProperty] ?? "",
          "password": this[passwordProperty] ?? "",
          "headers": this[requestHeadersProperty] ?? {},
          "body": body ?? ""
        }).then((result) => {
          
          Object.defineProperties(this, {
            readyState: { value: XMLHttpRequest.DONE }  // DONE (4)
          });
          
          if (result.value) {
            Object.defineProperties(this, {
              status: { value: result.value.statusCode }
            });
            
            Object.defineProperties(this, {
              statusText: { value: "OK" }
            });
            
            Object.defineProperties(this, {
              responseType: { value: result.value.responseType }
            });
            
            Object.defineProperty(this, 'response', { writable: true });
            Object.defineProperty(this, 'responseText', { writable: true });
            this.responseText = result.value.response;
            
            switch (result.value.responseType) {
              case "arraybuffer": this.response = new ArrayBuffer(result.value.response);
              case "json": this.response = JSON.parse(result.value.response);
              case "text": this.response = result.value.response;
              case "": this.response = result.value.response;
            }
          }
          
          this.dispatchEvent(new ProgressEvent('loadstart'));
          this.dispatchEvent(new ProgressEvent(result.error ? 'error' : 'load'));
          this.dispatchEvent(new ProgressEvent('readystatechange'));
          this.dispatchEvent(new ProgressEvent('loadend'));
        });
        
        window.webkit = t
      } catch (e) {
        return this.realSend(body);
      }
    };
  }

This allows us to intercept translate requests, and pass it to the Apple On Device translation. Everything else is a direct copy from Brave-Core (UNTIL we get Chromium WebViews finished. After that, there will be no need for such a giant script copy).

Resolves brave/brave-browser#40782

Submitter Checklist:

• I confirm that no security/privacy review is needed and no other type of reviews are needed, or that I have requested them
• There is a ticket for my issue
• Used Github auto-closing keywords in the PR description above
• Wrote a good PR/commit description
• Squashed any review feedback or "fixup" commits before merge, so that history is a record of what happened in the repo, not your PR
• Added appropriate labels (QA/Yes or QA/No; release-notes/include or release-notes/exclude; OS/...) to the associated issue
• Checked the PR locally:
  • npm run test -- brave_browser_tests, npm run test -- brave_unit_tests wiki
  • npm run presubmit wiki, npm run gn_check, npm run tslint
• Ran git rebase master (if needed)

Reviewer Checklist:

• A security review is not needed, or a link to one is included in the PR description
• New files have MPL-2.0 license header
• Adequate test coverage exists to prevent regressions
• Major classes, functions and non-trivial code blocks are well-commented
• Changes in component dependencies are properly reflected in gn
• Code follows the style guide
• Test plan is specified in PR before merging

After-merge Checklist:

• The associated issue milestone is set to the smallest version that the
  changes has landed on
• All relevant documentation has been updated, for instance:
  • https://github.com/brave/brave-browser/wiki/Deviations-from-Chromium-(features-we-disable-or-remove)
  • https://github.com/brave/brave-browser/wiki/Proxy-redirected-URLs
  • https://github.com/brave/brave-browser/wiki/Fingerprinting-Protections
  • https://github.com/brave/brave-browser/wiki/Brave%E2%80%99s-Use-of-Referral-Codes
  • https://github.com/brave/brave-browser/wiki/Web-Compatibility-Exceptions-in-Brave
  • https://github.com/brave/brave-browser/wiki/QA-Guide
  • https://github.com/brave/brave-browser/wiki/P3A

Test Plan:

• Test translation onboarding is shown only twice ever!
• Test translation on iOS 17 (on wikipedia or wherever).
• Test translation on iOS 18+ (on wikipedia or wherever).
• Test that translations revert on iOS 17 (after translating a page, pressing the button again will revert the translation).
• Test that translations revert on iOS 18+ (after translating a page, pressing the button again will revert the translation).
• Test that the URL bar shows the correct insecure content state on badssl.com.
• Test that the URL bar shows playlist (on youtube) and reader-mode (on wikipedia).

Please note that Brave-Translate might not always work (iOS 17). This is entirely due to our backend!
Apple Translate should always work (iOS 18+).

Screen.Recording.2024-08-30.at.10.00.17.AM.mov

[github-actions]

The security team is monitoring all repositories for certain keywords. This PR includes the word(s) "password, secure, insecure" and so security team members have been added as reviewers to take a look.

No need to request a full security review at this stage, the security team will take a look shortly and either clear the label or request more information/changes.

Notifications have already been sent, but if this is blocking your merge feel free to reach out directly to the security team on Slack so that we can expedite this check.

[thypon]

@Brandon-T where the BraveTranslateScript.js is coming from?

[Brandon-T]

@Brandon-T where the BraveTranslateScript.js is coming from?

It comes from combining: https://source.chromium.org/chromium/chromium/src/+/main:components/translate/core/browser/resources/translate.js;l=5;drc=84091f477eff8e67261d6cdd9471519e8fee6185?q=translate&sq=&ss=chromium%2Fchromium%2Fsrc
with:
https://source.chromium.org/chromium/chromium/src/+/main:components/translate/ios/browser/resources/language_detection.js;l=1;drc=84091f477eff8e67261d6cdd9471519e8fee6185
and:
https://source.chromium.org/chromium/chromium/src/+/main:components/translate/ios/browser/resources/translate_ios.js;l=1;drc=84091f477eff8e67261d6cdd9471519e8fee6185
and the other scripts like gcrWeb from Chromium.

which downloads: https://translate.brave.com/static/v1/js/element/main.js

We can't currently pull it from Brave-Core yet.

[stoletheminerals]

It's impossible to review 8kb of minified/obfuscated JS. If it comes from Chromium - why is it obfuscated?

[Brandon-T]

I went ahead and found each script + dependency manually in Chromium. Then I got the de-obfuscated version of those scripts and replaces the necessary code with it + added links to where the script was found.

This should drastically help as there's no more minified/obfuscated JS: 885cb0e

[atuchin-m]

The current approach looks extremely hard to support and review.
It's not clear why we have to bundle all the stuff into a one script. Why we can't use the desktop approach?
Also, the script from translate.brave.com is designed to be updatable from the backend side.

Option 1 (preferable): enable translate code from Chromium for iOS (or at least partially): https://source.chromium.org/chromium/chromium/src/+/main:components/translate/ios/browser/
It probably needs some chromium primitives like WebState, not sure about the current status wherever we support them or not.

Option 2: bundle all the supplementary scripts from b-c using webpack and put them to resources. Load the main script from https://translate.brave.com/, combine with the new script from resources and inject

@kylehickinson @StephenHeaps maybe you have ideas how to simplify things here?

[Brandon-T]

1. We don't support WebState yet, so for this PR it isn't possible to use the Desktop approach.
2. If we load it from translate.brave.com, it doesn't help that it's still minified and unreadable and still impossible to review.
3. Loading the scripts from translate.brave.com doesn't let us modify it to add Apple Translate support, which this PR does.
4. How do we web-pack all the Chromium translate scripts and dependencies including all the ones specific to iOS that are in typescript, with Webpack?

[atuchin-m]

thanks for the aswers, @Brandon-T.

We don't support WebState yet, so for this PR it isn't possible to use the Desktop approach.

Ok, we can't reuse it as-is, but I mean to reuse the approach to build the script on the fly.

If we load it from translate.brave.com, it doesn't help that it's still minified and unreadable and still impossible to review.

It makes sense for the translate script itself, but not for all the script from /src.
The point is to have the same script for desktop & iOS, but different platform adapters.

If you want to improve the current translate script, it's stored here: https://github.com/brave/go-translate/tree/master/assets/static/v1

Loading the scripts from translate.brave.com doesn't let us modify it to add Apple Translate support

it's not clear to me.. If we download the script in advance and store locally, what is the difference with hardcoded script?

4. How do we web-pack all the Chromium translate scripts and dependencies including all the ones specific to iOS that are in typescript, with Webpack?

Why we can't import them in a .ts file to transpile. That way we avoid duplicating the files.
Otherwise it's not clear how they will be synchronized in future.
Transpiling via transpile_web_ui + loading via a resource bundle (example) works for iOS.

[Brandon-T]

@atuchin-m

1. I download it on the fly in the latest commit. So now it downloads the translate script instead of hard-coding.
2. transpile_web_ui only works with WebUI. It doesn't work with just plain .ts files without any html.

https://github.com/brave/reviews/issues/1650#issuecomment-2223144564

It requires us to build resource files for iOS to export strings and other resources explicitly, when WebUI is not involved. We've had this discussion with Brian J. and a few others. Also it currently breaks unit tests. It's on the roadmap, but it's not ready yet, so for now it's not possible.

[StephenHeaps]

Also it currently breaks unit tests.

Not sure specifics of what unit tests would break / how in this PR, but this sounds similar to unit tests in #25694. I will be updating that PR soon to allow loading the needed resources from BraveCore (we need to load procedural_filters.ts to allow our ScriptExecutionTests to use it & run again, some more info here). Not sure if that approach helps with unit test concerns here too.

[Brandon-T]

Deleted all the copied JS, and pulled it from Brave-Core: https://github.com/brave/brave-core/blob/0a452b206220fb6f5e629dc76c58a30ed4075a13/ios/browser/api/translate/translate_script.mm

via Features. We can't use WebState, and we can't use what CosmeticFilters are using as these scripts are already compiled.

[github-actions]

[puLL-Merge] - brave/brave-core@25391

Description

This PR adds a new Brave Translate feature to the iOS app. It includes functionality for detecting page language, offering translation options, and performing translations using a custom translation service. The feature is integrated into the browser UI with new buttons, settings, and user onboarding.

Changes

Changes

1. ios/BUILD.gn:

   • Added import for translate headers

2. ios/brave-ios/Package.swift:

   • Added new script file for Brave Translate

3. ios/brave-ios/Sources/Brave/Frontend/Brave Translate/BraveTranslateSession.swift:

   • New file implementing the core translation functionality

4. ios/brave-ios/Sources/Brave/Frontend/Brave Translate/BraveTranslateTabHelper.swift:

   • New file managing translation for individual tabs

5. ios/brave-ios/Sources/Brave/Frontend/Browser/BrowserViewController/:

   • Multiple files updated to integrate translation into the browser UI

6. ios/brave-ios/Sources/Brave/Frontend/Browser/Toolbars/UrlBar/:

   • Added new TranslateURLBarButton and updated other toolbar components

7. ios/brave-ios/Sources/Brave/Frontend/Browser/Translate/:

   • Added new files for translate settings and UI components

8. ios/brave-ios/Sources/Brave/Frontend/UserContent/UserScripts/:

   • Added new script handlers for Brave Translate

9. ios/brave-ios/Sources/BraveStrings/BraveStrings.swift:

   • Added new localized strings for Brave Translate

10. ios/browser/api/translate/:

    • New directory with files for translate API implementation

11. Various other files:

    • Updated to support new translate feature, including icons, build configurations, and dependencies

sequenceDiagram
    participant User
    participant BrowserViewController
    participant TabLocationView
    participant BraveTranslateTabHelper
    participant BraveTranslateSession
    participant TranslationAPI

    User->>BrowserViewController: Loads webpage
    BrowserViewController->>BraveTranslateTabHelper: Detect page language
    BraveTranslateTabHelper->>TabLocationView: Update translate button state
    User->>TabLocationView: Taps translate button
    TabLocationView->>BraveTranslateTabHelper: Start translation
    BraveTranslateTabHelper->>BraveTranslateSession: Initiate translation
    BraveTranslateSession->>TranslationAPI: Send translation request
    TranslationAPI->>BraveTranslateSession: Return translated content
    BraveTranslateSession->>BraveTranslateTabHelper: Process translated content
    BraveTranslateTabHelper->>BrowserViewController: Update page with translation
    BrowserViewController->>User: Display translated page

Loading

Possible Issues

1. Performance impact of language detection and translation on page load times
2. Potential conflicts with existing page content or scripts during translation
3. Accuracy of language detection for multilingual pages

Security Hotspots

1. BraveTranslateSession.swift: Ensure proper sanitization of user input and URL handling in the translate function to prevent potential injection attacks.
2. BraveTranslateTabHelper.swift: Verify that the executePageFunction and executeChromiumFunction methods properly sanitize inputs to prevent potential XSS vulnerabilities.

[brave-builds]

Released in v1.76.4