# Interacting with Web content

## Interacting with Web content and WebExtensions

GeckoView allows embedder applications to register and run
[WebExtensions](https://developer.mozilla.org/en-US/docs/Mozilla/Add-ons/WebExtensions)
in a GeckoView instance. Extensions are the preferred way to interact
with Web content.

```{contents}
:local: true
```

### Running extensions in GeckoView

Extensions bundled with applications can be provided in a folder in the
`/assets` section of the APK. Like ordinary extensions, every
extension bundled with GeckoView requires a
[manifest.json](https://developer.mozilla.org/en-US/docs/Mozilla/Add-ons/WebExtensions/manifest.json)
file.

To locate files bundled with the APK, GeckoView provides a shorthand
`resource://android/` that points to the root of the APK.

E.g. `resource://android/assets/messaging/` will point to the
`/assets/messaging/` folder present in the APK.

Note: Every installed extension will need an
[id](https://developer.mozilla.org/en-US/docs/Mozilla/Add-ons/WebExtensions/manifest.json/browser_specific_settings)
and
[version](https://developer.mozilla.org/en-US/docs/Mozilla/Add-ons/WebExtensions/manifest.json/version)
specified in the `manifest.json` file.

To install a bundled extension in GeckoView, simply call
[WebExtensionController.installBuiltIn](<https://mozilla.github.io/geckoview/javadoc/mozilla-central/org/mozilla/geckoview/WebExtensionController.html#installBuiltIn(java.lang.String)>).

```java
runtime.getWebExtensionController()
  .installBuiltIn("resource://android/assets/messaging/")
```

Note that the lifetime of the extension is not tied with the lifetime of
the
[GeckoRuntime](https://mozilla.github.io/geckoview/javadoc/mozilla-central/org/mozilla/geckoview/GeckoRuntime.html)
instance. The extension persists even when your app is restarted.
Installing at every start up is fine, but it could be slow. To avoid
installing multiple times you can use `WebExtensionRuntime.ensureBuiltIn`,
which will only install if the extension is not installed yet.

```java
runtime.getWebExtensionController()
  .ensureBuiltIn("resource://android/assets/messaging/", "messaging@example.com")
  .accept(
        extension -> Log.i("MessageDelegate", "Extension installed: " + extension),
        e -> Log.e("MessageDelegate", "Error registering WebExtension", e)
  );
```

### Communicating with Web Content

GeckoView allows bidirectional communication with Web pages through
extensions.

When using GeckoView, [native
messaging](https://developer.mozilla.org/en-US/docs/Mozilla/Add-ons/WebExtensions/Native_messaging#Exchanging_messages)
can be used for communicating to and from the browser.

- [runtime.sendNativeMessage](https://developer.mozilla.org/en-US/docs/Mozilla/Add-ons/WebExtensions/API/runtime/sendNativeMessage)
  for one-off messages.
- [runtime.connectNative](https://developer.mozilla.org/en-US/docs/Mozilla/Add-ons/WebExtensions/API/runtime/connectNative)
  for connection-based messaging.

Note: these APIs are only available when the `geckoViewAddons`
[permission](https://developer.mozilla.org/en-US/docs/Mozilla/Add-ons/WebExtensions/manifest.json/permissions)
is present in the `manifest.json` file of the extension.

#### One-off messages

The easiest way to send messages from a [content
script](https://developer.mozilla.org/en-US/docs/Mozilla/Add-ons/WebExtensions/Content_scripts)
or a [background
script](https://developer.mozilla.org/en-US/docs/Mozilla/Add-ons/WebExtensions/Anatomy_of_a_WebExtension#Background_scripts)
is using
[runtime.sendNativeMessage](https://developer.mozilla.org/en-US/docs/Mozilla/Add-ons/WebExtensions/API/runtime/sendNativeMessage).

Note: Ordinarily, native extensions would use a [native
manifest](https://developer.mozilla.org/en-US/docs/Mozilla/Add-ons/WebExtensions/Native_messaging#App_manifest)
to define what native app identifier to use. For GeckoView this is *not*
needed, the `nativeApp` parameter in `setMessageDelegate` will be
use to determine what native app string is used.

#### Messaging Example

To receive messages from the background script, call
[setMessageDelegate](<https://mozilla.github.io/geckoview/javadoc/mozilla-central/org/mozilla/geckoview/WebExtension.html#setMessageDelegate(org.mozilla.geckoview.WebExtension.MessageDelegate,java.lang.String)>)
on the
[WebExtension](https://mozilla.github.io/geckoview/javadoc/mozilla-central/org/mozilla/geckoview/WebExtension.html)
object.

```java
extension.setMessageDelegate(messageDelegate, "browser");
```

[SessionController.setMessageDelegate](<https://mozilla.github.io/geckoview/javadoc/mozilla-central/org/mozilla/geckoview/WebExtension.SessionController.html#setMessageDelegate(org.mozilla.geckoview.WebExtension,org.mozilla.geckoview.WebExtension.MessageDelegate,java.lang.String)>)
allows the app to receive messages from content scripts.

```java
session.getWebExtensionController()
    .setMessageDelegate(extension, messageDelegate, "browser");
```

Note: the `"browser"` parameter in the code above determines what
native app id the extension can use to send native messages.

Note: extension can only send messages from content scripts if
explicitly authorized by the app by adding
`nativeMessagingFromContent` in the manifest.json file, e.g.

```json
"permissions": [
  "nativeMessaging",
  "nativeMessagingFromContent",
  "geckoViewAddons"
]
```

#### Example

Let’s set up an activity that registers an extension located in the
`/assets/messaging/` folder of the APK. This activity will set up a
[MessageDelegate](https://mozilla.github.io/geckoview/javadoc/mozilla-central/org/mozilla/geckoview/WebExtension.MessageDelegate.html)
that will be used to communicate with Web Content.

You can find the full example here:
{searchfox}`MessagingExample <mobile/android/examples/messaging_example>`.

##### Activity.java

```java
WebExtension.MessageDelegate messageDelegate = new WebExtension.MessageDelegate() {
    @Nullable
    public GeckoResult<Object> onMessage(final @NonNull String nativeApp,
                                         final @NonNull Object message,
                                         final @NonNull WebExtension.MessageSender sender) {
        // The sender object contains information about the session that
        // originated this message and can be used to validate that the message
        // has been sent from the expected location.

        // Be careful when handling the type of message as it depends on what
        // type of object was sent from the WebExtension script.
        if (message instanceof JSONObject) {
            // Do something with message
        }
        return null;
    }
};

// Let's make sure the extension is installed
runtime.getWebExtensionController()
        .ensureBuiltIn(EXTENSION_LOCATION, "messaging@example.com").accept(
            // Set delegate that will receive messages coming from this extension.
            extension -> session.getWebExtensionController()
                    .setMessageDelegate(extension, messageDelegate, "browser"),
            // Something bad happened, let's log an error
            e -> Log.e("MessageDelegate", "Error registering extension", e)
        );
```

Now add the `geckoViewAddons`, `nativeMessaging` and
`nativeMessagingFromContent` permissions to your `manifest.json`
file.

##### /assets/messaging/manifest.json

```json
{
  "manifest_version": 2,
  "name": "messaging",
  "version": "1.0",
  "description": "Example messaging web extension.",
  "browser_specific_settings": {
    "gecko": {
      "id": "messaging@example.com"
    }
  },
  "content_scripts": [
    {
      "matches": ["*://*.twitter.com/*"],
      "js": ["messaging.js"]
    }
  ],
  "permissions": [
    "nativeMessaging",
    "nativeMessagingFromContent",
    "geckoViewAddons"
  ]
}
```

And finally, write a content script that will send a message to the app
when a certain event occurs. For example, you could send a message
whenever a [WPA
manifest](https://developer.mozilla.org/en-US/docs/Web/Manifest) is
found on the page. Note that our `nativeApp` identifier used for
`sendNativeMessage` is the same as the one used in the
`setMessageDelegate` call in [Activity.java](#activityjava).

##### /assets/messaging/messaging.js

```JavaScript
let manifest = document.querySelector("head > link[rel=manifest]");
if (manifest) {
     fetch(manifest.href)
        .then(response => response.json())
        .then(json => {
             let message = {type: "WPAManifest", manifest: json};
             browser.runtime.sendNativeMessage("browser", message);
        });
}
```

You can handle this message in the `onMessage` method in the
`messageDelegate` [above](#activityjava).

```java
@Nullable
public GeckoResult<Object> onMessage(final @NonNull String nativeApp,
                                     final @NonNull Object message,
                                     final @NonNull WebExtension.MessageSender sender) {
    if (message instanceof JSONObject) {
        JSONObject json = (JSONObject) message;
        try {
            if (json.has("type") && "WPAManifest".equals(json.getString("type"))) {
                JSONObject manifest = json.getJSONObject("manifest");
                Log.d("MessageDelegate", "Found WPA manifest: " + manifest);
            }
        } catch (JSONException ex) {
            Log.e("MessageDelegate", "Invalid manifest", ex);
        }
    }
    return null;
}
```

Note that, in the case of content scripts, `sender.session` will be a
reference to the `GeckoSession` instance from which the message
originated. For background scripts, `sender.session` will always be
`null`.

Also note that the type of `message` will depend on what was sent from
the extension.

The type of `message` will be `JSONObject` when the extension sends
a javascript object, but could also be a primitive type if the extension
sends one, e.g. for

```javascript
runtime.browser.sendNativeMessage("browser", "Hello World!");
```

the type of `message` will be `java.util.String`.

### Connection-based messaging

For more complex scenarios or for when you want to send messages *from*
the app to the extension,
[runtime.connectNative](https://developer.mozilla.org/en-US/docs/Mozilla/Add-ons/WebExtensions/API/runtime/connectNative)
is the appropriate API to use.

`connectNative` returns a
[runtime.Port](https://developer.mozilla.org/en-US/docs/Mozilla/Add-ons/WebExtensions/API/runtime/Port)
that can be used to send messages to the app. On the app side,
implementing
[MessageDelegate#onConnect](<https://mozilla.github.io/geckoview/javadoc/mozilla-central/org/mozilla/geckoview/WebExtension.MessageDelegate.html#onConnect(org.mozilla.geckoview.WebExtension.Port)>)
will allow you to receive a
[Port](https://mozilla.github.io/geckoview/javadoc/mozilla-central/org/mozilla/geckoview/WebExtension.Port.html)
object that can be used to receive and send messages to the extension.

The following example can be found
{searchfox}`here <mobile/android/examples/port_messaging_example>`.

For this example, the extension side will do the following:

- open a port on the background script using `connectNative`
- listen on the port and log to console every message received
- send a message immediately after opening the port.

#### /assets/messaging/background.js

```JavaScript
// Establish connection with app
let port = browser.runtime.connectNative("browser");
port.onMessage.addListener(response => {
    // Let's just echo the message back
    port.postMessage(`Received: ${JSON.stringify(response)}`);
});
port.postMessage("Hello from WebExtension!");
```

On the app side, following the [above](#activityjava) example,
`onConnect` will be storing the `Port` object in a member variable
and then using it when needed.

```java
private WebExtension.Port mPort;

@Override
protected void onCreate(Bundle savedInstanceState) {
    // ... initialize GeckoView

    // This delegate will handle all communications from and to a specific Port
    // object
    WebExtension.PortDelegate portDelegate = new WebExtension.PortDelegate() {
        public WebExtension.Port port = null;

        public void onPortMessage(final @NonNull Object message,
                                  final @NonNull WebExtension.Port port) {
            // This method will be called every time a message is sent from the
            // extension through this port. For now, let's just log a
            // message.
            Log.d("PortDelegate", "Received message from WebExtension: "
                    + message);
        }

        public void onDisconnect(final @NonNull WebExtension.Port port) {
            // After this method is called, this port is not usable anymore.
            if (port == mPort) {
                mPort = null;
            }
        }
    };

    // This delegate will handle requests to open a port coming from the
    // extension
    WebExtension.MessageDelegate messageDelegate = new WebExtension.MessageDelegate() {
        @Nullable
        public void onConnect(final @NonNull WebExtension.Port port) {
            // Let's store the Port object in a member variable so it can be
            // used later to exchange messages with the WebExtension.
            mPort = port;

            // Registering the delegate will allow us to receive messages sent
            // through this port.
            mPort.setDelegate(portDelegate);
        }
    };

    runtime.getWebExtensionController()
        .ensureBuiltIn("resource://android/assets/messaging/", "messaging@example.com")
        .accept(
            // Register message delegate for background script
            extension -> extension.setMessageDelegate(messageDelegate, "browser"),
            e -> Log.e("MessageDelegate", "Error registering WebExtension", e)
        );

    // ... other
}
```

For example, let’s send a message to the extension every time the user
long presses on a key on the virtual keyboard, e.g. on the back button.

```java
@Override
public boolean onKeyLongPress(int keyCode, KeyEvent event) {
    if (mPort == null) {
        // No extension registered yet, let's ignore this message
        return false;
    }

    JSONObject message = new JSONObject();
    try {
        message.put("keyCode", keyCode);
        message.put("event", KeyEvent.keyCodeToString(event.getKeyCode()));
    } catch (JSONException ex) {
        throw new RuntimeException(ex);
    }

    mPort.postMessage(message);
    return true;
}
```

This allows bidirectional communication between the app and the
extension.

[geckoruntime]: https://mozilla.github.io/geckoview/javadoc/mozilla-central/org/mozilla/geckoview/GeckoRuntime.html
[runtime.sendnativemessage]: https://developer.mozilla.org/en-US/docs/Mozilla/Add-ons/WebExtensions/API/runtime/sendNativeMessage
[webextension]: https://mozilla.github.io/geckoview/javadoc/mozilla-central/org/mozilla/geckoview/WebExtension.html