# Debugging Firefox with Pernosco

This page is intended to help Firefox/Gecko developers get started using [Pernosco](https://pernos.co/)
to debug Firefox.

Mozilla employees have a paid subscription to Pernosco, and are encouraged to use it.
Contributors can use Pernosco as well. [A free account](https://pernos.co/pricing/)
comes with 5 uploads to see if it is suitable for your workflow.

## Prerequisites

- You need the ability to produce an {code}`rr` trace. See {doc}`debugging_firefox_with_rr`
- Employees, you can also generate traces for Pernosco for some CI job failures via
  Treeherder.

## Workflow

At a high level, working with Pernosco consists of recording a trace then uploading
the trace for processing.

### Install your Pernosco configuration

If you are using your own Pernoso account, follow the instructions
[here](https://pernos.co/account) to install your credentials.

Mozilla employees, see the [Pernosco install and configuration doc](https://docs.google.com/document/d/1cb-mkNyJYMudpS0a-R6zxy0kOMIRUSKrxxL-E9zFGKE/edit)
for credentials.

In general, you will want:

- Your email in {code}`~/.config/pernosco/user`
- Your group in {code}`~/.config/pernosco/group`
- Your key in {code}`~/.config/pernosco/user_secret_key`

### Gather a Local Trace

In general, to gather a trace follow the instruction for recording {code}`rr` traces
in {doc}`debugging_firefox_with_rr`. However, to gather a local rr trace that’s
compatible with Pernosco you need to disable some incompatible CPU features (currently
only AVX-512)

On an Intel machine, you can disable this at the command line. To simplify things a
shell script {code}`pernosco-record` can be created like so,

```bash
#!/bin/bash

rr record --disable-avx-512 "$@"
```

replacing plain {code}`rr` in command lines ({code}`mach --debugger pernosco-record`)

On AMD Zen Machines, you can disable AVX-512 and 5 level page tables at boot.

On a Debian derived distro this involves editing {code}`/etc/default/grub` and
adding {code}`clearcpuid=304,no5lvl` to {code}`GRUB_CMDLINE_LINUX_DEFAULT`, then
running {code}`sudo update-grub` after.

### Upload the trace

Find the trace you’re interested in {code}`~/.local/share/rr/<trace>`, and call {code}`pernosco-submit`

```bash
pernosco-submit upload ~/.local/share/rr/<trace> PATH_TO_CENTRAL --title "Name Of Trace"
```

where {code}`PATH TO CENTRAL` is the path to the source of your build, checked out
to the revision matching the build.

The last trace recorded is always symlinked, so you can use {code}`~/.local/share/rr/latest-trace` too.

### Wait for email

After the trace is processed, you'll get an email with a link to your trace. Good
hunting!

## Treeherder Workflow

You can also try to get traces generated via the Treeherder interface. On a failing
test that is eligible, in "Artifacts and Debugging Tools", you'll see a link to

```{image} ../img/record-this-failure.png
:width: 35 %
```

This link brings you to the self-service reproducer:

```{image} ../img/self-service.png
:width: 60 %
```

Select a test and a count, and whether or not chaos-mode should be used, then click
reproduce. At this point there is a **long** wait; the page will notify you when
reproduction has started with an {code}`alert` dialog.