Trace Processing

Design

Profilo comes with a Python library which can parse traces into a semantic trace object. This trace object can then be traversed and examined to extract meaningful metrics.

Trace model

The semantic trace objects follow a hierarchy (or trace model):

Traces consist of execution units.

Execution units consist of blocks. Execution units are usually individual threads in Profilo traces.

Blocks consist of points. A block always has a start and end point. Points within a block can share timestamps but are uniquely identifiable.

Edges can indicate change of control flow between any two points. There's an implicit edge between the start and end points of a block, as well as between the boundary points of consecutive blocks in an execution unit. All other edges must be explicitly created.

All of the above objects can have "annotations" (props in API) attached to them. Annotations are key-value pairs with very loose typing.

--Execution Unit 1--------------------
Block A Block B
[s.--> .e]-->[s --> .e]
| ^ |
v | |
[s-->e] |
Block C |
|
---------------------|----------------
|
--Execution Unit 2---|----------------
|
v Block D
[s --> e]
--------------------------------------

In the diagram above, the edges are as follows:

Edge SourceEdge TargetType
Block A, start pointBlock A, end pointImplicit
Block B, start pointBlock B, end pointImplicit
Block C, start pointBlock C, end pointImplicit
Block D, start pointBlock D, end pointImplicit
Block A, 2nd pointBlock C, start pointExplicit, child call
Block C, end pointBlock A, 3rd pointExplicit, child return
Block B, 2nd pointBlock D, start pointExplicit, async call

Workflow

The complete flow for tracing-downloading-analyzing is as follows:

  1. Ensure you have a environment which contains the dependencies listed in python/requirements.txt. For example, you can use virtualenv like so:

    virtualenv -p python3 env
    source env/bin/activate
    pip3 install -r requirements.txt
  2. Start and stop a trace in the app. (You can use the sample app to play with traces)

  3. Download the trace on a host. This can be done in a few ways:

    a. Navigate to python/ and run

    python3 -m profilo.profilo pull_traces --last <your_application_package>

    In this case:

    python3 -m profilo.profilo pull_traces --last com.facebook.profilo.sample

b. Navigate to python/profilo/device, start an interactive Python shell, and use the "device" module directly:

```
import device
device.pull_last_trace("com.example.package.name")
```

Both approaches will result in a file with the same name as the trace to be created in the current directory.

  1. Analyze the data. A sample demo script is provided in python/profilo/workflow_demo.py.

    You can run from within the python/ folder via python3 -m profilo.workflow_demo --help. Don't forget to activate your virtualenv first, if you are using one.

    The workflow demo contains logic to perform analysis on stacks, blocks, and system counters.

    a. Stacks: Parses the sampled stacks from a trace file into a format similar to that of DTrace. This allows for visualizations such as FlameGraph.

    Note that functions for which we couldn't find symbols will just show up as numbers.

    b. Blocks: Parses the instrumented start-end blocks from a trace file and outputs some per-thread statistics (mean, median, std) about execution times in csv format.

    c. System counters: Parses system counters from a trace file and sorts them by timestamp so as to obtain a time series. A command line argument must be used to define the output path for the generated graphs.