profiling - My CUDA nvprof 'API Trace' and 'GPU Trace' are not synchronized - what to do? -

i'm using cuda 7.0 profiler, nvprof, profile process making cuda calls:

$ nvprof -o out.nvprof /path/to/my/app 

later, generate 2 traces: 'api trace' (what happens on host cpu, e.g. cuda runtime calls , ranges mark) , 'gpu trace' (kernel executions, memsets, h2ds, d2hs , on):

$ nvprof -i out.nvprof --print-api-trace --csv 2>&1 | tail -n +2 > api-trace.csv $ nvprof -i out.nvprof --print-gpu-trace --csv 2>&1 | tail -n +2 > gpu-trace.csv 

every record in each of traces has timestamp (or start , end time). thing is, time value 0 in these 2 traces not same: gpu trace time-0 point seems signify when first operation on gpu triggered relevant process begins execute, while api trace's time-0 point seems beginning of process execution, or sometime thereabouts.

i've noticed when use nvvp , import out.nvprof, values corrected, say, start time of first gpu op not 0, more realistic.

how obtain correct offset between 2 traces?

it may not obvious the nvprof documentation, possible specify both --print-gpu-trace , --print-api-trace when requesting output nvprof, whether profiling app or extracting information captured profiler output file.

if profiling app, following should generate "harmonized" timeline both api activity , gpu activity:

nvprof --print-gpu-trace --print-api-trace ./my_app 

you can save output using --log-file option.

similarly, if extracting output captured output file (not same thing log file), can following:

nvprof -i profiler_out_file --print-gpu-trace --print-api-trace ... 

where profiler_out_file should name of file saved using the nvprof -o ... option.

printing both traces same command essential here 2 (combined) timelines begin @ same point in time; if issue 2 commands, each printing trace, may not 'harmonized'.