%\VignetteIndexEntry{Errors, Logs and Debugging}
%\VignetteKeywords{parallel, Infrastructure}
%\VignettePackage{BiocParallel}
%\VignetteEngine{knitr::knitr}


\documentclass{article}

<<style, eval=TRUE, echo=FALSE, results="asis">>=
BiocStyle::latex()
@ 

\newcommand{\BiocParallel}{\Biocpkg{BiocParallel}}

\title{Errors, Logs and Debugging in \BiocParallel} 
\author{Valerie Obenchain and Martin Morgan}
\date{Edited: May 13, 2015; Compiled: \today}

\begin{document}

\maketitle

\tableofcontents

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Introduction}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

This vignette is part of the \BiocParallel{} package and focuses on error
handling and logging. A section at the end demonstrates how the two can be used
together as part of an effective debugging routine.

\BiocParallel{} provides a unified interface to the parallel infrastructure in
several packages including \CRANpkg{snow}, \CRANpkg{parallel},
\CRANpkg{BatchJobs} and \CRANpkg{foreach}.  When implementing error handling in
\BiocParallel{} the primary goals were to enable the return of partial results
when an error is thrown (vs just the error) and to establish logging on the
workers. In cases where error handling existed, such as \CRANpkg{BatchJobs} and
\CRANpkg{foreach}, those behaviors were preserved.  Clusters created with
\CRANpkg{snow} and \CRANpkg{parallel} now have flexible error handling and
logging available through \Rcode{SnowParam} and \Rcode{MulticoreParam} objects.

In this document the term ``job'' is used to describe a single call to a
bp*apply function (e.g., the \Rcode{X} in \Rcode{bplapply}). A ``job'' consists
of one or more ``tasks'', where each ``task'' is run separately on a worker. 

The \Rpackage{BiocParallel} package is available at bioconductor.org
and can be downloaded via \Rcode{biocLite}:

<<biocLite, eval=FALSE>>=
source("http://bioconductor.org/biocLite.R")
biocLite("BiocParallel")
@ 

Load the package:

<<load>>=
library(BiocParallel)
@

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Error Handling}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\subsection{Catching errors} 

By default, \BiocParallel{} attempts all computations and returns any warnings
and errors along with successful results.  The \Rcode{stop.on.error} field
controls if the job is terminated as soon as one task throws an error. This is
useful when debugging or when running large jobs (many tasks) and you want to
be notified of an error before all runs complete.

\Rcode{stop.on.error} is \Rcode{FALSE} by default.

<<errors_constructor, eval=FALSE>>=
param <- SnowParam()
param
@

The field can be set when constructing the param or modified with the 
\Rcode{bpstopOnError} accessor.

<<errors_stopOnError, eval=FALSE>>=
param <- SnowParam(2, stop.on.error = TRUE)
param
bpstopOnError(param) <- FALSE
@

In this example \Rcode{X} is length 6. By default, the elements of \Rcode{X}
are divided as evenly as possible over the number of workers and run in chunks.
To more clearly demonstrate the affect of \Rcode{stop.on.error} the number of
tasks is set equal to the length of \Rcode{X}. This forces each element of
\Rcode{X} to be executed separately (6 tasks) vs chunked.

<<errors_6tasksA_stopOnError, eval=FALSE>>=
X <- list(1, 2, "3", 4, 5, 6)
param <- SnowParam(3, tasks = length(X), stop.on.error = TRUE)
@

The output list contains results for tasks 1 and 2 and an error for task 3.
Tasks 4, 5, and 6 are not attempted.

<<errors_6tasksB_stopOnError, eval=FALSE>>=
bplapply(X, sqrt, BPPARAM = param)
@

Next we look at an example where the elements of \Rcode{X} are grouped instead
of run separately. The default value for \Rcode{tasks} is 0 which means 'X' is
split as evenly as possible across the number of workers. There are 3
workers so the first task consists of list(1, 2), the second is list("3", 4) 
and the third is list(5, 6). 

<<errors_3tasksA_stopOnError, eval=FALSE>>=
X <- list(1, 2, "3", 4, 5, 6)
param <- SnowParam(3, stop.on.error = TRUE)
@

To simulate a longer running computation sleep time is added to 'FUN'. The
sleep forces task 2 to finish before task 3.

<<errors_3tasks_fun_stopOnError, eval=FALSE>>=
FUN <- function(i) { Sys.sleep(i); sqrt(i) }
@

The output shows an error in task 2 (vs 3 in the previous example) and a result
for '4' is included because it was part of the second task.

<<errors_3tasksB_stopOnError, eval=FALSE>>=
bplapply(X, FUN, BPPARAM = param)
@

Side Note: Results are collected from workers as they finish which is not
necessarily the same order in which they were loaded.  Depending on how tasks
are divided among workers it is possible that the task with an error completes
after all others. In that situation the output will includes all results along
with the error message and it may appear that \Rcode{stop.on.error} is not
doing much good. This is simply a heads up that the usefulness of
\Rcode{stop.on.error} may vary with run time and distribution of tasks over
workers.

\subsection{Identify failures with \Rcode{bpok()}}

The \Rcode{bpok()} function is a quick way to determine which (if any)
tasks failed. In this example the second element fails.

<<errors_bpok_bplapply, eval=FALSE>>=
res <- bplapply(list(1, "2", 3), sqrt)
res
@

\Rcode{bpok} returns TRUE if the task was successful.

<<errors_bpok, eval=FALSE>>=
bpok(res)
@

Once errors are identified with \Rcode{bpok} the traceback can be retrieved
with the \Rcode{attr} function. This is possible because errors are returned
as \Rcode{condition} objects with the traceback as an attribute.

<<errors_traceback, eval=FALSE>>=
fail <- !bpok(res)
tail(attr(res[[2]], "traceback"))
@

\subsection{Rerun failed tasks with \Rcode{BPREDO}}

Tasks can fail due to hardware problems or bugs in the input data.  The
\BiocParallel{} functions support a \Rcode{BPREDO} (re-do) argument for
recomputing only the tasks that failed. A list of partial results and errors is
supplied to \Rcode{BPREDO} in a second call to the function. The failed
elements are identified, recomputed and inserted into the original results.

The bug in this example is the second element of 'X' which is a character when
it should be numeric.

<<errors_BPREDO, eval=FALSE>>=
X <- list(1, "2", 3)
res <- bplapply(X, sqrt)
res
@

First fix the input data.

<<errors_BPREDO_input, eval=FALSE>>=
X.redo <- list(1, 2, 3)
@

Repeat the call to \Rcode{bplapply} this time supplying the partial results
as \Rcode{BPREDO}.

<<errors_BPREDO_recall, eval=FALSE>>=
bplapply(X.redo, sqrt, BPREDO = res)
@


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Logging}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

NOTE: Logging as described in this section is supported for SnowParam,
MulticoreParam and SerialParam.

\subsection{Parameters}

Logging in \BiocParallel{} is controlled by 3 fields in the 
\Rcode{BiocParallelParam}:

\begin{verbatim}
  log:       TRUE or FALSE
  logdir:    location to write log file
  threshold: one of "TRACE", "DEBUG", "INFO", "WARN", "ERROR", "FATAL"
\end{verbatim}

When \Rcode{log = TRUE} the \CRANpkg{futile.logger} package is loaded on 
each worker. \BiocParallel{} uses a custom script on the workers to
collect log messages as well as additional statistics such as gc, runtime 
and node information. Output to stderr and stdout is also captured.

By default \Rcode{log} is FALSE and \Rcode{threshold} is {\it INFO}.

<<logs_constructor, eval=FALSE>>=
param <- SnowParam()
param
@

Turn logging on and set the threshold to {\it TRACE}.

<<logs_accessors, eval=FALSE>>=
bplog(param) <- TRUE
bpthreshold(param) <- "TRACE"
param
@

\subsection{Setting a threshold}

All thresholds defined in \CRANpkg{futile.logger} are supported: {\it
FATAL}, {\it ERROR}, {\it WARN}, {\it INFO}, {\it DEBUG} and {\it TRACE}.  All
messages greater than or equal to the severity of the threshold are shown. For
example, a threshold of {\it INFO} will print all messages tagged as {\it
FATAL}, {\it ERROR}, {\it WARN} and {\it INFO}.

In this code chunk an {\it INFO}-level message is emitted when 
\CRANpkg{futile.logger} is loaded on the workers and a {\it ERROR}-level
message when attempting the square root of a character ("2"). 

<<logs_bplapply, eval=FALSE>>=
bpthreshold(param) <- "INFO"
bplapply(list(1, "2", 3), sqrt, BPPARAM = param)
@

All user-supplied messages written in the \CRANpkg{futile.logger} syntax
are also captured. This function performs argument checking and includes 
a couple of {\it WARN} and {\it DEBUG}-level messages.

<<logs_FUN, eval=FALSE>>=
FUN <- function(i) {
  flog.debug(paste0("value of 'i': ", i))

  if (!length(i)) {
      flog.warn("'i' is missing")
      NA 
  } else if (!is(i, "numeric")) {
      flog.info("coercing to numeric")
      as.numeric(i)
  } else {
      i
  }
}
@

Turn logging on and set the threshold to {\it WARN}.

<<logs_WARN, eval=FALSE>>=
param <- SnowParam(2, log = TRUE, threshold = "WARN")
bplapply(list(1, "2", integer()), FUN, BPPARAM = param)
@

Changing the threshold to {\it DEBUG} catches all {\it WARN}, {\it INFO} 
and {\it DEBUG} messages.

<<logs_ERROR, eval=FALSE>>=
param <- SnowParam(2, log = TRUE, threshold = "DEBUG")
bplapply(list(1, "2", integer()), FUN, BPPARAM = param)
@

\subsection{Log files}

When \Rcode{log == TRUE}, log messages are written to the console by default. 
If \Rcode{logdir} is given the output is written out to files, one per task.
File names are prefixed with the name in \Rcode{bpjobname(BPPARAM)}; default
is 'BPJOB'.

\begin{verbatim}
  > param <- SnowParam(2, log = TRUE, threshold = "DEBUG", logdir = tempdir())
  > res <- bplapply(list(1, "2", integer()), FUN, BPPARAM = param)
  INFO [2015-07-08 08:47:40] loading futile.logger on workers
  > list.files(bplogdir(param))
  [1] "BPJOB.task1.log" "BPJOB.task2.log"
\end{verbatim}

Read in BPJOB.task2.log:

\begin{verbatim}
  > readLines(paste0(bplogdir(param), "/BPJOB.task2.log"))
   [1] "############### LOG OUTPUT ###############"      
   [2] "Task: 2"                                         
   [3] "Node: 2"                                         
   [4] "Timestamp: 2015-07-08 09:03:59"                  
   [5] "Success: TRUE"                                   
   [6] "Task duration: "                                 
   [7] "   user  system elapsed "                        
   [8] "  0.009   0.000   0.011 "                        
   [9] "Memory use (gc): "                               
  [10] "         used (Mb) gc trigger (Mb) max used (Mb)"
  [11] "Ncells 325664 17.4     592000 31.7   393522 21.1"
  [12] "Vcells 436181  3.4    1023718  7.9   530425  4.1"
  [13] "Log messages:"                                   
  [14] "DEBUG [2015-07-08 09:03:59] value of 'i': 2"     
  [15] "INFO [2015-07-08 09:03:59] coercing to numeric"  
  [16] "DEBUG [2015-07-08 09:03:59] value of 'i': "      
  [17] "WARN [2015-07-08 09:03:59] 'i' is missing"       
  [18] ""                                                
  [19] "stderr and stdout:"                              
  [20] "character(0)"    
\end{verbatim}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Worker timeout}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

NOTE: \Rcode{timeout} is  supported for SnowParam and MulticoreParam. 

For long running jobs or untested code it can be useful to set a time limit.
The \Rcode{timeout} field is the time, in seconds, allowed for each worker to
complete a task; default is \Rcode{Inf}. If the task takes longer than
\Rcode{timeout} a timeout error is returned.

Time can be changed during param construction with the \Rcode{timeout} arg,

<<timeout_constructor, eval=FALSE>>=
param <- SnowParam(timeout = 20)
param
@

or with the \Rcode{bptimeout} setter:

<<timeout_setter, eval=FALSE>>=
bptimeout(param) <- 2 
param
@


A timeout error is returned if 

<<timeout_bplapply, eval=FALSE>>=
fun <- function(i) {
  Sys.sleep(i)
  i
}
bplapply(1:3, fun, BPPARAM = param) 
@


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Debugging}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

Effective debugging strategies vary by problem and often involve a combination
of error handling and logging techniques.  In general, when debugging
\R{}-generated errors the traceback is often the best place to start followed
by adding debug messages to the worker function. When trouble shooting
unexpected behavior (i.e., not a formal error or warning) adding debug messages
or switching to \Rcode{SerialParam} are good approaches. Below is an overview
of these different strategies. 

\subsection{Accessing the traceback}

The traceback is a good place to start when tracking down \R{}-generated 
errors. Because the function is executed on the workers it's not accessible for
interactive debugging with functions such as \Rcode{trace} or \Rcode{debug}.
The traceback provides a snapshot of the state of the worker at the time 
the error was thrown.

This function takes the square root of the absolute value of a vector.

<<debug_sqrtabs, eval=FALSE>>=
fun1 <- function(x) {
    v <- abs(x)
    sapply(1:length(v), function(i) sqrt(v[i]))
}
@

Calling ``fun1'' with a character throws an error:

<<errors_bplapply, eval=FALSE>>=
res <- bplapply(list(c(1,3), 5, "6"), fun1)
res
@

Identify which elements failed with \Rcode{bpok}:

<<errors_bpok_fun1, eval=FALSE>>=
bpok(res)
@

The error (i.e., third element of ``res'') is a \Rcode{condition} object:

<<debug_condition, eval=FALSE>>=
is(res[[3]], "condition")
@

The traceback is an attribute of the \Rcode{condition} and can be accessed with 
the \Rcode{attr} function. 

<<debug_traceback, eval=FALSE>>=
noquote(tail(attr(res[[3]], "traceback")))
@

\subsection{Adding debug messages}

When a \Rcode{numeric()} is passed to ``fun1'' no formal error is thrown
but the length of the second list element is 2 when it should be 1.

<<debug_fun1, eval=FALSE>>=
bplapply(list(c(1,3), numeric(), 6), fun1)
@

Without a formal error we have no traceback so we'll try adding a few debug
messages to ``fun1''. The \CRANpkg{futile.logger} syntax tags messages with
different levels of severity. A message created with \Rcode{flog.debug} will
only print if the threshold is {\it DEBUG} or lower.

``fun2'' has debug statements that show the value of `x', length of `v' and 
the index `i'.

<<debug_fun1_debug, eval=FALSE>>=
fun2 <- function(x) {
    v <- abs(x)
    flog.debug(
      paste0("'x' = ", paste(x, collapse=","), ": length(v) = ", length(v))
    )
    sapply(1:length(v), function(i) {
      flog.debug(paste0("'i': ", i))
      sqrt(v[i])
    })
}
@

Create a param that logs at a threshold level of {\it DEBUG}.

<<debug_param_debug, eval=FALSE>>=
param <- SnowParam(3, log = TRUE, threshold = "DEBUG")
@

The debug messages reveal the problem occurs when `x' is \Rcode{numeric()}.
The index for \Rcode{sapply} is along `v' which in this case has length 0.
This forces `i' to take values of `1' and `0' giving an output of length 2 for
the second element (i.e., \Rcode{NA} and \Rcode{numeric(0)}).

<<debug_DEBUG, eval=FALSE>>=
param <- SnowParam(3, log = TRUE, threshold = "DEBUG")
res <- bplapply(list(c(1,3), numeric(), 6), fun2, BPPARAM = param)
res
@

``fun2'' can be fixed by using \Rcode{seq\_along(v)} to create the index
instead of \Rcode{1:length(v)}.

\subsection{Local debugging with \Rcode{SerialParam}}

Errors that occur on parallel workers can be difficult to debug. Often the
traceback sent back from the workers is too much to parse or not informative. 
We are also limited in that our interactive strategies of
\Rcode{browser} and \Rcode{trace} are not available.

One option for further debugging is to run the code in serial with \Rcode{SerialParam}.
This removes the ``parallel'' component and is the same as running a
straight \Rcode{*apply} function. This approach may not help if the problem
was hardware related but can be very useful when the bug is in the \R{} code.

We use the now familiar square root example with a bug in the second element
of \Rcode{X}.

<<debug_sqrt, eval=FALSE>>=
res <- bplapply(list(1, "2", 3), sqrt, BPPARAM = SnowParam(3))
res
@

\Rcode{sqrt} is an internal function. The problem is likely with our data
going into the function and not the \Rcode{sqrt} function itself. We can
write a small wrapper around \Rcode{sqrt} so we can see the input.

<<debug_sqrt_wrap, eval=FALSE>>=
fun3 <- function(i) sqrt(i)
@

Debug the new function:

<<debug_sqrt_debug, eval=FALSE>>=
debug(fun3)
@

We want to recompute only elements that failed and for that we use
\Rcode{BPREDO}. The BPPARAM has been changed to \Rcode{SerialParam} so
the job is run locally (in the workspace) and in serial.

\begin{verbatim}
> bplapply(list(1, "2", 3), fun3, BPREDO = res, BPPARAM = SerialParam())
Resuming previous calculation ... 
debugging in: FUN(...)
debug: sqrt(i)
Browse[2]> objects()
[1] "i"
Browse[2]> i
[1] "2"
Browse[2]> 
\end{verbatim}

The local browsing allowed us to see the problematic input which was the
character "2".


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{\Rcode{sessionInfo()}}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

<<sessionInfo>>=
toLatex(sessionInfo())
@

\end{document}