"compile-in" method v.s. "sampling" method

There are two approaches to do the profiling, i.e. compile-in and sampling.
Compile-in is a method to record the timing of executing each component or function, and the one using the longest time is considered the bottleneck.
Sampling is a method to periodically check what the current active component is, and the one
appearing most frequently is considered the bottleneck.

Compile-in needs to instrument the source code, and may affect the performance due to the overhead of counting and logging. To implement compile-in method, one way is to instrument the source code directly, for instance, using macros to redefine the calls and recompile. We are thinking to use aspect of programming techniques to help the instrumentation so that we can keep the instrumented code and the original code separately.

Sampling involves interrupting the process, and grabbing a stack trace. Sampling may affect the performance less if the frequency is not intense. The implementation of sampling is more complicate, and we may reply on some system profiling functions.

Plan:
After the discussion with Paul, we decide to first implement a compile-in method for profiling file I/O, i.e. pread() and pwrite() in c_filesys.c . Our goal is to find out where in the code, a thread is "waiting". Note, when we count the wall time as the execution time, the context switch period is also included. We expect the impact of the context switch among processes is ignorable if the waiting duration is significant.