A new method to construct task graphs for -matrix arithmetic is introduced, which uses the information associated with all tasks of the standard recursive -matrix algorithms, e.g., the block index set of the matrix blocks involved in the computation. Task refinement, i.e., the replacement of tasks by subcomputations, is then used to proceed in the -matrix hierarchy until the matrix blocks containing the actual matrix data are reached. This process is a natural extension of the classical, recursive way in which -matrix arithmetic is defined and thereby simplifies the efficient usage of many-core systems. Numerical examples for model problems with different block structures demonstrate the various properties of the new approach.