Fractional Laplacians on domains, a development of Hormander's theory of μ-transmission pseudo differential operators

Let P be a classical pseudodifferential operator of order m epsilon C on an n-dimensional C-infinity manifold Omega(1). For the truncation P-Omega to a smooth subset Omega there is a well known theory of boundary value problems when P-Omega has the transmission property (preserves C-infinity ((Omega) over bar)) and is of integer order; the calculus of Boutet de Monvel. Many interesting operators, such as for example complex powers of the Laplacian (-Delta)(mu) with mu is not an element of Z, are not covered. They have instead the mu-transmission property defined in Hormander's books, mapping x(n)(mu) C-infinity ((Omega) over bar) into C-infinity((Omega) over bar). In an unpublished lecture note from 1965, Hormander described an L-2-solvability theory for mu-transmission operators, departing from Vishik and Eskin's results. We here develop the theory in Lp Sobolev spaces (1 < infinity) in a modern setting. It leads to not only Fredholm solvability statements but also regularity results in full scales of Sobolev spaces (s -> infinity). The solution spaces have a singularity at the boundary that we describe in detail. We moreover obtain results in Holder spaces, which radically improve recent regularity results for fractional Laplacians. (C) 2014 Elsevier Inc. All rights reserved.