Topological phase transitions in (Bi1-xInx)2Se3 and (Bi1-xSbx)2Se3

We study the phase transition from a topological to a normal insulator with concentration x in (Bi1-xInx)(2)Se-3 and (Bi1-xSbx)(2)Se-3 in the Bi2Se3 crystal structure. We carry out first-principles calculations on small supercells, using this information to build Wannierized effective Hamiltonians for a more realistic treatment of disorder. Despite the fact that the spin-orbit coupling (SOC) strength is similar in In and Sb, we find that the critical concentration xc is much smaller in (Bi1-xInx)(2)Se-3 than in (Bi1-xSbx)(2)Se-3. For example, the direct supercell calculations suggest that xc is below 12.5% and above 87.5% for the two alloys, respectively. More accurate results are obtained from realistic disordered calculations, where the topological properties of the disordered systems are understood from a statistical point of view. Based on these calculations, xc is around 17% for (Bi1-xInx)(2)Se-3, but as high as 78%-83% for (Bi1-xSbx)(2)Se-3. In (Bi1-xSbx)(2)Se-3, we find that the phase transition is dominated by the decrease of SOC, with a crossover or critical plateau observed from around 78% to 83%. On the other hand, for (Bi1-xInx)(2)Se-3, the In 5s orbitals suppress the topological band inversion at low-impurity concentration, therefore accelerating the phase transition. In (Bi1-xInx)(2)Se-3 we also find a tendency of In atoms to segregate.