Recently, topological insulators (TIs) have been studied intensively in condensed matter physics because of their topologically protected helical edge states, referred as topological surface states (TSSs) [1,2]. Bismuth selenide (Bi₂Se₃), which has a conducting, single Dirac cone of TSS at both its top and bottom surfaces and a relatively wide bulk bandgap (~0.3 eV), is an attractive candidate for future applications such as in spintronics, low-power electronics, quantum computing, and quantum mechanical platforms [3,4]. To enhance the use of TSSs, band structures of TI films with a thickness of only a few layers, wherein the bulk contribution is reduced, have been investigated theoretically and experimentally [5,6]. Because the TSSs of Bi₂Se₃ are distributed spatially at the top and bottom surface, and they are spread within 2.4 nm from each surface, when the film thickness is under 5 quintuple layers (QLs), the TSSs overlap each other, creating a surface bandgap [7]. Zhang et al. experimentally studied the band structures of the Bi₂Se₃ film grown by molecular beam epitaxy with angle resolved photoemission spectroscopy (ARPES) and observed an opened surface bandgap (130 meV at 3 QLs) of the film when the thickness was under 5 nm. In addition, effective models for bandgap opening in this 2D limit verified that the surface band has a bandgap and Rashba-type spin splitting caused by the hybridization of two surface states and structure inversion asymmetry (SIA) at the interface [7].

To overcome practical obstacles such as bandgap opening and topological softening in films with only a few layers, various approaches to control the TSSs have been investigated. Among them, the application of an external electric field is an attractive method to control the TSSs. The bandgap changes when a gate bias is applied through the ferroelectric substrate [8]. In addition, theoretical studies determined that without an additional field, the interaction at the interface (proximity effect) can modulate the spectral distribution of TSSs [9–11]. However, there are still obstacles to realize the modification of TSSs in TI films.

In this study, we investigated the suppression of the hybridization of surface states in a 3-QL Bi₂Se₃/graphene heterostructure. As a substrate, we used graphene, a monolayer sheet comprising a honeycomb lattice of carbon atoms, because of its low lattice match with Bi₂Se₃ and its proper band bending caused by the strong charge transfer at the interface [12,13]. We studied the transport properties in the 3-QL Bi₂Se₃/graphene heterostructure and verified the independent transport channels of Bi₂Se₃ and graphene

To obtain ARPES spectrum, we prepared Bi₂Se₃ films grown on epi-graphene that was pre-grown on a SiC substrate. We grew the Bi₂Se₃ film through a self-ordering process used in our previous work [14]. Band structures of the films were obtained using synchrotron light sources in the Pohang Accelerator Laboratory (PAL) 4A1-μ ARPES beamline. We used a Scienta SES-4000 electron energy analyzer at a pressure under 5 × 10⁻¹¹ Torr for detecting photoelectrons. The resolutions of energy and momentum were lower than 20 meV and 0.02 Å⁻¹, respectively. The measurements were conducted at 50–100 K, and the size of photon beam was under 100 μm.

We performed grazing incident small angle x-ray scattering (GISAXS) with synchrotron radiation from the 3 C beamline of PAL, using a 2D detector. The incidence angle was fixed to 0.2°, and the X-ray wavelength was 1.54 Å^−1. The X-ray photoelectron spectroscopy (XPS) data were obtained with high-resolution XPS (PHI 5000 Versa-probe) using a monochromatic Al Kα source (1486.6 eV). We used the in situ transfer vessels to avoid the surface oxidation of the films.

The surface band structures of Bi₂Se₃ films with and without monolayer graphene were calculated with the Vienna ab-initio simulation package (VASP) PAW-PBE, which considered van der Waals interactions with vdW-DF2 (Grimme). The Brillouin zone was sampled by a Γ-centered grid of 15 × 15 × 1. Spin-orbit coupling (SOC) was considered for all results, and the cutoff energy was −400 eV.

The transport data of the Bi₂Se₃/graphene heterostructures were obtained using the physical property measurement system (PPMS) with a Keithley Series 2400 source meter and a nanovoltmeter unit. Devices for transport measurement were fabricated using shadow masks with a wide and long transport channel (50 × 1000 μm²) to achieve a four-point geometry. The temperature varied from 2 to 300 K and the magnetic field was increased up to 9 T.

Bi₂Se₃ films with thicknesses under 5 QL and their surface bandgaps have been studied with a variety of measurement tools such as ARPES, scanning tunneling microscopes, transport measurements [7,15]. In our experiment, however, the surface band structure of 3-QL Bi₂Se₃ on a monolayer of graphene pre-grown on a SiC substrate did not have a bandgap, as shown in Fig. 1(a). The surface band was similar to the Dirac cone of the bulk films and the bulk conduction band was not observed because of the forbidden transition by the selection rule with a photon energy of 48 eV. Although the synchrotron light source was polarized, we could not find any symmetry in the surface band. This implied that the grain size of Bi₂Se₃ (a few hundred nanometers to a few micrometers) was smaller than the beam size. The charge neutral point of the surface state was positioned at 0.47 eV from the Fermi level. This position implied that the charge transfer between Bi₂Se₃ and graphene induced strong band bending. This bandgap closing was interesting compared with other reports.

In ultrathin TI films under 5 QLs, it is well-known that the spatial distributions of the two surface states overlap with each other, and the surface bandgap opens. However, some theoretical and experimental studies have verified the spatial modulation of the TSSs. Especially, Zhang et al. observed the surface bandgap tunability of 4-QL Sb₂Te₃ films by applying a gate bias through a SrTiO₃ (111) substrate [8]. In addition, they suggested that by the application of the SIA potential over some critical value, the surface bandgap of 3-QL Sb₂Te₃ could be closed. Following their report, more precise results were reported. Tsipas et al. used ARPES to observe closed TSSs in 3-QL Bi₂Se₃ films grown on an AlN (0001) substrate [16]. They proposed that low lattice mismatches with a minimum reaction at the interface can assist in the formation of the abrupt interface between Bi₂Se₃ and AlN (0001) substrate, and the asymmetry between the top and bottom surface states caused by the strong polarization of AlN (0001) induce the suppression of the hybridization and bandgap closing. In our experiments, we used monolayer graphene grown on a 4H-SiC (0001) substrate because both Bi₂Se₃ and graphene are layered structures with van der Waals bonding and their lattice mismatch has a relatively low value of 2.9 % [17]. Thus, Bi₂Se₃ films grown on monolayer graphene can achieve high-crystal quality to obtain a strong charge transfer.

To analyze structural effects such as strain and spin-orbit splitting (SOS) when Bi₂Se₃ films were grown on the graphene substrate, we employed GISAXS and XPS. We prepared two SiO₂ substrates and transferred graphene sheets to one of them. In Fig. 1(b), we plot the thickness dependent GISAXS data of Bi₂Se₃ film on graphene and SiO₂ substrates. The peak position near q = 0.66 Å that resulted from the diffraction peaks of the (003) planes in reciprocal space changed to lower values as the film thickness decreased from 30 to 2 QL. Without graphene, the d-spacing in the out-of-plane direction, which was influenced by surface relaxation, increased monotonically when the film thickness decreased below 5 QLs. With graphene, it showed a similar thickness dependency, except for 2 QLs. Interestingly, the strain of 2 QL Bi₂Se₃ film grown on graphene is smaller than that of 3 QL Bi₂Se₃ film grown on the graphene. It is very unnatural for the tensile strain increases to be caused by surface relaxation as thickness is reduced. This implies that the compressive strain from the substrate had a considerable effect on the total strain of the films with thickness below 3 QLs. Graphene led to relatively small increases in the Bi₂Se₃ films for each thickness, compared with the films without it. This suggested that in the ultrathin film, the tensile strain in the in-plane direction applied by the graphene substrate could induce the reduction of the tensile strain in the out-of-plane direction. The strain values of the 3-QL Bi₂Se₃ films grown on graphene and on SiO₂ were found to be 5.4 and 7.6 %, respectively. In these strain values, the drastic changes in the band structures could be induced by the SOC and crystalline properties, considering that these strain values were near 6 %, a point at which topological phase transition can occur [18,19].

To investigate the detailed changes in ultrathin films, we observed the Bi 5d spectra as a function of the thickness. As shown in Fig. 1(c), Bi 5d spectra of the 3-, 5-, and 20-QL Bi₂Se₃ films have clear SOS between the 5d3 and 5d5 peaks. This SOS has no exact relation with the SOC that can be explained as in XPS techniques. However, SOS could be a strong factor related to SOC. If the film thickness is below 5 QLs, and SOC could be a significant factor in the strain region from 5 to 7 %. Thus, we plotted SOS values of Bi 5d of Bi₂Se₃ both with and without graphene, as shown in Fig. 1(d). Although there were some variations between 20–30 meV, there were clear trends in thickness dependency. The SOS values in 5- and 20-QL Bi₂Se₃ films grown on graphene and on the SiO₂ substrate were similar to the reported values for bulk Bi₂Se₃; however, the SOS values showed a clearly different dependency on the substrate when the thickness was below 5 QLs [20]. SOS values of Bi₂Se₃ films grown on SiO₂ substrates were 3.015 and 3.002 eV for 4 and 3 QL, respectively, while those grown on graphene substrates are 3.041 and 3.038 eV for 4 and 3 QL, respectively. Because the strain can be strongly related to the crystal field and SOC, the difference in the SOS value trends of the films grown on two different substrates could be explained by the strain effect from the graphene [18,21]. In conclusion, the natural tensile strain in the out-of-plane direction of the 3-QL Bi₂Se₃ films caused by surface relaxation was reduced by the tensile strain in the in-plane direction from the lattice mismatch with graphene. Consequently, the bandgap of the Bi₂Se₃ film was reduced, as shown in Fig. 1(a), by suppressing the decrease in SOC and the improvement of crystal quality.

We conducted an ab initio study on the detailed mechanisms for bandgap closing, as shown in Fig. 2. The four band structures and spatial distributions in the conduction band minimum of bare Bi₂Se₃, Bi₂Se₃ without graphene, Bi₂Se₃ with graphene, and strained Bi₂Se₃ with graphene were calculated. Bi₂Se₃ without graphene had same relaxed parameters compared to Bi₂Se₃ with graphene. The bandgap of bare Bi₂Se₃ films was 0.18 eV, which was much higher than the reported values of ~0.13 eV [7,13]. For Bi₂Se₃ without graphene, the bandgap significantly decreased to 0.023 eV and with graphene, the bandgap decreased further to 0.018 eV. This shows that the strain from the graphene reduced the out-of-plane tensile strain to recover the SOC, which led to bandgap reduction. The graphene led to SIA in the films to cause Rashba-like splitting of the surface band and induce a small, additional reduction in the bandgap. Although graphene appeared to have little effect on bandgap reduction, interesting changes were observed in the spatial distributions of conduction band minimum. When there was no graphene, wavefunctions of the surface band were distributed in the overall region. However, for the film placed on the graphene, the conduction bands were localized at the topmost layer. This localization was related to the band bending in the film induced by charge transfer at the interface [9,16]. For example, in the heterostructures consisting of the TI film and conventional insulators of ZnM (M = S, Se, and Te), the TSSs, which were originally located at the interface, were spatially shifted to the interior of the TIs or to the conventional insulator side following their band alignment [9]. In general, the work functions of Bi₂Se₃ and graphene were reported to be 5.1 and 4.5 eV, respectively [22,23]. Thus, in our experiments, considering that an n-type SiC substrate can play a role as an electron donor, there was a significant difference between the work functions of Bi₂Se₃ and graphene, which induced a strong band bending, thereby leading to the suppression of the overlap of the top and bottom TSSs.

In Fig. 2(c), the bandgap decreased abruptly but remained open, which was not consistent with our ARPES results that showed linear dispersion and no Rashba-like splitting [Fig. 1(a)]. Thus, we calculated an additional structure considering a SiC substrate. We simulated the Bi₂Se₃/graphene heterostructure having 2 % tensile strain and found it to have no bandgap, as shown in Fig. 2(d). There were three bands in the bandgap of Bi₂Se₃. Because of the strong band bending, an additional band that interacted with bulk state was formed at the middle layer [24]. The surface state of the conduction band, which we focused on, was localized at the topmost layer more than that in the case of relaxed Bi₂Se₃/graphene. The clear localization of the TSS induced the suppression of band hybridization. Thus, the ab initio studies showed that the bandgap closing was induced in 3-QL Bi₂Se₃/graphene heterostructures and was assisted by both the strain effect from the graphene and the band localization due to charge transfer.

We obtained temperature dependent resistance (RT) and magneto-conductance (MC) curves by using a PPMS, as shown in Fig. 3, to investigate the practical influence of bandgap closing in the 3-QL Bi₂Se₃/graphene heterostructure on the transport properties. In the RT curves, the resistance of 5-QL Bi₂Se₃/graphene was higher than that of 3-QL Bi₂Se₃/graphene. The difference in resistance implies that the Fermi levels of the two graphene sheets transferred onto the SiO₂ substrates were different from each other. The resistances linearly decreased from 300 to 45 K for the 5-QL sample and to 26 K for the 3-QL sample; the resistances, however, exponentially increased from those temperatures to 3.5 K. In both samples, resistances below 3.5 K were saturated owing to the defect-assisted charge transports, which were related to the Se vacancy [25]. The first region of linear decrease resulted from the thermal interference with the conducting states of both the Bi₂Se₃ and graphene films; the second region of exponential increase was from the two-dimensional electron–electron interaction (EEI) in both Bi₂Se₃ and graphene [26]. Considering that both Bi₂Se₃ and graphene have a similar temperature dependency, there appeared to be no significant reason for the interaction between graphene and the Bi₂Se₃ film. Further, we measured the temperature dependent MCs in both heterostructures, as shown in Figs. 3(b) and 3(d). Although the two sets of data exhibited some differences, they had similar temperature dependencies. At 2 K, the MC curves had a trend containing both weak antilocalization (WAL) and weak localization (WL). Similar to how WAL is well-known to be observed in TI films because of its Berry phase, WL is well-known to be observed in the graphene film. In fact, graphene can show WAL in MC curves; however, it does this under very strict conditions, wherein the electron dephasing time is smaller than the elastic intervalley and intravalley scattering time, i.e., graphene should be almost defect-free to show WAL in the MC curves [27–29]. Our graphene was prepared by chemical vapor deposition and then transferred to the SiO₂ substrate, and therefore, it was natural for our graphene to show WL in the MC curves. At 10 K, WL remained but WAL was weakened significantly. At 40 K (near the cross point between EEI and the electron–phonon region), WAL was not observed and only WL remained; at 80 K, WL also was weakened. This implies that the WAL of Bi₂Se₃ and WL of graphene have independent temperature dependencies, i.e., the Bi₂Se₃ and graphene in Bi₂Se₃/graphene heterostructures have independent transport channels.

To analyze WAL and WL in further detail, we fitted the two MC curves of 3-QL Bi₂Se₃/graphene and 5-QL Bi₂Se₃/graphene at 2 K, as shown in Fig. 4. We considered two independent transport channels of Bi₂Se₃ and graphene. For the WAL of Bi₂Se₃, we use the simplified Hikami–Larkin–Nagaoka equation and for the WL of graphene, we use the general equation described by McCann [30,31]. The equations are

where Ψ(z) is a digamma function and F(_z) = lnz + Ψ(0.5 + z⁻¹). The coefficients α and β are pre-factors for the TI film and graphene, respectively, which are related to the contribution for the transport. Bφ=ℏ/4DeτWAL τB-1=(4De/ℏ)B, and τx-1 (x=φ,i,*) are associated with phase-breaking (φ), intervalley (i) and intravalley (*) scattering, where τ*-1 is divided into single valley chirality-breaking and trigonal warping rates. In addition, using the coherence lengths Ly-1(y=WAL,φ,i,*)=(Dτy)1/2, we can extract each coherence length of L_WAL, L_φ, L_i, and L_* from the fitting data (we refer to the phase coherence length in Bi₂Se₃ as L_WAL to avoid confusion with the phase coherence length L_φ of graphene). We fit the graphene sheet first to investigate graphene quality and obtained L_φ = 434 nm, L_i = 296 nm, and L_* < 1 nm, which implied that there were strong scattering sources such as defects and grain boundaries in the graphene. For the 3-QL Bi₂Se₃/graphene heterostructure, the L_* value was still very low and for the 5-QL Bi₂Se₃/graphene heterostructure, the L_* value was 56.7 nm, which was a reasonable value. This implies that the quality of graphene can be improved in the stack structure of Bi₂Se₃/graphene [12]. Detailed fitting parameters for the samples are listed in Table 1.

The pre-factor α values in the two samples had the same value, 0.18. The ideal value is 0.5, which comes from the surface conductance channel. The lower values could have resulted from Anderson localization, which is used in ultrathin and disordered TI films with low conductivity [32]. The phase coherence lengths were found to be very high values of 534 and 269 nm for the 3- and 5-QL Bi₂Se₃/graphene heterostructures, respectively. These values were higher than other reported values and even higher than the 20-QL Bi₂Se₃ film [15,33–35]. In addition, it was interesting that L_WAL of 3-QL Bi₂Se₃/graphene was higher than that of 5 QL Bi₂Se₃/graphene. This can be regarded as the single channel effect. In our film, as shown in Fig. 1(a), the hybridization of the surface states was fully suppressed so that the 3-QL Bi₂Se₃ film had a closed bandgap of TSSs. As a result, the coupling of surface and bulk channels was reduced, and this induced a thickness dependency of phase coherence length in the ultrathin film. Thus, the large phase coherence lengths of our Bi₂Se₃/graphene heterostructures provided strong evidence for gapless TSSs, which was consistent with the ARPES data.

We investigated the TSSs of Bi₂Se₃ and the transport properties of ultrathin Bi₂Se₃/graphene heterostructures through experimental and theoretical methods. We observed a closed bandgap in a 3-QL Bi₂Se₃ film grown on a monolayer graphene sheet, previously grown on a SiC substrate. Graphene reduced the tensile strain in the out-of-plane direction, which resulted from surface relaxation, thereby causing a bandgap reduction. From ab initio studies, we determined that the graphene caused not only a strain effect via lattice mismatch between Bi₂Se₃ and graphene but also induced band bending effects via charge transfer suppressing the hybridization of the top and bottom TSSs and closure of the bandgap. Transport measurements showed two independent transport channels of the Bi₂Se₃ and graphene layers. In addition, by fitting the MC curves, we obtained very long coherence lengths, with the coherence length of 3-QL Bi₂Se₀/graphene reaching 534 nm. The long coherence length implies that with graphene, Bi₂Se₃ can exhibit improved crystalline quality and TSSs can be recovered by suppressing their hybridization. Therefore, the closed bandgap and long coherence length in the 3-QL Bi₂Se₃/graphene heterostructure implies that the proximity effect of strain and band bending in the TI/non-TI heterostructure can be an attractive system for future applications, beyond the practical and topological thickness limit.

This work was supported by the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIP) (Grant No. 2018R1A2A1A05023214). The authors would like to thank Byeong-Gyu Park from the Pohang Accelerator Laboratory for technical assistance with ARPES and the Korea Institute for Advanced Study for providing computing resources (KIAS Center for Advanced Computation Linux Cluster System).