There are two interesting books that deal with the finances of the Third Reich, if you're interested:
"The Wages of Destruction" by Adam Tooze and "Hitler's Beneficiaries: Plunder, Racial War and the Nazi Welfare State" by Gotz Aly.
The Nazis didn't tend to borrow from abroad - they espoused a form of autarky, aiming for financial and industrial self-sufficiency. Foreign dealings were limited to purchasing the raw materials needed to rearm the country. They borrowed from within the German banking sector - from the public and private banks - which were placed under strict controls to ensure that the Nazi government got the best possible deal.
Much of the Nazi war machine was funded by MEFO bills issued by a private, fake company ((Metallurgische Forschungsgesellschaft) that purchased weapons from the main arms manufacturers such as Siemens and Krupp and paid in these promissory notes. The Reichsbank was then authorised to exchange those bills for real money - charging 4% interest (although in reality, these bills were converted into short term government bonds). The scheme, dreamed up by the president of the Central Bank, and a good friend of Hitler, Hjalmar Schacht, and it was an amazingly successful scheme in creating work etc: it was basically a shadow currency off the books and made the national debt look much lower than it actually was.
This scheme meant that financial disaster was looming - if it hadn't been for the money raised by confiscating property from German Jews and foreigners, and the theft of the gold and silver reserves and other assets of the countries the Third Reich invaded (such as Austria and Czechoslovakia) along with the stripping of their natural resources, many historians believe that the finances of Germany would have soon collapsed.