Tuesday, April 21, 2009

The Arms Race in High Frequency Trading

If the calls I am getting from headhunters are any indication, the hot area now is high frequency trading. And no wonder. There are two areas that were spared in the 2008 debacle: macro and high frequency trading. Macro funds on average were up ten percent or so last year because most of them skirted the edge of the major dislocations; their strategies focus on liquid instruments and are not oriented toward credit. High frequency trading did well because it thrives in an environment of high volatility and demand for liquidity, and 2008 was a hot house for both. Every year, people pile on to whatever strategy did well the previous year – this tendency is worth a book or two on its own – and so this year high frequency is destined to be the darling of the fund of funds.

But I think the days for high frequency trading are numbered. For one thing, high frequency trading is capacity constrained like few other strategies. The high frequency trader is basically a stand-alone market maker; he is sitting there to provide liquidity to others. And one way he provides it is to pull in the positions that others will shortly be demanding – thus the need for speed. If the footprint for high frequency traders gets too large, they become liquidity demanders themselves, and the gig is up. The Renaissances of the strategy will make their way through, but generally we will see a lot of shooting stars.

A second reason is that high frequency trading is embroiled in an arms race. And arms races are negative sum games. The arms in this case are not tanks and jets, but computer chips and throughput. But like any arms race, the result is a cycle of spending which leaves everyone in the same relative position, only poorer. Put another way, like any arms race, what is happening with high frequency trading is a net drain on social welfare.

In terms of chips, I gave a talk at an Intel conference a few years ago, when they were launching their newest chip, dubbed the Tigerton. The various financial firms who had to be as fast as everyone else then shelled out an aggregate of hundreds of millions of dollar to upgrade, so that they could now execute trades in thirty milliseconds rather than forty milliseconds – or whatever, I really can’t remember, except that it is too fast for anyone to care were it not that other people were also doing it. And now there is a new chip, code named Nehalem. So another hundred million dollars all around, and latency will be dropped a few milliseconds more.

In terms of throughput and latency, the standard tricks are to get your servers as close to the data source as possible, use really big lines, and break data into little bite-sized packets. I was speaking at Reuters last week, and they mentioned to me that they were breaking their news flows into optimized sixty byte packets for their arms race-oriented clients, because that was the fastest way through network. (Anything smaller gets queued by some network algorithms, so sixty bytes seems to be the magic number).

If we get out of the forest and look at what is going on, some questions come to mind. Does anyone really get a benefit in having the latency of their trade cut by milliseconds – except for the fact that their competitor is also spending the money to cut his latency? Should anyone care if a news event hits market prices in twenty-nine milliseconds rather than thirty milliseconds? Does it do anything to make the markets more efficient? Does it add any value to society?

We usually do not think about trading in terms of social value, but trading often does have social value, and it should. The objective of trading is to provide liquidity to the market, and to make sure that prices best reflect all available information – the usual efficient market argument we all grew up with. The solution? How about having everyone agree to standards in terms of hardware and related configurations. A high-frequency arms limitation treaty. We could call it HALT.

70 comments:

  1. Interesting but I have to disagree. Surely high frequency trading has two big beneficial effects, below.

    Firstly, markets are more efficient and capital is allocated most effectively - as HF traders smooth the bumps and take on ever smaller profits, the proceeds of trading are distributed more evenly across the market rather than in the hands of a few.

    Secondly, like weapons arms races; the investment, technology, and research these people are putting in benefit not just the trading, banking, and FS industry but the wider world too (including but not limited to creating the initial demand for newer, better chips, making computers faster and cheaper for us all).

    ReplyDelete
  2. Right. And the Space Race gave us Tang and GPS, and the Cold War arms race gave us DARPA and the internet.

    ReplyDelete
  3. Given those examples, why are you advocating for a halt in this case?

    ReplyDelete
  4. Just because you get some positive externalities doesn't mean that they outweigh the costs. And as far as fast chips go, I think the gaming industry and visualization demands are more of an impetus.

    ReplyDelete
  5. What about liquidity? HF traders bring additional liquidity (a lot) into the market which absolutely benefits society. This allows investors to get into and out of positions at prices which aren't as likely to dislocate as they would in a less-liquid market.

    ReplyDelete
  6. I agree that some firms are better off not entering the space, but why try to stop them if they want to? When they stop making money, they may decide that they wasted money on the hardware and network infrastructure. Or maybe they'll continue to make money.

    Either way, I don't see why you have a problem with it. And you're kidding about mandating a standard server that all firms have to use, right? This isn't car racing which has artificial constraints on engines in order to maintain fairness - this is business, where companies should be able to get any (legal) advantage they can.

    ReplyDelete
  7. I am not seriously thinking a HALT would occur; it is meant more to make the point that this activity is a dead-weight loss for society, and people are forced to spend the money because other people are spending the money.

    ReplyDelete
  8. These traders have yet to realize the processor isn't the bottle neck. The slowest piece of hardware on the machine is the hard drive. I suggest Intel's new SSD drives. Though I digress, this is a shell game that no one can really win....

    ReplyDelete
  9. The heck with milliseconds, does anyone care that there's 70 kinds of gum, or 30 kinds of toothpaste? (i made up those numbers, but you get the point). It seems to be the age of a planned economy, managed by opinion, thought experiment, and political power. But, to paraphrase my favorite physicist, who happened to be Roger Penrose's mentor back in the day, "I'm not intelligent enough to know the answer to how to plan economic activity or determine value".

    If 4 billion shares are traded daily on the nyse, and competition amongst HF market makers results in only a pip ($0.0001) of reduction in liquidity cost per share, that's about $100M of savings for market participants per year. Modest, but why prevent that by some sort of cartel agreement or law? I agree that improvements from here are going to be smaller and smaller, but it's not societal dead weight, it's savings to be deployed elsewhere, due to improved informational efficiency. Amazingly enough, the volume of trading worldwide makes milliseconds worth something. And if enterprising, risk-seeking souls want to try to deliver that ever smaller value to the marketplace, why not? It's not like they are going to keep trying if they are losing money, so society is safe as long as entrepreneurs are allowed to operate.

    ReplyDelete
  10. Yes but exchanges could agree, or regulators could impose, a floor on the frequency of trading, eg, 1 second ticks for messages in and out. You can't stop people analysing information for a trading advantage but you might redirect their energies.

    ReplyDelete
  11. high frequency trading will plunge in irrelevancy once markets become more liquid: it will take ever greater amounts of capital in chase of tiny profits.
    from that perspective, it does not add liquidity, it exacerbates intraday volatility, while having no impact on day to day volatility.
    any daytrader would know the risk/reward in such strategies is higly skewed to the risk side, and could be profitable only when playing someone else's money.

    ReplyDelete
  12. You can well argue that the drive for ever-faster CPUs is not beneficial to society, so that's not even a positive externality. That drive only comes from a small segment of the market really.

    A drive towards more energy-efficient CPUs (and computers in general) would be of greater benefit to society at large than the drive to increase GHz.

    Software companies would drive towards more efficient software if their ever-increasing bloat wasn't offset by Moore's law.

    etc

    ReplyDelete
  13. Nice to find your blog. Read your book. I want to thank you for writing one of the best and most important contributions to the market literature in recent years.
    --Newton Fawcett

    ReplyDelete
  14. Not sure about the social value of trading. But if I accept your point that some good comes of the increased liquidity then what of the good that the consumption of technology and employment for engineers that it creates? To be honest all us science types would be creating more social good working in more traditional engineering appointments but if we are to waste our time in the financial markets we might as well be buying up supercomputers and tweaking networks as taking a punt on the next payroll's figures, don't you think?

    ReplyDelete
  15. No-one is forced to spend the money. If you don't want your money spent on this stuff, then take it out of your high-freq hedge fund and buy Intel/Cisco shares with it instead.

    In fact those funds (or their investors) are doing me 2 big favours - they provide liquidity to the market, and they subsidize Intel to make better, faster chips, which eventually find their way into my desktop, so I can run MS bloatware at a reasonable speed.

    ReplyDelete
  16. One further point is that in my experience of HFT the bottlenecks are the networks and the kind of latencies that exist there are many orders of magnitude greater than those introduced by the type of processor. Optimising routers, network cards and most importantly network distance to the exchange are the factors that really make a difference.

    ReplyDelete
  17. I love when gamblers...umm, sorry I meant to say bankers...talk about benefits to the society. What benefits did using highly leveraged instruments to wager on future events provide to the society? These instruments were made for HEDGING, that is risk reduction, not risk increase. Limiting technological research by cutting off upgrade expenditures would be the worst possible case now. You greedy bunch just keep quiet for a decade or so, you had your chance and blew it big time.

    ReplyDelete
  18. this post is fun to read especially since I quit my day job (deeply embedded software engineer) to start writing trading tools. I've watched a ton of arms races play out within my industry and it's never obvious what is deadweight loss and what is continual refinement within a hypercomplex set of constraints. I would probably argue the deadweight loss is in articulating and maintaining compliance to "reasonable" standards of restraint. Practically speaking though I suspect the issue is moot since making the software that makes good hyperfast decisions is the true expensive part of it. My instinct is that anyone with the wits to live who is investing in the difference between 40 and 30 ms has already spent 50x that on being able to pull the trigger intelligently. From that perspective, upgrading to state of the art off the shelf equipment is a really easy and cheap call: just do it.

    ReplyDelete
  19. I agree that bankers and traders do not spend much time reflecting on if and how their work provides value to society. But that doesn't mean they shouldn't consider this, or that their work doesn't from time to time have social value. I agree that much of the work with innovative products, derivatives etc. has not hit the mark in terms of social value added. I make this point in my book, and also in past blogs, such as the one last year on "My non-testimony...". Maybe this is a yardstick that should start to be applied in the inevitable emerging regulatory regime.

    ReplyDelete
  20. Rick,

    I think your concerns are unfounded. It's an arms race, but it has always been one and getting expensive hardware is a very small part of it. Most of the money and effort is still expended on developing strategies and software. Anyone with a hardcore CS background can tell you that the difference between a great algorithm and good algorithm can still amount to about 5-10x difference in performance. No matter how fast your systems may be if you do not have profitable strategies, you will not survive in the market. The market is a changing landscape and players have to constantly adapt. So right behind your $100 million for hardware there are billions more spent on paying for top talent like yourself ;) It's just the cost of doing business...

    ReplyDelete
  21. Not only are there negligible positive externalities to trading, trading is most likely an activity that has NEGATIVE externalities, much like pollution. The reason is that the traders live for noise and price action, not some "noble" objective like arbitrage. They act for their own individual interests, and collectively, they destabilize the system by increasing volatility.

    If you don't believe me, just do a thought experiment: what would happen if we stopped continuous trading on all the exchanges and merely had one Dutch auction per day for all securities? Most of these HF quant "traders" would be out of business, BUT more importantly there would be no negligible social welfare costs.

    I mean do we really need to trade more than once per day to correctly allocate resources? The whole industry is just one big tax on the economy.

    ReplyDelete
  22. Certainly HFT enterprises enhance market efficiency by providing better liquidity and information.

    The question here: At what performance point are HFT enterprises no longer increasing market efficiency, but instead just trying to beat their HFT competitors to the punch?

    In the abstract there is some amount of profit to be made by providing liquidity and information. But once HFT enterprises have extracted all of that intrinsic profit they can find themselves in a zero-sum arms race where HFT competitors just try to steal the pieces of that limited efficiency premium from each other, without providing any benefit to the market.

    Is there a way to determine when the HFT industry has crossed that threshold to zero-sum warfare?

    ReplyDelete
  23. The HFT arms race will ease when it no longer pays to get faster. An electronic options market maker is dependent on being very close to the edge in speed. running a book of options means every price change in the underlying leads to a refresh in every put and call at every strike in every expiration all at once. If you are slow you will get plucked in a hurry. arbitrage across the same instrument trading on multiple exchanges or ecn's depends on speed. But all of these actions increase liquidity and reduces frictional costs of transactions for directional traders and large funds that trade in huge size. The reduction in frictional costs are a real benefit to society in my opinion. The need for all the frenetic trade by "investors" is another debate entirely.
    Plus the space4 program produced a little more than Tang, like most of the electronics industry based on NASA research.

    ReplyDelete
  24. by generating demand for cutting edge hardware aren't we encouraging better and cheaper hardware to be built that can be used for much "better" purposes?

    ReplyDelete
  25. "it is meant more to make the point that this activity is a dead-weight loss for society"

    It's unfortunate that you, like Jim Cramer, have chosen to devalue your prior profession. Who is to decide what is and is not "dead-weight" for society? Not you, I hope.

    I once was a military physician and am now a L/S fund manager, and I can honestly say that it is very difficult to determine which did more good for society as a whole. Government is, in most cases, an extremely inefficient allocator of assets.

    If quants are increasing the demand for super computing power, and therefore giving commercial support to developing better supercomputers, than they are helping us all. And the best part is, the free market, not some self-hating retired fund manager, decided how the resources should be allocated.

    ReplyDelete
  26. I don't know which is worse, being compared to Jim Cramer or being called self-hating. And I am not retired (I hope).

    ReplyDelete
  27. Have you read Daniel Pinks book: "A Whole New Mind - Why Right-Brainers Will Rule The Future"? -- premise is that raw compute power (number-crunching) is of secondary importance to right-brain focus on recognizing larger shifts in the big-picture and recognizing patterns that get created by competing quants.

    Rick, I enjoyed an exchange we had in early 2008 here on your blog where you mentioned how leverage had increased liquidity to the point of low volatility. In effect, the VIX was in a bubble in that it was far too LOW. Now that has unwound but I can't see leverage going back to where it was so VIX should not go back to where it was (the teens). Any thoughts on this highly useful subject??

    ReplyDelete
  28. I wrote an article back in the 1980's in the FAJ on volatility. I argued that the reason volatility is so stable in the long run, even though it is not in the short run, is that the sources of information in our society generate information at a stable rate, and the shocks from that information are controlled. Think about Fed policy announcements and earnings announcements, for example. If these change -- if we get information more rapidly or if the information 'packets' are of different amplitude -- then volatility could shift.

    Maybe this is happening, but I think it is more likely that what we are seeing now is higher amplitude due to the crisis and liquidity effects that force larger price movements to clear demand. That is, prices also have to move more to elicit liquidity, because traders are fewer, they demand a higher return to take on the higher liquidity risk and they have greater constraints on financing.

    ReplyDelete
  29. By purchasing a new chips (say chip 1) that provides a better latency by 2 milliseconds from the current 40 milliseconds, you are contributing to further research. Chipmaker who had invested in the chip design and manufacturing would be able to use the money they gain from sales to invest in next generation chip (say chip 2) that would allow you to be faster by 3 milliseconds.

    Now imagine not investing in Chip 1 and in return not providing further research money to the chipmaker. Few cycles down the years there will be no Chip 5, one that could have improved the latency further. Compared to the latency rate at time of Chip 1 (40 milliseconds), Chip 5 could give you latency of 20 milliseconds. Do you see the difference? Its 50% faster from few years ago. Could you imagine checking your email through a 56k connection? AOL did not invest in DSL, Cable, etc. Where are they now?

    ReplyDelete
  30. The difference between improving speed if you are an internet provider versus a high frequency trader is that if internet speed goes up, society gains. Your competitive advantage in the internet provider industry might be short term from making the investment, but the impact of your efforts will accrue to society.

    In my example with high frequency trading, the investment is made, and the end result is no change anywhere. I am assuming -- an important point that I make explicit in the post -- that there is no practical value to having a trade execution or an information flow drop from thirty to, say, twenty milliseconds.

    ReplyDelete
  31. I suppose that back in the day you would have complained about the increased volatility that the telegraph brought, with all those tech savvy electrical engineers trying to get more bandwidth so they could do intermarket arbitrage. What did they know about finance anyway? They were just ruining everything for those guys who had to use the mail.

    I think that HF trading escaped the current pandemonium precisely because it deals in the most liquid instruments, therefore the most transparent. We got into trouble with instruments with no liquidity. Had there only been HF systems in the credit markets, we probably wouldn't be in this mess.
    Volatile markets are not the problem. "Marking to Model" is the problem. At least every one knows that now.


    Something like "HALT" is a ridiculous idea.

    ReplyDelete
  32. I think you make a sound point. Beyond a certain point, increases in execution speed only serve to facilitate those trying to manipulate the market, not those serving a legitimate purpose of speculation and liquidity. It doesn't make my life, or the life of 99% of the rest of Americans, any better--not that that's the only measure of validity. But for a small class of people, they can eke a bit more out of the market each day (and some must fail for that to happen). What value is "created" as a result? Better trading algorithms? Tighter inner loops? Efficiency doesn't matter if you're headed in the wrong direction.

    There is a ton of money being made by trading firms (e.g. Goldman) and it has nothing to do with "providing liquidity" or some societal good. They do it to make money, period; liquidity is an externality, not a goal. It makes them richer, at the cost of increased risk to the economy and the overall system due to feedback and coupling effects. Right now we're paying for their greed. And I say this as someone who trades in the market every day.

    Nice essay, glad I stumbled on it.

    ReplyDelete
  33. It is really microseconds not milliseconds that HFTF are dealing with. Many newer exchanges respond within hundreds of microseconds.. the networking speed being a large part of the sum.

    ReplyDelete
  34. "BBL Jr said...
    The HFT arms race will ease when it no longer pays to get faster."

    This is precisely the point.
    Economic fundamentals dictate that EVERY market functions as an "arms race," where producers fight to acheive the lowest cost and profits are a zero-sum game. Those who cannot lower their marginal costs will ultimately fall out of the market (with the exception of favored groups and market intervention; GM and AIG as prime examples.)

    I tailed to this blog (now bookmarked) from Ars, where they critique your claim to lower marginal benefits. The financial sector is one of extremely few areas of which this bleeding edge has import. Cutting that out of the equation is slicing a limb off the technological cash-cow. It might still be a cow, but it's not as stable or pretty as the others.

    Being the hands that guide such technological development, as prior commenters pointed out, has led to SIGNIFICANT development in processing ability, bandwith, latency, and other things. To cap HFT would essentially serve to deaden the growth we have been experiencing. While the field itself seems to benefit the public at large very little, History has shown that such things do much to generate indirect benefits for society at large.

    ReplyDelete
  35. Impose a tiny tax on all transactions to reduce the profit from HFT. Use the proceeds from the tax to pay down the debt incurred from once again saving Wall St. ass.

    ReplyDelete
  36. Maybe along the lines of a Tobin Tax. This has been suggested by some people with a more academic bent.

    ReplyDelete
  37. speaking of values...who are we to judge what the appropriate limits of competition are?

    in sports, we attempt to define steroid usage as a limit because of the example it would set for kids, etc - dangerous health consequences. that's a real "social value" where we impose rules on competition.

    What's the parallel for HFT? where's the harm? you seem to want to prevent people from hurting themselves. nanny state mentality.

    It might seem wasteful to you, but maybe that's just age/experience talking. why can't we assume that competitors in the space will continue funding this investment until marginal spending does not yield marginal profits?

    the market will force traders to acknowledge it when they reach it. plus, in the interim, these entrepreneurs fund ever more sophisticated technology that might serve as (if we're lucky) the basis for the artificial intelligence of 2020...or gene sequencing...also known as an unanticipated benefit.

    so the last folks to unsuccessfully eke money out of the hardware/HFT space will lose money ... it's always that way ... what's it to you?

    ReplyDelete
  38. Fun thread... Back again for some more.

    Let me add one vital factoid that isn't readily apparent- a consortium of chipmakers (sorry no source) observed years ago that cell phone chips would be the demand side of Moores law for decades. This is where R&D money for hardware has gone. The desktop chip designs simply coattail the foundry innovations for handsets. Who actually appreciates 64 bit multicore laptops? If more did then they would cost more than my last desktop.

    There have always been different scalping/trading/investment timeframes and so what of it if a new timeframe can be colonized by specialists? There will still be head and shoulders as well as massive commodity trends. Appreciate the observance that an economy is composed of arms races.

    As for social good, all of this does seem a far cry from health and happiness. My favorite angle is that there are few drivers for producing robust national technical infrastructures as those that must move tens of millions of dollars extremely reliably with millisecond turnarounds. Sure these techniques are proprietary but yesterdays VMWare is todays VirtualBox. Imagine that at the level of a finance network for a third world country jumping to second world.

    You know, this thread is really about fear of loss and jealousy of those who seem to be benefitting. Anyone serious about 'social good' issues should consider fundamentals not technicals because that's a million times more relevant to how we will define 'social good'. Go on, head over to Naked Capitalism and really get acquainted with the costs of regulatory capture by the extremely powerful financial industry. Do not pretend that Congresspersons do not know who has money.

    ReplyDelete
  39. Doesn't popularity of a method guarantee its eventual demise because of the dilution of the effectiveness? It seems like investing methods are much like restaurant concepts; at one time the popular one is steak houses, and then once there are too many and it is common, sushi becomes the next big thing....

    Are we innately this way. The only difference is whether you are a leader or a follower?

    ReplyDelete
  40. Let me just say that every incremental gain in computing power _does_ benefit society as a whole.

    If, say, the research of Folding@home (http://folding.stanford.edu) leads to a breakthrough which occurs a few days earlier due to the increased speed of these chips, and those few days are the difference in saving the life of one of your loved ones, would you feel differently?

    ReplyDelete
  41. don't think high frequency trading necessarily is what got several hedge funds in trouble---------I think everybody being in the same trade along with fund managers not understanding taking big positions in illiquid markets was a mistake has cost there clients money-------the last few hedge funds goldman sachs have run is a poster child for my thesis---------hedge funds just trading US equities through the quant models i think are at it big disadvantage win they need to liquidate-------there are ways after all to hedge the currency implications of being in other markets----------the last point i will make is i don't think the disadvantage of managing to much money through near term strategies is properly understood for the most part----regardsMJJ

    ReplyDelete
  42. I think my friend 'Jim Simons' and some others commenting here have it wrong. They are getting off topic by thinking about HFT as a game with no collateral damage and with some glimmer of value to society through a possible technological leap that might coincidentally come from the activities.

    If you want to improve computer speed or develop gene splicing or AI, then go and do it. Don't play the HFT game with the idea that maybe it will lead to something good. Maybe it will, maybe it won't. But whether it will, and whether it will lead to anything near the level of money wasted is an open question.

    The point is whatever does come of it is coming in a haphazard and inefficient way.

    ReplyDelete
  43. Stephen McCrackenApril 24, 2009 at 2:30 PM

    I've seen HF traders justify their business to outsiders by saying that they reduce bid/ask spreads, which ultimately benefits retail investors.

    However, somewhere in my studies (John Hull's book?) I learned that trading itself is a suspected source of volatility. Surely that can't benefit retail investors. Does anyone know of current research on the relation between HF trading and volatility?

    Recently I interviewed at a HF firm, and I asked how sensitive their performance was to volume -- if panicked trading subsided, would it hit their bottom line? The answer was: yes, but keep in mind that 2/3 of trading volume is now other HF firms. 2/3! That really raised my eyebrows. They've gotta be nearing zero-sum land (even ignoring technology costs).

    ReplyDelete
  44. thanks rick, always looking forward to your posts. in my view, this particular arms race is nothing to worry about. unlike the subprime crisis or CDOs this cannot grow for 10 years before blowing up like a supernova wiping out trillions. main reason is that current accounting standards, as they are, will tell you very soon when you are losing money, which will inevitably happen, and management will soon get the message and stop investing. with the structured products, on the other hand, thin markets and accounting based on mark-to-something mentality maintained the illusion of profit for years and years. if there is an early warning mechanism that tells you when you are doing something stupid, then it's fine, let them try whatever with their money, no need to halt the arms race.

    ReplyDelete
  45. Don't play the HFT game with the idea that maybe it will lead to something good.The practitioners of it probably won't do anything significant. The performance experts they employ might.

    10 years of doing OS performance work at IBM suggests to me that looking to new chips for an edge is probably the wrong place to look, since everyone can buy the same chips to get the same incremental gains.

    Algorithmic improvements usually trump hardware improvements (often by an order of magnitude or more) and can be deployed on any schedule you want, unlike new hardware.

    As a general statement, most code out in the field is crap. Often, even code you believe to be pretty good can be coaxed to run 50% (or more) faster.

    There's a saying in the software business. You can have it on time, on budget, or on spec. Pick any two. The sad reality is that most programmers are pressured to deliver on a schedule, so something always suffers as a result.

    If quality (in terms of functional points working) can't be compromised, then performance is invariably the easiest thing to throw overboard when the deadline approaches, because non-optimal code isn't something that a user notices unless its really egregious.

    Real performance isn't something one applies after the fact like Bondo either. It has to be architected in and continuously tested for through the whole development (and ongoing post-deployment maintenance) phase.

    If critical path analysis and latency measurements aren't part of your architecture and testing process, then you'll wind up with something that has genetic problems and will never really be fast.

    ReplyDelete
  46. The difference between improving speed if you are an internet provider versus a high frequency trader is that if internet speed goes up, society gains. Your competitive advantage in the internet provider industry might be short term from making the investment, but the impact of your efforts will accrue to society.If Intel sees that Wall Street can spend $1 billion in new technology if Intel delivers they will invest in it. It's true that Cnn.com loads fine with my 3 year old computer but that new technology might also be used for medicine research, global warming, and god knows what.

    Your point is valid up to a point, but companies need an incentive to invest in research. Not every cent invested will yield benefits, but one in 1500 might make up for everything. If we had a perfect way of directing research it would be much better, but we don't and many inventions came as accidents while working on a different thing.

    My point: not everything is wasted.

    ReplyDelete
  47. I guess looking at HFT as loading up on technology is one way, the other perspective is to ask why the current technology of computers is designed to be optimized for a PC basically, with a million an one processes running and no structure of the machine, all generalists. As Sherlock Holmes character claimed to specialize himself for what he needs, HFT efforts on computing are something similar, nothing as deep and revolutionary as people not in HFT think. Trust me, it is hard to get a generalist specialized.

    ReplyDelete
  48. HFT is the ultimate free market whereby the end result is that all players end up making less and less untill they are unprofitable. Looking at Bersteins Capital Ideas Evolving you can see that in order for this to be profitable you need: cheap money, huge technological expenditures, and adequate leverage to generate multiples of small gains. Think of a typical grocery store, small margins and lots of volume. However, the downside is that this liquidity requires a defined rangeness in order to generate returns without a blowup risk. Too directional, or too random and these systems are ineffective and counterproductive.

    We have iconic movies dedicated to the invasion of aliens, and being taken over by machines and cyborgs. Ironically it has already happened and it is HTF, and many remarkable esoteric innovations that are redefining our relationship with technology.

    Perhaps the rocket scientists will find a way for machines to run all companies independent of human input, that way none of us will be necessary, and our value to society will be clearly defined as: zero.

    ReplyDelete
  49. I agree with Rick's argument that competing for microsecond advantages have no benefit to the society. The fact that, such arbitrages are possible is a flaw in the way electronic trades are conducted. The flaw is that the trading protocols are based on absolute time and not based on time stamps. This has nothing to do with free market. It is a matter of designing the correct communication protocol for electronic trades. I don't understand why fixing a bug in the trading system goes against the spirit of free market.

    Just like internet traffic protocols and time sharing protocols are all designed well to avoid race conditions, the electronic trading system also need to be designed to avoid such systemic flaws. It is a simple matter of enforcing a minimum time granularity with time stamps.

    Another reader pointed out that it is the strategy that matters and speed is only a small fraction of the cost. In HFT, speed **is** the strategy. As I understand, the key driver is the latency arbitrage..other strategies on top of this are not exactly rocket science.

    ReplyDelete
  50. Rick,

    Isn't this a question of scale too?

    I'm a prop trader/high frequency trader. While I generally agree with the premise of your post I think scale is really the determinant. For example, individual traders can often make money on short term strategies that aren't scaleable beyond a certain trade size, but are still quite profitable. Such strategies would not be profitable for a large fund, but they can be for an individual trader...

    ReplyDelete
  51. I agree. I know people who have strategies that can only work for ten or twenty million dollars, and probably are not of interest to large players. But if you have 100 people doing these, each in their separate garages, it still adds up.

    ReplyDelete
  52. What is all of this talk about benefits to society? If the HFT industry employs some 5000 people without creating any real systematic risk, is that not a benefit to society? Aren't those folks going out and acting like good consumers, saving some, giving philanthropically and raising well educated, technically minded offspring?

    ReplyDelete
  53. Assume a high street serves the retail shopping needs of a village community. Along this high street are 3 shops specializing in selling swim wear. Assume that the demand for swimwear in this village is constant.

    Shop 1 starts with an inexpensive mannequin in the shop window. Shop 2 responds by having mannequins and hiring someone to mock up a beach. Shop 3 decides it is going to one up Shop 2 by adding a water tank to create a fake sea.

    Shop 1 is losing market share, so it copies the fake beach and sea, AND adds a wave machine for the sea. Shop 2 adds a spotlight to mimick the sun .... shop 3 hires sexy girls to replace the mannequins ... ad infinitum.

    The same quantity of swimwear is being consumed in this micro-economy, yet depending on how much the stores can pass on these costs to the consumers, the margins on the sales could be shrinking due to increasing marketing costs.

    This is also an arms race we might call a halt on. Why not simply ban shop windows and save on all the "unnecessary" expense?

    Just as the shop window arms race created employment in the village for someone to haul sand, sculpt a beach and other esoteric pursuits, the HF arms race is pulling people from the hardware and software fields to innovate in their respective niches.

    Should we jump in and tell the market that it's allocating resources inefficiently? Should we make judgements about whether certain pursuits undertaken by an occupation is socially unproductive? That brilliant IT people should not be allowed to spend their time squeezing latency for Wall St's behalf?

    Is this any different from arguing that brilliant cameramen should not be allowed to waste their time working on forgettable summer blockbusters? Or going back to the example, that shops should just sell their goods behind a desk and not waste effort on shop windows?

    Seems like we're back to arguing whether we trust the market to do the allocation for society or whether we need a bureaucrat telling us what's good for us.

    ReplyDelete
  54. I agree with the last two posts. Unless there is some risk to the wider economy involved (and I believe there is none in this case - correct me if I am wrong), people should go about their business and try out good and bad business ideas.

    What we should aim for is improving the way economic reality is reflected and evaluated, like focusing on realized gains/losses rather than unrealized mark to something dreams that last for years. This way management can make the right call early enough without the need for guidance from some super-intelligent regulator. But, as I said previously, HFT is economically extremely simple, no valuation issues there because they don't sit on hundreds of billions of one way risk - again, correct me if I am wrong.

    ReplyDelete
  55. High frequency, what about low frequency? Is anyone willing to discuss the common installation of low frequency noise generators in UK merchant banks? And don't tell me they don't exist, I installed five.

    ReplyDelete
  56. Quoting Reuters for anything high frequency related is like quoting computer expert from the 50th in relation to the latest Intel chip. Reuters is probably by far the worst, slowest and least technologically advance trading platform in HF world. I also love than some quant discusses "social value" , not providing any himself. Over all I appreciate the article - it gives perceptive of the people we are leaving behind on the trading floor.

    ReplyDelete
  57. I think you should implement HALT. Companies shouldn't have a "legal" advantage, just because they can throw a bunch of money, buy the best computers and stick them closest to the exchange in order to allow some front running algo (GS...here) to scalp the market on "technical liquidity" movements... Companies values (and the core thesis of investing) has always been in the value of things and not the constant change in sentiment. These HF institutions are slowly concentrating wealth so that they can in turn spin it out to a few thereby damaging society... that's my 2 cents~

    ReplyDelete
  58. Any activity that involves growth ends up being an arms race. The arms race will limit the growth. Equilibrium will ensue.

    ReplyDelete
  59. While I haven't read every comment, it seems that none have touched on the point that HFT is not really providing liquidity.

    Goldman is receiving a rebate via the SLP, and that is perhaps a large part of why they are a main player.

    When participants are paid by the exchanges to churn, is that really liquidity?

    And more importantly, when a huge percentage of overall volume is from HFT, what happens when there is a dislocation and a fund that thought a stock trade 10mil shares per day, and had built a large position, tries to exit the position? Are the HFT going to be there, taking the other side of the trade? Not likely. Yet the fund doesn't know that 50% of the volume was simply churn from the HFT firms.

    ReplyDelete
  60. Woodshedder,

    It's true that HF players do not add liquidity, but they do shift liquidity in the time domain, fungible products domain or semi-equivalent products domain for a small risk premium. Different players in the market need liquidity in different times and ways. HF players simply shift the natural buys and sells to where there is greatest demand. Many slower classic hedge fund strategies (e.g. convert arb, merger arb, vol arb and stat arb) exploit the same set of phenomena. Contrary to the tinfoil hat crowd that seem to hang out at Zero Hedge believe, HF players are not out there to manipulate the market. They're there to earn risk premium via shifting liquidity to where there is high demand.

    ReplyDelete
  61. Let's call it for what is is - frontrunning. Folks, don't let the lables mislead you. "Helping liquidity" is the trojan horse, but inside it is frontrunning. Let's kill this evil quickly.

    ReplyDelete
  62. @ Anonymous, if they do not add liquidity, as you state (glad that we agree there) then why is the NYSE paying Goldman via the SLP?

    And I made no mention of manipulation of the market. Why would you bring that into a discussion of liquidity?

    ReplyDelete
  63. At a macro level, HFT (arms race) will demand and pay for innovation, innovation will spur jobs in R & D, engineering, manufacturing, marketing, etc... We hope this innovation will flow from the U.S., ready for export ($), tipping the scale in our favor - monetarily speaking. This same technology can be applicable to other industries (e.g. web 2.0), accelerating the technical adoption even further. Value to society? Plenty. Not just some kind of dream sequence, this is happening today, HFT can only help propel this further. Great thread btw.

    ReplyDelete
  64. It might be that there are coincidental positive externalities from HFT. Just as was argued for the space race. Or, for that matter, the hundreds of millions spent on the Americas Cup.

    Throw enough money at anything in a competitive race, and various industries will benefit, and that will in turn help yet other industries. But by that argument, we ought to also be running around picking wars to start.

    ReplyDelete
  65. Woodshedder,

    SLP just encourages folks to shift liquidity to that venue. I believe the program also comes with certain obligations. Rebates in general compensates the limit order placer for the adverse selection that he faces. Placing a limit order provides an option to the taker (the taker exercises the option by lifting or hitting the market). That optionality has a small risk premium associated with it. It is also generally the case that the taker has more information, hence to encourage people to place limit orders, the exchange gives out rebates to compensate for the risk limit order placers take.

    ReplyDelete
  66. amazing how sheeple mouth n mime corporate PR departments: "liquidity shifting/latency arb/positive electronic hardware externalities for society/" gimme a break.

    All these people should encourage their depository banks where they stash their meager checking balances to take a few dimes out of every accout each month for eternity (to avoid greenhouse gases caused by the extra heat used in computing interest to two decimal places!) that too is a positive externality...they may please go and RAH RAH that capitalism.

    HTere is no HFT without dark pool/off market transaction. IT is FRONT RUNNING (but like a good mafiosi...it is politely done...: they'll only take a millisec and it will only be one or two bips away from the "current spot" price.

    gimme a break. goldman has many devotees sucking the golden calf.

    ReplyDelete
  67. anon from may 8th 3:37pm. what the ****? low frequency noise generators! isn't that CIA speak for the latest in torture devices or is that the newest building safety perimeter practise in the UK? WTF? what reason did they give u for installing this junk? please follow up!

    ReplyDelete
  68. As Anonymous posted, HFT and the flash orders used, are nothing more than FRONT RUNNING.

    Goldman and others are currently reaping billions through HFT. The sharks have done the math and figure a few million spent on giving them a nanosecond on their competitors will be well worth the expense in this particular arms race, when billions are stake in a bullish or bear market. Either way they win over retail and institutional investors.

    It's not about liquidity nor has it ever been. Liquidity is the excuse used to employ the skim.

    ReplyDelete
  69. Earlier you said: "Right. And the Space Race gave us Tang and GPS, and the Cold War arms race gave us DARPA and the internet.
    April 21, 2009 12:07 PM"

    Actually, the Space race gave us the PC. Because NASA needed to get as many electronics into as small a place as possible, they offered contracts to firms willing to make smaller, lighter, and more durable electronics. By 1969 when Apollo was in full swing, there were many electronic components in existence that hadn't been aroung in 1962. One of them was the motherboard.

    By 1972, the motherboard and associated chips had found a place in Apollo, but more importantly, they were ready to be used by someone who could create the PC industry. Enter Bill Gates, Steve Jobs, and Steve Wosniak. see:
    http://space-exploration.suite101.com/article.cfm/the_nasa_moon_landings_and_the_personal_computer

    ReplyDelete
  70. it would be much easier if every exchange would take quotes with random delay (30-120ms).
    Then, no one cares about this light-speed-latency-war :)

    ReplyDelete