- The most prominent form of automated or algorithmic trading today is known as high-frequency trading (HFT). For a good introduction, this documentary provides some insight into the way that the shadowy world of HFT operates. Essentially, this kind of trading relies on buying and selling assets just moments ahead of slight changes in price up or down. By acting on informational disparity at exceptionally small time intervals, these these HFT firms can make a tiny bit of profit on each trade. In the world of HFT, those with the fastest information get the profit, and the profitability of these firms is driving a scramble for faster data transmission.
- But this is where it gets a bit scary. The institution of algorithmic trading as the main force driving modern markets means that the decisions of algorithms are increasingly the basis of market price. As these algorithms start to go up against eachother, they are also being used to discover ways to manipulate the market itself. Scarier still, it is unclear how connected the gaming between algorithmic traders is to the fundamentals of economic function (like how many people can afford bread tomorrow).
mk kleinbl00 flagamuffin mhr b_b If you guys didn't know, I've been working in HFT for about three years now. There are a lot of assumptions and misconceptions in this article. I don't want to nit pick the article post a comment here and then not have anyone read it. So rather, if you have the desire, feel free to ask me anything about the HFT industry. Speaking on my own behalf I'll do my best to answer it. (edit: also note I am leaving for work in ten minutes so I will probably answer it later tonight or tomorrow morning).
Cool, thanks. Personally, I'd like to hear a bit more about what you said in another comment here: I have some doubts about this, because AFAIK, HFT often works in a way that is agnostic to the market itself. That is, the goal is to profit from predicting price fluctuations, and this might have little to do with reflecting market conditions. It would seem if you could manipulate the market with some trades, and then profit from that manipulation in others, that would be an approach that would be superior to predicting the market or getting reliable advanced signals of market movement. In that case, this could actually make the market less efficient. EDIT: Another thought came to me: Is more efficiency a good thing anyway? I can imagine that some inefficiency between market changes and stock prices could actually be beneficial for the health of the economy. Inertia is a stabilizing factor of many systems. And, even if we were to decide that increased efficiency was a goal, then we would have to determine what that efficiency was being measured against. If it's measured against news of market conditions, then to the extent that the news is biased or inaccurate or inefficient, then to that extent the market is modeling the wrong thing. I guess I see HFT as a possible creator and amplifier of noise. If news was most correct at time x after the first report, then slowing trading to that time scale could actually improve the market's efficiency.Just consider this a force that pushes the market towards efficiency and not toward inefficiency.
This really begs at the question are Capital Markets good? I would say a resounding yes. That's not to say that our capital markets are perfect or couldn't be improved upon, but its better than not having an equity market. Now what does the stock market do for the company and the investor and how is the stock market and in particularly the accuracy of the price at any one moment have to relate to that. Equity is one way for a company to fund itself. You sell a portion of company ownership for cash in an IPO or another offering. Then going forward you can also use the equity you retain and the current price in the market as one of your assets. For an investor equity is part ownership in the future cash flow of the company. If the company has more money they might issue a higher dividend or do a stock buyback thereby increasing the value of what you own. The Efficiency of Price Discovery I can try to answer the question in the context of something material happening to JCP. Let's say their holiday sales are released and it's bad. If you are an owner of JCP stock you probably want to compare your expectation of future cash flow with the current price of the stock. Would you rather sell now for cash or hold it for future appreciation. If you think you are much better off selling now then you go to the market and sell. This is going to affect the price in the market, because the level at which someone was willing to buy at may no longer be there. Only so many people are going to be willing to buy at price X until they move the price down to X-1. So the stock price moves down eventually it stops moving down and market participants are willing to push it back up again. This process is called price discovery and it's something that market makers help to achieve. Now here's the kicker, if you as an investor want the option to be able to buy or sell in a reasonable time after material information is released then price discovery has to happen! The price can't sit at it's last price and everyone is allowed to execute at that price. Clearly that doesn't work because who is actually going to receive the sold securities? There has to be real time buyers and sellers to facilitate this movement. The Efficiency of Alternate Markets Now above I described the efficiency of price discovery and how market makers help with the process. Now there is also the efficiency of price being equal across different exchanges. This is easier to understand, if there is a resting offer to sell BAC for 12.99 at NYSE and a resting bid to buy BAC at 13.00 at NASDAQ then it only makes sense that someone is going to come along and try to buy BAC at 12.99 at NYSE and sell it for 13.00 at NASDAQ. If you want more than one market center then this has to happen! And you do want more than one market center, because competing market centers help to keep exchange fees low via competition among the exchanges. Not to mention that competition among the exchanges helps them to improve their market technology being able to handle more bandwidth and coming up with new trading technologies that help market participants. The CME futures market is an interesting counterexample to the US equity markets, because although there are about 7 lit equity markets there are only 2 big futures markets. So CME has a lot more power. I cannot say definitively whether it's cheaper to trade in equities or futures because its like comparing apples to oranges. With CME they are also a central clearing house so they help facilitate the risk if a counter-party defaults and cannot fulfill an order (imagine Smithfield goes broke and cannot deliver on its pork futures). The problems with trading equities are not equivalent. The Efficiency of Related Securities Many of you are likely familiar with ETFs. Exchange Traded Funds are a security backed by a corporation that is designed to back this security with specific other securities. The simplest one is SPY, which is made up of the securities of the S&P 500. So the person who issues SPY buys the basket of stocks that relate to the S&P 500 then they issues an equivalent amount of SPY shares. When they get dividends from the basket that they own they issue dividends to the owners of SPY. Now why is this a good thing, because owning SPY instead of owning 500 stocks is a cheaper way to diversify your portfolio. Why do you want to diversify your portfolio, because mathematically you have higher expected returns with lower risk (check out modern portfolio theory). Now just with the alternate exchange example above, if I could calculate the fair price of SPY and I see that the price SPY is being traded lower than the cost of its constituents then I'm going to buy SPY and sell the constituents. This pushes the price SPY to the true price of SPY. Aside, now one other thing that could happen that is different than it works today is there could be mini auctions every few milliseconds instead of continual trading. I wouldn't necessarily object to this idea. The idea is similar to what happens in the morning or closing auction. Everyone who wants to buy or sell a stock issues the price they would buy or sell at. If the market crosses then the exchange finds the fair value and matches as many buyer and sellers as possible at the price. You would have to match everyone pro rata though, because if the match still occurred based on time priority you wouldn't have solved any of the negative sides of the industry such as high technology cost. I don't consider the high cost of technology to be a huge negative of the trading industry. What automated trading has accomplished is take the job of 1000's of human market makers and automate them. The old human market makers used to make a lot of money for what they did. They also couldn't handle more than a few stocks at a time so there had to be a lot of them. A modern moderately sized team can handle the entire universe of stocks though. This should make it clear why it's cheaper to trade now than it ever has been. In the process they have brought the spread between buyers and sellers down to penny and subpenny levels as well as enable you as an investor to click on your Scotttrade account and actually own the shares you want to own before the page even finishes loading. That's amazing if you think about it.Is more efficiency a good thing anyway?
This is all way over my head but I appreciate the time and effort you've put into discussing this. :)
Yay!!! I got a badge and insomniasexx followed me and it only took me 863 days to do it. I can start dishing out these 4 badges I'm holding now that I know the joy of getting one. On a serious note, It was worth my time to comment on this subject. Imagine you were a psychologist and I posted "all psychologists are quacks and this is why they are hurting our society". You probably wouldn't feel contempt, but you would see it as an opportunity to explain to others why you think psychology is good because you're a psychologist and you know the benefits first hand.
In response to the this In my experience strategies do not work in a way that is agnostic to the market. Usually strategies are formulated off of some premise related to the market. One example would be if there are more orders resting on the buy side then the market might move up. Or if the price has been moving up keep moving it up until reach some reversion point that you have calibrated by backtesting. These two strategies here may seem like they are agnostic to the fundamental price information of the stock, but they are not agnostic to the market itself. As a market maker you want to facilitate trading between buyers and sellers and using information that is coming into the market (like resting orders or recent trades) allows you to get a better idea of what the market participants are going to do next. Now I imagine that there are other things that traders could do that are way out there. Attempting to correlate twitter traffic to price fluctuations seems particularly hard to me. Similarly doing anything that is deep machine learning like Neural networks or Genetic Algorithms run a bigger risk in my opinion because if the premise that you trained your learning algorithm on changes then your strategy might work in the complete wrong way. As an analogy imagine that you trained a neural network facial recognition algorithm in a room with only yellow incandescent light and then you bring in a test corpus to see if your training has worked. It will probably work pretty well but what if the yellow light gets changed to fluorescent light. If it's a neural network it will be hard to know if it will continue working. Similarly in the market there might be a certain market sentiment all of 2013 but if something big happens in 2014 like a sovereign default or another externality you have no idea how the market is going to behave moving forward. If you at least tie your strategy to something simple like order flow you are likely to run less risk.HFT often works in a way that is agnostic to the market itself.
The slight inefficiency of the market creates a small time buffer that means a lot of the meaningless fluctuations of the market can be ignored (have to be), which would seem to lend stability. HFT lets computers exploit these fluctuations instead, perhaps blowing them out of proportion. And the other thing you said is true as well -- if you have the computing power it seems much more foolproof to work the market via HFT than it would be through doing actual investment research and accumulating knowledge.EDIT: Another thought came to me: Is more efficiency a good thing anyway? I can imagine that some inefficiency between market changes and stock prices could actually be beneficial for the health of the economy. Inertia is a stabilizing factor of many systems.
I want to dispel a common misconception that I think is displayed here. You can't exactly manipulate the market without taking on risk and that's without even saying that manipulating the market is illegal and it's something you can go to jail for. The rule is something along the lines that every order that you put out into the market has to be an order that you are willing to trade on. If the SEC can prove that you have a strategy where the orders you put into the market are not genuine then you are in big trouble. The legality of it aside, you can't manipulate the market without taking risk on. Think about it. To change the price of a stock upwards you would have to buy all the resting orders at all the exchanges until the price was actually at a higher level. Now if you want to profit from that upturn guess what? It's kind of hard because now your holding a whole bunch of shares. So buy 10,000 at 12.99 and sell 100 at 13.00 doesn't exactly work. The manipulation that I am more familiar with (usually reported by nanex.net are algos mesing with other algos. Picture putting a bunch of resting bids on one side of the book this might make some dumb strategies think that the market is going to move up so they buy then the algo that put the false resting bids sell to the confused algo willing to buy at a higher price level. Then the algo buys back the shares when the price comes back down to its normal level. This is illegal but it's also a reason why strategies don't usually act on anything but trades, because you don't know if a bid is a true willingness to buy. Aside from what I described resting bids could just be a holding spot for a trading strategy that is waiting for separate market centers to converge or waiting to fulfill an order for one of its clients.
The article also mentions that transaction fees approach zero. This seems a critical factor too for a high-frequency trader. How close to zero, and how significant are transaction costs overall?feel free to ask me anything about the HFT industry
Thanks for the offer. I would like to know what kind of factors your algorithms consider. Are they mostly indicators that traditional investors look for, like product announcements and earnings statements, but HFT aims to react faster than others, or do you put in variables day traders might ignore? Not just Twitter trends, but obscure things like weather events, stock price proximity to multiples of ten, ease of pronunciation of company names.
I can't comment on anything I've specifically been involved in and even that is a small subset of the motivation of all the market participants. The price discovery that I described in mk's answer above is a combination of everyone coming together with their own strategies. These strategies come from a wide variety of ideas and time horizons. Trading on P/E on a month to month basis might be one. While trading on twitter traffic in a minute to minute horizon may be another. Trading on order flow on a microsecond by microsecond basis is another. Predicting how the weather is going to affect Oil and Utility companies is probably something else going on. Ha it's interesting that you mentioned stock price proximity to multiples of ten because that is something I wouldn't expect someone outside the industry to even know about. Are you a shark trying to extract info? It's true stock prices do weird things when they get close to whole numbers. I've pictured it a matter of psychology, I think it's the same reason that queues at restaurants get busy at 12:00 and 1:00pm but not at 12:23. It's because everyone told their friends to meet at the restaurant at 12:00 not 12:23. I think people do the same thing for stocks. They say "OK, I will finally sell TSLA if it hits 180." Not "I will sell TSLA if it hits 179.97." except me that's exactly what I say because I know that's going to get my order executed before everyone at 180. Some stocks can even trade along 1 penny from a whole number all day long. It's weird and I don't get it completely. IMO as I described above, in the smaller time horizon space the simpler and more fundamental of a signal you can act on the lower risk you are going to have and the quicker you are going to be able to act on that signal. Here are the costs of all the regular lit exchanges NYSE, NASDAQ, BATS, and Direct Edge. It's less than a penny to to take liquidity of a liquid stock and you also get paid to provide liquidity of that same stock. How much you get paid for taking less how much you get paid for providing is clearly positive resulting in profit for the exchange. Additionally you might be interested to know that every big lit market has an inverted exchange where you get paid for taking and have to pay for providing. There tends to be less trading on these exchanges. Executing a trade, whether that trade came from a signal or a client is all about finding the cheapest liquidity in the market. If you can provide liquidity and get paid by NASDAQ then do that first. If that doesn't exist then go to the dark pool where you might be able to match between the Best bid and ask. If that doesn't work then go to a lit exchange and take liquidity. You can see from the cost lists above that it's not going to be more than a fraction of a penny to execute in any of these areas. As far as their significance even though its small how much you are going to get paid for a trade from your client is also pretty small. So you want to be as frugal as possible. Trading is a numbers game. You want to make each fraction of a penny of profit you can make in a transaction as big as possible while doing as many of those transactions as possible while also taking on as little risk as possible.Thanks for the offer. I would like to know what kind of factors your algorithms consider. Are they mostly indicators that traditional investors look for, like product announcements and earnings statements, but HFT aims to react faster than others, or do you put in variables day traders might ignore? Not just Twitter trends, but obscure things like weather events, stock price proximity to multiples of ten, ease of pronunciation of company names.
The article also mentions that transaction fees approach zero. This seems a critical factor too for a high-frequency trader. How close to zero, and how significant are transaction costs overall?
Thank you for the detailed information. Every time I came here to reply I got sidetracked reading about something you mentioned. Your description of price discovery was illuminating and reminded me of the Law of One Price. It sounds very official when they call it a law. I had not heard of "lit" markets, but found a source that says "a market is lit if its orders and quotes are viewable by the general public and dark if they are not." Simple enough. I learned my lesson on the stock market riding the early downward slope of the dot com bust. Watching my brighter friends try their hand at investing has mostly reinforced my feeling that it is something best left to the professionals. I bought my first "For Dummies" book on the advice of a friend and have since then parked my 401(k) in an index fund. The multiples-of-ten idea comes into play as soon as you apply a little game theory (what do I know that they know that I know...) to bidding on eBay. I have also noticed peculiar discontinuities around the round numbers of the Bitcoin market depth charts. There are big jumps at round numbers like $1000 and $980, and also little parasitic clusters a few cents away from those numbers. Thanks for the links on transaction prices. I am still a bit in the dark about the market maker's role compared to that of the individual investor, but still found your overview a fascinating look at how HFT works, especially the parts about algorithms anticipating and competing with one another.it's interesting that you mentioned stock price proximity to multiples of ten because that is something I wouldn't expect someone outside the industry to even know about. Are you a shark trying to extract info?
Of course you will understand that the Shark Code prevents me from answering in the affirmative.
Also, the bit about pronunciation of company names came from Thinking, Fast and Slow:Companies with pronounceable names do better than others for the first week after the stock is issued, though the effect disappears over time. Stocks with pronounceable trading symbols (like KAR or LUNMOO) outperform those with tongue-twisting tickers like PXG or RDO — and they appear to retain a small advantage over some time. A study conducted in Switzerland found that investors believe that stocks with fluent names like Emmi, Swissfirst, and Comet will earn higher returns than those with clunky labels like Geberit and Ypsomed.
This is sweet. What do you do for a living? Are you involved with statistics or computers?
I work in software development, but regret that I use very little math on the job. There is a current fad in the "agile" style of project management for using Fibonnaci numbers to estimate level of effort. It is all I can do to ignore the incorrect series "0, 1, 2, 3, 5, 8 ..." scrawled on the whiteboard.
The article is pretty uninformed, no doubt. If you feel like talking about HFT, okay. Keep in mind, though - my distaste for markets in general and HFT in particular is not due to imperfect knowledge. Better to say I've done enough research and had enough experience to have made up my mind already.
I just want to point out that this philosophy is a really poor way to approach a conversation on hubski. Hubski allows me to get insight into so many areas and opinions that are not my own or that I would have never considered to contemplate. I always approach my conversations wanting to get as much as I can out of the person I am talking to not reject as much as I can with prejudice.
Go ahead and point it out - but keep in mind: you approached ME. I didn't seek you out for a conversation on HFT. I responded in a decidedly lukewarm way because, honestly, your familiarity with HFT is not likely to impact my antagonism towards HFT. Put another way - "I will entertain your statements on HFT but yours is not knowledge I am seeking." Your audience isn't necessarily hostile, but it's not necessarily inviting, either. Would you have preferred I hid that attitude from you?
No matter what the topic I think trying to keep an open mind even if you have entrenched views is important. I'm doing my best to dispel falsehoods that I see said about electronic trading, point out the benefits that electronic trading has had for society, and engage anyone who has an honest question.
You know, I'm not appreciating getting beat up on this. I'm keeping an open mind. I said I'd listen. I told you my viewpoint. And here you are, kicking me in the shins for having that viewpoint. I didn't ask you to "dispel falsehoods." I don't have any questions. I don't feel any particular attraction to listening to your take on HFT; I've done plenty of reading on my own and had lots of my own experience with markets which - to put it bluntly - you're doing your best to denigrate. So basically, I said "go ahead, but recognize that I'm skeptical" and you're saying "you're an ass for being skeptical." Skepticism intact. If this is how you evangelize, I suggest you work on your bedside manner.
I'm in but lets do it for bullion cubes or sarsaparilla extract
Hahaha. I think its going viral. Buy buy buy!!!!
At first, I thought that Heinz was a private company, but then I researched and found that Berkshire Hathaway owns Heinz, so funnily enough, the second richest man in American would just get more rich from your proposed campaign to point out the danger of algorithmic trading.
Full disclosure: 1) I think HFT has brought nothing good to the world. 2) I'm of the opinion that stock trading in general should be illegal. THAT SAID: If I were to list "immediate existential threats to the modern world" I would not include HFT. Yeah, the flash crash was no bueno but it was nothing compared to, say, LTCM. Yeah, algorithms can manipulate the market but nothing compared to, say, Enron. And yeah, a lot of money can vanish in the stock market if things go wrong but it's a fly on the ass of the elephant that is the bond market. Quoth Michael Lewis, comparing his book The Big Short with his debut from 20 years earlier, Liar's Poker: I mean, the editor in chief of Business Insider, Henry Blodgett, is only an editor because he's been banned for life from trading because of manipulating investors. Clearly, that worked out poorly for him. So yeah - the algorithms fuck up. But humans fuck up worse, they fuck up consistently, and they often fuck up on purpose.…exactly twenty years after Howie Rubin became a scandalous household name for losing $250 million, another mortgage bond trader named Howie, inside Morgan Stanley, would lose $9 Billion on a single mortgage trade, and remain essentially unknown, without anyone beyond a small circle inside Morgan Stanley ever hearing about what he'd done, or why.
I think it's when you pair the two fuck ups that you run into problems. I've just spent ten minutes defending the author to my only intelligent facebook friend so I won't go into detail, but his point that botnets could conceivably -- just through sheer reaction time -- do significant damage to the stock market shouldn't be ignored just because people also from time to time fuck everything up. I'm not sure where he's going with such grandiose language and his use of the word 'existential' was probably to his own detriment. EDIT: apropos of nothing, I've been meaning to ask you if you've read George Friedman' America's Secret War.So yeah - the algorithms fuck up. But humans fuck up worse, they fuck up consistently, and they often fuck up on purpose.
A number of reasons. For one, the only proven, verifiable way to profit long-term is insider trading. Which basically means it's a con game whereby those in power fleece those without. For another, I've seen nothing destroy a company faster than "shareholder concerns." It isn't an "always" thing, but it's a lot harder to execute long-term planning when you're beholden to quarterly earnings reports. I've also never been close (working for or friends with those working for) to a company whose stock price had anything to do with that company's performance or viability. The most striking example for me was when I asked a buddy who worked for Real Networks how his stock was doing: "We're down five percent. It's amazing, really - we released a report yesterday that not only beat our earnings estimates, but actually declared a dividend. We're one of the only profitable dot coms out there. But because our numbers didn't beat moody's, we're down twenty bucks." So really, you've got a distracting game that exists to fleece fools. If Wall Street had to follow the same basic rules as Vegas, I'd be a lot cooler with it.
Yeah but if you give a fancy, technical sounding name like "information asymmetry" then it just sounds like we're not as smart as the other guys, and we deserve to lose. I love the power of manipulative language....the only proven, verifiable way to profit long-term is insider trading. Which basically means it's a con game whereby those in power fleece those without.
kleinbl00 the market is not that simple. If it was everyone would make money. Analysts estimates are a poll done by a financial news source and companies estimates are just where the board thought they would be at that time. Each investor in the market though is going to have their own expectations and perhaps they weren't polled when the estimates were taken. On top of that there are a lot of participants that are going to be trading on momentum or are going to try to arbitrage individual stocks with ETFs their a member of. You have to look at it this way. Thousand of participants all with their own opinions trying to make money. The ones that are right make money and survive. The ones that are wrong lose money and exit the market.
Except that's not how it works. At all. Not even vaguely. You have to look at it this way: You have to look at it this way. Thousand of participants all with their own opinions trying to make money. The ones that are popular make money and survive. The ones that are unpopular lose money and exit the market. In the words of the immortal William Munny, "Deserve's got nuthin' to do with it."You have to look at it this way. Thousand of participants all with their own opinions trying to make money. The ones that are right make money and survive. The ones that are wrong lose money and exit the market.
Hmm. I don't really get what you are saying here kleinbl00. Can you explain it a different way? What does popularity have to do with making money in the market?You have to look at it this way. Thousand of participants all with their own opinions trying to make money. The ones that are popular make money and survive. The ones that are unpopular lose money and exit the market.
Sure - you are arguing for the efficient market hypothesis, which is positively retro of you in light of 2008. Your statement: The ones that are right (whose opinions are correct) "make money and survive." Which is where I go - Worldcom - Enron - Countrywide - WaMu …and point out that up until a couple days before doomsday, the people making money were the ones who thought these companies were sound. So the first problem with the efficient markets hypothesis is windowing - yeah, overall the universe will die of accumulated entropy but that doesn't mean you have a handle on the temperature of Sirius precisely 80 million years from now. I also pointed out my friend at RealNetworks, who were absolutely killing it in 2000 yet the market just hated on 'em. Their lack of valuation impacted their financing and ended up curtailing their business because the market decided they weren't cool enough. So what I'm saying, in very simple terms, is the stock market is a popularity contest, not a "perfect information" contest, particularly as "perfect information" is considered insider trading and is banned by the SEC. It's kind of ironic, if I may say so, to use the efficient market hypothesis as it pertains to HFT. The whole raison d'etre of HFT is to take advantage of the inefficiencies of the markets as experienced by the rubes and marks. As I recall, wasn't Goldman Sachs busted hard-core for basically using the gimmick from The Sting in their HFT - in other words, delaying their reports to the floor and arbitraging the difference?
I'm not arguing that a market mechanism makes a perfect predictor. As I highlighted in one of my responses above you are giving an investor a fair price to trade in their stock ownership for what it is selling at the moment in the market. I dunno how you suppose to fix the problems with funding RealNetworks. If there is no public market are they going to get funding from the government or private investors???? I mean isn't that even more unfair than having a place where any individual can invest in a company and trade what they think is worthy of their valuation?
You asked for a clarification. I gave you one. Do you have a question? Keep in mind - you asked me based on a flippant comment I made several days ago. I did not set out to lay out a treatise on why the stock market should be banned. My interest in this is tertiary at best - the more antagonistically you treat this, the less likely I am to respond.
A lot of you might be interested in Knight Capital Group losing a whole bunch of money trading. Knight Unhorsed via Dealbreaker I use it as an example of when an HFT outfit made some huge mistakes and is now crippled because of the money they lost. Just consider this a force that pushes the market towards efficiency and not toward inefficiency. And what did the smart guys do while they were throwing money away? They caught it, and made all sorts of profit. It's like the flash crash, some person fat fingered in Kansas and what did the smart people do? They traded through it and made tons of money! Bringing the market back up to where it should've been quicker than you would expect otherwise.