.
Nice, I too am experimenting with docker. A couple of code changes to your Dockerfile though, 1) always use the USER command to switch to a non root user, a vulnerability was discovered just yesterday about not using SELinux and root users. 2) your pIp installs should be on one line that way if you ever need to install a new package the entire image will be rebuilt. 3) if you deploy more than one container it might be wise to invest some time in learning YML so your container cluster can be deployed a little better and more efficiently.
Other than that it really is cool to see how your using it.
Hi Mark. I will add the USER command in (will take me a few minutes to update both hubs and the blog post). For (2) I'm not sure I agree entirely as I can see use cases where I want to update pandas and not scipy since the former is in development but the latter is more stable. For (3) I already use YAML (assume that's what you mean eg in) for pysytemtrade configuration so that's one less thing I need to learn (thank goodness).
Problem: running as a specific USER gives me permission issues (sudo can easily acccess the bound /home/rob/results directory, specific user cannot). Actually even in its present form the code isn't ideal since it creates a .pck file that has superuser only privileges.
After a lot of googling it seems the best option is to create a data volume container, however it isn't clear to me if that will get rid of the permissions problem...
Problem: running as a specific USER gives me permission issues
I believe this is due to UNIX filesystem access rights. User and group ownership of the directories and files is not the same on the host as in the container.
This is solved by setting the same numeric user id and group id on the directories and files on the host filesystem as the user have in the container. You can change it recursively on the host filesystem with: chown -R : , the uid and gid numbers you can find by logging into the container with "docker exec" and issueing "id ", the actual info is stored in /etc/group and /etc/passwd text files.
Another way to do it is to make the directories and files on the host filesystme writable for everyone, chmod -R o+w , this is of course less good from a security standpoint, but it will solve your problem.
Thanks. This is obviously the best solution.
Nice one.
Passing messages between containers is fairly straightforward. When using "docker links", it effectively maps the local IP address of the linked container to /etc/hosts within the container, so you simply address it by the linked name (e.g. would give you the IP of a container linked with the name 'mongo').
IB gateway, either within a container or standalone, should hook onto the host systems' network interface, so your trading container should also be able to do the same thing.
My understanding is that Interactive Brokers allows you to run one TWS/Gateway instance at a time. How do you handle this limitation if you use Docker to run multiple instances of your trading system in parallel?
There would only be one Gateway instance to which multiple processes would connect.
As someone who works with docker daily, I'd recommend against it in this case. You have a mostly static environment and the added complexity well just add to your maintenance work. My recommendation would be a solid config management tool like ansible to provision your servers and for you to standardise on an os, like centos 7.
Thanks for commenting; I guess I'll be able to evaluate when I start using it in anger.
I would object. Using Docker and following the paradigm of "Infrastructure as Code" brings a lot of benefits and flexibilty. Using a build tool like make/cmake/ansible/ant/puppet/etc to automatically manufacture your container and uploading it to a remote registry, you are then able to run your software from anywhere in the world on any computer, cluster or cloud supporting Docker. Not only good from a development standpoint but aklso great from a disaster recovery perspective if you physical machine goes down you will be rapidly up and running on some other place with minimal downtime.
What I wanted to say is that you can treat your infrastructure as disposable and you throw it way and rebuild it anytime you like. "Pets vs. Cattle"
As opposed to manually installing the OS, manually installing the software, manually configuring the software, time consuming installation and setup process, dependent on one server and that is seen as being fragile and need be careful taken care of.
Hi Rob,
Great article!
Just a basic question about backtesting strategies, particularly monthly rebalanced strategies.
Say you have a strategy that gets rebalanced the last day of each month. Typically, you just run the weights on next months returns and have a monthly returns stream. However, what if you'd wanted insights about how the strategy performed on a daily basis. To do that would you simulate it as though you rebalanced daily using the most recent weights? Or would you simulate it mark-to-market on a daily basis?
If the latter, is there an efficient way to do this?
Thank you so much.
A: Simulate it mark-to-market on a daily basis.
Otherwise you'll be changing the behaviour of your strategy.
You basically want to forward fill your positions from month start to month end after matching them against the data frame of prices.
Thank you for your response.
I hate to be a pest, but would you mind demonstrating how this is done in python? There's nothing on the internet about how to do this(atleast that I've found). It would be greatly appreciated. It also gets tricky because I don't rebalance on the end of each month e.g the 10th trading day of the month.
import pandas as pd
positions=pd.DataFrame(...monthly...)
prices=pd.DataFrame(...daily...)
positions=positions.reindex(prices.index, method="ffill")
Will work for any day of rebalancing.
Yes, but isn't that just daily rebalancing. For instance, if I rebalance on 1/30 to a weight of 50% for market A and 50% for market B and I have $100, for example. Then on 2/1 the price for market A goes up 10% and market B is flat. So market A is now $50 * 1.1 = $55 and market B is still $50. So now my weight for market A is $55 / $105($55 + $50) is ~52% and market B is ~48%. Using your method it would say 50%/50% which is just daily rebalancing.
Maybe I asked poorly. But I am trying to simulate what my weights and returns would be on a daily basis if I rebalance once a month without rebalancing at all between those periods.
Using the example above, for instance, on 2/1 to find out what my weights "should" be I would see 52%/48% not 50%/50% and the returns for 2/2 would be based on the 52%/48% weights at EOD 2/1.
The code is correct. You need to forward fill your positions, not your weights.
Pardon my naivete, but what do you mean by positions?
Thank you for responding, btw.
Position = how many actual shares or futures contracts you hold.
Okay great. Then each day during the month its just multiplying the # of contracts by the contract values?
Yes
Thank you so much. It is more than appreciated.
So, I've been trying this entire day to make this work, unfortunately I cannot figure this out. I know your very busy and have more important things to do but I can I email you my code snippet n sample of the data I am working with for some help?
This is an act of desperation.
This comment has been removed by the author.
There's a recent discussion thread over at nuclearphynance about Docker, and seems the sentiment about its actual ease of deployment is mixed, and one comment linked to this blog (a fellow Londoner, Rob, got to count for something right):
Seems he's quite sour to Debian/Ubuntu, Cent OS, and so on.
My dedicated server is running CentOS7 due to RHEL-only Linux support for Rithmic API...I'd considered using Docker but abandoned the idea until later...maybe Xen/LXD would be a good alternative?
That's a really interesting article. I guess the key takeaway for me is not to build something that relies on docker being present; or to put it another way to create services that can run inside or outside of a container. I'd planned to do this anyway, as I'm releasing everything open source I don't think its fair to force a particular architecture on other users. I'm using Linux Mint which is Ubuntu based FWIW. I'd also always planned to create daily copies of the data normally inside any docker volumes on to a real file system somewhere, since I never really trusted the idea of a docker volume wholeheartedly.
While it's not clear they use it for front office systems, your former employers competitor posted recently about containerised services recently. Clearly smart minds think alike!!