Did I just get Hacked? Thanks Docker?

TLDR: My EC2 instance got hacked, I think it is related to the Docker Hub hack. I left some clue and what I did to tighten security.

A few years ago, I finally upgraded from Godaddy to AWS EC2. In fact everything has now been moved away from Godaddy (thank goodness). Route 53, isn’t completely user friendly, but it works fine.

Regardless, I notice that I was getting a bunch of Jetpack notices that my website was down. This was not characteristic of AWS. So ssh’ed into the server and checked the “top” function. I should have taken a screen shot, but my server was being maxed out 99% with numerous “dockerd”. (docker daemon) This load was making my server crash / clogged computing power, etc.

This was a little unnerving, but I do remember installing docker and then abandoning the project. Anyways, what was definitely clear is that dockerd was trashing my server.

Not exactly sure what was accomplished by taking over my server. The instance is a t1.nano, so it doesn’t offer much in the way of crypto mining. My site doesn’t get enough hits, so I doubt it was advertising malware. My only thought is that they might have used it for some kind of SEO / traffic bounce scheme?

Then I found this article, so I’m now guessing it was part of the issue?

Here is what I did immediately tighten up security. If anybody has any more suggestions, please place them in the comments.

Here is just some of the code I found on the server, if anybody is interested.

root:/var/www/html/stocks# docker rmi $(docker images -q)
Untagged: postgres:10-alpine
Untagged: [email protected]:3d9c62b0b614fb7e21afa752b4e84ff83cbbde5b84ea0b533799cc4c44ea3789
Deleted: sha256:787004db496a65b949515e809683667ed7c4b566fd27b3dc6ed46b322575f278
Deleted: sha256:2e6c28a94b8898ff48ed5c425150a7074dcfa2b461bee071836846055ebc398e
Deleted: sha256:7568dcf4070e53f38076d2ea02a5691a30323e3ce897c57e12d2c5c7d60a1cae
Deleted: sha256:f31fd8e4dd027a25da196eeb7e119e5e570203b92f14cc6fbfa662a1c3fbc881
Deleted: sha256:de54b53c627c545176a86b66f1f88f688291e2db4aff5184714f8b802b5ac071
Deleted: sha256:fa489a85d1cce60b84491f3566eaf484a93753108e77a4e3aac52fa8157f36d0
Deleted: sha256:5af665f72deff595b9bb0afeea5a944e7e63c0b40d05371f2ff1f23d14cdccb3
Deleted: sha256:de258e6e7b50a26bc997ffb2161daaf1f07485c8ef9623318b3aeb16e31d1019
Deleted: sha256:af2a306c8e66e784cefaca9cbef5d63893892ac0717cf8df9de3cf24405059cd
Deleted: sha256:df64d3292fd6194b7865d7326af5255db6d81e9df29f48adde61a918fbd8c332

and in /etc/docker I found the following.

/etc/docker

{"crv":"P-256","d":"e_7N92oVLU-ZnRPVKYvfVYpHbK2IfXWk4ftPVWLeqgQ","kid":"HDIK:LFQO:OYHE:NLHJ:DJ46:AWUM:BCCR:K2MZ:UXVD:NLBC:4KGQ:DMQ5","kty":"EC","x":"JJ2UCLb4ODabtzh_NAiKBYja9hRmn8USXcVQZE2mk7g","y":"eQVXO0ouR-XFFxLvojXM2p1qEiAReUoaC3wuKogC_Uo"}

I will be reporting this to the FBI, as soon they are done with the Muller report and or fall out.

How to Scrape and Parse Stock Earnings Reports

For years I looked for a simple way to get earnings reports from Wall St. It seems they would change their earning reports on a regular basis and the dates would be inconsistent. Then one day when I wasn’t paying attention, BAM. Earnings report and the stock would take off or crash on the numbers and I would be left holding the bag or missed the boat. You get the point.

It has been my experience that Yahoo Financial calendar provides the most consistent and update earning report for listed companies. (If there any better ones out there please do email or do PR for the repo.)  So now that I have found a source, how to do I make take that information and standardize and normalize it into a database. Obviously once I have that information in a database, I can use that data to purchase calls, puts or evaluate the earnings date based on a host of other information I have available to “cross reference” or analyze.

Luckily, through the power of scraping + python + github anybody can now have standardized earnings date reports and now I can scrape and standard the data and manipulate it to my needs.

In my repo I added the file earnings.py.  This is where the magic occurs..

Just a simple Mysql table to capture date, stocks, eps estimates, etc…

q = "truncate yahoo_earnings"
cursor.execute(q)

Then we have to a little “wonky” stuff with the dates that we are going to query. It seems like weather I’m in PHP or Python, I’m always getting entagled in dealing with DATE or TIME, if anybody can parse those dates better please send at PR.

#set dates
now = datetime.now()
startDate = now.strftime('%b %d %Y %I:%M%p')
print(startDate)
endDate = now + timedelta(days=60)
endDate = endDate.strftime('%b %d %Y %I:%M%p')
print(endDate)
startDate = datetime.strptime(
startDate, '%b %d %Y %I:%M%p')
endDate = datetime.strptime(
endDate, '%b %d %Y %I:%M%p')

Then as you can see from the code it’s pretty simple after that, call the function, parse the data, load it into your database or where ever you need it. Oh I forgot to mention, Make a ton of Money.

All this code can be found in my stock and options API repo, specifically with the Earnings-Scraper/earnings.py file that I’m discuss here. If you want the raw repo from wenboyu2

 

The Best and Worst Stock and Option Trading APIs

In my quest to program and build my own trading system, I have discovered a lot of conflicting information on the “Internets” about trading APIs and stock and option price quotes.  In the past, I posted on HN news about some of my findings, only to get some great new insights. One thing I can’t find is a simple location for all trading APIs and I have stumble along some rabbit holes when dealing with the APIs, trying to see what works and what is no longer supported. With that said, I will will be launching a General information Git Repository, to hopefully provide links to SDK for trading API and price quote APIs, Etc. I will obviously do pull requests, but my opinions and finding on certain trading systems will be detailed below and on this site.

Finally, having built successful trading systems and algorithms, some of my work can be found here. I’m available for hire at Upwork or via email.

The repo can be found here.

Ally Financial (used to be Trade King api):

Pro: Open an account with as little as $50

  • Access to real time stock quotes and option chainss
  • Simple REST api calls, takes 5 minutes to get up and running.
  • Support email is responsive.
  • Documentataion is simple and concise.
  • Oauth2

Cons:

  • no paper trading account
  • no historical quotes
  • actually trading a stock or options is documented poorly

eTrade

Etrade has and an API, it doesn’t seem to be very well supported. Meaning it came out in I think 2012 and it hasnt been updated recentley (like the last few years).

Pros:

Sand box environment: I only made it to the sand box. Which really isn’t a test/development environment. It’s an environment that no matter what stock price you query it will return quotes for Apple, Google or Microsoft. This really isn’t a true testing environment.

  • SDK available

Cons:

  • Can take weeks to set up and become operational
  • Very little support from eTrade
  • documentation is clearly outdated
  • Very little documentation on Stackoverflow or google searchs

TD Ameritrade:

TD Ameritrade does infact have an brand new API, it seems to be a stealth launch.

Pro:

  • Documentation is pretty good, not great.
  • TD ameritrate has the best trading platform, which would be the Think or Swim (TOS) desktop application.
  • Can create API calls on the website.
  • Email support is responsive

Cons:

  • Still in soft launch phase
  • documentation is not complete, spelling mistakes, inconsistent.
  • No paper trading account.

Interactive Brokers:

Pro:

  • Probably the best API.
  • Recommend IB_insync wrapper, well supported with a large community.
  • Offers historical quotes so you don’t have to rely on Yahoo
  • Options, FX, Bonds, Stocks
  • There are infinite amount of ways to trade
  • Live Trading & paper trading

Cons:

  • At times overly complicated
  • Only works with Java, C++, Python, .NET (C#), C++, ActiveX, DDE
  • The Trader Work Station, which is not required, is grossly outdated and cumbersome.

How do you get a job in algorithmic trading?

TL;DR: Looking for employment resources in Finance Programming / Trading algorithms? I’m in a rock and hard place…

I have the ability to write stock trading algorithms, portfolio trading algorithms. Also Cryptocurrencies, Options, and if need be futures and FX.

I blame the great depression for my ability to code, infact I call myself a “self taught depression coder”. Meaning that during the great recession I couldn’t find a job, so I took off my MBA in Finance on my resume and started teaching myself PHP and MySql. I also know Javascript, python, node, etc, to name a few.  So I don’t have the Computer Science Degree or Statistics PhD , but I’m right in the middle with an MBA and proven coding skills.

Currently, I’m a fully employed programmer. Why you might ask? Well because I can’t seem to walk down the street without someone trying to hire me for a programming job for the last few years. The pay is great, the risk is low and generally the stress is low. I have made lot of money programming other peoples ideas. Oddly enough, a lot of them failed, not because of me, but mostly because the ideas where not that great in the first place.  For example, a new paypal system for marijuana or a website to list trailers? Hmm… last time I checked it was free to list trailers on craigslist. Oh well, even though the money is great, there is a limit to how much you can get paid versus boredom and intellectual challenge.

So know I have slowed down any side work and only work on my algorithms at night. My day programming job is not intellectually challenging, so I have to spend my nights writing algorithms to remain intellectually stimulated. I’m also an SEC register online adviser (RIA), but I don’t have any clients as marketing is not necessarily my strong suit.

Basically, I’m looking for people or websites that might help me find a job in the area of quantitative finance / algorithm trading. Not interest at working at one of the banks, or investment firms cause I’m sure they got it all covered. But maybe a smaller shop. The only other stipulation is that I have made my home in Denver, with no desire to move to Co-Location alley (NJ,NY).

I was wondering if any can point me to a website/recruiters/firms/portfolio manager  that specialize in algorithmic trading or if there are an Adviser or RIA that would like to discuss some opportunities.

A click and show resume can be found here.   Some of my trades can be found here.

Think or Swim on ubuntu 17.10

I really wish TOS would distinguish themselves from the other trading service and support Linux. Regardless, I have TOS running on 17.10. Here is how I did it. The majority of the instructions are from the TOS website with a minor change or two.

Just to be clear this the same as all the other ubuntu directions, but I just want to make sure if someone was having an issue on 17.10 they could find it.

  1. Load your system with Java 8

sudo apt-add-repository ppa:webupd8team/java
sudo apt-get update
sudo apt-get install oracle-java8-installer

2. Download the installer 

3. install

cd /Downloads
sh ./thinkorswim_installer.sh

4. When you are going thru the wizard, install the thinkorswim software here: I believe that is the trick that worked for me, was to install the app into:

    /usr/local/bin/thinkorswim

The installer should start the application for you.

The TOS small box will pop up and say that it is intalling updates. This seems to take a long time, like 5 minutes? So go do something else for a few. Then you should get login screen after a while.

When it’s time to shutdown and start from stratch use:

    sudo ./thinkorswim

Note: If you find any discrepancies or tips please leave them in the comments. I will do my best to post them, as this blog post and the other Ubuntu / TOS get a lot of hits, there are people out there that need hits.

Drupal Vs. Laravel

A few months ago I posted on HackerNews a question about Laravel vs. Drupal. Although an unofficial poll, One can the overwhelming response in favor of Laravel. However, gather feedback from the Internets wasn’t enough.

A little background, I’m working on a contract for a large organization and the administrator of the contract who has a background in system administration decided to use Drupal 8 as project to prevent people from uploading directly to the production server from Dreamweaver, SCP, FTP. terminal or some archaic method. (There is absolutely no deployment methodology) Prior to the sys admin leaving in July, I asked the sysadmin why he chose Drupal? His response was “Well I programmed in it briefly, like 10 years ago so I think it will help us.” So that was the rational. I started to learn to program in Drupal, I quickly learned that essential it is a pain in the @ss to program in Drupal. In fact it is downright excruciating.

Spending a few weeks trying to implement a simple group permission module that would allow certain writers access to certain articles and deny writers access to different articles, etc. What I would consider very basic content moderation / group access, there only two modules out there, Groups and Organic Groups and neither of them seem to work very well. Most importantly, most developers haven’t moved over to Drupal 8, so there doesn’t seem to be a rush to support these modules.  In addition, I had coded out content moderation / group access features in php / Codeignitior / MVC with no problems in the past.

With the  Sys Admin, who did a little coding 10 years ago, now gone, I saw opportunity to change direction, as I didn’t want to get stuck coding in Drupal. I address the issue with the new administrator and instead of telling him that I was going to switch to a new framework, I thought it was only fair to a present my case. I believe that it is important for administrators or senior leadership to make informed decisions. All to often I have seen developers just tell non technical people what they are going to and then start programming in their flavor of the week coding language. With that said I would put together a presentation on Drupal vs. Laravel and make non technical person to understand. I was excited about this because writing code is fun, but sometimes I like add a little more challenge and this would provide an opportunity.

The key issue really is 40% of the project really setting up a simple CMS system to publish articles. The other 60% of the project is completely unknown. It could be something like converting oracle data and displaying in a custom CMS view.

The one thing I did get to Sys Admin was to get a simple mission statement as to lock him down on the specifics of the project. So here is the presentation:

Mission Statement / Directive: To build a comprehensive Content Management System (CMS) using the Drupal framework to allow different stakeholders and pre approved groups the ability to create appropriate content (news, embeded video, images) and static pages for publication on the website. The public facing will only have read access and all content management (CRUD) will occur behind a protective firewall.

With that here is my presentation:

Laravel vs. Drupal

Problem: Develop a CMS for an unknown project description and make sure the CMS system is adaptable enough that it can implemented across a variety of project / websites that have not been defined.

Current situation: Currently we are attempting to implement Drupal 8, which was was released on Nov 2015. Drupal 8 is significantly different than Drupal 7 and does not have an end of life date. This is important to note that current web developers are not converting sites to Drupal 8 at an accelerated pace because they can continue to use Drupal 7. Hence the development on Drupal 8 and drupal modules has been slower and spotty. Implementing Drupal 7 would be an issue as we would be confronted with an upgrade to Drupal 8 in the future. Using Drupal 7 would only be an effort to kick the can down the road.

Proposed solution: After working with Drupal over the course of a few months, it appears the community is stuck in Drupal 7 and possibly abandoning the platform. Solutions to Drupal 8 issues appear to be sparse. In addition some key modules are in beta. Modules for group permissions, there are only two modules that support this feature. Which really makes this project dependent on the two modules. Permission to manage content is standard with WordPress.

After careful consideration and research. It is recommended we move over to the Laraval / PHP framework without altering our November or new October 1st deadline, maybe even decreasing it. Laraval is a robust framework that allows us to implement the most basic requirements of the “Drupal – RISE – Project” of creating a simple blog/article ability with basic Admin capabilities. The beauty of laravel allow to adapt and build features that we don’t know exist, for example connecting the water levels with mussel data should be able to develop without being bound to constraints that Drupal might place on creating new features. Although all constraints of a Drupal or Laravel can be overcome through programming, Laravel is just built in a way, that you can get up and running faster to implement new features.

PRO Laravel:
-Up and coming community
-Laracast (well documented support and video learning)
-Well received / received in the community
-Eloquent ORM (immediate protection against Mysql attacks)
-Lumen (micro service / API)
-Far greater admin user interface than Drupal
-Follows traditional MVC framework
-LDAP integration
-Super easy documentation:
https://laravel.com/docs/5.4/
https://laravel.com/api/5.4/
-Super easy and fast
-Implements a simple Model-View-Controller architecture

CON Laravel:
-New system
-Start from the beginning
-Started in 2012
-Calls it’s programmers, “Web Artisans”

PRO Drupal:
-Work already exists, LDAP, admin
-The traditional and established “work horse”, built in 2001
-Empirical evidence appears that drupal is used for newspapers format.

CON Drupal:
-Open source community seems to be stuck between Drupal 7 and Drupal 8, minimal development
– https://www.drupal.org/project/dbfm (Module Derick needs is stuck on Drupal 7)
– https://www.drupal.org/project/group
-Fading use as illustrated in charts below.
-Not traditional MVC environment (although moving towards it)

Google:
https://trends.google.com/trends/explore?date=today%205-y&geo=US&q=drupal%208,drupal%207,drupal,laravel

Stack Overflow:
https://data.stackexchange.com/stackoverflow/query/697097#graph

Reditt:
Drupal Users Group: https://www.reddit.com/r/drupal/ = 7,066 readers
Laravel Users Group: https://www.reddit.com/r/laravel/ = 11,028 readers

Epilogue: In the end, I wasn’t able to persuade the new Sys Admin to move over to Laravel, we are stuck with Drupal.  Not sure if I really lost as much as I might not have hit the right selling points with the audience. That can be difficult, when the mission statement is very open and the end goals are not defined.  I did have some competition as another developer presented the case for Drupal.  Any frame work is going to be flawed, but the lack of support in the Drupal 8 community is a serious issue. On the bright side, I was able to gain exposure to Laravel as I got to play around with it for a week or so. Although my background is Codeigniter, Laravel is super fast to pick up super quick to learn.

Leave back here or on hackernews, be interesting to see the communities thoughts.

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

The Best or Worst way to decline a technical interview.

Hiring practices in the tech field seem to be topic of discussion on HN. I’m truly, amazed at some of the hiring practices  in the tech field.  Passion and desire are overlooked for technical knowledge. It truly is like no other field… With demand for developers, it kind of shocks me there are not better methods. CTO / Sr. developers should not be the sole gate keeper of hiring developers. (I’m sure that will get me in trouble…  notice that I used the word “sole”, yes they should have some input in the hiring decision.)

Recently (last week, to be exact), I applied for a position Senior Software Development position. I was intrigued by the salary, working remote and some of the leadership opportunities. However, I knew at some point it was going to lead a stereotypical technical interview. (This might come as a shock, but the first question on the tech interview was “Describe Object Oriented Programming?”) Being a self taught programmer, I unfortunately take great pride in knowing that I will fail miserably. I also seem to have lot of desire not to speak the lingo of the tech interview either, as I would rather spend more time learning new code. With that I really never should have taken the interview, as I honestly believe I’m qualified, but don’t really care to have a pop quiz over the phone about what I consider academic programming questions. My work, my side projects, code examples, desire to learn should illustrate that on a resume (7 years of paid programming experience), that was never discussed in the technical interview.

So I went thru the culture fit with a non technical / culture interview with no problems. Then came the technical interview…. as you can see from below it was an epic fail. As laid out below it’s clearly my fault. In fact it might have been comical to some degree. But regardless weather I wanted the job or not, rejection is frustration and annoying.  So I shouldn’t have agreed to the interview in the first place as I knew the outcome. While at the same time I feel extremely comfortable there wasn’t anything in their code that I couldn’t have done. At the end of the day it was my fault for being frustrated and I don’t want to be angry at myself. I would rather save my anger for injustice in the world.

After the interview, I sent the following email to hopefully relieve myself of some my anger. After clicking send, I do feel better… So there is in a opening in Denver, maybe I will get a referral fee…

 

Greetings,

I’m going to respectfully withdraw my application for employment for HxxxxXxxxx. I don’t feel that meet the technical requirements as laid out in the technical interview.

This might not come as a surprise as the technical interview was a interesting adventure in clearly explaining I don’t know the answers to academic programming questions. Since I’m not a typical programmer and self taught programmer, I believe that might have surprised the interviewer with my willingness to clearly state that I had no experience with a particular type of procedure and politely explain that I didn’t have answer. (I could almost feel the pause on the phone as I answered honestly, that I didn’t know the answer. I also mentioned that I wasn’t going to try to google question while on the phone either, which seems to be a new trend.) Honesty, can be a terrible curse. It’s 4 years since I retired from the Marine Corps it’s concept of honestly and responsibility are still haunting me.

Regardless, please be assured that I have absolute no animosity towards the HxxxxXxxxx or the people that interviewed me. I’m not lashing out or frustrated. The animosity lies completely with myself. Yes, that’s right. The responsibility lies completely with myself that has placed me in a frustrated mood. Unlike our current political / societal environment, it’s all me. I should have never taken this interview as I generally winds up with a stereo-typical technical interview and I know that I’m not going to pass. Which ultimately meaning that, I shouldn’t have applied for the position. Yeap, I should have never taken the interview. I’m writing this email for three reasons 1. Unburden some of my personal frustration. 2. This will also serve as a “Memento” to myself, not to fall into this self inflicted trap I successfully get myself into every few months as I seem to have mild form of amenisa regarding this issue. 3. Sometimes I like to step away from the code and practice some of communication skills and self awareness. For this self inflicted wound, I apologize for expending your time and mine.

Sincerely,

Chad Humphrey

Part of the reason I like to post things like this, is I get the see other people reactions on HN. Which can be good and bad, but I’m willing to see things from another perspective.

How to get option prices for free – API Yahoo

I don’t know why this was so difficult to find this on google, but eventually I found it in the code of a node js library.

Deep inside the nodejs library is the url to query option data. I don’t think this is a secret but yahoo sure wants it that way with there use of YSQL nonsense.

Regardless, here is the url: https://query1.finance.yahoo.com/v7/finance/options/AAPL

Obviously, AAPL is apple computers. This will return nicely formed JSON data. It will also return all the existing expiration dates at the top of the json object.

If you want a specific date the you just add the ?date=1505433600 to the end of the URL.  The date is converted into unix time, which you can translate here.

https://query1.finance.yahoo.com/v7/finance/options/CMG?date=1505433600

I will use php to parse the json object, drop it into database, which wasn’t that difficult. The key take away is the formation of the URL call to Yahoo. One can use any language to parse the data.

Surprisingly, this is pretty simple.

Leave questions or concerns in the comments. I’m also available for hire.

 

 

 

Think or Swim on ubuntu 16.04

I really wish TOS would distinguish themselves from the other trading service and support Linux. Regardless, I have TOS running on 16.04. Here is how I did it. The majority of the instructions are from the TOS website with a minor change or two.

  1. Load your system with Java 8

sudo apt-add-repository ppa:webupd8team/java
sudo apt-get update
sudo apt-get install oracle-java8-installer

2. Download the installer 

3. install

cd /Downloads
sh ./thinkorswim_installer.sh

4. When you are going thru the wizard, install the thinkorswim software here: I believe that is the trick that worked for me, was to install the app into:

    /usr/local/bin/thinkorswim

The installer should start the application for you.

The TOS small box will pop up and say that it is intalling updates. This seems to take a long time, like 5 minutes? So go do something else for a few. Then you should get login screen after a while.

When it’s time to shutdown and start from stratch use:

    sudo ./thinkorswim

Note: If you find any discrepancies or tips please leave them in the comments. I will do my best to post them, as this blog post and the other Ubuntu / TOS get a lot of hits, there are people out there that need hits.

Why won’t Think or Swim work on ubuntu 15.10

Once again desperately trying to install TOS on ubuntu 15.10. But once again I get a big fail from TOS.

My previous post on 14.04 worked fine, but I haven’t been trading that much and I upgraded my linux box. 

Here is what I have done so far:

Thinkorswim needs Java7 to run. To install java7 on the machine:

  1. sudo apt-add-repository ppa:webupd8team/java
  2. sudo apt-get update
  3. sudo apt-get install oracle-java7-installer

To install Thinkorswim:

  1. Click “Install thinkdesktop” to download the thinkorswim installer to a directory on your PC.
  2. After downloading open a shell and CD to the directory where you downloaded the installer.
  3. At the prompt type: sh ./thinkorswim_installer.sh

Once I get the program installed I get a new error message. There is no proxy to chose from as it just comes up blank. I’m not doing any thing fancy with proxy or firewalls so this shouldn’t be an issue.

think or swim fail

Any thoughts Think or Swim? or anybody else?

 

Update: 11/30 – 12/1

 

 

Screenshot from 2015-11-30 11-01-53

So if you go to https://216.105.249.176/suit/index.xml?whitelabel=tos  you get the following error message

{"error":"symbol is not futures"}

So if I getting a return message from the server then how can it be a 400 response code. Also I concerned that an organization such as TD Ameritrade isn’t using all https? I’m not an expert in this area but even my twitter bootstrap cdn’s are using https. (Of course I’m not but that’s a whole other story…)